User Identification for Needs Assessment - Draft

User research needs to cover a broad and diverse range of users.  We want to make sure we are hearing from a comprehensive set of users in Higher Education Campuses.  In order to limit scope and allow us to gain understanding at various levels -- from user's goals (what drives them) to the details of a particular workflow we will complete the contextual inquiries in iterations.  Our initial focus will be on schools close to home for UC Berkeley and U Toronto where the research teams are located.  Each location has a variety of institution types in the near vicinity which allows great coverage.  We will also recruit some volunteers from various international locations to make sure we understand their users, context and work practices.

Our main goals in choosing users to work with are (also see overall research goals):

• Comprehensive understanding of complex user population and system - how does managing content fit into the larger context of the work
• Get users involved early in fleshing out the problem(s)
• Understand the various workflows, needs & goals off Sakai users
• Be able to identify patterns of use -- what's the 80% for various personas
• Understand where variances occur so we can build in the flexibility at the appropriate places
• Gain user empathy

Strategy / Plan:

• Identify minimum number of users to meet above goals.  Using the matrix below to think through what the different user groups are, how various characteristics give them distinct needs and goal (ex. a user's institution has particular business practices they need to follow, the type and style of their class drives needs around sharing, collaboration, communication, etc.).  We'll make some educated guesses about how many users in what roles and where we need to schedule our initial interviews.
• Recruit / organize teams of 2 in the identified locations.  Berkeley & Toronto are givens.  We'll likely need to recruit from UK, South Africa & Australia.  Each team will be responsible for identifying, recruiting and scheduling their contextual inquiries with the appropriate users.
• Complete Contextual Inquiries (CI).  In teams of 2 the researchers will go to the place the user most often does their work.  In Higher Ed this gets tricky since users likely have several contexts in which they do their work.  We'll want to make sure we cover a variety of the likely contexts:  office, home, computer lab, coffee shops, etc.  A look at the contextual inquiries:
  • Always do the CI in teams of 2 if at all possible - one researcher should drive the conversation while the other takes detailed notes.  Switching back and forth is a good way to mix it up, make sure everyone gets a good mix of experience and have different perspectives in note taking and facilitating.
  • Contextual inquiries should be as conversational as possible.  We'll use guides to help us keep the conversation focused around content and our areas of interest but we don't want it to be like an interrogation where we just run down a list of questions.  It should be a natural conversation and we want users to show us their work as much as possible.  As with any user participation, it's important to introduce yourself, make the participant feel as comfortable as possible and let them know that they are not being tested and get a consent form signed.
  • 1 1/2 hours with the participant.  Start with global questions like "tell me about a typical day?", "what do you do first thing in the morning?", "what frustrates you about your work?"  "what is the favorite part of your work?".  This allows us to fit the rest of the conversation into context, early on identify some areas to probe around in more detail, and let the participant warm into the conversation.  As the conversation continues we want to get more and more detailed, asking the participant to show us what they do, or better yet if there is any real work to do, to do it.  Also be sure to ask for screen shots and/or copies of work  artifacts along the way (they can always forward to you later).  We'll shoot for 1 1/2 hours of inquiry time.  I usually like to stick to an hour to respect busy schedules but we have a lot to cover.
  • 2 hours of analyzing and digitizing what was learned -- immediately (or as immediately as possible).   Rather than just transcribe the notes, the researchers should analyze and make meaning of the notes and the experience with the participant while digitizing the notes.  I've found this collaborative processing to be invaluable in the past.
  • Share, discuss and access where we are at and what we know -- often (at least weekly).  With a distributed team it will be important for us to connect often to get synched up -- assess where we are, what we've been learning, what patterns are arising, what gaps are we seeing, etc.  We'll use one of our various on-line modes.
  • Face to face meeting to kick off modeling.  Once we've completed our initial round of contextual inquiries and feel we've begun seeing patterns, we hope to get together face to face for some period of time to work through some initial models of what we've learned -- personas, activity diagrams, use cases, scenarios, communication models, etc.

Other ways to slice it to minimize scope:

• Supplement with some sort of user self report-- surveys, models and assumptions to react and add to, diaries, etc.
• Focus on particular sets of user groups in an iteration.  This is tricky & won't give us the big picture.  For instance, if we focus only on Instructors first we'll miss the understanding of how Instructors pass a lot of work off to TAs and what the TAs role is in the processes.  We'll also miss the connections between how instructors share content with students and then students study, view & act on it.
• Focus only on particular components.  This means we lose the goal of understanding a better global UI architecture for our systems based on organizing around content & activities rather than tools.  We could shorten the inquiries and focus on say uploading files.  I think this is also a mistake.  We won't understand how the components fit into the larger context of work.  We'll have to rely on what they do now rather than understanding their goals and what drives them to find better, innovative ways to support them.
• Spread the inquiries out over a longer period of time (prior to modeling).  This can also be a bit dangerous.  Deep processing is going on for the researchers so the cost of switching tasks can be even higher than usual.  We also need to think about the outward appearance.  If the research drags on it can give onlookers the feeling often misunderstood perception that user research takes a long time.

Distinctions in process, structure, workflows, goals

Factors to consider:

• Regional distinctions (particularly international)
• Institution types (i.e. large research versus community college)
• Departments (i.e sciences versus humanities)
• Course type / structure (see below)
• Job roles
• Learning styles
• Number & size of processes
• Number of people involved in a process / collaboration involved
• Time at institution / moving across institutions
• Pedagogy styles
• Computer interaction practices (i.e. keyboard only, screen reader user, etc.)
• Course content type / subject matter
• Also see the individual user matrices for more detail on some of these factors

Institutions identified

• UC Berkeley
• Stanford - Can the UX team at Stanford help out here?
• Distance learning?
• University of Toronto
• York University
• Cambridge
• Unisa
•  

Overall Matrix of Sakai Teaching & Learning Users

Working Area

Regions 

• Australia
• US (Do we need to distinguish between different regions here?)
• Canada
• South America
  • Columbia
• South Africa
• Egypt
• China
• Mexico
• Europe -  what distinctions do we need to make here?  
  • Germany
  • Netherlands
  • Italy
  • Spain
  • Portugal
  • UK
  • Belgium
  • Ireland
  • Sweden
  • Turkey
  • Denmark

Types of Higher Education Institutions

How many of these are relevant in different regions?

Other types not so US-centric? 

• Public
  • large research
  • extensions (not degreed programs)
  • state college (smaller)
  • community college (more mature student population?)
• Private (what distinctions do we make here?)
  • Elite?
  • Small
  • Distance learning (also public?)

User Roles

• Faculty
  • Lecturer
  • Tenured and/or tenured track
  • Other?
• Teaching Assistants
• Students
  • undergrad
  • grad
  • phd

Class Types

• Large class
• Small class
• Class with multiple sections
• Cross-listed class
• Multiple instructors
• Seminar
• No term

Departments

• Humanities
• Languages
• Sciences
• Professional Schools 

Pedagogical styles