(Floe) OER Commons authoring tool user testing protocol

Owned by James Yoon
Last updated: Jun 28, 2012Version comment
4 min read

(Work in progress)

Methodology

Time required

This protocol is intended to be used for very fast (~10-15 minutes), in-the-field hallway user testing.

Session structure

The session is composed of three parts:

  1. Task-based user testing (tab or panel variation)
  2. Impressions on untested variation
  3. Impressions on high-fidelity mockups
  4. Post-test questionnaire

In task-based testing, the moderator asks the user to fulfill specific tasks, specifically, tasks pertaining to the primary intended design/aims of the OER Commons authoring tool. This part of the test is intended to study the usability and intuitiveness of specific features and the relationship between the different areas of information for one variation of the tool.

Test half the users with the tab variation, and another half with the panel variation.

After the user testing is complete, the user is shown the untested variation for their impressions (e.g., better/worse, and in what ways). Following this, the user is given a quick demographics questionnaire. The questionnaire is followed by showing early renderings of the authoring tool to acquire first impressions on aesthetics, adoptability, likability, and/or persuasiveness. 

Data gathering

Gather data on the user testing and impressions using:

  • Pen & paper notes
  • If possible, audio recording

Materials

Print or show on the computer screen these versions of the authoring tool:

0. Protocol Introduction

General things to keep in mind for the test

  • Reassure the user that we are not testing the user--we are testing the designs, and there are no wrong answers.
  • Ask the user to think aloud.
  • Don't offer help; let the user attempt to perform the task themselves. If they ask for help reply with:
    • "What do you expect that would do?"
    • "What do you think that means?"
  • Be sure that the user knows what an OER is before administering the test. If he/she doesn't know, the interviewer should explain it briefly and clearly.
  • If the user has already talked through a task during their own exploration of the design, skip it.

1. User Testing

Instructions for user (script)

We're about to show you a few blueprints of the application we're designing. They won't be coloured or look complete, but we want to get your input on how it should work. We will be recording sound and video of your screen during this session. These recordings are just for the design team and will not be distributed. Is it okay if we record you during this session?

We'll start by asking you to complete a few tasks. Please speak your thoughts aloud. Let us know what pops out at you first, what you're looking at, what you're reading, what you think the different parts of the design do, what you're expecting to happen, what options you're thinking of and how you come to your decision. Let us know if anything seems awkward or doesn't seem to flow the way you'd expect or want it to.

The scenario (script)

<ask user about what subject(s) they teach; if they don't teach, what subjects/disciplines/fields of study they're interested in>

Suppose you want to author a lesson plan, assignment, or activity around _subject_ that you want to share with other educators over the internet. You've discovered OER Commons as a means to do this, and you've clicked on a button on their website that reads, "Create a new educational resource".

This is the screen you've arrived at. <show tab variation screen 1>

Task List

Beginnings

  • What does the user first do?
    Screen: 1
    You are about to start preparing a new lesson plan, assignment, or activity on __subject__. How would you start?

Content editing

  • Content structure
    Screen: 1
    Suppose you know that you want your resource to have multiple sections, the first of which you want to call "Introduction". How do you go about authoring that section?
  • Insert image
    (Note: Let the user explain what he would do in each one of the steps of this process)
    Screens: 2-3-4-5-6-7
    You want to complement your content with some useful diagrams and images, which you have on your computer. How would you do this?
  • Embed a video
    (Note: Let the user explain what he would do in each one of the steps of this process)
    Screens: 7-8-9
    You want to complement your content with a YouTube video you recently saw. How would you do this?

(Talk through Add a File, screen 10)

  • Move image
    Screens: 11-12-13-14
    You realize that the image is misplaced, and want to have it between the first and the second paragraph. How would you do it?
  • Resize image
    Screens: 14-15-16
    The image looks too small. How would you go about making it bigger?
  • Describe image
    Screens: 17-18
    How would you add a description for the image?
  • Modifying content structure
    Screens: 19-20
    Suppose you want to add a subsection to the first section, before the third paragraph. How would you go about doing that?
  • Finished editing content
    Screen: 20 (if the user clicks on the Metadata tab)
    Suppose now you've finished writing your content. What do you do next?

Metadata editing

  • Filling in required fields
    Screen: 21
    What fields would you fill out on this screen? And once you've completed those fields, what is your next step?

Finished

  • Alert about required fields
    Screens: 22 or 23 for Tabs variation, 22 for Panel variation.
    Upon finishing your work, you click on "Done". This alert appears. What do you suppose you should do?

2. Impressions on untested variation

We've also designed an alternative interface for this content editing environment. Take a look at it here. Based on your first impressions, which do you feel would be easier to use? Why?

3. Impressions on high-fidelity mockups

We also have a coloured, more polished design of this content editing environment. It should give you a better idea of what it might look like when it's finished. Take a look at it here. Based on your impressions of this, or the blueprints that you've seen, do you think this is an application you'd feel comfortable using? If not, why not?

4. Post-Test Questionnaire

General demographics

1. What is your gender?

  • Male
  • Female

2. What is your age group?

  • 18-25
  • 26-35
  • 36-45
  • 46-55
  • 56-65
  • 65+

3. What is your occupation or main activity?

________________________________________________________________

Internet experience

4. In general, how would you rate your comfort level with using the internet/web?

  • Very comfortable
  • Comfortable
  • Uncomfortable
  • Very uncomfortable

5. Have you created, edited, or published documents, resources, media, etc. on the web before? (e.g., Google Docs, YouTube/Vimeo, Flickr, SlideShare, etc.) If so, using which service(s)?

________________________________________________________________

Pedagogy

6. If you teach (or used to teach), what subjects do you teach and at what educational level?

________________________________________________________________

7. If you teach (or used to teach), do you use online resources to support your teaching? If so, what kinds and in what way?

________________________________________________________________

8. Do you have any initial ideas for materials you would like to share online? If so, what?

________________________________________________________________

Finishing Up

Ask the user if he/she has any questions. Thank the user for participating, an give an honorarium if we have one.