(Floe) UI Options user testing protocol (February 2012)

Owned by Tona Monjo
Last updated: Sept 17, 2012 by James YoonVersion comment
5 min read

This page contains the protocol for User Interface Options user testing (see mockups here).

See also: 

Methodology

Session structure

The session is composed of four parts:

  1. Pre-test interview
  2. User testing
    a. Undirected, "reality" testing
    b. Task-based testing
  3. Post-test interview
  4. Post-test questionnaire (demographics)

The first part of the session, pre-test interview, is intended to explore the user’s needs and expectations before the user gets into the testing. The goal is to avoid a biased opinion, influenced by the use of UI Options. 

The second part, user testing, is composed of two tests: 

  • An undirected, "reality" testing, in which the moderator provides a document and lets the user act freely. 
  • In the second test, task-based testing, the moderator asks the user to fulfill specific tasks. This part of the test is intended to study the usability and intuitiveness of specific features of UI Options not covered in the undirected testing.

After the user testing is complete, the user is interviewed about his/her experience with the application, including problems encountered, potential areas of improvement, and possible new functionality. This interview is followed by a demographics questionnaire.

Data gathering

Data for user testing will be gathered using the following tools:

  • Audio and video recording
  • Screen recording + eye tracking
  • Pen & paper notes

Data for the post-test interview will be gathered using the following tools:

  • Audio recording
  • Pen & paper notes

Materials

Document with integrated UI Options: 

0. Protocol Introduction

General things to keep in mind for the test

  • Reassure the user that we are not testing the user--we are testing the application, and there are no wrong answers.
  • Ask the user to think aloud whenever possible.
  • Probe them on expectations, frustrations, and general thoughts.
  • Avoid influencing the user's decision-making and deflect questions to gain further insight into the user's thoughts (e.g., "What does this do?", "What do you think it does?").
  • Don't offer help; let the user attempt to perform the task themselves. If they ask for help reply with:
    • "What do you think you should do?"
    • "What do you think that means/would do?"

Greeting script

Greeting script
Hi [user's name]. I'm [your title] with the Floe project. Today we are looking for ways to improve the user experience of User Interface Options, a tool used to customize the experience of reading a document on the web.

This is a test of UI Options; we are not testing you. If you find something difficult or unintuitive to use, chances are that others will as well. This test of the UI Options is simply a means of evaluating the design and to discover any issues we need to address.

The study is composed of four parts: we'll start with a short interview and there will come the main part of the study, where you get a chance to play with UI Options. Then you will be asked to perform a few specific tasks. Afterward, we'll have a short interview about your experience, and wrap that up with a quick demographics questionnaire.

The study will take about 30 minutes, and you can stop at any time during the study for any reason.

Do you have any questions before we start?

1. Pre-test interview

  1. When you're using the web, do you ever find webpages that are difficult to read or use? If so, in what way?
  2. Do you ever have to fiddle with options or settings to make webpages easier to read or use? If so, which ones?
  3. Is there anything that you’d like to be able to change on websites, but can't?

2. User Testing

2A. Undirected testing

Instructions for user (script)

You have been provided with this document to learn about ______, but the current design is not completely comfortable for you. Do you think that you can customize it? How would you? 

2B. Directed testing

Instructions for user (script)

Now we'll be asking you to complete a few tasks using the application. As before, please speak your thoughts aloud. Let us know what pops out at you first, what you're looking at, what you're reading, what you think the different items do, what you're expecting to happen, what options you're thinking of and how you come to your decision. Let us know if anything seems awkward or doesn't seem to flow the way you'd expect or want it to.

Instructions for moderator

From the full task list, ask user to complete tasks that weren't completed during the previous reality testing part. You may also want to repeat a task you missed observing during reality testing.

Note: To avoid performing a too structured testing, there are two groups of tasks:

  • Tasks to perform in random order: each one of the tasks will be printed in a different card, and picked up in random order. During the reality testing, remove the tasks that are being performed (excepting those that you want to repeat). 
    Download tasks cards here: UIoptions_testing_cards.pdf
  • Tasks to perform at the end of the testing: after the random tasks have been performed, come up with these tasks to conclude the directed testing.

Task List (to perform in random order)

Text & Display

  • Looking at this document, you realize that the text looks too small for you. Can you increase its size? 
    Text size
  • The text in this document looks too dense for you. The space among lines is to narrow and disturbs you reading. Can you think on some way to fix it? 
    Line spacing
  • You usually prefer to read in Times New Roman. Try to change the text style. 
    Text style
  • The current contrast doesn’t result enough confortable for you. Can you increase it?
    Color & contrast 

Layout and navigation

  • Given the document’s design, you don’t achieve a clear idea of the contents of the page. Can you find some way to know the contents' structure?
    Show Table of Contents

Links and buttons

  • You don’t see clearly where are the links in the text. How would you make them more visible? 
    Emphasize links
  • You’d prefer to have larger buttons. How would you get it?
    Make inputs larger

Task List (to perform at the end of the testing)

Tasks for versions with Save option

  • How would you return to the document without applying anyone of the changes you’ve done? 
    <Just question, don't allow the user to click on Cancel, to be able to perform the next task>
    Cancel
  • Now return to the document applying all the changes you’ve done. 
    Save and Apply

For all versions

  • Figure that you don’t like the changes done, and decide to return to the original design. How would you do it?
    Reset all/reset and apply
  • You’ve finished customizing the visualization, how would you close the options? 
    Hide/Other options in panel

3. Post-Test Interview

Use the following questions as guidance during the semi-structured interview. Probe deeper into any key issues that arise.

  1. How would you describe your experience using UI Options overall?
  2. What did you find confusing or frustrating using UI Options?
  3. Would you add any option or functionality?
  4. Do you think that the integration of UI Options with the document is comfortable enough? 
  5. What might we do to improve the application?

4. Post-Test Questionnaire (Demographics)

General demographics

1. What is your gender?

  • Male
  • Female

2. What is your age group?

  • 12-17
  • 18-25
  • 26-35
  • 36-45
  • 46-55
  • 56-65
  • 65+

3. What is your occupation or main activity? ________________________

4. What is your primary language? ________________________

5. If you have any disabilities that affect your use of computers, list them here: ________________________

6. If you use any custom devices to use computers (even glasses), list them here: ________________________

Technology questions

7. In general, what is your comfort level with technology?

  • Very comfortable
  • Comfortable
  • Uncomfortable
  • Very uncomfortable

8. How often do you read documents on screen?

  • Daily
  • Often (2-3 times a week)
  • Occasionally (once a week)
  • Not often (once a month)
  • Only if I can not print them

9. If you read on screen, do you make any customization? List them here ______________________

Finishing Up

Ask the user if he/she has any questions, thank the user for participating, and give an honorarium if we have one.