(Floe) UI Options user testing protocol (September 2012)

Owned by James Yoon
Last updated: Apr 10, 2013 by Joanna VassVersion comment
7 min read

Download mock-ups used for testing

See Also:
(Floe) UI Options user testing results (November 2012)
(Floe) UI Options user testing report (November 2012)

Methodology

Session structure

The session is composed of four parts:

0. Introduction
1. User testing

    1. Undirected testing for top/side panel + browser extension (half the users test top panel, half the users test side panel)
    2. Directed testing for top/side panel + browser extension (same users distribution as previous)
    3. Undirected testing for mobile
    4. Directed testing for mobile

2. Post-test interview (during this interview, the user will compare the tested and the alternative versions -top/side panel-). 
3. Post-test demographics questionnaire

Expected duration: 60 min.

Testing formats

All formats of UI Options must be tested: 

  • Top panel
  • Side panel
  • Browser extension
  • Mobile

To avoid tiring the participants, the tests will be divided into two combinations (3 users for each one): 

  • Top panel + Browser extension + Mobile
  • Side panel + Browser extension + Mobile

Materials

Print or provide on screen via Wizard-of-Oz methods the following:

Data gathering

Consider using the following tools to gather data:

  • Pen & paper notes
  • Audio and video recording

User profiling

This testing will be performed with 6 users. Suggested profiles:

  • 2 users with no disabilities
  • 2 seniors
  • 1 user with low vision
  • 1 user with motor constraints (?)

Protocol introduction

General things to keep in mind for the test

  • Reassure the user that we are not testing the user--we are testing the application, and there are no wrong answers.
  • Ask the user to think aloud whenever possible.
  • Probe them on expectations, frustrations, and general thoughts.
  • Avoid influencing the user's decision-making and deflect questions to gain further insight into the user's thoughts (e.g., "What does this do?", "What do you think it does?")
  • Don't offer help; let the user attempt to perform the tasks themselves. If they ask for help, reply with probing questions such as:
    • "What do you think you should do?"
    • "What do you think that means/would do?"
Greeting script

Hi [participant's name]. I'm [your title] with the Fluid Project. The Fluid Project is a project aiming to build more inclusive, usable web experiences for everyone. Today we are looking at ways to help tweak and customize your web experience.

This is a test of our designs; we are not testing you. If you find something difficult or unintuitive to use, chances are that others will as well. This test of the design is simply a means of evaluating our work and to discover any issues we need to address.

The study is composed of three parts: we'll start with the main part of the study, where you get a chance to see printed screenshots of our designs and complete a few tasks with them. Afterward, we'll ask you a few questions about your experience, and then end with a demographics questionnaire.

The study will take about 60 minutes, and you can stop at any time during the study for any reason. We will answer any questions you have now and at the end of the study.

Do you have any questions before we start?

First we'll need you to read over and sign this consent form.

User testing

Introduction to UI Options context

  • In the real version, UI Options will be available after installing it as a browser extension. The user needs to be introduced to this fact, to be able to understand the background performance and establish same mental relationships as in the real  experience, between the Browser Extension and the UIO Panel. At this point, inform the user that in the real world: 
    • He will install the UI Options from UIO site
    • He must sign up/log in from the brower extension, to start using UI Options 

Discovery/undirected testing for top/side panel + browser extension

  • You arrive at this difficult-to-read page (low contrast, small text) <show screen>, and the article on the page is something you really need to read. How do you proceed?
    Screen: Default top/side panel, scrollable, included in a browser window. Other screens must also be available, to follow user's actions.

Directed tasks for top/side panel + browser extension

Ask only for the tasks that the user has not been able to perform in the undirected testing.

Basic preferences (same for top or side panel)

  • 1. Suppose you've been finding the page difficult to read: the text is too small, the contrast isn't ideal. Make some adjustments to make the page easier to see.
    Test focus: expanding UI Options panel, using basic preferences
    Screen: 1. Web page, with Display Preferences collapsed. 2. Default top/side panel, scrollable, included in a browser window. 
  • 2.Suppose that the page was cluttered, and that you were interested only in its content without being distracted by the clutter of other features. How would you do this?
    Test focus: basic preferences, understanding options
    Screens: 1. Default top/side panel, scrollable, including Simplify widget. 2. Previous screen, with Simplify customizing panel expanded. 
  • 3. Suppose you were on a page that used the same styling between links and regular text, so it was difficult to tell what was linkable, and what wasn't. Make some adjustments so that you could better distinguish them.
    Test focus: basic preferences, understanding options, scrollbar perceiving and using
    Screens: 1. Default top/side panel, scrollable, including Emphasis Links & Buttons widget beyond the scrollbar. 2. Previous screen, scrolled panel with Emphasis Links & Buttons visible. 

Advanced features

  • 4a. Suppose you want to go momentarily back to the original design, to check if your changes have really made the page easier to see. Can you do this?
    Test focus: reset to default and undo reset
    Screens: 1. Default top/side panel (ensure that Reset to Default option is visible). 2. Previous screen with the Reset Undo option available. 
  • 4b. Now go back to your customized design. 
    Test focus: undo reset to default
    Screens: 1. Screen with the Reset Undo option available (same as previous). 
  • 5. Suppose you find the images on this page distracting, and you want to replace them with text. How would you do this?
    Test focus: search field
    Screens: 1. Default top/side panel. Ensure that the search field is available. 2. Panel with results, Replace Images with Text option available.
    Take note of: the exact text that the user would enter in the search field (ask him if he doesn't specify it during the task).
  • 6. Remember that you had previously changed the contrast. Do you think that you should be able to change also the page background colour? How?
    Test focus: customize options
    Screens: 1. Default top/side panel, scrollable, including Contrast widget with the Customize options bar visible. 2. Previous screen, with Contrast customizing panel expanded. 

Browser extension

  • 7. Suppose you want to save the changes you have done, to apply them to other devices. How would you do this?
    Test focus: understanding Browser Extension function
    Screens: 1. Default top/side panel, scrollable, included in a browser window (with Browser Extension option available) 2. Browser Extension expanded.
  • 8. Suppose you want to share your preferences with a friend, because it may be useful for him. How would you do this?
    Test focus: sharing option in Browser Extension
    Screen: Browser Extension panel
  • 9. Imagine you'd like to create another set of preferences, to use when you are reading at night. Could you do this?
    Test focus: New Theme option in Browser Extension
    Screen: Browser Extension panel
  • 10. Now you have finished customizing your preferences, and want to continue reading the web page. How would you close all the UI Options items?
    Test focus: closing UI Options
    Screen: 1. Browser Extension panel (so the user can say howto close it), 2. Top/left panel
  • 11. Now you are leaving this computer and want to disable all customizations, so the next user can see the original designs. How would you do this?
    Test focus: using the Browser Extension menu
    Screen: 1. Browser Extension menu (check if they would click on off)

Discovery/undirected testing for mobile

  • You arrive at this difficult-to-read page (low contrast, small text) <show screen>, and the article on the page is something you really need to read. How do you proceed?
    Screen: Default mobile screen. Other screens for mobile must also be available, to follow user's actions.

Directed tasks for mobile

Ask only for the tasks that the user has not been able to perform in the undirected testing.

  • 1. Suppose you've been finding that the the text in this page is too small and you'd like to make it larger. How would you do this?
    Test focus: open display preferences, use basic preferences
    Screen: 1. Mobile default screen. 2. Previous screen with Display Preferences bar visible. 3. Previous screen with Make Thinks Larger visible. 
  • 2. Suppose you also need to adjust the page contrast. How would you do this?
    Test focus: switching between preferences
    Screen: 1. Mobile default screen with Display Preferences bar visible. 2. Previous screen with Adjust Contrast options visible.  
  • 3. Suppose you want to go momentarily back to the original design, to check if your changes have really made the page easier to see. Can you do this?
    Test focus: reset to default
    Screens: Screen: 1. Mobile screen with Display Preferences and Reset to Default option available. 

Post-test interview

Use the following questions as guidance during the semi-structured interview. Probe deeper into any key issues that arise.

  1. How would you describe your overall experience using display preferences?
  2. What did you find confusing or difficult to understand in the desktop version?
  3. What did you find confusing or difficult to understand in the mobile version?
  4. Are there any features you would have liked to find in Display Preferences?
  5. How did you find the integration of Display Preferences with this page?
    Goal: check the user feelings about the desktop version that he has tested (top/side panel)
  6. Now let's see an alternative location for Display Preferences. Which one do you feel can more comfortable?
    Show the user an image of the alternative design: if he tested the top panel, show him the side panel; if he tested the side panel, show him the top panel.
    Goal: check the user feelings when comparing the known and the new version (this is an explorative question, so results can not be considered conclusive). 
  7. What else might we do to improve Display Preferences?

Post-test demographics questionnaire

General demographics

1. What is your gender?

  • Male
  • Female

2. What is your age group?

  • 12-17
  • 18-25
  • 26-35
  • 36-45
  • 46-55
  • 56-65
  • 65+

3. What is your occupation or main activity? ________________________

4. If you have any disabilities that affect your use of computers, list them here: ________________________

5. If you use any custom devices to use computers (even glasses), list them here: ________________________

Technology questions

6. In general, what is your comfort level with technology?

  • Very comfortable
  • Comfortable
  • Uncomfortable

7. How often do you read documents on a computer screen?

  • Daily
  • Often (2-3 times a week)
  • Occasionally (once a week)
  • Not often (once a month)
  • Only if I can't print them. 

8. If you read on a computer screen, do you make any customization? List them here ______________________

9. Do you use a smartphone or an Internet-enabled mobile device?

  • Yes
  • No

10. How often do you read long texts on the mobile screen?

  • Daily
  • Often (2-3 times a week)
  • Occasionally (once a week)
  • Not often (once a month)
  • Never. Why?______________________

11. If you read on the mobile screen, do you make any customization? List them here ______________________

Finishing Up

Ask the user if he/she has any questions, thank the user for participating, and give an honorarium if we have one.

On this page