Methodology

Session Structure

The testing session is composed of two parts:

Undirected "Observational" Testing
Directed "Keypad Functionality" Testing
Expected duration: 60min

Testing formats

Kiosk exhibit formats for testing:

single kiosk (insight stations)

Materials

Paper prototypes of the following may be used:
- various interactive museum storyboards
- keypad
Wireframes as provided by CMHR will also be used for testing where available. A simplified interactive may need to be created based on these wireframes.
A prototype keypad will be constructed using 3D printing.
Rubber keypad base
Materials to build a functional keypad (Makey Makey kit or programmable keyboard)
Computer monitor
Screen-reader audio will also be incorporated into the testing (manual or automated TBD)
Alternate keypad layout prototype

Data gathering

Consider using the following tools to gather data:

Pen & paper notes
laptop
Audio and video recording

Test Participants

This testing will be performed with 5-7 users.

1. Protocol introduction

General things to keep in mind for the test

Probe the user on expectations, frustrations, and general thoughts.
Avoid influencing the user's decision-making and deflect questions to gain further insight into the user's thoughts (e.g., "What does this do?", "What do you think it does?")
Don't offer help; let the user attempt to perform the tasks themselves. If they ask for help, reply with probing questions such as:
- "What do you think you should do?"
- "What do you think that means/would do?"

Reassure the user that we are not testing the user--we are testing the application, and there are no wrong answers.
Ask the user to think aloud whenever possible.

Greeting script

Hi [participant's name]. I'm [your title] with the Fluid Project. The Fluid Project is a project aiming to build more inclusive, usable web experiences for everyone. Today we are looking at ways to help tweak and customize your experience of interacting with a museum-based, touch-screen kiosk and associated keypad.

This is a test of our designs; we are not testing you. If you find something difficult or unintuitive to use, chances are that others will as well. This test of the design is simply a means of evaluating our work and to discover any issues we need to address.

The study is composed of three parts: we'll start with an undirected, observational test where you can explore the keypad without any functional interaction with an interface. The second part is a keypad functionality test where the keypad will be connected to a simplified interface. After each section we'll ask you a few questions about your experience, and then end the testing period with a demographics questionnaire.

The study will take about 60 minutes, and you can stop at any time during the study for any reason. We will answer any questions you have now and at the end of the study.

Do you have any questions before we start?

First we'll need you to read over and sign this consent form.

2. User testing

Context

Before the test begins the user will be provided with information about the context of the keypad as well as a representative screen-shot of a kiosk interface.

The context description will include the following:

that the goal of the keypad is to provide alternative input controls for touch-screen kiosks that will be installed in the Canadian Museum of Human Rights
that the kiosks are an integral part of the interactive exhibits in the museum which present information about various aspects of human rights in Canada and beyond
that keypads will be installed on the right-hand, bottom side of touch-screen kiosks which are designed at a set height and angle
that interaction with both the keypad and the touch-screen will be possible in the museum, but for the purposes of our testing there will be no touch screen functionality
that the keypad for testing is a prototype only, such that
- the final key material will be different (but will likely be made of hard plastic with a smoother surface)
- the force required and physical/audio response to pressing the keys may be different in the final product
- the keys in the production keypad will likely be illuminated (either at all times or to indicate active keys)
- the size, height and shape of the keys will be the same (dependent on the results of this testing)
- the layout of the keys will be the same (dependent on the results of this testing)
- the current screen-reader content is for testing purposes only, and in many cases is non-specific

a) Undirected "Observational" Testing

Test focus: The goal of this phase of the testing is to determine whether the function of each key in the keypad is clear to the user, as well as to determine if the design of the keys or layout of the keypad introduces any concerns or problems for the user. This phase of testing will allow us to observe the initial reaction of the user upon engaging with the keypad for the first time.

Introduction

Let the user know that the goal of this phase of the testing is to allow them to explore the keypad in order to determine if the function of each of the keys is clear, and to get any feedback they may have about the design of the keys and the keypad layout.

Let the user know that during this phase of testing the keypad will not be functional (that is, pressing the keys will not activate any response), but that the user is free to press the keys as desired.

Test

First, the user will be encouraged to observe the keypad, in both a visual and/or tactile manner, without any specific guidance. The user will be encouraged to verbalize any observations or questions that come up and these will be recorded.

Next, the user will be asked to name the function of each of the keys as best they can (if they haven't already). If they are uncertain of any of the functions, they will be asked to take a best guess, and this uncertainty will be recorded.

After the observational period is over, the user will be asked to provide any further feedback or observations about the keypad that they might have.

b) Directed "Keypad Functionality" Testing

Test focus: Basic functionality of the keypad.

Introduction

In order to isolate keypad functionality testing from issues related to the graphic interface structure of the museum interactives (which is outside the scope of this project), we will limit this phase of the testing to interaction with a highly simplified navigational interface based loosely on the interface mockups provided by the museum

Test

Inform the user that the interface is a very simplified version of what will be seen in the museum, and is only for the purposes of testing keypad functionality. Navigatable/selectable items will be highlighted with focus-state boxes - these do not represent the final design (design of focus state TBD). Furthermore, many buttons, though focussable, are not currently active; selecting these buttons will result in no action (we'll let you know what these are). Inform the user that this audio content is for the purposes of testing only and will be different in the final product.

Testing will proceed as per the following:

open wireframe home page and let the user know that only the English language content is available at this time
ask the user to complete the following tasks:
Task 1:
- a) Find the video and information related to Mass Atrocities story #4
- b) Now choose a different story to explore
- c) Go back to Welcome/first page
Task 2:
- a) Navigate to the Geography view page
- b) Go to the Stories view page
- c) Go back to Geography view page
Task 3:
- a) Turn on the screen reader
- b) Increase the volume
- c) Decrease the volume
- d) Turn off the screen reader
Task 4:
- a) Zoom in on the page
- b) Return to original zoom level

Post-Test Questions

At this time the user will be asked a series of post-test questions and the answers should be recorded (audio recording or note-taking).

Post-test questions will include the following (if the user has not already answered them in their feedback):

How would you describe your overall experience and comfort in using the keypad?
Were you at any point confused about what each key does?
The following are specific questions about the key and keypad design:
- Do you have any thoughts about the grouping of the keys? Would you group them differently?
- Is the shape of the key helpful in distinguishing the function of that key?
- Are the raised symbols on the keys helpful in distinguishing the function?
- How do you feel about the size of the keys?
- How do you feel about the distance between each key?
- If you need additional instructions on how to use the exhibit, what would you do?
Are there any functions you wished to perform that you couldn’t? (that didn’t appear to be available to you through the keypad?)
Would you add a key or take away a key and if so which key?
Do you have any other concerns or observations about the design of the keys or the layout of the keypad?
What do you like about the keypad?
Although the design of the interface is not in the scope of our work, we may be able to provide some input into the final design. Do you have any ideas or feedback about the interface design or navigation features?
If you could design an alternative input device for the kiosks (keypad or not), what would it be?

3. Post-test demographics questionnaire

1. If you use any custom devices to use computers (even glasses), list them here: ________________________

Technology questions

2. In general, what is your comfort level with technology?

Very comfortable
Comfortable
Uncomfortable

Finishing Up

Ask the user if he/she has any questions, thank the user for participating, and give an honorarium if we have one.

On this page

Test Results

Browser not supported