Mobile user testing protocol (Draft 9, Engage 0.3)

Owned by Tona Monjo
Last updated: Dec 08, 2011 by James YoonVersion comment
8 min read

Methodology

Session structure

The session is composed of four parts:

1. User testing
   a. Undirected, "reality" testing
   b. Task-based testing
2. Post-test interview
3. Post-test questionnaire
4. Post-session survey

The first part of the session, user testing, is composed of two tests. During undirected testing, the user is given the mobile device with the application pre-loaded and asked by the moderator to roam freely throughout the museum exhibition space with the device in hand as he/she pleases. The moderator should avoid influencing the user's decision-making and deflect questions in such a way to gain further insight to the user's thoughts (e.g., Q: "What does this do?" A: "What do you think it does?"). This part of the test is intended to study the instinctive, uninstructed use of the application without priming.

In the second part, task-based testing, the moderator asks the user to fulfill specific tasks, specifically, tasks pertaining to the primary intended design/aims of the application. This part of the test is intended to study the usability and intuitiveness of specific features of the application.

After the user testing is complete, the user is interviewed about his/her experience with the application, including problems encountered, potential areas of improvement, and possible new functionality. This interview is followed by a demographics questionnaire covering some basic vitals, museum experience, and technological savvy.

Finally, after the session, the user is emailed a request to fill out an online survey about their experience with the application, and some specific features.

Data gathering

Data for user testing will be gathered using the following tools:

  • Audio and video recording: videographer following the user around during his/her session, capturing interaction between user, device, and space, and capturing user's facial and upper body expression
  • Screen recording: recording of interactions on the screen (using VNC + Camtasia)
  • Pen & paper notes

Data for the post-test interview will be gathered using the following tools:

  • Audio recording
  • Pen & paper notes

0. Protocol Introduction

General things to keep in mind for the test

  • Reassure the user that we are not testing the user--we are testing the designs, and there are no wrong answers.
  • Ask the user to think aloud.
  • Don't offer help; let the user attempt to perform the task themselves. If they ask for help reply with:
    • "What do you think you/that would do?"
    • "What do you think that means?"
  • For the pilot, we'll have one facilitator/note-taker, and one remote screen observer/task recorder.

Greeting script

Greeting script

Hi [visitor's name]. I'm [your title] with the Fluid Engage project. Fluid Engage is a project aiming to build software for museums to create engaging visitor experiences. Today we are looking for ways to improve the user experience of a museum mobile application.

This is a test of the mobile design; we are not testing you. If you find something difficult or unintuitive to use, chances are that others will as well. This test of the mobile design is simply a means of evaluating the design and to discover any issues we need to address.

The study is composed of four parts: we'll start with a demographics questionnaire, and then move on to the main part of the study where you get a chance to play with a functioning prototype of our mobile museum application. Afterward, we'll ask you to complete a questionnaire about the experience, and, finally, have a short interview about your experience.

The study will take about x minutes, and you can stop at any time during the study for any reason. We will answer any questions you have now and at the end of the study.

Do you have any questions before we start?

First we'll need you to read over and sign this consent form.

Let's get started!

1. User Testing

Part A: "Reality" Testing

See Section 0, "General things to keep in mind for the test".

Instructions for user (script)

In this part of the test, we'll give you a mobile device running our prototype application on it. For the next 10-15 minutes, we'd like you to use the application and walk around the space as you feel befitting.

Please speak your thoughts aloud. Let us know what pops out at you first, what you're looking at, what you're reading, what you think the different parts of the screen do, what you're expecting to happen, what options you're thinking of and how you come to your decision. This helps us better understand what sorts of choices our designs lend to, and where our designs are working or coming short. Let us know if anything seems awkward or doesn't seem to flow the way you'd expect or want it to.

Instructions for moderator

While the visitor goes about on his/her exploration, mark down the completed task lists.

Part B: Task-based Testing

Instructions for user (script)

Now we'll be asking you to complete a few tasks using the prototype application. As before, please speak your thoughts aloud. Let us know what pops out at you first, what you're looking at, what you're reading, what you think the different parts of the screen do, what you're expecting to happen, what options you're thinking of and how you come to your decision. Let us know if anything seems awkward or doesn't seem to flow the way you'd expect or want it to.

Instructions for moderator

From the full task list, ask user to complete tasks that weren't completed during the reality testing step. You may also want to repeat a task you missed observing during reality testing.

Task list

Language selection

You need to change the language of the application. How would you do this?

Finding and viewing artifacts

Approach the picture Flood, Bonaventure Depot, Montreal. Point it out.
How would you go about finding more about this artifact using the mobile device?

Collecting an artifact
You want to mark this object somehow so you can come back to it in the future, maybe at home. How would you do this? <Collect a few>

Uncollecting an artifact
Suppose you want to remove one of the artifacts from your collection. How would you go about doing that?

Reading full description of Flood, Bonaventure Depot, Montreal.
How would you find a more comprehensive description of this artifact?

Collapsing full description.
How would you hide the full description?

Reading comments on artifact.
You'd like to know the opinion of other visitors about this artifacat. How do you go about doing this?

Add a comment to an artifact page.
Now you want to add a comment. How do you go about doing this?

Approach curling stone (#09). Entering code.
How can you find information about this object?

Accessing a movie clip (curling stone).
Is there something to watch for this object? How would you go about watching it?

Viewing a movie (curling stone).
Go ahead and watch the video.

Approach hockey trophy (#10). Accessing a movie clip.
Is there something to watch for this object? How would you go about doing that?

Viewing a movie (hockey trophy).
Go ahead and watch the video.

Accessing related objects (hockey trophy).
Suppose you want to see other objects related to this trophy. How would you go about doing that?

Approaching the foot warmer (#06). Watching a movie clip.
Go ahead and watch the video for this object.

Exhibitions

Browsing through a catalogue of a current exhibition.
Suppose you'd like to see a list of the artifacts in the exhibition. How would you go about doing this?

"My Collection"

Viewing "My Collection" with artifacts in it.
How can you check if your collection contains any objects?

Going to an artifact in one's collection.
How can you retrieve more information about a particular artifact in your collection?

Sending the collection to an email address.
Suppose you want to send your collection to your email. How would you go about doing this?

Conclusion of the visit

Reading comments in an exhibition.
Suppose you'd like to know what other visitors thought of the exhibition. How do you go about doing this?

Add a note to the guestbook of an exhibit.
Suppose you'd like to share your opinion about this exhibition with other visitors. How do you go about doing this?

2. Post-Test Interview

Use of the application

1. How easy did you find it to use the application?
2. How long did it take to understand the application?
3. What sort of difficulties or problems did you have in using this application, technical or otherwise?
4. What can we do to improve the application?
5. What sorts of features would you have liked to see in a mobile application like this?

Visit experience

6. Were you able to make discoveries with this application? If yes, what were they?
7. What do you think about visiting museums with this type of smartphone/device? Was it helpful? If yes, how?
8. You saw some video clips about objects (curling stone and brooms, hockey trophy, foot warmer). Which one(s) did you prefer?
9. The clips associated with these objects are presented differently. Which type(s) did you prefer?
   a. The one about curling, with images featuring the use of the objects
   b. The one about the hocky trophy, showing details of the object
   c. Any of the three others, with fixed images

Feelings about the application

10. What is your general feeling about the application?
11. Would you like to revisit the museum with the tool once the application is developed?

3. Post-Test Questionnaire (Demographics)

General demographics

1. What is your gender?

  • Male
  • Female

2. What is your age group?

  • 12-17
  • 18-25
  • 26-35
  • 36-45
  • 46-55
  • 56-65
  • 65+

3. Where do you live?

  • Montreal area
  • elsewhere in Quebec
  • elsewhere in Canada
  • United States
  • Europe
  • Other: ____________

4. What is your occupation or main activity? ________________________

5. What is your primary language?

  • English
  • French
  • Other: ____________

Museum experience

6. Before today, have you ever been to the McCord Museum?

  • Yes
  • No (and skip to question #8)

7. Have you ever visited the Simply Montreal exhibition?

  • Yes
  • No

8. Have you been to the McCord Museum website?

  • Yes
  • No (and skip to question #10)

9. Do you have a My McCord account?

  • Yes
  • No

10. How often do you visit museums or heritage sites?

  • Never
  • Less than once a year
  • 1-2 times a year
  • 2-5 times a year
  • 5+ times a year

11. In general, do you use audio guides when it is offered for your museum visits?

  • Yes
  • No

Mobile experience

12. In general, what is your comfort level with technology?

  • Very comfortable
  • Comfortable
  • Uncomfortable
  • Very uncomfortable

13. Do you use any of these applications/technologies on the web? Please check all that apply.

  • Facebook
  • Flickr
  • Twitter
  • Youtube
  • RSS feeds

14. Do you own a smartphone or an Internet-enabled mobile device?

  • Yes
  • No (and skip the remaining questions)

15. What kind of device do you own?

  • iPhone
  • iPod touch
  • Blackberry
  • Android
  • PDA
  • Other: ____________

16. Do you use it in order to go on the Internet? (e.g., check your emails, Facebook, Twitter, etc.)

  • Yes
  • No
  • I don't know

4. Post-Session Survey

1. Did you receive your collection of objects, which was sent to you by email?

  • Yes
  • No

2. In general, do you think it was easy to find objects from the "Simply Montreal" exhibition in the application?

  • Yes
  • No

3. How useful do you think it is...

Q.

Very useful! I'd use it frequently.

Useful. I'd use it sometimes.

Not very useful. I wouldn't use it very often.

Not useful at all. I wouldn't use it very often.

... to retrieve additional information about an object on display?

 

 

 

 

... to watch video clips?

 

 

 

 

... to see objects related to the one selected?

 

 

 

 

... to collect an object?

 

 

 

 

... to send your collection by email?

 

 

 

 

... to comment on an object?

 

 

 

 

... to see comments other visitors left for objects?

 

 

 

 

... to comment on exhibits?

 

 

 

 

... to see comments other visitors left on exhibits?

 

 

 

 

3. Would you want to rate an object by using a scale such as "Like" or "Dislike"?

  • Yes
  • No

4. Would you like to see other features in this mobile application? If so, what sort of features?

  • Yes
  • No

5. Does this mobile application change your experience of visiting a museum? If so, how?

  • Yes
  • No

Finishing Up

Ask the user if he/she has any questions, thank the user for participating, and give an honorarium if we have one.

On this page