Mobile user testing protocol (Draft 8)
General things to keep in mind for the test
- Reassure the user that we are not testing the user--we are testing the designs, and there are no wrong answers.
- Ask the user to think aloud.
- Ask the user to think about what they would do in the scenario, without necessarily being tied down to the particular device/system being tested (i.e., if they would use a different resource, let them lead us through that).
- Don't offer help; let the user attempt to perform the task themselves. If they ask for help reply with:
- "What do you think you/that would do?"
- "What do you think that means?"
- Start with general, non-leading tasks; gradually lead to the user to more specific tasks with specific screens to test screens and functionality that the user hasn't naturally gravitated to.
- One facilitator, one note-taker.
Greeting script
Hi [user's name]. I'm [your title] with the Fluid Engage project. Fluid Engage is a project aiming to build software for museums to create engaging visitor experiences. Today we are looking for ways to improve the user experience of a museum mobile application.
This is a test of the mobile design; we are not testing you. If you find something difficult or unintuitive to use, chances are that others will as well. This test of the mobile design is simply a means of evaluating the design and to discover any issues we need to address.
If you feel uncomfortable you can stop at any time during the study.
Please speak all your thoughts aloud as you go through the tasks. Let us know what pops out at you first, what you're looking at, what you're reading, and what you think the different parts of the screen do. This helps us better understand what sorts of choices our designs lend to. Let us know if anything seems awkward or doesn't seem to flow the way you'd expect it to.
The study will take about 30 minutes. We will answer any questions you have at the now and at the end of the study.
Do you have any questions before we start?
First we'll need you to sign this Contributor License Agreement.
Let's get started!
Demographic questionnaire
General demographics
Gender:
- Male
- Female
Age:
- 18 - 29
- 30 - 39
- 40 - 49
- 50 - 64
- 65 or older
McCord's age divisions:
- 6-13
- 14-18
- 19-24
- 25-44
- 45+
What is your occupation? (if student, what is your field of study?)
What is your comfort level with technology?
- Uncomfortable
- Somewhat uncomfortable
- Neither comfortable nor uncomfortable
- Somewhat comfortable
- Comfortable
Museum experience
How often do you visit a museum or a gallery?
- Regularly (at least once a month)
- Occasionally (once a month to once a year)
- Seldom (once a year to once every two years)
- Hardly ever (less than once every two years)
Have you ever used a museum's mobile application, or a museum's website created specifically for mobile?
- Yes
- No
If yes, what museum was it for? How was your experience? Would you use it again?
Mobile experience
Do you own a smartphone or internet-enabled mobile device? (e.g., iPhone, iPod touch, Blackberry, any phone with a touch screen or QWERTY keyboard)
- Yes
- No
- Don't know
How often do you use it, for any purpose? (calls, music, applications, web browsing, etc.)
- A few times a day
- A few times a week
- A few times a month
- Hardly ever
How often do you surf the internet or use some internet-enabled application (e.g., Facebook, Twitter, mail, etc.) using your mobile device?
- A few times a day
- A few times a week
- A few times a month
- Hardly ever
Usability test
<Actions for mediator in blue and within angle brackets.>
Tasks in green.
[Reflective questions in swamp green and within square brackets.]
Pre-visit scenario
You are at a bus stop, waiting for the bus to come, and are thinking about visiting the McCord Museum of Canadian History sometime this week. You pull out your iPhone, go to the McCord Museum's website, and are greeted by their mobile webapp.
<present language selection screen>
General tasks:
What do you do to decide whether or not you really want to go?
Suppose now that you've decided to go later today or this week. How would you prepare, if at all, for your visit?
Specific leading tasks, if not already covered by general tasks:
- What do you do to learn about the exhibitions that are on right now?** [Are you satisfied with the depth/breadth of information shown? What else would you want to know about the exhibition? What did you not care for?]
- Find out more about an exhibition you're interested in.** How much will it cost you to see the exhibition?
- What sorts of artifacts, objects, or works are in that exhibition?*** Suppose you find a few artifacts that are really interesting that you want to make sure you see when you get there. How would you remember to do this? How would you make a list of artifacts you really want to see when you get to the museum?
- Can you find out what others thought about the exhibition?
- What events are coming up related to that exhibition?
- How do you find out more about the museum? e.g., where it is, and what time it opens and closes, and how much it costs.
In-museum scenario
Now suppose you arrive at the McCord Museum of Canadian History. The museum provides free access to an online mobile application, and you browse to it on your iPhone to learn more about it. Here is what it looks like:
<present the language selection screen>
- Suppose you want to see whether there are any events happening today. How do you go about doing this? How would you go about reminding yourself about the event's time?
- [if the user "collected"/"bookmarked" artifacts for pre-visit planning] How do you go about finding the artifacts you bookmarked from your pre-visit?
You are walking through one of the exhibitions, and find a particularly interesting object that captures your attention.
<present picture of artifact with label; make sure label has object code!>
General task:
Intrigued by the object, how do you go about learning more about it?
Leading tasks:
Task |
Potential screens |
|---|---|
How would you go about finding more about the object using the mobile device? |
Browse sequence |
Now that you've found the object and have read a bit more about it, you want to mark it somehow so you can come back to it in the, maybe at home. How would you do this? |
Favorites |
The object reminds you of something, and you decide to share a note about it for other visitors or potential visitors to read. How would you do this? |
Tagging |
You're really interested in this object, and would like to see other objects related to it. How do you go about doing this? |
Related artifacts |
Is there anything you can listen to or watch about this artifact? How would you go about doing that? |
A/V panel |
Is there more than one image you can look at on this artifact? How would you go about doing? |
Artifact image viewer |
Suppose you the image is too small. How would you go about seeing this artifact, but bigger? |
Artifact image |
This object is really exciting, and you want to somehow share the experience with your friends. What do you do? |
"Send" |
Post-visit scenario
You leave the museum feeling satistied. On your way home, you are still playing with the mobile app. You remembered some of the objects you "collected", and remember seeing some objects in the same area that you didn't collect but, in retrospect, would have liked to.
Task |
Potential screens |
|---|---|
How would you go about finding the objected you collected? |
My Collection |
Exit questionnaire
How was your overall experience using this application?
- Very pleasant--it's great!
- Not bad--I approve.
- Didn't like it--needs improvement.
What would *you* use this application for? Choose as many as are appropriate.
- Planning a trip to a museum
- Research
- Fun
- Social
- Education/learning
- Other:
What sorts of features or functionality would you have liked to see in a mobile app like this? (e.g., what's missing?)
What would you have changed in this app? (e.g., what can be improved? What would make it more fun or engaging?)
How useful do you think it is to "collect" artifacts, objects, and paintings?
- Very useful! I'd use it frequently.
- Useful. I'd use it sometimes.
- Not very useful. I wouldn't use it very often.
- Not useful at all. I'd probably never use it.
How useful do you think it is to see artifacts, objects, and paintings other people "collected"?
- Very useful! I'd use it frequently.
- Useful. I'd use it sometimes.
- Not very useful. I wouldn't use it very often.
- Not useful at all. I'd probably never use it.
How useful do you think it is to tag artifacts, objects, and paintings?
- Very useful! I'd use it frequently.
- Useful. I'd use it sometimes.
- Not very useful. I wouldn't use it very often.
- Not useful at all. I'd probably never use it.
How useful do you think it is to see the tags of artifacts, objects, and paintings?
- Very useful! I'd use it frequently.
- Useful. I'd use it sometimes.
- Not very useful. I wouldn't use it very often.
- Not useful at all. I'd probably never use it.
How useful do you think it is to comment on exhibits?
- Very useful! I'd use it frequently.
- Useful. I'd use it sometimes.
- Not very useful. I wouldn't use it very often.
- Not useful at all. I'd probably never use it.
How useful do you think it is to read comments other visitors left for exhibits?
- Very useful! I'd use it frequently.
- Useful. I'd use it sometimes.
- Not very useful. I wouldn't use it very often.
- Not useful at all. I'd probably never use it.
How useful do you think it is to comment on artifacts, objects, and paintings?
- Very useful! I'd use it frequently.
- Useful. I'd use it sometimes.
- Not very useful. I wouldn't use it very often.
- Not useful at all. I'd probably never use it.
How useful do you think it is to read comments other visitors left for artifacts, objects, and paintings?
- Very useful! I'd use it frequently.
- Useful. I'd use it sometimes.
- Not very useful. I wouldn't use it very often.
- Not useful at all. I'd probably never use it.Any other thoughts or ideas?
Anything else you'd like us to know?
Finishing up
Ask the user if he/she has any questions, thank the user for participating, and give an honorarium if we have one.