Mobile user testing protocol, notes (Engage 0.3)

Test Objectives

This formative study is designed to gather usability and usage data about the Engage 0.3 Mobile application. More precisely, the objectives are:

To measure the usability of the application (efficiency, effectiveness and user satisfaction);
To measure the usefulness of the application for the different categories of target users;
To measure the desirability of the application;
To measure the performance of the audio & video content (podcasts) as tools of cultural mediation;
To inform the design of next iterations of the application;
To serve as a benchmark for futur user evaluations.

Type of Study

This study is a field experiment (also known as "in the wild", "quasi-realistic" or "in-situ" experiment). A field experiment is equivalent to a laboratory study transposed in the actual context of usage of the application (in this case, a specially prepared section of the permanent exhibition of the McCord Museum).

Engage 0.3 test sessions will be moderated and recorded, and use a mix of free reign and directed tasks. Field experiments like this one cannot be considered ecologic since they usually involve a moderator, directed tasks, partial functionality and some type of visible recording (audio / video / note taking). A reality experiment (ecologic) is considered for the Engage 0.5 user testing. This testing phase would be truely free reing, and users would be observed from afar, without their knowledge.

Components of this study

Pre-test questionnaire (demographics)
Free reign exploration
Directed tasks (tasks that were not completed during free reign)
Post-test questionnaire
Exit semi-structured interview

Participants

Participant Characteristics

Characteristic	Desired Number of Participants
Domain knowledge
Knowledgeable in canadian history / museums (includes museum professionals, museology or history teachers & students, history buffs)
No particular knowledge of canadian history / museums
Familiarity with Mobile Technology
Familiar with mobile technology (owns or has previously used a mobile device)
Unfamiliar with mobile technology
Place of Residence
Greater Montreal Region
Outside of Montreal Region
Age
0-11
12-17
18-21
22-26
27-40 (ATTENTION: there's a gap between this and the next range)
51-61
62+
Gender
Male	10
Female	10
First Language
English
French
Other
Total number of participants	20

Additional participant characteristics (like internet use, experience with audio guide, etc) will be collected via the background questionnaire but will not influence the selection of participants.

About the required number of participants

If we segment the target audience by Domain Knowledge and Familiarity with Mobile Technology, we obtain 4 distinct classes of users :

	Domain Knowledgeable	Not Domain Knowledgeable
Familiar with Mobile Technology
Unfamiliar with Mobile Technology

There is an agreement that for a formative study, 4-5 participant per main user category is sufficient, hence the current target number of participants (20).

"In our experience, five participants per significately different class of user is usually enough"
Measuring the user experience, Tullis & Albert, p.119

"for the purpose of conducting a less formal usability test, research has shown that four to five participants who represent one audience cell will expose about 80 percent of the usability deficiencies of a product for that audience, and that this 80 percent will represent most of the major problems."
Handbook of Usability Testing, Rubin & Chrisnell, p.72

Recruiting Strategy

Participant will be recruited among :

the McCord museum personnel & volunteers;
immediate network of family and friends;
museum Facebook fans;
on site museum visitors.

Evaluation Method

Situation Scenario

Data Collection

Logistics

User tests will take place in a specially prepared section of the Permanent Exhibit.

The participant will be be accompanied by a moderator and a note taker / audio / video recorder.