(Floe) Universal Subtitles user testing protocol

This page contains the protocol for Universal Subtitles user testing.

See also: 

Methodology

Session structure

The session is composed of three parts:

  1. User testing
    a. Undirected, "reality" testing
    b. Task-based testing
  2. Post-test interview
  3. Post-test questionnaire (demographics)

The first part of the session, user testing, is composed of two tests. During undirected testing, the user is given a selection of (short) videos to watch, and selects one for subtitling. The user is then sent to the home page of Universal Subtitles (US), and asked to attempt subtitling the selected video without any other instruction from the moderator. This part of the test is intended to study the instinctive, uninstructed use of the application without priming.

In the second part, task-based testing, the moderator asks the user to fulfill specific tasks, specifically, tasks pertaining to the primary intended design/aims of the US. This part of the test is intended to study the usability and intuitiveness of specific features of US, and the relationship between the different areas in the website.

After the user testing is complete, the user is interviewed about his/her experience with the application, including problems encountered, potential areas of improvement, and possible new functionality. This interview is followed by a demographics questionnaire.

Data gathering

Data for user testing will be gathered using the following tools:

  • Audio and video recording
  • Screen recording + eye tracking
  • Pen & paper notes

Data for the post-test interview will be gathered using the following tools:

  • Audio recording
  • Pen & paper notes

Materials

ALERT: Deleting subtitles is an action that requires administrator-level permissions. It is not available through the user interface. The moderator must be sure that the videos selected for each testing session have not been subtitled previously. The moderator must be sure to have at least as many videos as participants + 1.

0. Protocol Introduction

General things to keep in mind for the test

  • Reassure the user that we are not testing the user--we are testing the application, and there are no wrong answers.
  • Ask the user to think aloud whenever possible.
  • Probe them on expectations, frustrations, and general thoughts.
  • Avoid influencing the user's decision-making and deflect questions to gain further insight into the user's thoughts (e.g., "What does this do?", "What do you think it does?").
  • Don't offer help; let the user attempt to perform the task themselves. If they ask for help reply with:
    • "What do you think you should do?"
    • "What do you think that means/would do?"

Greeting script

Greeting script

Hi [user's name]. I'm [your title] with the Floe project. Today we are looking for ways to improve the user experience of Universal Subtitles, a tool on the web used to add subtitles to any video.

This is a test of Universal Subtitles design; we are not testing you. If you find something difficult or unintuitive to use, chances are that others will as well. This test of the Universal Subtitles is simply a means of evaluating the design and to discover any issues we need to address.

The study is composed of four parts: we'll start with the main part of the study where you get a chance to play with Universal Subtitles. Then you will be asked to perform a few specific tasks. Afterward, we'll have a short interview about your experience, and wrap that up with a quick demographics questionnaire.

The study will take about 45 minutes, and you can stop at any time during the study for any reason.

Do you have any questions before we start?

1. User Testing

1A. Undirected testing

Instructions for user (script)

There are a few browser windows open here, each on a video clip. Go ahead and watch the clips, and select one that you'd like to subtitle.

<Wait for user to watch and select a video>

Now go ahead and subtitle this video with Universal Subtitles.

<Wait for user to reach subtitling application screen>

By looking at the screen in front of you, what can you surmise about how you should add subtitles to your video? What is your expectation of how this works?

Instructions for moderator

Give the user a selection of two videos to view (make sure that these two videos have not been subtitled before). After the user has selected a video to subtitle, send the user to the Universal Subtitles home page. Ask the user to go ahead and work toward subtitling the video they selected.

SKIP THE TUTORIAL VIDEO.

At the subtitling screen, ask the user to explain how they think the subtitling process might work, in a general way (i.e., probe them on their initial mental model of US).

Ask the user to go ahead and subtitle the video. While the user is paused/typing, probe the user's thoughts. Avoid probing the user while the video is playing.

1B. Directed testing

Instructions for user (script)

Now we'll be asking you to complete a few tasks using the application. As before, please speak your thoughts aloud. Let us know what pops out at you first, what you're looking at, what you're reading, what you think the different parts of the screen does, what you're expecting to happen, what options you're thinking of and how you come to your decision. Let us know if anything seems awkward or doesn't seem to flow the way you'd expect or want it to.

Instructions for moderator

From the full task list, ask user to complete tasks that weren't completed during the previous reality testing part. You may also want to repeat a task you missed observing during reality testing.

Task List

Activity breakdown: 1) general navigation; 2) creating new subtitles + editing; 3) editing another user's subtitles; 4) translating existing subtitles; 5) "other" (social, profile, etc.)

Preparing the process

  • Select a video to subtitle (the moderator will provide an online video to perform the test)
    You want to subtitle this video. How would you do this using Universal Subtitles?
  • Choose the language to subtitle
    You want to subtitle the video in English. How will you do this?

Typing

  • Write the subtitles
    Go ahead and add subtitles to this video (the moderator waits until the user ends up writing all the subtitles)
  • Missing some subtitles
    You've missed a few words to subtitle near this mark (point to specific time). How would you fix that?
  • Split one of the subtitles
    This subtitle (point to a specific one) is too long. How would you go about fixing that?
  • Correct typographical errors
    This subtitle (point to a specific one) has some spelling mistakes. How would you go about fixing that?
  • Go ahead in the process
    Now that the subtitles contain no mistakes, what would you do next?

Syncing

  • Sync the subtitles
    Try to sync the subtitles with the video (since there are various ways to sync the video, the moderator must observe what is the process followed by the user)
  • Save the work
    Now you need to do a pause. How can you be sure that you won't lost the work done? 

Reviewing

  • After syncing, you see that two subtitles are too short. Gather them into one.
    You realize that these two subtitles (point to them) are too short. Try to convert them into one. 
  • Add periods at the end of each paragraph (the purpose is to observe how the user edits the text in this phase)
    Now review the subtitles and make sure that each paragraph ends with a period. 
  • Go directly to a subtitle to review it.
    I think that there's some mistake in the second subtitle, can you go directly there and review it?
  • Review the syncing to make fine adjustments.
    Now check if the subtitles sync well with the video, and make all the adjustments you need (the moderator must observe if the adjustments are done using the scrubbing bar, or the time codes in the subtitles)
  • Ask the user to complete the subtitling process
    Now that all the subtitles are ready, what would you do next?
    (If the user clicks on "Submit the work" the moderator must explain him that he won't be able, since he's not logged. Check if the user is able to return to the initial page)

Sharing

  • Embed this video in your web page.
  • Imagine that you have a website, and you want to include this video in one of its pages. How would you do it?
  • Email a link to this video.
    This video can be interesting for a friend, so you want to email him the link. How would you do it?
  • How would you send the subtitles to someone to translate them into another language? (download the subtitles)
    You know a French translator, and want to send her your English subtitles in a document so she can prepare the French version. How would you do this?
  • Correct a mistake that you have seen after sharing the video. 
    Now you see that there's a mistake in the third subtitle. How would you correct it?

Adding a new translation

  • Start a new translation.
    Now you want to prepare a translation in Italian. How would you start?
  • Write the translation (does he use the automatic translation option?)
    The fact is that you just know a little Italian, but you want to try to write the translation. How will you do this?
  • Your professional translator has sent you a document with a new translation. How would you add this to the video? (does he use the option to upload subtitles?)
    Your French translator has already sent you the doc with the French subtitles. How will you add this version to the video?

2. Post-Test Interview

Use the following questions as guidance during the semi-structured interview. Probe deeper into any key issues that arise.

  1. How would you describe your experience subtitling using Universal Subtitles overall?
  2. What did you find confusing or difficult to understand about subtitling?
  3. What did you find difficult or frustrating?
  4. What might we do to improve the application?

3. Post-Test Questionnaire (Demographics)

General demographics

1. What is your gender?

  • Male
  • Female

2. What is your age group?

  • 12-17
  • 18-25
  • 26-35
  • 36-45
  • 46-55
  • 56-65
  • 65+

3. What is your occupation or main activity? ________________________

4. What is your primary language? ________________________

5. If you have any disabilities that affect your use of computers, list them here: ________________________

Technology questions

6. In general, what is your comfort level with technology?

  • Very comfortable
  • Comfortable
  • Uncomfortable
  • Very uncomfortable

7. How often do you watch video clips (e.g., embedded into blogs, news sites, YouTube, Vimeo, etc.) or movies on the web?

  • Very often (7+ videos a week)
  • Often (4-6 videos a week)
  • Occasionally (1-3 videos a week)
  • Not often (less than once a week)

8. Where do you watch these video clips? Check all that apply.

  • Video clip sites (YouTube, Vimeo, etc.)
  • Blogs
  • News sites
  • TV or movie sites (e.g., Netflix, Hulu, local/national TV stations, etc.)
  • Other: ____________

8. How often do you use closed captions or subtitles on videos? (e.g., when watching a video in a foreign language, or in a noisy environment, or in an environment where you can't play audio)

  • I use CC/subtitles for most videos
  • I use CC/subtitles from time to time
  • I use CC/subtitles on the rare occasion
  • I never use CC/subtitles

9. Have you ever added, created, or edited captions or subtitles for videos before? If so, which tool(s) did/do you use?

  • Yes: ________________________
  • No

10. How often do you use video editing software? (e.g., iMovie, Final Cut, Adobe Premiere, etc.)

  • More than once a week
  • Once a week to once a month
  • Once a month to once a year
  • Less than once a year

Finishing Up

Ask the user if he/she has any questions, thank the user for participating, and give an honorarium if we have one.