(Floe) Universal Subtitles user testing report (February 2012)

This report gathers results and suggestions coming from Universal Subtitles user testing.

Summary of results

Design issues found in testing

On the landing screen
- Most users find no issues and enter the video URL into "Subtitle a video now" field.
- Some of the users are a bit confused and note that there's no information about where to put the URL, but intuitively paste it into the "Subtitle a video now" field.
- A user clicks on "Subtitle a video now" button without entering any URL. He eventually enters the URL in the error page.
On the subtitling stage screen
- Most users focus directly in the video image, without paying attention to the other elements in the page.
- Most users take some seconds to discover what to do.
- All the users click on the "Subtitle me" option below the video.
- When selecting Spanish language, some users have been confused because "Spanish" appeared at the top of the list, while Spanish variations appeared much below, in alphabetical order (in the "S"). Other times, Spanish appears in its alphabetical order. There's no apparent reason for this change.
- The "embed" option is clearly understood by experienced users, but not by the others (the don't recognize what "embed" means).
- Some users don't notice the options at the left of the screen, and search ways to embed or share the video in the right side of the page, around the video preview.
- When a user clicks on some of the options above the video, he doesn't fully understand their contents.

On the "typing" screen
- General comments:
  - There has been observed a different user's behavior depending on the previous experience in captioning or video-editing software. Experts users almost don't read the instructions, whilemost of the non-expert read them carefully.
  - Captioning/video-editing software experts seem to expect that the application will behave the same way that their usual software. In most cases, they go on trial and error until they learn the differences between the expected and the real application's behavior.
  - Some users (mainly the experts in technology) use the keyboard to play/pause the video, and some others (mainly the less experts) click on the play and replay buttons. Both ways (keyboard and buttons) are useful.
  - The captioning/video-editing software experts tend to use the play/pause in the video bar, or Tab or Space keys to play/pause the video.
  - All the users click on the "Done? Next step..." button to continue. Only one of them clicks on the numerical buttons at the top of the screen.
- About the instructions:
  - A contradiction has been found: says "press play" to start video in one place, and "press Tab" in another.
  - A user comments that the screen area spent on the instructions is excessive. He also comments that the text is too long.
- About saving:
  - A user says that she would login right away to don't risk loosing the work done.
  - Another user is very annoyed by the fact of not being able to save and continue. The "save and exit" button is really disturbing for him, because he'd like to save and continue working.
- About subtitling process:
  - 8 seconds interval is too long for some users, but they can not select any smaller interval.
  - Some users like to type all the text without breaking (they write now and expect to break and sync later).
  - Almost any user has understood clearly the function of the text field border turning red, when subtitle goes too long. When they try to understand this effect, they give different arguments, and sometimes they are right, but not sure of their interpretation. The most of them review the instructions looking for some explanation.
  - Some of the users type looking at the keyboard while listening the video, so they don't notice that the subtitle is becoming too long.
  - When the application automatically creates a new line (a too long subtitle), the users are really confused. They don't know why that new line was created if they had not pressed Enter.
  - A user suggests having more than one line visible while editing.
  - A user suggests having an audio alert when the subtitle is going too long (he types without looking at the screen).
  - Clicking on the subtitle to edit is very intuitive for all the users.
  - Most users understand without difficulties the function of the "+" (add new subtitle) and the "x" (delete) buttons.
  - All the users find easy to split a subtitle or gather two of them, using the copy/paste and add new subtitle/delete options.
  - Several users expect being able to drag the subtitles to reorder them.
  - Several users expect to go directly to a video segment when they click on the corresponding subtitle.
  - A few users have spent some time to find the text field to type the subtitle (they comment that it's not enough visible).
  - A few users confuse the "+" button with a handle to drag the subtitle.
  - A user doesn't find how to copy/paste the contents of a subtitle.
  - A user doesn't find the subtitle after pressing Enter (she doesn't see the subtitle below).
On the "syncing" screen
- General comments
  - As in previous screen, video editing experts don't read the instructions, while most others read them.
  - Almost all the users (excepting one) understand clearly the purpose of this screen, even if they haven't read the instructions.
  - A user suggests having a label "subtitle 1", "subtitle 2" to provide a more clear idea of what the list of subtitles is.
- Inserting the subtitles
  - Most users press the down arrow when next subtitle must appear. They do even if they haven't read the instructions, because this command is very visible in the screen.
  - Some users look anxious when press the down arrow too early or too late. They usually don't pause the video, and continue confused until the end.
  - Two users are confused by blank subtitles entered by mistake. Instead of realizing that these subtitles are empty, they ask themselves why the subtitle doesn't appear on the video, if they are pressing the down arrow.
  - A user comments that would be great if he could click the mouse when next subtitle should appear (alternative to the down-arrow key).
  - A user comments that pressing the down arrow to make the subtitle appear can be felt as playing a competition with the computer (but she doesn't like competing).
- Reviewing sync
  - Most users review syncing in this screen, without waiting to the next step.
  - For most users, fine adjustment is confusing at the beginning, but they learn pretty quickly on a trial and error process.
  - Most users express pleasing surprise when discover that when move a caption's timing on the timeline, it automatically shifts the start or the end of the other.
  - Many users search how to review the sync looking at the subtitles list, and spend some time on this.
  - A few of them see the left/right arrows to move the position of the subtitle. When eventually discover the draggable boxes in the timeline, they look relieved because is much more intuitive.
  - Two users try to edit the number between the left/right arrows to move the position of the subtitle.
- Moving the timeline
  - Most users try to drag the numbers in the timeline to move back or forward the video. They are confused when see that the position of the timeline has no relation with the play header in the scrubbing bar.
  - Depending on the position of the cursor, the user can not drag the numbers in the timeline. He must be very precise.
  - Some users (specially those experts in video-editing software) try to drag the red line to move the video playback. They are frustrated when discover that the line is not draggable.
  - Some users miss having an option to rewind the video.
On the reviewing screen
- Most users don't see any relevant difference between this screen and the previous one (the "syncing" screen).
- Most users appreciate having an opportunity to review the work done, and being still able to adjust any mistake.
- Most users comment that this screen is too similar to the previous one. Although appreciate being still able to make modifications, don't see much point to this step.
- Some users comment that they expected to have a screen without an editing interface, simply to preview what the video would look like with the subtitles.
- Some users comment that having the whole interface it's a bit distracting to focus on the video.
- Some users notice some mistakes and fix them in this screen. They also do some fine adjustments. The process of fixing and adjusting is quick, because is the same that have learnt before.
- A few users have downloaded subtitles. These meet their expectations (note from the facilitator: the subtitles window is really small, so the users always must resize the window to see the text).

Design issues found in post-test interview

1. Describing the overall experience

For most users, the overall experience is good.
Most users look satisfied.
Most users find the process reasonably straightforward.
Some users felt disoriented at first, not knowing what to do.
A user comments that the interface is nice and simple. Appreciates not having too many buttons.
Most users (implicitly or explicitly) find the division between three stages useful.
Some users say to be confused by the 3rd screen.

2. Found confusing, difficult or frustrating...

Most users find difficult scrubbing on the timeline.
Many users feel annoying that the scrubber doesn't seem to sync with the video.
Many users want to be able to move the red line to a specific time code.
Some users want to be able to type the time code manually (in the subtitle field).
Some users find the instructions too long: too much text to read. Suggest more visual instructions. A user said that he didn't read main instructions, just the color-coded visuals (note from the facilitator: expert users tend to read only the color-coded visuals)
Some users comment that using the Tab to play the next video segment is quite confusing at the beginning.
A user found a number of the buttons and clickables too small using a mouse.
A user comments that the 8 seconds interval is too long, and that she should be able to adjust it.
A user comments that adjusting the subtitle with the handles is difficult.

3. Users suggestions on how to improve the application

Let the user save the work and continue (without exit)
Make the look-and-feel of the website and the application more consistent.
Add some instruction in the first screen (in the "Subtitle a video now" field)
Add different colors for the subtitles, to distinguish two people talking.
Add an acoustic signal to alert when a subtitle is getting too long, for people that type looking at the keyboard.
In the 3rd step, put the subtitles directly over the video (keeping them editable) so the video can be bigger.

Technical issues

On the subtitling stage screen
- In the list of language to translate, "Spanish" appears sometimes at the top of the list (below English) and sometimes at the "S" position (heading all the Spanish variations).
- In some occasions, the application indicates that the original language is English although the language is another.
- In some cases, the original language can not be modified (appears as a normal text, not as a drop-down menu).
On the "typing" screen
- Some videos from Youtube disallow embedding (not known if the same occurs for other video sources). No error message appears, so the user is not able to see the video, but doesn't know why.
- In one occasion (Barcelona user 1) and with no apparent reason, the video plays without pausing after 8 seconds.
On the "syncing" screen
- Seems that in one occasion, the video play automatically when the user get to the syncing screen.
- In one occasion, the last subtitle disappears between the step 1 and 2.
- Clicking on the timeline and then pressing play again resets the play to the beginning (observed in the Review Screen).

Detailed user testing notes

The complete notes from user testing are available at http://wiki.fluidproject.org/display/fluid/Universal+Subtitles+user+testing+results+%28January+2012%29

Design suggestions

General issues

Increase consistency between the website and the application.
Increase visibility of the main actions in the website (add new translation, improve subtitles,...)
Make sure that all the text fields contain suggestions to help the user understand what he's expected to do (e.g. "Subtitle a video" field)
Unify the way to return to the home from any screen.

In the application

Allow the user to save and continue.
Allow the user to customize keyboarding.
Allow the user to freely select the play segment time (lower than 8 seconds).
The color change is not enough clear to alert that a subtitle is going too long.
Add acoustic alert when a subtitle goes too long.
Note that some users like to write all text at once, and then break it into segments.
Enable visualization of two or more lines in the text field. The text field should be elastic to accomodate the subtitle text.
To determine when next subtitle should appear, enable mouse click as an alternative to down-arrow key.
The third step should consist on a preview screen, although the user must still be able to edit and sync the captions.
Question whether the distribution in 3 steps is really useful.
On "download subtitles", increase the default size of the text field.
Clarify what "submit subtitles" implies.
For technical jargon as "embed", add a description.

Instructions

Keep brief the instructions.
Make the instructions more visual.
Attach the instructions to the interface elements.
Tutorials should be available as a help option (not intrusive).

Subtitles and timeline

Enable color change for subtitles (to distinguish different speakers).
Simplify the relationship between timeline and scrubbing bar.
Sync the timeline and the scrubbing bar.
When a subtitle is selected, play the video to its time code.
Make the time code in the subtitles editable.
Enable precise selection of a time code.
Enable dragging subtitles to reorder.

On this page