Page Comparison

This page gathers the results of Universal Subtitles user testing, performed in Toronto (Canada) and Barcelona (Spain) in January 2012. 

To complete this information, see also: 

• (Floe) Universal Subtitles user testing protocol
• Universal Subtitles user testing report (with the summary of results and design suggestions)

Toronto, Canada

Toronto, user 1 (from the pilot)

Demographics profile

• English-speaking female software developer, 36-45
• Very comfortable with technology
• Doesn't often watch video clips on the web
• Watches video clips mainly from YouTube and news sites
• Uses closed captioning/subtitles for on the rare occasion
• Has used captioning/subtitling software before (CapScribe)

Testing data

• On the landing screen:
  • No issues with entering video URL into "Subtitle a video now"
• On the subtitling staging screen:
  • Initially saw left side column (metadata and some social features), but ignored as it didn't seem relevant
• On the "typing" screen:
  • Carefully read instructions on the side, and concluded with, "It looks like you just type to subtitle, and press enter when you're ready."
  • Confused by contradiction between instructions: says press play to start video in one place, and press 'Tab' in another
  • Fiddled a bit with speed mode, and found options confusing/surprising
  • Saw the notice indicating you need to login to save
    • Normally, wouldn't risk losing data, so would login right away--wouldn't want to waste time risking not doing it
  • User attempted subtitling two videos. In the first video, YouTube disallowed embedding (this was not known to the facilitator prior to testing). User pressed 'Tab', but to no effect. Kept pressing 'Tab', but again to no effect. No error messages or messages of any kind appeared. Remainder of testing notes for this user hereon apply to a second video, on a video where YouTube allowed embedding.
  • User's understanding/mental model of how it's going to work: hit the tab key to play 8 seconds of video, and then it stops
  • User consistently notes throughout the typing experience that the 8 second interval felt too long
    • "Some people might type faster, but I wish I could choose a smaller interval of time."
    • After a couple of intervals, looked at other speed options, but didn't change as she felt the others must be more harder to use (they weren't labeled as "Beginner", as the default 8 second interval was)
  • Successfully discovered and used 'Shift-Tab' to play previous interval again without any prompting
  • User typed for a while without breaking, and the application created a new line; user noticed it
    • "I don't remember pressing enter, but it seems to have created a new line. Is there a limit to the number of characters?"
    • Eventually noticed that the border around the text field turns red after typing--interpreted as a warning that she's reaching the limit of characters
    • "Is there a way to adjust the [text] breaks afterward? This isn't the way I'd actually like to break it up--I'm just typing 8 seconds at a time."
      • At the end of each interval, user reviews and edits typographical errors
      • After editing an interval, user replays because she sometimes forgets if she captured all the text
  • 8 seconds is way too long!
  • "Would be nice if it showed more than one line while editing."
  • Once the user was done going through the video once:
    • Reviews the work, looks over the typed text
    • Hovers over the text, and noticed the "+" button
      • "Is that for inserting new text? And the 'X' is for deleting?"
    • "How would I go back and play that 8 seconds again?"
    • Experimented with clicking on the text--pleased that it puts her into edit mode so she can fix typos
    • In reviewing her text, noticed that she must have missed a word in a sentence. Not sure how to play video again for that text.
    • Plays the video again from the start to do a full review of the subtitles, even before syncing
      • Hopes that it does play in 8 second chunks when she uses the play button on the regular video player... but it does stop at 8 second intervals.
    • Noticed music afterward, was so focused on text when she was first typing it in
      • Successfully edited and added that in ("[music playing]")
    • Some text feels too long, but can't seem to insert line breaks with this interface
      • Eventually used injections of new subtitles with text editing to break it up successfully
    • At some point, when reviewing, the video had paused after an 8 second interval with a perfect break in a sentence, and the user was waiting for it to continue, forgetting that there were 8 second intervals.
  • On the "syncing" screen:
    • Read instructions thoroughly
    • User has experience with other video editing software before, and based on instructions, thinks she just needs to press down arrow whenever the subtitles appear
    • Occasional accidents:
      • Pressing it too early
      • Pressing it too late
      • Tapped down twice while the video was playing
      • User is unsure whether these can be fixed
    • User observes that when thinking about when to press the down arrow, she's looking at the subtitles list and listening to the audio, not looking at all to the video or timeline
    • After a run through, the user is reviewing the placement of subtitles on the timeline
      • Tries to drag the timeline, but is unexpectedly realizing that it has nothing to do with the scrubbing bar above--or is it? Confusion about relationship. Green bar on scrubber doesn't line up with the timeline. The two seem inconsistent.
    • User spending a good deal of time reviewing and editing the subtitle sync, iteratively
      • During this time, user expresses being increasingly comfortable with the interface, what can and can't be done
      • Noticed how if you move a caption's timing on the timeline, it automatically shifts the start or end of the other (before, the user was manually changing the adjacent one first to make room for the caption of interest)
    • By the end of syncing stage, finding that she's becoming increasingly comfortable with the controls, and has a good sense of what can and can't be done.
  • On the "reviewing" screen:
    • Before getting to the screen, there was an expectation that "review" would be a screen without an editing interface, simply there to watch and preview what the video would look like with the subtitles.
    • "Looks like the same interface, again. I just did all this."
    • "Might as well check it one last time."
    • During final review, was focusing on the timeline as in the syncing phase--corrected herself and focused on video afterward
      • Found the timeline and other interface a bit distracting when she was supposed to be focusing on the video
    • Found an error in her subtitle syncing, and fixed it
      • "I take that back--now I'm glad the controls are here."

Post-test interview

• Overall experience not bad, found it reasonably straightforward
• Previous video experience may have helped, but not sure how it'd be for another user without video experience
• Found it easier to use than Capscribe, which she's had prior experience with
• Found the interface nice and simple, appreciated not having too many buttons
• Found the default 8 second intervals too long, and not being able to adjust that was a downer
  • Really wishes she could adjust the timing
  • Also wishes there were a way to stop the playback yourself, to play it until your brain reaches capacity
  • Didn't know what to expect in some of the other typing options like "Magical autopause"
    • Expectation is that it would wait for vacant pauses and pause automatically for you
• Instructions were reasonable, but had to sit and read all of it. There was a lot to read. Instruction visuals could be improved.
• Scrubbing on the timeline was difficult--area to click/drag was very small.
• Scrubber doesn't seem to sync with the video--why is that?
• Felt like she should've been able to grab the blocks of subtitles in the timeline to move them.

Toronto, user 2

Demographics profile

• English-speaking male web designer, 36-45
• Mobility issues affects use of computers
• Very comfortable with technology
• Watches video clips on the web very often
• Watches video clips from video clip sites, news sites, and TV/movie sites
• Never uses CC/subtitles
• Never added/created/edited CC/subtitles before
• Uses video editing software less than once a year or never

Testing data

• Landing screen:
  • Successfully inserted URL into "Subtitle now" field without issues
• Staging screen:
  • Took a few moments to scan page, then clicked "Subtitle me" beside the video
• Typing screen:
  • Took several moments to read instructions carefully
  • First interaction was to press 'Tab', without putting cursor focus anywhere; second interaction was to repeat the interval using 'Shift-Tab'
  • Plays video, listens, and types at the same time
    • Successfully using 'Tab' to play 8 seconds, but often using 'Shift-Tab' to replay last 8 seconds
    • While typing, focus is on the keyboard, not on the screen
    • Filled the text field to its limit without pressing enter, despite the red indicator
    • Attention is still primarily on the keyboard--user has not noticed that a new text field has been created after the previous one was filled
      • Upon noticing on a glance up at the screen: "I didn't expect the text to jump down"
  • Upon finishing entering all text, user presses 'Tab' to play again and review what he's done. Presses 'Tab' after each 8 seconds without breaking, but doesn't appear to be annoyed by it
  • Upon finishing review, presses 'Enter' to submit text that was left in the field (did not think to submit it after typing the text the first time; it was sitting in the field until the very end of the review)
• Syncing screen:
  • Played the video before attempting any syncs, and decided she "did it wrong--should have broken the text up" (in the previous stage, she had not pressed enter after entering text; the system had automatically broken up the text)
    • Went back to the typing screen (no issues going back), successfully broke up text by inserting new text fields, and cutting and pasting text into it.
  • No issues with understanding and doing syncing (successfully sync'd by pressing down arrow), but a number of subtitles weren't well-sync'd
  • Did not play again to review
• Review screen:
  • Successfully moved subtitles on timeline bar to edit syncs
  • Workflow consisted of playing the video, pausing just after when the next subtitle would appear, and resize the subtitle bar on the timeline to just before the pause point
  • After making all the edits, played all over again to review
    • "The hiccup at the beginning is annoying" (takes a moment for the subtitles to sync in)
    • Spent time doing micro-edits down to the split second for syncing both the start and end of a subtitle
  • Reviewed three or four times in total
    • After making a single edit, user continues to play instead of going back to watch and review the edit he just made
    • User finishes making a round of edits before reviewing again

Post-test interview

• Found the experience "Pretty good overall"
• Found the division between the three stages useful, and can't think of how one would change it
• In the syncing stage, found pressing the down arrow easy, and liked not having to fumble around with the mouse, or having to look at two places on the screen (e.g., video and the button to sync)
• On the automatic splitting of text in the typing field: found it unexpected, but in hindsight makes sense. At the time, user was worried about getting everything typed, and not so much about the structure.
• Cutting and editing of text into sentences made sense with the video and audio--found it pretty easy to do, and understood how to insert text (using the '+' button). It made sense.
• Found a number of the buttons and clickables too small using a mouse

Toronto, user 3

Demographics profile

• English-speaking female student, 18-25
• Comfortable with technology
• Watches video clips very often on the web
• Watches video clips mainly from video clip sites and TV/movie sites
• Uses CC/subtitles on the rare occasion
• Has used captioning/subtitling software before (CapScribe)
• Uses video editing software between once a month to once a year.

Testing data

• Landing screen:
  • User notes that it doesn't say where to put the URL, but presumes she can copy and paste it into the text field
• Staging screen:
  • User trying to figure out what to do on this screen, spends several moments scanning and reading page
  • Asks, "Where did the other page go?"
  • After several moments, presses 'Subtitle me' and continues
• Typing screen:
  • Reads instructions casually and not thoroughly
  • Presses 'Play next 8 seconds' and presses it again to pause (notes that it should maybe read "Play or pause")
  • Types in text, and presses enter
    • When the text disappears from the video, asks, "Where did the text go?" (did not see subtitle panel below)
    • Clicked on scrubber to where she thinks the text should go, and starts typing the text again, but doesn't press enter yet
    • Presses play to continue, with text in the field and on the preview screen
    • User getting frustrated, and doesn't know how to move ahead
    • Reading instructions carefully now
      • "Key controls below? What are those?"
    • Thinking to herself: "Press enter for a new line sounds like putting a new sentence under an existing line, and not a new subtitle..."
  • <facilitator injected instructions on pressing enter for each new subtitle, and how it appears in the subtitle panel below>
    • "Okay!" (user understands now)
    • Manually clicking 'Play next 8 seconds' and clicking again to pause--filling up her mental memory of content
    • User consistently tweaking the scrubber to where she thinks the text should be entered
    • Notices change of colours when the lines are getting long, understands and acknowledges text length indicator
  • Presses play to review the subtitles, but the subtitles don't appear--clicks next to move on
• Syncing screen:
  • User plays the video immediately without reading instructions, expecting the captions to appear, but none appear
  • User is confused and reading instructions more carefully
    • "Oh--would've been nice to know that earlier"
  • Noticed that she missed a subtitle, and tried to use '+' button, thinking it'd insert a place for entering subtitles into the video player above, but instead it just entered a blank in the subtitles panel
    • Ignored, and tried to start syncing by playing and using down arrow, but it didn't seem to be working (it was inserting the blanks from previously)--no luck
    • Stopped trying to use the down arrow during play, and instead selected a subtitle and pressed down--no luck
    • Tried to move the subtitles in the timeline using the anchors--no luck
    • Getting frustrated again
    • Tried deleting blanks from subtitles panel to see if that helps, and then retried syncing by playing and using down arrow, and that worked
      • Started to get into a flow, "Kind of like a video game, pressing down at the right time. But, I don't play video games, and it's getting all competitive with the computer."
        • Can see it being fun for some people, but different than what she's used to doing
    • Eventually figured out that added subtitle blanks were meant to be edited in the panel, and not on the video player
  • On trying to correct syncs:
    • Clicked on subtitle, and pressed down, expecting it to sync to the current time on the video--no luck
    • Randomly clicking on the timeline, trying to find out where/how to sync and cut subtitles
      • Success with click and drag of subtitle edges; initially tried to click it to see if it's react to her somehow, but no feedback
      • Editing via resizing the subtitle bars on the timeline was reassuring and familiar
  • On trying to add a subtitle to the very end: had difficulty finding how to do it, but eventually found the separate "Add subtitle" link. Noted that it was very different from the rest.
  • User looks like she's done, with nothing else to do.
• Reviewing screen:
  • User eventually pressed next.
  • User played the video once more, and said she was done.

Post-test interview

• Good experience overall, but was initially very frustrated. Didn't know how to begin.
• Felt that the instructions were not explicit enough
• Felt that better instructions, with some visuals, would make it easier to learn
• Once she played around and figured out how it works, felt it was pretty easy
• It was like a video game

Toronto, user 4

Demographics profile

• English-speaking male instructor, 36-45
• Low vision affects use of computer
• Comfortable with technology
• Often (4-6 times/week) watches videos on the web
• Watches videos on video clip sites, blogs, news sites
• Uses CC/subtitles from time to time
• Has never added/created/edited captions/subtitles before
• Uses video editing software between once a week and once a month

Testing data

• Pre-landing screen:
  • User using browser zooming to increase magnification of everything
• Landing screen:
  • User successfully/instinctively copied and pasted video URL into textfield on landing page
• Staging screen:
  • After a quick scan, user clicked on the "Subtitle me" button
• Typing screen:
  • Read instructions casually, did not spend much time reading it
  • Started playing the video, and started typing into the appropriate textfield
    • Used tab to pause midway to finish typing
    • User is not on manual pausing mode (user is in default automatic 8 second interval pausing mode)
    • After unpausing the manual pause, video automatically paused at completion of 8 second interval--user pressed tab to continue
    • User finished subtitling very, very quickly--single, unproblematic run through
  • User started reading instructions more carefully to get an idea of what to do next
• Syncing screen:
  • <did the video play automatically when he got to the syncing screen???>
  • User reading instructions thoroughly
  • Started to play to place syncs, but missed start of first subtitle; pressed "Restart this step" to try again
    • First two attempts failed
    • On third attempt, successfully figured out how to sync
  • Restarted and did a fourth attempt because he wanted to get it perfect
    • Still had a few mis-syncs
• Review screen:
  • Decided one of his subtitles was too long
    • Divided the subtitle into two successfully (using '+' twice, deleting the original, and some text editing)
    • But, has sync'd the new subtitles yet
  • Making successful edits of punctuation and capitalization
  • Using the spacebar to play/pause
  • Successfully making micro-adjustments in syncing by resizing bars
  • "Would love to be able to drag that timeline"
    • User tried clicking and dragging the red marker (both line and arrow) on the timeline without success
    • Clicking on the timeline, and then pressing play again resets the play to the beginning <bug?>
    • User expresses wanting to be able to click on the timeline to be brought to that time code
    • User expresses wanting to type in time codes to jump to a specific time
  • User downloaded subtitles--text file of subtitles met expectations

Post-test interview

• User found the experience overall surprisingly easy
• Having instructions nearby was convenient
  • Did not read main instructions, mostly ready the colour-coded visuals (keys, etc.)
• Liked it having step-by-step
• No hesitation at any point that something was wrong
• Question to facilitator: "If it were dialogue-based audio, how would you position subtitle for persons speaking offscreen?"
• Subtitle previews on video were readable
• On the timeline:
  • Really want to be able to move around on the timeline (click on timeline and be brought to that frame; move back and forth on timeline, etc.)
  • And type in time codes manually
  • <facilitator explained that users were intended to click and drag on red line>: makes sense now, but make that more explicit

Barcelona, Spain

Barcelona, user 1 (from the pilot)

Demographics profile

• Spanish-speaking male project manager (high relationship with video editing software), 36-45
• Very comfortable with technology
• Views video clips on the web often (4-6 videos a week)
• Watches video clips mainly from YouTube, Vimeo and blogs
• Uses closed captioning/subtitles for on the rare occasion
• Has used captioning/subtitling software before (doesn't remember its name)
• Uses video editing software one a month to once a year (Premiere, After Effects)
• Test performed on a Mac

Testing data

• On the landing screen:
  • No issues with entering video URL into "Subtitle a video now". Finds the option quickly.
• On the subtitling staging screen:
  • Goes directly to the "Subtitle Me" option, to select the language. Doesn't pay attention to the rest of the screen.
  • He doesn't find the "Spanish" option to subtitle into this language, and selects "Spanish, Mexican" (through the various tests, it's observed that in some cases "Spanish" appears next to the Spanish variation and in other cases appears at the top of the drop-down menu, just below English. Doesn't seem to be any reason for this variation)
• On the "typing" screen:
  • Looks over the instructions on the side -without reading them, excepting the text on the buttons-, and starts by pressing Tab. This participant uses the keyboard during all the test, doesn't seem to perceive the green and blue buttons as clickable. 
  • After pressing Tab, the video starts to play, but after 8 seconds doesn't pause. The user is confused, because the instructions specify clearly "Play next 8 seconds". Pauses the video pressing Tab key again.
  • Tries pressing the Shift+Tab and the video re-plays from the beginning. 
  • Tries again the Tab from the beginning, and this time it plays only 8 seconds. The user looks confused. 
  • The user begins to subtitle, using the Tab key to play/pause the video at his own pace, without waiting for the pause of 8 seconds.
  • Takes some seconds to find the field to start typing, "Oh ok, I was looking where to enter the text". 
  • He understands that now he only must type the text, because the button "Done? Next Step: Syncing" indicates him clearly that he will adjust subtitles in another screen. 
  • Doesn't understand the change of the background color of the text field when entering a too long subtitle: "I see that is changing to red, but I don't know why... Well, usually red indicates that there's some mistake... Maybe I should save the work I'm doing...". 
  • Then he sees the option "Save and Exit", but feels uncomfortable: "Much better if I had an option to save and continue". 
  • Finally he reads the instructions and realizes that the subtitle is too long. He doesn't divide it at the moment, just hits "Enter" and continues subtitling. 
  • Before clicking on the button to the next step, says he would like to save, to protect the work done. But insists that he doesn't like the idea of "save and exit", because he wants to continue working. 
• On the "syncing" screen:
  • Now he looks carefully the new screen. He sees that a timeline has appeared, but the rest of the interface is apparently the same (recognizes the context). He sees clearly that the purpose of this step is to sync the subtitles. 
  • Starts by pressing Tab to play the video, and when hears the narration, pauses the video and drags the first subtitle to the starting point (in the timeline). Continues dragging the second subtitle. He seems to be following a trial-and-error behavior to understand how the scrubbing bar and the subtitles boxes work. Hasn't noticed the instruction "tap when next subtitle should appear" (didn't read the instructions when entered in this step). 
  • Says to be confused because he expected to be able to move the timeline by dragging the frame numbers: "Obviously the video has its control here (points to the video bar), but now it's not only the video, it's the video + the text". He tries dragging again the timeline and it works (seems to have an irregular behavior, the dragging seems not to be precise). 
  • Says to be confused, and continues trying by dragging the subtitles boxes, using the handlers.  
  • Finally he decides to read the instructions, and then realizes that he can press down-arrow to make appear the next subtitle. Comments that he was trying to follow the same behavior that he used in another subtitling application, but now notes that this have a different procedure. After this, the process of syncing becomes much faster. 
  • When reaches the end, the user realizes that a subtitle entered in the previous screen is missing. He adds a subtitle (using the option at the end of the subtitles list), types again this subtitle and places it <Seems that a subtitle has been lost from the Step1 to the Step2>
  • The user adjusts the position of the subtitles in this same screen, dragging the subtitles boxes in the scrubbing bar. 
  • The user realizes that a subtitle is too long, and adds a subtitle (using the option at the end of the subtitles list), cuts a segment and copies it into the new field. 
  • The user is prepared to go to the next step, but insists on saving the work before. 
• On the "reviewing" screen:
  • When entering in this screen, the user comments "I'm trying to see what is the difference between this screen and the previous one". 
  • He sees that he must adjust a subtitle that was not well placed, and drags it. 
  • When asked about adding a subtitle between two existing ones, he sees the "+" button and uses it without any difficulty. 
  • Sees that he can correct typing mistakes by clicking on the field. 
  • Now realizes that he can also adjust the position of the subtitles with the left-right arrows that appear on mouse over a subtitle. 
  • Once the adjusting process is finished, the facilitator asks him what to do next. He says that he would download the subtitles (with the Download Subtitles button) to save them. He likes how this works.  
• Back to the subtitling staging screen:
  • When asked about including the video in a blog, he directly points the "embed" option. 
  • When asked about sharing the video with a friend, he directly points to the sharing options.
  • When asked about correcting a mistake, the user doubts, and finally points to the list of languages in the "View Subtitles" section. But when he clicks on a language, he doesn't see the way to do a correction. 

Post-test interview

1. How would you describe your experience subtitling using Universal Subtitles overall?
   A bit confusing, because I don't understand the 3rd step (refers to the "reviewing screen")
2. What did you find confusing or difficult to understand about subtitling?
   Adjust the subtitle location with the handles. 
3. What did you find difficult or frustrating?
   Anything. 
4. What might we do to improve the application?
   Let the user save the work without being forced to exit.

Barcelona, user 2 

Demographics profile

• Catalan/Spanish-speaking female web designer, 36-45
• Comfortable with technology
• Views video clips on the web ocasionally (1-3 videos a week)
• Watches video clips mainly from YouTube, Vimeo and blogs
• Uses closed captioning/subtitles for most videos
• Never has used captioning/subtitling software before
• Uses video editing software less than once a year
• Test performed on Windows

Testing data

• On the landing screen:
  • No issues with entering video URL into "Subtitle a video now". Finds the option quickly.
• On the subtitling staging screen:
  • Goes directly to the "Subtitle Me" option, to select the language. Doesn't pay attention to the rest of the screen.
  • Selects "Catalan" as language to subtitle. No problems found. 
• On the "typing" screen:
  • Reads carefully the instructions on the right side. 
  • Starts typing the subtitles. Uses the "Enter" key to send a subtitle and start a new one. No problems found. The user learns how to type subtitles immediately. 
  • In an occasion, the user enters a too long subtitle. When asked by the moderator about the changing background color, she says "I don't know what is this for. I didn't realized that this color was changing!". The user doesn't understand the meaning of the background color change.  
  • Without being asked, she realizes that two subtitles are too short, and gathers them into one by copy/paste and deleting the leftover subtitle. 
  • When asked by the moderator to correct some mistake, she directly clicks on the subtitle and edits it. No problems found. 
  • When asked to split one of the subtitles, she initially has some problems to find how to add a new subtitle. She looks confused by the "+" option, because she expected that the new subtitle was created after the existing one, not before. But once she checks it, understands it without difficulties. Then she follows the copy/paste sequence to split the contents of the subtitle. 
  • She clicks on the button to go to the next step.  
• On the "syncing" screen:
  • She reads carefully the instructions. Initially says being confused by the instruction about pressing Tab when next subtitle appears. However, she tries it and starts to sync the subtitles using the tap, without any problems: "Ok! Now I understand what I must do". 
  • After a syncing all using the Tab option, she starts to make fine adjustments. She sees immediately that the subtitles boxes in the timeline are draggable. She drags the subtitles to adjust it. Says this is an easy method. 
  • The user misses having an option to rewind the video. She finds this lack very disturbing, because she returns to the beginning several times to review the adjustments. 
  • Once finished, she clicks directly on the button to go to the next step.  
• On the "reviewing" screen:
  • She reads the instructions and observes carefully the screen. When asked by the moderator to think aloud, she says "I don't see the sense of this screen. It looks like the previous one". 
  • When asked if she still can correct a mistake in one subtitle, she clicks on it and says: "Yes, I can correct it, and delete it or add a new one. The same things that I did before".
  • The moderator explains the user that she won't be able to submit the subtitle because she's not logged on, and heads to the subtitling staging screen.
• Back to the subtitling staging screen:
  • When asked about including the video in a blog, the user takes some time to think and finally clicks on "URLs" at the top of the video. She doesn't see (or doesn't understand) the Embed option at the left. 
  • When asked about sending a link to the video, the user takes again some time to think and finally says that she doesn't know how to do it. 
  • At the end, the moderator indicates that there two options are at the left side of the page:
    - M: "Haven't you seen them?" 
    - U: "No, I was focusing on the video area (the right side of the page), but I didn't expect to find anything about the video in this side (pointing to the left and the top of the page). That area is for me about the tool, not about the video. For me, elements related to the video are just the ones I see around it."

Post-test interview

1. How would you describe your experience subtitling using Universal Subtitles overall?
   I like it! It's useful and easy 
2. What did you find confusing or difficult to understand about subtitling?
   The last screen (refers to the subtitling staging screen, because of the problem that she explained with the different areas of the screen and its functionality)  
3. What did you find difficult or frustrating?
   Scrubbing, but I've learned quickly.
4. What might we do to improve the application?
   The last screen, when you have done all (refers again to the subtitling staging screen). And graphically, the look-and-feel of the website and the tool is too different.

Barcelona, user 3 

Demographics profile

• Catalan/Spanish-speaking male developer, 26-35
• Very comfortable with technology
• Views video clips on the web often (4-6 videos a week)
• Watches video clips mainly from YouTube, Vimeo, and also from the mobile
• Uses closed captioning/subtitles on the rare occasion
• Never has used captioning/subtitling software before
• Uses video editing software once a month to once a year 
• Test performed on a Mac

Testing data

• On the landing screen:
  • No issues with entering video URL into "Subtitle a video now". Finds the option very quickly.
• On the subtitling staging screen:
  • Goes directly to the "Subtitle Me" option, to select the language. Doesn't pay attention to the rest of the screen.
  • Selects "Spanish" as language to subtitle. Spends some seconds to find Spanish, because he sees that there are some variations of Spanish available (Mexican,...) but doesn't find Spanish. Finally he finds it at the top of the list, below English. 
• On the "typing" screen:
  • Doesn't spend any time reading the instruction, but sees quickly the options to Play next 8 seconds, and to Replay. 
  • Starts typing the subtitles. Uses the "Enter" key to send a subtitle and start a new one. No problems found. The user learns how to type subtitles immediately. 
  • The user comments "I thought that I had to type and sync at the same time, but I see that this doesn't work so. I'll sync it later". This doesn't seem to annoy him, it's just a comment. 
  • Continues typing all the subtitles with no problem. 
  • When asked to split a subtitle, he directly adds a subtitle, clicks on the original one, cuts a segment and pastes it into the new subtitle. No problems found, and quick sequence.
  • When asked to correct some mistake, directly clicks on the subtitle and edits it. No problems found.
  • When finished, he clicks on the button to go to the next step.  
• On the "syncing" screen:
  • He takes a quick look at the instructions. Mainly, he searches how to synchronize. When he finds the "Tap down-arrow" button, he immediately starts playing the video and pressing down-arrow each time a new subtitle must appear. 
  • Continues the process with no problems. 
  • The user comments that it would be great if he could also click on the mouse each time he wants a subtitle to appear (an alternative to the down-arrow key option). He thinks it would be more direct. 
  • Once finished, clicks directly on the button to go to the next step.
• On the "reviewing" screen:
  • The user drags the subtitles in the timeline to adjust it. Does it quickly and with no problems. 
  • Comments that when the subtitles handlers are too difficult to drag when they are below the red line (the one that indicates the current time).
  • Finishes the sync process quickly and without doubts. 
  • When asked about gathering two subtitles into one, he directly copies one of the subtitles, deletes it (x button) and copies it into the previous one. 
  • When asked about editing some mistake, he clicks on a subtitle to edit it. No problems found. 
• Back to the subtitling staging screen:
  • When asked about including the video in a blog, the user points to the Embed option. 
  • When asked about sending a link to the video, the user points to the "Share" group of options. 

Post-test interview

1. How would you describe your experience subtitling using Universal Subtitles overall?
   It's fine, logical.  
2. What did you find confusing or difficult to understand about subtitling?
   Maybe the most confusing is using the Tab to play the next video segment. 
3. What did you find difficult or frustrating?
   Anything, maybe the handlers to extend a subtitle. 
4. What might we do to improve the application?
   Anything (finds it alright). 

Barcelona, user 4 

Demographics profile

• Catalan/Spanish-speaking female administrative, 36-45
• Comfortable with technology
• Views video clips on the web often (4-6 videos a week)
• Watches video clips mainly from YouTube, Vimeo, and also from searching results.
• Uses closed captioning/subtitles for most videos (if the video is in English, she uses English subtitles to help her learn this language).
• Never has used captioning/subtitling software before
• Never has used video editing software 
• Test performed on Windows

Testing data

• On the landing screen:
  • No issues with entering video URL into "Subtitle a video now". Finds the option very quickly.
• On the subtitling staging screen:
  • Goes directly to the "Subtitle Me" option, to select the language. 
  • Selects "Catalan" as language to subtitle.  
• On the "typing" screen:
  • Reads carefully the instructions, and then comments "Ok, let's go". Doesn't seem very convinced about knowing clearly what to do.
  • Presses Tab to play the video, and then says surprised "Oh, I though that the subtitles would appear automatically!". She thinks for some seconds and says "Ok, let's type it". She doesn't look annoyed, she looks as if she where building a mental map of the application. 
  • Goes on typing until the end of the video. No problems found.
  • When asked about splitting a subtitle, she adds a subtitle but doesn't find how to copy/paste a segment of the original subtitle into the new one. She finally types the segment in the new subtitle, and deletes it from the original.  
  • When asked by the moderator to correct some mistake, she directly clicks on the subtitle and edits it. No problems found.
  • When finished, clicks on the button to go to the next step.  
• On the "syncing" screen:
  • She starts by reading the instructions. 
  • Plays the video and clicks on the "tap down-arrow" button each time a new subtitle must appear. 
  • Sometimes she has not clicked on the button on time, so she tries to replay last 8 seconds. She finds the play+playback+tap down-arrow key sequence very confusing. 
  • Once finished, she clicks on the button to go to the next step.
• On the "reviewing" screen:
  • She uses the left-arrow/right-arrow buttons (that appear on over the subtitles boxes) to adjust the subtitles. 
  • When asked about the subtitles boxes in the timeline, the user says that she doesn't see clear how to adjust the subtiles there. She finds easier to adjust them with the left-arrow/right-arrow buttons.
  • The user comments: "It would be great that when you click on a subtitle, the video goes there". 
  • After some times of play/playback using the buttons at the right, realizes that she can also scrub the video from the timeline, and finds it much easier.
• Back to the subtitling staging screen:
  • When asked about including the video in a blog, the user points to the "Share" group of options.
  • When asked about sending a link to the video, she doesn't see how to do it (she tries to find the subtitled video URL). 

Post-test interview

1. How would you describe your experience subtitling using Universal Subtitles overall?
   It's been great (despite some difficulties, the user looks satisfied). 
2. What did you find confusing or difficult to understand about subtitling?
   Synchronizing.
3. What did you find difficult or frustrating?
   Anything. 
4. What might we do to improve the application?
   It's alright. Only that the interface is in English and it makes more difficult to understand how it works. And some words are very technical.

Barcelona, user 5 

Demographics profile

• Spanish-speaking female student, 18-25.
• Comfortable with technology.
• Views video clips on the web very often (7+ videos a week).
• Watches video clips mainly from YouTube, Vimeo, and on News sites.
• Uses closed captioning/subtitles for time to time.
• Never has used captioning/subtitling software before.
• Uses video editing software less than once a year (Adobe Premiere)
• Test performed on Windows

Testing data

• On the landing screen:
  • No issues with entering video URL into "Subtitle a video now". Finds the option very quickly.
  • Comments: " I didn't see it very clear, I did it intuitively. It would be better if the text field had a helping text, like "URL" or so". 
• On the subtitling staging screen:
  • Goes directly to the "Subtitle Me" option, to select the language. 
  • Selects "Spanish" as language to subtitle.  
• On the "typing" screen:
  • When entering in the subtitling process, she clicks on the "x" button to skip the tutorial video (returns to the initial page and repeats the process).
  • Once in the typing screen, reads carefully the instructions.
  • She takes some seconds to find the text field: "Oh, it's here! I though I should click on the green area" (refers to the green box with the text "To begin, press TAB to play and start typing")
  • Starts playing the video using the Play button in the scrubbing bar (doesn't use the Tab key). Pauses the same way. 
  • Notices that the text field background is becoming red: " Why is this red? (thinks for some seconds). I guess that the subtitle is too long, isn't it?" When asked why she thinks so, she says "Because of the instruction "don't let subtitles get too long"". She doesn't look very sure about this.  
  • She continues typing until the end of the video. No other problems found. 
  • When she accidentally inserts a blank subtitle, she directly removes it (finds by herself the Delete option without being asked). 
  • When asked by the moderator to split a subtitle, she adds a subtitle by adding a new one, and cutting/pasting a segment of the original. She finds confusing the button to add a new subtitle ("+" button), because of its shape: "I thought it would be for moving the subtitle up, because seems an up-arrow). 
  • She comments: "Would be great if I could reorder the subtitles by dragging them up and down".   
  • When asked to correct a mistake, she directly clicks on the subtitle and edits it. No problems found.
  • When finished, she clicks on the button to go to the next step.  
• On the "syncing" screen:
  • She starts by carefully reading the instructions, twice. 
  • Plays the video and clicks on the "tap..." button each time a new subtitle must appear. No problems found, learns the process immediately. 
  • Reverses the order of two subtitles by adding a new subtitle, copying the text in the old into the new one, and deleting the old subtitle. 
  • She starts to adjust sync using the left-arrow/right arrow buttons (in the subtitles boxes). "I suppose that with this I can adjust to the exact second". 
  • She tries to drag the red line (the one that indicates the video current position), but she looks frustrated because she can't. 
  • She uses the timeline to see exactly when a subtitle should start, and then tries to adjust with the left-arrow/right arrow buttons. Frustrated because she can't edit the time number (at the left of the arrow buttons). 
  • After several attempts, she discovers the handlers in the subtitles boxes (in the timeline) and uses them to adjust: "This is much easier!". 
  • She comments that it's difficult to scroll through the scrubbing bar. 
  • Once finished, she clicks on the button to go to the next step.
• On the "reviewing" screen:
  • She starts this step by playing the video. 
  • She repeats how annoying is not being able to scroll through the scrubbing bar.
  • When asked about this screen purpose, she comments "It's about showing you what you have done. It's a last chance to adjust it". She says "It's useful to make fine adjustments, but much better if the video screen was bigger to let you see you creation, with a button to edit. That is, like a preview before ending. Now this screen looks too similar to the previous one (refers to the syncing screen)".   
• Back to the subtitling staging screen:
  • When asked about including the video in a blog, the users takes some seconds to cover the entire screen, and finally points the "URLs" tab. "Here? I don't know. I guess it's here, but I don't see it". When the testing finishes, the moderator indicates her the "Embed" option; then she comments: "Oh, I see... but it would be easier to understand if it had some explanation". 
  • When asked about sending a link to the video, she points to the "Share" options. 

Post-test interview

1. How would you describe your experience subtitling using Universal Subtitles overall?
   I've found it good, after doing it a couple of times, sure you get familiar with it. At the starting screen, a bit of explanation is needed for some issues, like the URL indication in the "Subtitle a video now" text field.  
2. What did you find confusing or difficult to understand about subtitling?
   Adjusting subtitles and inserting a new subtitle between two others. Also reordering the subtitles, much better if I could drag them.
3. What did you find difficult or frustrating?
   Anything. Maybe at the beginning, I didn't know clearly where to start (refers to the "Subtitle a video now" page). 
4. What might we do to improve the application?
   Make the red line draggable (refers to the line that indicates the video current position in the timeline). 

Barcelona, user 6 

Demographics profile

• Catalan/Spanish-speaking male, digital video expert, 36-45.

Testing data

• On the landing screen:
  • Carefully scans all the screen. 
  • After some seconds, clicks on the "Subtitle" button without entering anything in the text field. 
  • Heads to a screen with an alert about the required field (http://www.universalsubtitles.org/es/videos/create/).
  • Enters the video URL into the text field. 
• On the subtitling staging screen:
  • Goes directly to the "Subtitle Me" option, to select the language. 
  • Notices that the popup indicates that the video's original language is English, although is actually Spanish: "Ops, this is wrong and I can not edit it" <the "video language" field is wrong and not editable>. 
  • Tries to find "Spanish". After some seconds, locates it over the list of Spanish variations: "Selecting the language is always boring". 
• On the "typing" screen:
  • Doesn't read the instructions. 
  • Clicks on "Play" in the video scrubbing bar. 
  • Sees the text field and starts to write immediately. 
  • Notices that the text that he's writing appears also over the video: "Look, it appears here, how cool!"
  • Without entering the field, clicks on play/pause (video bar) several times, to correct typing mistakes. 
  • Clicks on play to continue writing. 
  • Text field start to become red. He doesn't realize because types looking at the keyboard. 
  • When it's too long, the subtitle enters automatically. He doesn't realize and continues typing (the application empties the text field, so the typing process can go on without any problem). 
  • When he looks at the screen, realizes that the text field is becoming red (again), and doesn't know why. 
  • He tries to enter this new subtitle, but sees that the first one entered automatically, and doesn't know what to do now. He's totally confused and is unable to carry this task out: "I don't see this intuitive enough". 
  • The moderator explains him that he should press Enter to save the subtitle (so the process can go on). 
  • When asked about what does he suppose the next step is about, says "To check the subtitles, I guess". 
  • Clicks on the second step button in the top-right margin of the screen (not in the "Done? Next step..." button in the bottom). 
  • During all this phase, he has never used nor noticed the buttons in the right area of the screen (play and re-play 8 seconds). 
• On the "syncing" screen:
  • Very confused. Doesn't know what to do. Doesn't read the instructions.
  • Sees the list of subtitles. Adds a subtitle to put it in the beginning (instead of moving the first one), but sees that the new subtitle appears at the end of the list. Tries to drag it upwards, but he can't. Looks frustrated: "If I could drag the subtitles it would be easier" (refers to the subtitles in the list). 
  • Looks at the timeline and tries dragging the subtitles handlers. He's positively surprised. 
  • Uses the Space bar to play/pause the video. 
  • Doesn't use the down-arrow button any time to sync the subtitles. 
  • Tries to drag the red line to find the exact frame, but can't: "But with video editors, we can". 
  • Looking at the subtitles list, says: "Would be great if a "Subtitle 1", "Subtitle 2"... text would appear, to indicate they are subtitles, because I didn't understand what were these small windows" (referring to the list of subtitles boxes below the video). 
  • Finished adjusting the subtitles from the timeline, dragging the subtitles boxes. When ready, clicks on the "3" button (top-right margin) to continue the process. Before entering the next step, comments: "This reminds me the purchasing process, where you first order, then check and finally pay. It's something similar, isn't it?"  
• On the "reviewing" screen:
  • Scans the screen: "This reviewing step provides me nothing, because we can do the same here as in previous screens. I expected to find a screen where you were asked to save and go".
• Back to the subtitling staging screen:
  • When asked about including the video in a blog, points directly to the "Embed" option.
  • When asked about sending a link to the video, points directly to the "Share-mail" option. 

Post-test interview

1. How would you describe your experience subtitling using Universal Subtitles overall?
   Interesting. I've found it easy because, unlike in a video editor, you only must fill some small boxes (refers to the subtitles boxes) and drag or move it. For other thinks, it's very similar to a video editor. 
   I've missed that the red line (in the timeline) was draggable. 
2. What did you find confusing or difficult to understand about subtitling?
   Marking the initial and final point of a subtitle: press Enter and so... I need some kind of help, because I'm focused on what I'm listening form the video and on writing it. I don't look at the screen when typing, I always look at the keyboard, so I don't know if a subtitle is getting too long. 
3. What did you find difficult or frustrating?
   I don't see that the 3rd step is necessary. You review your work while you are doing it. 
4. What might we do to improve the application?
   I've missed different colors for the subtitles. When there are two people talking we depict it in different colors, for the deaf people. 
   Also, an acoustic signal when the subtitle is getting too long would be helpful, because if you are focused on the keyboard, you don't realize that the field is becoming red. 
   Also, I would prefer having the subtitles directly over the video, so that I can edit directly there. Then, the video view could be bigger. I don't need that much screen percentage spent on the instructions. 
   The draggable boxes (refers to the elements in the timeline) could also appear over the video. You see, I have the subtitle three times, too many (points to the subtitle over the video, in the text field and in the bottom list).  

Barcelona, accessibility test

Demographics profile

• Catalan/Spanish-speaking male developer
• Accessibility expert
• Blind user
• Age 36-45
• Very comfortable with technology
• Test performed on Windows with JAWS

Testing data

• On the landing screen:
  • About accessibility: 
    • The announcement "Universal Subtitles is hiring..." is on the top and is read in the first place. This interferes in the user's conceptual map of the page. 
    • This announcement should have a title or header to let the user know what it is about. 
    • In this page there are two H1. The subtitle should be H2. 
    • In "Most popular" and "Featured videos" there is no H3: it jumps directly from H2 to H4. 
    • "Subtitle a video now": the input has no label. The label could be the same "subtitle a video now". 
    • The search input should have a label, and also a "send" button. 
  • Task performance: 
    • The user has found the field to paste the video URL, and has done the process of copy/paste the URL without any problem. 
• On the subtitling staging screen:
  • About accessibility: 
    • In most of the cases, the video title will be too long to use it as the page title. The page title should be other, more descriptive.
    • The page title should appear before the instruction box, to inform the user where he is located.
    • In the instructions box, the "x" link doesn't provide information about its function. In this case, the user can only infer it from the context. All the links must be able to be understood out of the context. 
    • The instructions text shouldn't be H2 (too high in the page hierarchy). 
    • The user doesn't find for himself the "Subtitle me" option. Once indicated by the moderator, he says that he has been confused by the previous icon. 
    • In the video area, JAWS reads the Play button (from Universal Subtitles) and also all the buttons from the original video, which is confusing. 
  • Other tasks performance: 
    • The order of the languages in the pull-down menu can be confusing: sometimes, "Spanish" appears heading all its variations, and sometimes it appears at the top of the menu, below English. 

• On the tutorial screen:
  • After clicking on "Skip", the user is confused by the "Continue" button. He believes that "skip" implies continuing automatically, and that the button "Continue" will lead to a different action (like playing the tutorial video). 

• On the "typing" screen:
  •  About accessibility: 
    • In the top of the page, the buttons 1, 2, 3 (Steps) don't give information out of the context. They should be labelled in a descriptive way, like "Step 2, syncing". 
    • The interface uses Tab to play/pause, while JAWS uses Tab to navigate between links in the page. As Intro is used in the typing process, the user suggests a key combination as Ctrl+Tab.
    • The user doesn't find the way to edit a subtitle after being typed (eg, to correct a mistake). 
    • The user can not delete or add a subtitle using the options in the subtitles boxes, because all the options that appear by mouse over are not readable by JAWS.
    • The moderator asks the user if there is any way to find the options inside the subtiles boxes. Then he activates the virtual cursor and he spends quite time looking for an option. Finally he finds the delete options, but he doesn't know its function (only reads "x").  
    • The subtitles in the list should be links, to make them clickable. In the current design, they are mouse-dependent. After clicking on a subtitle-link, its related options should appear in a list. 
  • Other tasks performance: 
    • The user is well oriented in the general interface. He can read the instructions, presses Tab and enters subtitles without difficulties. 
    • He can find without difficulties the list of subtitles that has been adding.
• On the "syncing" screen:
  • In the timeline, JAWS reads each frame of the video as a bullet. This is confusing. 
  • The user doesn't know how to sync the subtitles with the video. 
  • The user can drag the subtitles, but he has not feedback about where is he dragging to. 
  • The user doesn't listen the subtitle, to know where does it appear in the video. 
  • The user comments that this step is really confusing for him. He can not adjust the subtitles position. 
• On the "reviewing" screen:
  •  Same comments as in the previous screen.
• Final comments:
  • In the initial pages, the main questions are related to the structure of the page (header, sub-header and on). Sometimes headers are used to determine the visual appearance of the page, while this should be defined by styles, to preserve the readability of the page.
  • All the options and text fields must be labelled. They must be comprehensible out of context. 
  • Any of the options should be device-dependent. 
  • The main problems appear when the user tries to edit and sync the subtitles. He looks satisfied for being able to do the initial part of the process: "Universal Subtitles have done a great job, this is difficult. But I should type and sync well all the subtitles at the first attempt, otherwise I've lost my chance!".