Uploader User Testing - Round 1 Results

Summary

The user testing on the Uploader was carried out in a "quick-and-dirty" fashion. Ten users (5 in Toronto, 5 in Berkeley) were asked to spend less than 30 minutes with us to complete 4 upload tasks. Below are the results thus far.

Toronto:

All users were able to successfully upload files
3 of 4 users asked if selecting multiple files was possible before being asked to add multiple files. Given the selected users were rather tech-savvy, this demonstrates the users' high expectations.
All users acknowledged the list of selected files and proceeded to click on Upload

Berkeley:

All users were "wowed" by being able to browse and select multiple files at once. Even though they didn't expect it, all quickly selected multiple files.
While files were uploading, 4 of 4 users chose "browse collections" to return to the image gallery thumbnail view. They expected the upload to continue "behind the scenes". There was no feedback that the upload would stop (but that's what happens). This happened on one to several tasks for each user.
One user mentioned it would be nice to see thumbnails in the upload queue. He said the thumbnail could even get bigger as the user mouses over it.
3 of 4 users said they would not stop the upload in progress to remove files from the queue. They would wait until all files were uploaded and then remove them from the Gallery. One user did mention that if she was doing it on her own she probably would have played around with the pause.
4 out of 4 users didn't immediately notice when the files were done uploading. For a couple it took prompting for them to realize. A couple users mentioned that it was pretty subtle. They expected to be taken back to where they started from (browse images & collections) when the upload was complete.
- Increase the visibility of the "File Uploaded" message. How about "Upload Complete" in bold? There is no reason to grey it out. Users immediately think greyed out information is irrelevant.
The one user that used "remove" in the upload queue said it is too subtle. She had trouble seeing that the file was actually gone (files name with default camera numbers). She ended up removing extra files because she didn't think it was removing.

Demographics

The users were selected at random by the tester rather than through a formal recruiting process. Below table shows the demographics and previous experiences of each user we spoke to. We felt that such criteria as areas of study has little relevance to users' behaviour with file uploader, so we made conscious decision to leave them out.

Demographic Background

User	Gender	Age	Role	Type of Computer
UT User 1	Female	25-30	Grad Student	Windows laptop
UT User 2	Male	31-35	Grad Student	Windows laptop
UT User 3	Male	25-30	Grad Student	Windows laptop
UT User 4	Female	19-24	Student	Windows desktop
UCB User 1	Male	25-30	Staff	Mac laptop
UCB User 2	Female	35-40	Staff	Mac
UCB User 3	Female	41-50	Staff	PC
UCB User 4	Female	31-35	Staff	PC at work; no home computer

Do you do any of the following online and if so how often?

User	Email	IM	Shop Online	Online Banking	Research Online	Online Classes	Social Networking
UT User 1	All the time	A few/week	A few/week	A few/week	All the time	A few/month	All the time
UT User 2	All the time	All the time	A few/month	A few/month	All the time	Hardly ever	A few/week
UT User 3	All the time	All the time	A few/month	A few/week	All the time	Hardly ever	A few/week
UT User 4	All the time	All the time	A few/month	A few/month	All the time	Hardly ever	A few/week
UCB User 1	All the time	A few/week	A few/week	Never	All the time	Hardly ever	Hardly ever
UCB User 2	All the time	All the time	A few/month	A few/week	All the time	Hardly Ever	Hardly Ever
UCB User 3	All the time	All the time	A few/month	Never	A few/month	Never	A few/month
UCB User 4	All the time	All the time	A few/moth	Never	Never	Never	Hardly Ever

Do you upload files on the web and if so how often?

User	Pictures	Media	Documents to course sites	Documents to social networking sites	attachment in email
UT User 1	A few/month	Hardly ever	A few/month	Hardly ever	All the time
UT User 2	A few/month	A few/month	A few/month	A few/month	All the time
UT User 3	A few/week	A few/week	A few/week	Hardly ever	All the time
UT User 4	A few/month	Never	Never	Never	A few/week
UCB User 1	A few/month	A few/month	A few/month	A few/month	A few/month
UCB User 2	A few/month	A few/month	A few/month	Hardly ever	All the time
UCB User 3	All the time	All the time	All the time	A few/month	All the time
UCB User 4	A few/month	Never	A few/week	Hardly ever	All the time

Legend
All the time - A few/week - A few/month - Hardly ever - Never

Interaction Notes

Setup:

These tests were done in "hallway" style. We grabbed people community areas and asked them to participate for a few minutes.
All tests done on a Mac.
Removed emphasis placed on the key convention for selecting multiple files, since the users were PC users
Added a question at the end of Task 4 "Once you're done adding all the files you want, what would you do next?" to see how they exit the File Uploader

Although the PC users were asked to perform the tasks on Erin or Daphne's Macbook, they had little problems completing the tasks. If they were obviously looking for the ctrl key to choose non-consecutive multiples, they were prompted that on a Mac the apple key is the parallel. We assume they would normally be completing this task on their own computer and we were not testing their knowledge of the keyboard shortcuts.

Buttons the users clicked on are notated in [fluid:brackets].

Task 1: Upload a file from the Photography101 folder on your desktop.

Optimal Path: Add Images > Browse > Select file > Upload > Browse Images & Collections

Task 2: Upload the 4 most recent files in the Photography101 folder.

Optimal Path: Add Images > Browse > Select files > Upload > Browse Images & Collections

Task 3: Upload all files in Photography101. <while upload progress is going> You realized that IMG_9409.jpg and IMG_9407.jpg are the wrong files and you didn't intend to upload those to the Photography class site. What would you do?

Optimal Path: Add Images > Browse > Select file(s) from desktop > Upload > Pause > Remove both files > Resume Upload > Browse Images & Collections

Task 4: If you wanted to add a couple files from Photography101 and some files from your desktop, how would you go about doing that?

Optimal Path: Add Images > Browse > Select files from Photography 101 folder > Add more files > Select file(s) from desktop > Upload > Browse Images & Collections

User	Task 1	Task 2	Task 3	Task 4	General Observations
UT User 1	- [fluid:Browse] - Double-clicked on file - [fluid:Upload] - "It's weird it doesn't show me the files uploaded"	- [fluid:Shift] to select multiple files - "Nice that it tells me the progress" - "This is pretty self-explanatory"	- [fluid:Pause] - Sees that the files she wants removed have already been uploaded, "Here I can't do anything I guess."	- To exit, "I would click on the breadcrumb" - Expects to go back to the "main page" to "reorder, add descriptions, pick cover photo, send it to people"
UT User 2	- [fluid:Browse] - Picked the folder - Picked an image - [fluid:Select] - acknowledges "Ready to Upload" - acknowledges "file uploaded"	- "files are already sorted by dates modified" - [fluid:Shift] to select multiple - Four files are "ready" - [fluid:Upload]	- "If it's just two wrong files, I would wait till the upload is completed. If it's more than half of the files that are wrong, I would pause and do the selection again." - Looked for the two wrong files mentioned and said "I can't select these files" (expected to be able to remove even the already-uploaded files from this screen) - Would go to "Browse Image Collection" to check to see where these files are	- "Remove is right here! for the previous task, I would've used this remove button." - Feels that it should support two different kinds of remove: 1) remove from the "intention list" 2) if they are uploaded already "remove from collection" - "It's surprising the rows can't be clicked on" referring to the rows in the table
UT User 3	- [fluid:Browse] - "So you can upload all of these?" to which I answered Yes, and the user skipped Task 1	- [fluid:Shift] to select multiple - [fluid:Upload]	- [fluid:Pause] - [fluid:Remove] files he doesn't want - [fluid:Upload] the rest	- Would click on "Browse Images" - "Would be nice to see what the images look like on the actual site, like thumbnails of images, so I can see exactly what I've uploaded and remove some if needed"
UT User 4	- [fluid:Browse] - "Can I upload all of these?" to which I answered Yes, and the user skipped Task 1	- Click and drag to select multiple - [fluid:Upload]	- [fluid:Pause] - When asked to take out the two images whose names start with "IMG_", the user observed the table and said "The ones I wanted to delete happened to be one of these ones (ones already uploaded)" and expressed she'd like to have the ability to remove uploaded files	- Added them separately. Clicked on [fluid:Browse], added files from Photography101, and clicked [fluid:Upload]. Then, clicked [fluid:Browse] again, added files from desktop, and clicked [fluid:Upload] - Would like to see a "OK" or "Back to main" button for when she finishes uploading
UCB User 1	Task completed via optimal path	Task completed via optimal path	Task NOT completed Path: - [fluid:Add images] - [fluid:Browse] - Selected all files from Photography101 folder using the [fluid:shift key] - After being told there were a couple files in the the upload queue that he actually didn't want to upload, the user went to [fluid:browse all images] expecting the images to show up there as they were uploaded. He said he would just remove them after they were uploaded. He mentioned that he just really used to using breadcrumbs. - User expected the upload to continue behind the scenes while he went to [fluid:browse all images]. - User didn't notice the Pause button. Not sure he would have used it if he had.	Task completed Path via optimal path	Would like to see thumbnails of images in the upload queue. Even if they were small, they could be "blown up" on rollover.
UCB User 2	Task completed via optimal path. Other comments: A second of hesitation before she realize she needed hit the upload button. She thought she was done after browsing. A few seconds after upload was done she said, "Am I done?" Not enough indication that something has changed after the files are uploaded. She wasn't sure it was done. Then she said, "It's all about once you know, you know".	Task completed via optimal path. Other comments:	Task NOT completed because user went to "Browse Images & Collections" before upload was complete. Had she not done that she would have completed (assuming entire system was working) but not as expected. User said she would let all the files upload and then delete the ones she didn't want. Other comments: Assumed she could just upload the entire folder. Went to "browse	Task completed. Other comments: She tried to select files from folder and then from desktop on same browse. As soon as she realized the first files weren't in the queue she realized what happened and completed as expected.	Wanted to upload folder but could not
UCB User 3	Task completed via optimal path. Other Comments: On upload page, she was looking for her desktop -- "didn't think of browse" She asked "how do I get back" but pretty quickly noticed the link at the top for Browse images & colllecitons. Image Gallery feedback: Took a long time to find 'Add Images'. * Tried to use "Find" to browse for files (to upload).* Throughout tasks she wanted to go to Add Collections for add files. She said she was thinking "add to collection". Perhaps we could use "Create new collection" to make it clearer.	Followed optimal path but Task NOT completed because user went to Browse Images & Collections before upload was complete. Other Comments: She was trying to get into 'Recently Added" playlist since the task mentioned recently added files. Task should be updated not to user terminology in the interface. Immediately used keyboard to select multiple at once. Went to "Browse Images & Collections" before upload was complete. She expected it to continue.	Task completed (assuming entire system was working) but not as expected. User said she would let all files upload and then go back and delete the ones she didn't want. She followed optimal path through the first Upload then let the system upload them all. Other commments: She mentioned that had she been on her own she would have probably tried "pause".	Task completed via optimal path. Other Comments: "I would upload them from the folder first and then go back and upload the ones on the desktop". After choosing from the folder, she quickly noticed the "browse" below the queue and used it to choose additional files.	Seems like a clean layout Image Gallery - icons aren't clear.
UCB User 4	Task completed via optimal path although it took awhile to realize uploading was done. Other Comments: Mentioned the progress message was very dark. Status went away and she said, "and now it's gone and I'm still on the same page". She said she expected it to take her back to where she started once the upload was done (Browse Images & Collections). She "felt stuck". She suggested that if not taken back then adding an additional button on the upload page to 'take be back'. She had a slight hesitation before realizing she needed to select "Upload" after selecting the images.	Task completed via optimal path. Other Comments: "Oooh, it worked, that is spiffy" she said after selected multiple images at once. Chose "Browse Images & Collections" before upload was complete.	Used optimal path but did NOT complete task. She removed more images than expected because she wasn't seeing them get removed from the queue (too quick and subtle). Other Comments She said, "easy to remove but visual cue that file has been removed is too subtle." She noticed the countdown during this task. She said since she was focused on the remove column her eye was drawn to it (it is left aligned in the table like the remove).	Task completed via optimal path. Other Comments:	She mentioned that that first thing she tries when uploading is selected multiples even though she doesn't expect it to work in many places. Image Gallery Issues: She said she found herself drawn to the "Add Collections" so it took a few minutes before she found the "Add Images" button.

Post-test Questionnaire Responses

The users were asked to evaluate their experience on Likert scales after the test. The questions ask how easy or difficult it was to do each of the following:

User	Uploading single	Uploading multiple	Uploading additional	Cancel upload	From 2 different locations
UT User 1	Very Easy	Easy	Easy	Very Easy	Easy
UT User 2	Very Easy	Very Easy	Very Easy	Neutral	Easy
UT User 3	Very Easy	Very Easy	Very Easy	Easy	Easy
UT User 4	Very Easy	Very Easy	Very Easy	Very Easy	Neutral
UCB User 1	Easy	Easy	Very easy	Easy	Easy
UCB User 2	Very Easy (once noticed "Add Images")	Very Easy	Easy	N/A	Easy
UCB User 3	Between Difficult & Neutral. She said it was easy after first time.	Easy	Easy	N/A	Easy
UCB User 4	Very easy	Very Easy	Very Easy	Easy	Very Easy

Potential Design Improvements (based on testing)

File Uploader Component

Users expected items (entire row) in upload queue to be selectable to remove them.
Users would like to be able to remove files from upload queue even if they have been uploaded. Can we do this?
Display thumbnails in the queue.
Include a lightbox-window style for next round of testing.
Allow users to upload a folder and automatically "unpack" it for them.
Make the pause button pop a bit more.
Increase the visibility of the "File Uploaded" message. How about "Upload Complete" in bold? There is not reason to grey it out. Users immediately think greyed out information is irrelevant.
Removing from queue is too subtle. Particularly when files are named with camera defaults. Need to be more obvious that something happened and the file is gone.
Need to help users know what to do after the upload is complete OR take them back to where they came from once the upload is complete. If we don't take them back and we use the lightbox overlay pattern, perhaps a prominent message at the top could help, "Upload complete, you may close the window to continue your work."
After upload is complete, take user back to where they came from. The main use case is upload a set of files so it makes sense to first design for it. In the case where users want to upload more than one batch, it only requires them 1 click to choose "add files" (or whatever took them to the upload).
Button pattern usually has the main action left of the cancel button (or right is buttons are right-aligned). Change the order of buttons so Upload is far left.

Design Pattern

Design Pattern - newly added files should be in clear view when return to page they were adding from. Users should easily be able to remove files from there.

User Test

Check for leading task language. We really should not be using "upload" in task. We also don't want to use any terminology in the interface.
Post-test questionnaire should ask about ease of uploading the 1st time and then repetitive times. Users mentioned that it wasn't as easy the first time but once they have done it, they knew.