Uploader User Testing - Round 2 Results

Summary

4 of 5 users found uploading a single file very easy. (DO - included Erin's users in my numbers, AB added mine)
3 of 5 users weren't sure which keys would allow them to select multiple files. (DO - included Erin's users in my numbers, AB added mine)
- A user added files individually and said that's what she did in email too because she "didn't know what controls or things to use"
4 of 5 users did not use the Pause / Remove workflow when told they really didn't want to upload 2 of the files in a big upload. They waited for the files to be uploaded and then removed them from the aTutor interface.
Users are generally quick to recognize multiple files can be uploaded. EY
Users easily recognized the Remove buttons and felt free to use them to remove files (often without pausing), but one user thought they meant "upload didn't work." EY
One user could not initially find the "Browse files" link AB
One user never noticed any confirmation/error messages, and instead confirmed that the files were uploaded in the application's File Manager. AB
3 of 3 users found Uploading a single file very easy JC
3 of 3 users were immediately able to select multiple files and without asking for instructions. JC
2 of 3 users did not use the Pause / remove workflow when attempting to correct file selection task. JC
All users recognized and employed the Remove buttons. JC
- Two users immediately employed the Remove buttons to delete files as the uploads occurred.
- One user initially decided to wait for uploading to finish but when asked to explain why she reconsidered and looked for avenues to remove the files; she immediately employed the Remove button.

Demographics

User Number	Location	Gender	Age	Role	Tech-savvy	Do you own a personal computer?
User 1	Toronto	Female		Undergraduate	Low	Mac
User 2	Toronto	Female		Graduate	Low to medium	PC
User 3	Toronto	Male		Undergraduate	Low to medium	PC
User 4 (RL)	Berkeley	Female	51-60	Staff	N/A	Mac
User 5 (MS)	Berkeley	Male	35-40	Staff	medium	Customized PC
User 6 (JH)	Berkeley	Male	35 - 40	Staff	high	Mac laptop
User 7	Vancouver	Male	19-24	Student	High	PC
User 8	Vancouver	Male	19-24	Student	High	PC
User 9	Vancouver	Female	19-24	Staff	Medium	PC

Do you do any of the following and if so how often?

User	Checking email	Instant message	Shop online	online banking	Internet research	Take class online	Social networking
User 1
User 2
User 3
User 4 (RL)	all the time	hardly ever	a few times a month	all the time	all the time	hardly ever	hardly ever
User 5 (MS)	all the time	hardly ever	a few times a month	never	N/A (Never)	a few times a month	a few times a week
User 6 (JH)	all the time	all the time	a few times a week	a few times a week	all the time	hardly ever	all the time
User 7	all the time	all the time	a few times a month	a few times a month	a few times a month	all the time	all the time
User 8	a few times a week	a few times a week	hardly ever	a few times a month	all the time	a few times a month	hardly ever
User 9	all the time	all the time	a few times a month	a few times a month	never	hardly ever	all the time

Do you upload files on the web and if so how often?

User	Pictures	Media	Documents to course site	Documents to social networking site	Attach a file in email
User 1
User 2
User 3
User 4 (RL)	hardly ever	a few times a week	a few times a month	hardly ever	all the time
User 5 (MS)	a few times a month	a few times a month	N/A (never)	hardly ever	all the time
User 6 (JH)	a few times a month	a few times a month	all the time	a few times a week	all the time
User 7	hardly ever	hardly ever	hardly ever	hardly ever	a few times a month
User 8	a few times a month	a few times a month	a few times a month	hardly ever	all the time
User 9	hardly ever	hardly ever	never	a few times a month	all the time

Interaction Notes

User Number	Previous LMS Experience	Uploading a file	Uploading multiple files	Removing files from queue
User 1	Blackboard	Selected a file > pressed Select > pressed Upload	"I don't know. I would do it one by one" I showed user how to select multiple files. User selected files and clicked Select "And then... none of them worked," looking at the red x's. She quickly realized, "Oh, do I need to press Upload?" "Ones with checkmarks are uploaded."	Pressed the remove buttons to remove files "I wouldn't know how to do it with the check marks or the yellow ones" meaning she wouldn't know how to remove files that are already uploaded or that have failed to upload.
User 2	-	Selected a file > pressed Select > pressed Upload	Clicked Browse Files and selected multiple by pressing shift Pressed Select > Pressed Upload	Pressed remove buttons to remove
User 3	-	"That's it?" User found it extremely straight forward.	With a mouse, clicked and dragged cursor to select multiple files Pressed enter key to select "Oh, that one didn't work," pointing to a yellow warning icon amonst green checkmarks "Pretty easy. Pretty straight forward."	Pressed Pause "Kill the ones I don't want" Pressed Resume "Oh, you can skip ahead and kill the other ones." While the upload was in progress, user scrolled down and removed files in the queue.

NOTE: We changed the tasks after user 3 so we've created a new table for the remainder of testers. See the protocol for the most recent version of tasks.

User	Task 1 - Upload single file	Task 2 - Upload 4 files	Task 3 - Upload/pause/remove	Task 4 - Upload 3 more	Other general comments
User 4 (RL)	Successful in 1 attempt Tried clicking on "add files or folder" header several times while saying "this would be my first choice". After a few seconds she looked around and saw the upload files buton and use it. Uploading 1 file was fine.	Successful by uploading individually & then using multi upload after I told her she could. For uploading multiple files (task 2) she uploaded 1 at a time. She mentioned I would just upload 1 at a time because I don't know what the controls or whatever are". Then she added that's how she adds email attachments too and is how she would probably do it in bspace. I decided at that point to tell her she could upload multiples to see if she could do the task (we knew discoverability was lacking at thtat point) After I told her she said, "can you highlight?" as she started selecting them one at a time with the shift key. This worked fine. And it was an additional task since I wanted to see how she uploaded multiples.	Did not use the pause & remove workflow She waited until they were done and then deleted them from atutor. There was plenty of time for her to pause and remove but she just didn't. Not sure if she noticed she could but she really wasn't looking either. She had noticed the remove icons previously and mentioned them so I was a little surprised she didn't try to use them. She mentioned that she thought it would have been easier if she had her own computer. She didn't actually use pause or remove but answered those questions in the questionnaire with "easy". I think this was "not wanting to feel stupid" syndrome.	Success in 1st attempt	Thought it would have been easier on her own computer -- although she was a mac user
User 5 (MS) (see attached Word version of results with screenshots)	There was a problem with the environment and the user was not presented with the File Uploader. They just had regular (single file) Browse and Upload buttons (see screenshot below). He had no trouble finding Browse, selecting the files in the OS file browser then hitting upload (but this was not our File Upload). Upon completion of the upload, user immediately scrolled down to the File Browser window to confirm his file was uploaded and takes no notice of the confirmation message at the top.	At this point the screen changed to one that allows the user to access the multi-file Uploader (see screenshot). He clicks on the Upload files button with no problem. He can't see what to do here (doesn't see "Browse Files" or the message about it) and clicks "Cancel." Back at screen #2 above, he comments that the "Create a New File" and "Upload Files" buttons are unclear. He goes back into "Upload Files" and looks around for a few seconds. He finally notices the "Browse Files" button. He says, "Cancel is so prominent but Browse is hidden." He selects multiple files from the OS File browser and returns to the Uploader. He looks at the files for a few seconds and finally clicks Upload. He says, "It's interesting that I selected them already and now I have to confirm I want to Upload. It's an extra step I don't want to have." He wondered why he had to look at the files again when he'd already selected them. (AB Note: I think this problem may be because he looks at a similar list of files in both cases.) He further said that maybe the system could be smart and not ask him to confirm by activating "Upload" automatically if there were smaller files, and only if there were large files ask him to confirm. This user was also afraid to hit "Pause" at first (which he later got over), though, so this request this may be a symptom of his fear at that point of stopping the upload. User almost seemed not to want to see the files in the Uploader, and instead to just check in the File Manager to see whether the files were uploaded. He seemed confused that he had to process where the files were in three places: the OS File browser, the Uploader, and the application's File Manager (all of which are potentially viewable at the same time). User wondered whether he needed to upload into the Assignments and Lecture Notes folders, but I told him not to worry about it (the folders weren't supposed to be there.)	• User was confused at first because he had added some files between tasks as we were discussing something and had to delete them before beginning the task. He quickly realized he could delete files before the upload had started with the red X. • As he selects files to upload, user mentions that he'd like the OS File browser to tell him what files he'd already uploaded. • User said that the upload is slow and he'd like to "kill" the files, but prefers to just wait until everything is uploaded. "I'm reluctant to interrupt the process." • Removed them from application File Manager after upload was complete	• We did this as a new task since I didn't ask the question in time. • User says he thinks the Uploader file browser window "is annoying." • User finally did try the Pause, saying, "Since I've played with it a bit, I think it may be robust enough to give it a try." • When asked what he was thinking, at first he said, "nothing," then as the upload progressed, "I'm thinking it takes so much time to upload it!" • User received an error about duplicate files (he'd uploaded duplicates at this point), and he said, "I don't know why" and tried to upload the same files again. He doesn't notice in the application File Manager that the file is already there. • Now I'm thinking I need a secretary, I don't want to deal with this again." • Once again the user pauses and resumes.	• He would like to see a thumbnail of what the file looks like in the File Manager. • When I asked, he said he noticed the progress bar changing, but didn't think it was very prominent • When I asked, he said he never noticed the green confirmation message that files had been uploaded. He commented that it would be helpful for that message to say how many files were uploaded.
User 6 (JH)	immediately understood the concept wanted to click "Add File or Folder" header at the top of the screen since it looked "clickable" but realized that it was a header before clicking clicked on "Upload Files" button immediately wonders whether you can upload multiple files, even though the task is only upload single selects two files and successfully adds them to the list clicks the Remove button before uploading with out prompting	used Date Modified sort in the OS File dialog to determine the 4 most recent files not completely sure where the files have gone didn't understand that you have to be inside a folder to put files into that folder expected the application to ask him where to put the files Progress initially confusing because of the two different progress bars, but on further inspections got it and thought that it was cool	successful clicked the Remove button before pausing, wondered if that was OK	successful

User	Previous LMS Experience	Task 1 - Upload single file	Task 2 - Upload 4 files	Task 3 - Upload/pause/remove	Task 4 - Upload 3 more
User 7	WebCT Vista	clicked on Upload Files to open the uploader window user does not know where file is but is told the location. To confirm he navigates to desktop to confirm the location returns to uploader window clicked on Browse Files clicked on a file > clicked on Open > clicked on Upload clicked on Done to close the uploader window	clicked on Upload Files to open the uploader window clicked on Browse Files held down Shift and selected multiple files > clicked on Open > clicked on Upload clicked on Done to close the uploader window	clicked on Upload Files to open the uploader window clicked on Browse Files held down Shift and multi-selected files 5-12> clicked on Open > clicked on Upload clicked on the Remove buttons to remove files as other files were being uploaded did not notice or use the Pause button	waited for the previous files to finish uploading user clicked on Add More button held down Shift and multi-selected files 3 files > clicked on Open > clicked Resume clicked on Done to close the uploader window
User 8	WebCT Turn it in Vista	clicked on Upload Files to open the uploader window clicked on Browse Files selected wrong file> clicks Open to add to queue>promptly deletes from the queue using the Remove button clicked on Browse Files selected a file > clicked on Open > clicked on Upload did not click on Done	clicked on Add More to browse files Held down Shift and selected multiple files > clicked on Open did not click on Done but does click on Resume to initiate the uploads uploads are completed did not click on Done	clicked on Add More to browse Files Held down Shift and multi-selected files 5-12> clicked on Open > clicked on Resume clicks on the Pause button clicks on the Remove buttons to remove files as other files were being uploaded clicks on Resume to complete the uploading	user clicked on Add More button held down Shift and multi-selected files 3 files > clicked on Open > clicked Resume clicked on Done to close the uploader window
User 9	WebCT Turn it in	clicked on Upload Files to open the uploader window clicked on Browse Files Selected a file > clicked on Open > clicked on Upload did not click on Done but File Uploader closes by itself as if Done was clicked on.	clicked on Upload Files to open the uploader window Held down Shift and selected multiple files > clicked on Open > clicked on Upload did not click on Done but File Uploader closes by itself as if Done was clicked on.	clicked on Upload Files to open the uploader window clicked on Browse Files selected wrong file> clicks Open to add to queue>promptly deletes from the queue using the Remove button clicked on Browse Files Held down Shift and multi-selected files 5-12> clicked on Open > clicked on Upload was going to let them all complete downloading but upon noticing the Remove buttons, decides to use those to delete files clicked on the Remove buttons to remove files as other files were being uploaded did not notice or use the Pause button clicked on Done to close the uploader window	clicked on Upload Files to open the uploader window clicked on Browse Files held down Shift and multi-selected files 3 files > pressed Open > clicked on Upload clicked on Done to close the uploader window

Post Test Questionnaire Responses

Question	User 1	User 2	User 3	User 4 (RL)	User 5 (MS)	User 6 (JH)
How easy or difficult was it for you to realize you could upload multiple files?	Easy	Very easy	Very easy	easy	very easy	easy
How easy or difficult was it for you to upload a single file?				very easy	very easy	very easy
How easy or difficult was it for you to upload multiple files?				very easy	neutral	very easy
If you paused during upload, how easy or difficult was it for you?	N/A	Very easy	Very easy	neutral (she didn't do this task)	easy ("but I wasn't sure I could do it at the beginning")	very easy
How easy or difficult was it for you to add more files to upload? Removed this task from the questionairre after user 3	Easy	Very easy	Very easy	N/A	easy	very easy
If you removed files from the list of files, how easy or difficult was it for you to re?	Neutral	Very easy	Very easy	easy (she didn't do this task)	easy	very easy
How easy is it to see which files are uploaded and which files are not?	Easy	Neutral	Very easy	easy	very difficult	very easy
How confident were you that the files you uploaded were actually uploaded?				confident	uncertain	neutral

Question	User 7	User 8	User 9
How easy or difficult was it for you to realize you could upload multiple files?
How easy or difficult was it for you to upload a single file?	Easy	Very Easy	Very Easy
How easy or difficult was it for you to upload multiple files?	Easy	Very easy	Very Easy
If you paused during upload, how easy or difficult was it for you?	Very easy	Very easy	N/A
How easy or difficult was it for you to add more files to upload? (Removed this task from the questionairre after user 3)	Very Easy	Easy	Very Easy
If you removed files from the list of files, how easy or difficult was it for you to remove?	Very easy	Easy	Easy
How easy is it to see which files are uploaded and which files are not?	Easy	Very Easy	Very Easy
How confident were you that the files you uploaded were actually uploaded?

Potential Design Improvements (based on testing)

(DO) Add the instructions back in for how to multi-upload
(DO) Should we rethink pause, remove, add more files? So far people aren't using them. They do add complexity to the interface. If those tasks don't fall in the 80/20 rule I might vote for getting rid of them since users can accomplish both in other ways.
(DO) Why do we have to have pause. It's weird to have to know you have to pause first when the big red delete icon is right there in your face. Why not let them remove but keep the file in the queue and make it way greyed out or somehow very different visually from the others. I know it's cleaner technically but I think it's a real stretch to expect users to know to pause first. Couldn't we leave removed files in queue and make them look distinctly removed? (+1 AB, continue to let users remove files while the upload is taking place.)
(EY) +1 regarding pause. There is no need to make users wonder if it's okay to press remove before pausing. Pause is intended to be used for one of the following: 1) adding more files 2) removing files from queue 3) canceling upload. The experience would be smoother if we allowed the user to do these things directly without having to pause.
(eli) simplify the user experience by doing an immediate upload and de-emphasize pause and remove. (AB: I like this idea, but think users should still be able to remove files from the queue when it is uploading, which would essentially allow them to cancel the upload if necessary, and make it very clear what is and is not being uploaded.)
(eli) only have one progress bar with one pause button (+1 AB)
(eli) in addition to the progress bar include a more active hourglass or spinner so that when Progress is stalled because of network activity or file size there is still some feedback that the Uploader is still uploading. (+1 AB)
(AB) Fix test environment, or do an upload first yourself to ensure the user gets the Fluid Uploader on their first task.
(AB) Consider centering "Done" button (if it's the only button because we get rid of Pause/Resume) to make it more prominent.
(AB) Make the progress bar more prominent.
(AB) Put confirmation and error messages in one place, and add guidance for doing this to pattern, tutorial, etc.. Currently there are error & confirmation messages in two different places the Uploader and confirmation messages in the application itself. In testing my user did not see the application's confirmation messages, the Uploader's error messages, or even the Uploader's confirm message (I don't think--I actually didn't notice it).
- In most cases, I think the application will want to handle the error messages (for consistency's sake), but there may be situations in which it makes sense to display they error messages in the uploader itself (e.g. when an application or website has no other error messages) to better ensure users see them.
- Make the "Uploaded" message in the Uploader more prominent. Consider putting the number of files *not* uploaded there too.
- Add information to the design pattern and component tutorial/documentation suggesting a very prominent "files uploaded" message (with the number of files uploaded, too, if possible--do we send this information back to the originating app?) if the messages are being handled by the application.
(AB) If we retain the files' individual error messages in the Uploader, make them more prominent, and if possible put the name of the file in the message.
(AB) Launch OS File Browser when user opens Uploader (or make "Browse files" link more prominent).
(AB) Think about whether there are recommendations (e.g. for the design pattern)/modifications(e.g. config options for the Uploader) we'd like to make when using the Uploader with a (standard) File Manager. Is it overkill/too many steps to have to press "Done" in this case since the user can also just scroll down to see if the files were uploaded? Or is it too much in general since users are dealing with 3 different views on their files (OS File Browser, Uploader, Application File Manager)"?
(JC)change text in blue section where it says "click browse files to add files to the queue" to include "hold down the shift key to select multiple files."
(JC) the Resume button is not needed. Would prefer just these buttons : Done/Cancel and Upload/Pause. A button should have no more than 2 states or it could get confusing. I notice the Done button actually acts like a Cancel button because if I pause the upload and Click done, the Uploader closes and the file inventory remains unchanged.
(JC) using red (Cancel) buttons next green(Upload) is an intuitive choice but not ideal since people with deuteranopia colour-blindness will interpret those as the same shade. In our case it's not a huge deal but I'm sure for a person working in a nuclear powerplant it would!