Layout Reorderer User Testing - Round 3 Results

Summary

4 of 4 users were able to drag and drop portlets. The one user who took a longer time to realize she could do this (and she did eventually do it) was confused by the task.
1 user suggested that when mousing over a movable portlet it might be helpful for it to say "click & drag to move."
4 of 4 users understood what a locked portlet was
Performance was markedly improved

Demographics

User Number	Location	Gender	Age	Role	Tech-savvy	Do you own a personal computer?
User 1 (BF)	Berkeley	Female	41-50	Staff	Medium	Yes, Macbook
User 2 (AL)	Berkeley	Male	19-24	Student	Medium-high	Yes, Toshiba laptop (PC)
User 3 (JV)	Berkeley	Female	19-24	Student	Medium	laptop, windows XP
User 4 (DB)	Berkeley	Female	51-60	Staff	Medium-low	Yes, a PC
User 5
User 6
User 7

Do you do any of the following and if so how often?

User	Checking email	Instant message	Shop online	Online banking	Internet research	Take class online	Social networking
User 1	All the time	All the time	All the time	All the time	N/A	A few times a month (webinar)	No answer
User 2	All the time	A few times a month	A few times a month	A few times a month	A few times a week	Hardly ever	A few times a week
User 3	All the time	A few times a month	Hardly ever	A few times a month	A few times a week	Never	A few times a week
User 4	All the time	Hardly ever	A few times a month	All the time	A few times a month	A few times a month	A few times a week
User 4
User 5
User 6
User 7

Do you upload files on the web and if so how often?

User	Pictures	Media	Docs to course site	Docs to soc. ntwkg	File to email
User 1	Hardly Ever	Never	Never	Hardly Ever	All the time
User 2	A few times a month	Hardly Ever	Never	Hardly Ever	A few times a week
User 3	A few times a week	A few times a month	Hardly ever	Never	All the time
User 4	A few times a week	Hardly ever	Never	Hardly Ever	All the time
User 5
User 6
User 7

Interaction Notes

User Number	Task 1 - Move a particular box/portlet	Task 2 - Change the placement of other boxes/portlets	Task 3 - Deal with a Locked box	Task 4 (if not done already) - drag and drop a portlet	General comments
User 1	The Announcements portlet wasn't there as it should have been, so we had the user move the Weather portlet. User seems to have thought we wanted her to move it to another tab, as she said "I never customize the tabs." Daphne had to explain that we wanted her to "move that box" to make it clear what we wanted. She had no problem dragging and dropping the box, but still didn't think that's what we wanted. She didn't understand at first that when you moved a box to a new spot, the box underneath it moved down--she was looking for this box for a bit and couldn't find it.	She had no trouble dragging and dropping portlets, but she still wanted to interact with the tabs. She tried "Add content" and "Add tab." She said "I would put my bookmarks under a tab; I wouldn't put them up front."	She dragged the portlet way up to the upper left and read the pop-up message. Then she said, "What's considered a locked box?" She realizes "Welcome" is locked when she sees the lock icon and quickly realizes she can't move the Bookmarks portlet up any higher because of it.	She had no problem moving the image lightbox down.
User 2	Scrolls down to Weather and drags and drops it with no problem. Says it's similar to the Facebook interface.	Tries to drag Weather up. Moves Bookmarks to the left. Moves Image Gallery to the right and there's a little weirdness when he tries to move it to a space where there is no portlet - he can't drop it.	Tries to drag there two times, then quickly sees that it's locked. Tries to click on the word "Welcome" in the portlet header to unlock it.	Easily performs this task.
User 3	Easily drags and drops the portlet right away. Tries to put it on a locked portlet and fails so moves it to upper right. Says right away, "I didn't see this lock so I tried to put this on top of it. Now the lock makes sense."	Calendar drags a little jerkily as she moves it up, but she still has no problem moving it.	She'd already done this in task #1 so we didn't ask again.	She moves Weather down across/below a somewhat bit portlet with no problem.
User 4	Initially asks, "is that considered a folder?" Then says, "no, it's an object on the page. Shouldn't there be something that tells me I can move things?" She sees "Page Layout." Then she says, "Can I move it (the portlet)? I can move it maybe?" Sees the red box and drops it and says, "I can!" Realizes she can and moves the portlet pretty quickly from start to finish.	Easily drags Calendar between Weather and Bookmarks portlets.	User tries to drag the portlet to the upper-left. Says, "it won't change with the Welcome so that's the top-most left."	User easily moves the Weather portlet to the left.	Thought it might be helpful when mousing over something for it to say "click & drag to move."
User 5
User 6
User 7

Post-test Questionnaire Responses

Question	User 1	User 2	User 3	User 4
How easy or difficult was it for you to realize you could move boxes in the portal by dragging a box from one location and dropping it in a new location?	Neutral	Very easy	Very easy	Difficult
How easy or difficult was it for you to actually move a box in the portal (by dragging and dropping) to a specific location?	Very easy	Easy	Very easy	Very easy
How easy or difficult was it for you to distinguish between boxes in the portal that could be moved and those that were locked in place?	Neutral	Easy	Very easy	Easy

Post-test Question Responses (verbally asked)

Question	User 1	User 2	User 3	User 4
1) Did you notice a visual cue for picking up one of these boxes? How helpful was it to you? (looking for cursor change to hand)	Didn't notice a visual cue. Did see borders on a box come up when she started to drag and drop.	He says he didn't notice a visual cue because "you said pick up the box" which told him he could do that. Says the bars on the top of the portlets are modeled "kind of like Windows." Also, the bar at the bottom of the portlet gives it sort of a 3-D feel, which makes him think he can move it.	"I just assumed you could (drag and drop) because you can do that on Google things." She didn't notice a visual indicator.	First things she saw were "Customize" or "Page Layout," like in MyYahoo! and a couple other places and thought that's what we wanted. (Though she says maybe MyYahoo! will allow you to drag and drop, too -- she doesn't know.) When I clicked I saw the four-cornered arrow and when I see that I feel that I can move something. It would have been easier if it had said "you can click & drag." The box also changed to an outline, which I think means it's active and I can do something with it.
2) Did you notice a visual cue for dropping (and placing) a box? How helpful was it to you? (looking for green box drop target)	She noticed the shadow of the box when the image is moving with you. Didn't realize it would move things down, however.	The translucent view was a clue. I've seen that at deviantart.com. After thinking about it, he says, "I guess what helps with placement is the pink box." "It's sort of like solitaire--you pick up a card and move it."	"Yeah, I like the red boxes. It makes it very clear."	I should have read that Welcome is locked, I see it's locked now. I just moved it elsewhere and everything shifted around it. I felt comfortable about placement...after I tried to move it where it wouldn't go, I understood where things would go.
3) Further questions about drop target, which can be skipped if answered in question #2: How did you know where the box you were moving would land? Did you notice a green outline box? What do you think it is for? Was there anything confusing about this box? If so, what? (If confusing) Are there any changes you would suggest to make it less confusing?	It was confusing because I wasn't sure what the task was. It would have been helpful to tell her she was "customizing a web page for my own information... "I didn't realize that's what we were doing." I'm not sure how you'd add tabs.
4) Did you see a visual cue that helped you determine whether a box could be picked up or not? How helpful was it to you? (looking for lock icon to show the portlet is locked.)		The "combination lock thing."	"It made sense; the box is locked. I can't move it." Said she saw nothing else on the screen (the message) that told her she couldn't move it.	When I let it go and it went down below Welcome I was nervous (at the beginning of the test) so I didn't notice the lock.

Potential Design Improvements (based on testing)

There still seems to be a slight issue if you try to drag a portlet into whitespace way below other portlets--it can't be dropped until you drag it close to the bottom portlet on a column. You should be able to drop anywhere in that whitespace.