OSDPL User Testing - Round 1 Results

Summary

2 of 3 users assumed that "Contributor" by default meant them.
2 of 3 users did not use the "Upload" button to upload an image
4 of 4 users (3 formal users, 1 informal user) weren't sure what happened after submission of a pattern.
3 of 3 users found their patterns relatively quickly using "View/Edit My Patterns."
2 of 3 users wondered how the material on the OSDPL related to what was on the Fluid wiki and seemed somewhat unsure about the distinction between a component and design pattern (these 2 users already knew about Fluid).
2 of 3 users said they would like to be able to save a design pattern without all the required fields (presumably as an in-progress draft).
User registration was relatively painless. All users were able to register without any major problems. (Successful registration feedback message helped user #1).
Single page pattern entry form seemed to work well. None of the users remarked on the length of the form, but one user did mention that the tabs helped her see what the high-level categories were without having to scroll.

Demographics

User Number	Location	Gender	Age	Job Title	Role	Tech-savvy	Do you own a personal computer?

User Number	Location	Gender	Age	Job Title	Role	Tech-savvy	Do you own a personal computer?
User 1	Berkeley	Female	25-30	Graduate Student	Student	Medium-high, unfamiliar with design patterns	Yes, Mac
User 2	Berkeley	Female	31-35	User Experience Specialist	Staff	Medium, slightly familiar with design patterns	Yes, Mac OS 10.5
User 3	Berkeley	Male	Over 60 years	Professor	Faculty	Medium-high, very familiar with design patterns	Yes, Mac

Do you do any of the following and if so how often?

User	Checking email	Instant message	Shop online	Online banking	Internet research	Take class online	Social networking

User	Checking email	Instant message	Shop online	Online banking	Internet research	Take class online	Social networking
User 1	All the time	All the time	A few times a month	A few times a month	All the time	Hardly ever	All the time
User 2	All the time	A few times a week	A few times a month	A few times a month	Never	Hardly ever	Never
User 3	All the time	Hardly ever	A few times a month	Hardly ever	All the time	Never	A few times a week

Do you upload files on the web and if so how often?

User	Pictures	Media	Docs to course site	Docs to soc. ntwkg	File to email

User	Pictures	Media	Docs to course site	Docs to soc. ntwkg	File to email
User 1	A few times a month	A few times a month	A few times a month	A few times a month	A few times a week
User 2	A few times a month	Hardly Ever	Never	Hardly Ever	A few times a week
User 3	All the time	A few times a week	All the time	Hardly Ever	All the time

Post-test Questionnaire Responses

Question	User 1	User 2	User 3

Question	User 1	User 2	User 3
How easy or difficult was it for you to find a design pattern on the site?	neutral	easy	easy
How helpful did you find the design patterns you looked at on the site?	somewhat helpful	somewhat helpful	helpful
How helpful did you find the other information available on the site?	somewhat helpful	somewhat helpful	N/A
How helpful did you find the left navigation of the site?	helpful	helpful	helpful
How easy or difficult was it for you to discover how to create a design pattern?	neutral	easy	easy
How easy or difficult was it for you to create a design pattern in general?	difficult	easy	easy
Which version of the design pattern data entry form did you prefer?	slightly preferred tabbed (would have greatly preferred tabbed if it didn't have submit on every page)	slightly preferred single page	highly preferred single page
How easy or difficult was it for you to upload an image?	easy	easy (as long as it was the right image size)	very easy
How easy or difficult was it for you to describe and/or list the source of an image?	easy	difficult (didn't have one)	easy
If you encountered an error, was it easy or difficult to understand the feedback given?	easy	easy	easy
After submitting a design pattern, how easy or difficult was it to find your pattern on the site?	easy	easy (wasn't indexed, but "my pattern" worked)	easy
After creating the design pattern, how easy or difficult was it to edit it?	easy	easy	very easy

Post-test Question Responses (verbally asked)

Question	User 1	User 2	User 3

Question	User 1	User 2	User 3
1) Why did/didn't you explore the various links in the navigation of the OSDPL? Could you tell me what you think the links in the left navigation mean? How could we make these pages or the link names of this navigation more helpful to you?	not sure what the News section is. Thinks it's customer comments. Clicked on comment for one of the news items and tries to add a comment. Input format text appearing below text field. Scanned it over. Initially failed at Captcha, but then succeeds. After submitting comment, she is surprised not to see it on the front page. Clicks "Read More" and doesn't see it there. Goes back and clicks "1 comment" and finally finds it. User assumed comments were for the whole site, but now realizes these are comment on each News item.	"I didn't feel I needed to read the FAQ." Would like to have the pages easily available "if I need them in context to what I'm doing." "How to write a pattern was easy to find but I sort of wish it was built into the form somehow because I wanted to make sure I was writing this in a standardized way. Could have put the directions in the text field itself that could be edited away." "Doesn't really say how many patterns are in there"...then she sees all patterns. She wants to know total number but only two pages. "Can't imagine anyone wanting to come in and browse, searching on design problems. Inline edit -- I wouldn't have used this word." "If there are people who are new to this, there could be people who want to experiment with this and set up accounts and it gets filled with crap, especially if people don't realize their drafts are out there. Which are reviewed, favorited? What if two people submitted design patterns for the same problem? Would you expect people to collaborate or would there be competing designs." "Assuming people who come to this site have a design background."	Recently Added Patterns is good.
2) What tasks would you expect to perform on the OSDPL? How helpful was the general organization and navigation of the pattern library in supporting you in performing these tasks? What could we improve to make it easier?	"Who is the target of this site? Someone (that) is familiar with design patterns, that kind of person can figure it out easily, but (for) someone like me it could be difficult to solve the problem I have." Feels that someone like her would be confused - coming to the site, wasn't clear how the site can help her with solving her problem / task. Comments that if she can see visual examples first, can better orient her.	Would like to be able to submit incomplete drafts "I think the navigation is fine--there are certain things I haven't explored."	It was pretty good overall. Might be nice to have a way to create diagrams for the patterns.
3) Do the design patterns on the site have the type of information you need? What information could we add, subtract, reorganize, etc. to make it easier for you to use the patterns?	At first it took a while, but once she understood, she can now get to the information she wants. Remarked at how it is not clear how it works after submitting the pattern.	"I'd want to give more thought to that. (I'd) compare it to the wiki." (Again, confusion about components vs. design patterns.)
4) Did you have all the information you needed at the right time to perform your tasks? What could we change to make it easier?		"Other than 'how to write a pattern," yes."	Make it clearer what to do and when I was done. I thought Submit was on a per tab basis.
5) Could you tell us which version of the pattern data entry form you preferred and why?	Liked the tabbed version because it looked more simple than the long version. But if the submit button is changed to be on the last tab only, then perhaps it would be even better.	I would have liked to be able to submit by answering the required questions Preferred long form mostly because she couldn't tell whether she could submit or not - the button was always there. If the buttons weren't on every page, she did like that the tabs told her what would be further down in the form without scrolling.	Preferred the single page. Even if there weren't submit buttons on every tab I would have preferred the single page.
6) What was your overall impression of the pattern creation and editing process? How can we improve it?	Make it clear what happens after a pattern is submitted. Doesn't know why the search isn't working.	"I want more information about how to write a problem and the other pieces. Wasn't clear what the difference was between the examples and solution image."	It was pretty good. Would be nice to know how you could invite colleagues to edit your patterns.

Interaction Notes

User Number	Task 1 - Locate a Design Pattern	Task 2 - Create a Design Pattern	Task 3 - Edit your Design Pattern

User Number	Task 1 - Locate a Design Pattern	Task 2 - Create a Design Pattern	Task 3 - Edit your Design Pattern
User 1	In performing this task, user isn't sure if an account is required. Clicks on "Login" first, enters her email address and a password not realizing she doesn't have an account on the system. After getting a login error, she finds the Register tab quickly and user registration process goes smoothly. Given task b (inline edit of a profile). Thought she should find patterns related to profile and searched on "profile" with no results. User is unfamiliar with design patterns. After reading "What is a Design Pattern" she says it's "still a bit vague." She goes to the "Recently Updated Pattern" on the front page and says the picture in the pattern helped her better understand what a pattern is. User doesn't appear to see "Browse Patterns" until now. She clicks on "Content Management" and when she doesn't see the pattern she needs, she opens but doesn't use "Filter by Category" at first. She then chooses Category->Profile Management and Status->Draft and finds nothing. She then changes Status-> Done and still finds nothing (there are no patterns in this category). User goes to front page and chooses "All Patterns" but again sees there are no "Profile Management" category patterns. She then goes to "Create a Pattern" and reads through the tabs. Skims the "How to write a Pattern page" and says she usually doesn't resort to "how to's" like this--she'll try to figure it it herself first before she reads them. When asked, user says if this were a real life situation she'd probably give up at this point.	(tabbed) Though user had briefly looked at "How to Write a Pattern" page, she doesn't go back as she's entering the pattern. Pattern entered: My Profile. Generally she doesn't enter much content in any of the fields throughout the process of entering patterns. In this first pattern, she enters required fields only: Title, Design Problem, Category, Solution, Image, Solution Image Source ("desktop"), How, Rationale. User enters title and category and tries to Preview right away. When she receives errors she says, "I should fill in all required." User submits again on the Problem tab. Goes to Solution tab, then Examples. Comments, "this Publishing Actions box is a bit confusing. If I click Submit I see errors. It should be clicked after I submit the whole thing, not each section." Goes to last tab and submits. User publishes and skims pattern. Other remarks: - did not click the "upload" button when specifying images.	(tabbed) On her own she moves to the next task without being asked. "I want to try to change my design pattern." Viewing her pattern, she clicks on one of the "Edit" buttons in the Related section which takes her directly to that tab. She adds "List Builder" as a related pattern, and says she expected this to change the images for the pattern--presumably to add the List Builder image as an example for her pattern. User comments that she usually starts by adding only required info in a form, then changing options. She also says she doesn't usually read the "how to" guide. Remarks that she usually just confronts the application to solve it herself
User 2	"Curious to see what's in here now...I don't know if pager is in here, but it's come up so I'm browsing patterns, content management, forms IO sounds promising, maybe navigation. Let's try information org, there is a search." (but she didn't use it.) "I don't think pager is in there." Goes to navigation and then sees Pagination. User had been at All Hands discussion of Pager and seen it on the wiki. "I wanted to see what's in here and if it's as complete as what's in the wiki right now. Is it complete or is it a subset? Maybe I have to log on to get more info."