OSDPL User Testing - Round 1 Results

User 1

Berkeley

Female

25-30

Graduate Student

Student

Medium-high, unfamiliar with design patterns

Yes, Mac

User 2

Berkeley

Female

31-35

User Experience Specialist

Staff

Medium, slightly familiar with design patterns

Yes, Mac OS 10.5

User 3

Berkeley

Male

Over 60 years

Professor

Faculty

Medium-high, very familiar with design patterns

Yes, Mac

User 1 

All the time

All the time

A few times a month

A few times a month

All the time

Hardly ever

All the time

User 2

All the time

A few times a week

A few times a month

A few times a month

Never

Hardly ever

Never

User 3 

All the time

Hardly ever

A few times a month

Hardly ever

All the time

Never

A few times a week

User 1 

A few times a month

A few times a month

A few times a month

A few times a month

A few times a week

User 2 

A few times a month

Hardly Ever

Never

Hardly Ever

A few times a week

User 3

All the time

A few times a week

All the time

Hardly Ever

All the time

How easy or difficult was it for you to find a design pattern on the site?

neutral

easy

easy

How helpful did you find the design patterns you looked at on the site?

somewhat helpful

somewhat helpful

helpful

How helpful did you find the other information available on the site?

somewhat helpful

somewhat helpful

N/A

How helpful did you find the left navigation of the site?

helpful

helpful

helpful

How easy or difficult was it for you to discover how to create a design pattern?

neutral

easy

easy

How easy or difficult was it for you to create a design pattern in general?

difficult

easy

easy

Which version of the design pattern data entry form did you prefer?

slightly preferred tabbed (would have greatly preferred tabbed if it didn't have submit on every page)

slightly preferred single page

highly preferred single page

How easy or difficult was it for you to upload an image?

easy

easy (as long as it was the right image size)

very easy

How easy or difficult was it for you to describe and/or list the source of an image?

easy

difficult (didn't have one)

easy

If you encountered an error, was it easy or difficult to understand the feedback given?

easy

easy

easy

After submitting a design pattern, how easy or difficult was it to find your pattern on the site?

easy

easy (wasn't indexed, but "my pattern" worked)

easy

After creating the design pattern, how easy or difficult was it to edit it?

easy

easy

very easy

1) Why did/didn't you explore the various links in the navigation of the OSDPL? Could you tell me what you think the links in the left navigation mean? How could we make these pages or the link names of this navigation more helpful to you?

• not sure what the News section is. Thinks it's customer comments.
• Clicked on comment for one of the news items and tries to add a comment. 
• Input format text appearing below text field. Scanned it over. 
• Initially failed at Captcha, but then succeeds. 
• After submitting comment, she is surprised not to see it on the front page. 
• Clicks "Read More" and doesn't see it there. Goes back and clicks "1 comment" and finally finds it.
• User assumed comments were for the whole site, but now realizes these are comment on each News item.

• "I didn't feel I needed to read the FAQ." Would like to have the pages easily available "if I need them in context to what I'm doing."
•  "How to write a pattern was easy to find but I sort of wish it was built into the form somehow because I wanted to make sure I was writing this in a standardized way. Could have put the directions in the text field itself that could be edited away."
• "Doesn't really say how many patterns are in there"...then she sees all patterns. She wants to know total number but only two pages. "Can't imagine anyone wanting to come in and browse, searching on design problems. Inline edit -- I wouldn't have used this word."
• "If there are people who are new to this, there could be people who want to experiment with this and set up accounts and it gets filled with crap, especially if people don't realize their drafts are out there. Which are reviewed, favorited? What if two people submitted design patterns for the same problem? Would you expect people to collaborate or would there be competing designs."
• "Assuming people who come to this site have a design background."

Recently Added Patterns is good.

2) What tasks would you expect to perform on the OSDPL? How helpful was the general organization and navigation of the pattern library in supporting you in performing these tasks? What could we improve to make it easier?

"Who is the target of this site? Someone (that) is familiar with design patterns, that kind of person can figure it out easily, but (for) someone like me it could be difficult to solve the problem I have."

Feels that someone like her would be confused - coming to the site, wasn't clear how the site can help her with solving her problem / task. Comments that if she can see visual examples first, can better orient her.

• Would like to be able to submit incomplete drafts
• "I think the navigation is fine--there are certain things I haven't explored."

It was pretty good overall. Might be nice to have a way to create diagrams for the patterns.

3) Do the design patterns on the site have the type of information you need? What information could we add, subtract, reorganize, etc. to make it easier for you to use the patterns?

At first it took a while, but once she understood, she can now get to the information she wants.
Remarked at how it is not clear how it works after submitting the pattern.

"I'd want to give more thought to that. (I'd) compare it to the wiki." (Again, confusion about components vs. design patterns.)

4) Did you have all the information you needed at the right time to perform your tasks? What could we change to make it easier?

"Other than 'how to write a pattern," yes."

Make it clearer what to do and when I was done. I thought Submit was on a per tab basis.

5) Could you tell us which version of the pattern data entry form you preferred and why?

Liked the tabbed version because it looked more simple than the long version.
But if the submit button is changed to be on the last tab only, then perhaps it would be even better.

• I would have liked to be able to submit by answering the required questions
• Preferred long form mostly because she couldn't tell whether she could submit or not - the button was always there. If the buttons weren't on every page, she *did* like that the tabs told her what would be further down in the form without scrolling.

Preferred the single page. Even if there weren't submit buttons on every tab I would have preferred the single page.

6) What was your overall impression of the pattern creation and editing process? How can we improve it?

Make it clear what happens after a pattern is submitted. Doesn't know why the search isn't working.

"I want more information about how to write a problem and the other pieces. Wasn't clear what the difference was between the examples and solution image."

It was pretty good. Would be nice to know how you could invite colleagues to edit your patterns.

User 1

In performing this task, user isn't sure if an account is required. Clicks on "Login" first, enters her email address and a password not realizing she doesn't have an account on the system. After getting a login error, she finds the Register tab quickly and user registration process goes smoothly.

Given task b (inline edit of a profile). Thought she should find patterns related to profile and searched on "profile" with no results.

User is unfamiliar with design patterns. After reading "What is a Design Pattern" she says it's "still a bit vague." She goes to the "Recently Updated Pattern" on the front page and says the picture in the pattern helped her better understand what a pattern is. 
User doesn't appear to see "Browse Patterns" until now. She clicks on "Content Management" and when she doesn't see the pattern she needs, she opens but doesn't use "Filter by Category" at first. She then chooses Category->Profile Management and Status->Draft and finds nothing. She then changes Status-> Done and still finds nothing (there are no patterns in this category).

User goes to front page and chooses "All Patterns" but again sees there are no "Profile Management" category patterns. She then goes to "Create a Pattern" and reads through the tabs. Skims the "How to write a Pattern page" and says she usually doesn't resort to "how to's" like this--she'll try to figure it it herself first before she reads them.

When asked, user says if this were a real life situation she'd probably give up at this point.

(tabbed)
Though user had briefly looked at "How to Write a Pattern" page, she doesn't go back as she's entering the pattern.

Pattern entered: My Profile. Generally she doesn't enter much content in any of the fields throughout the process of entering patterns. In this first pattern, she enters required fields only: Title, Design Problem, Category, Solution, Image, Solution Image Source ("desktop"), How, Rationale.

User enters title and category and tries to Preview right away. When she receives errors she says, "I should fill in all required." User submits again on the Problem tab. Goes to Solution tab, then Examples. Comments, "this Publishing Actions box is a bit confusing. If I click Submit I see errors. It should be clicked after I submit the whole thing, not each section." Goes to last tab and submits.

User publishes and skims pattern.

Other remarks:
- did not click the "upload" button when specifying images.

(tabbed)
On her own she moves to the next task without being asked.

"I want to try to change my design pattern." Viewing her pattern, she clicks on one of the "Edit" buttons in the Related section which takes her directly to that tab. She adds "List Builder" as a related pattern, and says she expected this to change the images for the pattern--presumably to add the List Builder image as an example for her pattern.

User comments that she usually starts by adding only required info in a form, then changing options. She also says she doesn't usually read the "how to" guide.

Remarks that she usually just confronts the application to solve it herself

User 2

"Curious to see what's in here now...I don't know if pager is in here, but it's come up so I'm browsing patterns, content management, forms IO sounds promising, maybe navigation. Let's try information org, there is a search." (but she didn't use it.) "I don't think pager is in there." Goes to navigation and then sees Pagination.

User had been at All Hands discussion of Pager and seen it on the wiki. "I wanted to see what's in here and if it's as complete as what's in the wiki right now. Is it complete or is it a subset? Maybe I have to log on to get more info."

(single page)
User goes right to "Create Pattern.""It's not clear whether I should be logging in, I think I might but you haven't supplied me with a login." Sees message on front page that she needs to register to create a pattern and does so.

Uses "Create" link from main content area. Says "I'm not sure if I should create a pattern or how to write a pattern first and I assume I can toggle between the two." "I'm thinking of (creating a pattern for) watching an audio recording happening, but don't see a category for that. Am I allowed to create categories? I'm kind of stuck a bit, I can't ignore it. I have to choose one. Maybe I can choose (a wrong) one and change it later."

Enters title "Recording audio response" and says "I wonder ho descriptive it has to be." Entered tags and hit the "working" indicator on that field -- seemed to be thinking it was a refresh. Easily finds her name in contributors. Enters design problem, and comments, "is there a specific way I should be writing this?" She opens "how to create a design pattern" in a new tab and wonders if she'd opened it in the same tab if she would have lost the information she'd already entered into her form. [I tested and using the back button, she keeps most fields but does lose auto-complete fields.]

User carefully reads "how to create a design pattern," and says, "I would like a brief description of the expectation of how the problem is worded on the form itself, a pop-up or something." User then enters info from "How to create a design pattern" directly into design pattern entry form's Solution field. Says, "I really hope there is an opportunity to get feedback...did I include enough detail, do I really understand what these elements are meant to do, but I'll give it a shot." Finally sees solution image under solution and realizes "Solution" field isn't supposed to contain everything.

User wonders how the example images are different from the solution image. User reads Open Examples Guide quickly and thinks she needs to make sure she has permission to use someone else's work. Think "Example Image Source" may mean that you could link directly to an example on the web (it should be just the URL of the uploaded image). In "Relationships" the URL field is too wide for the page.

User looks for info on how design patterns and components relate on "How to write a design pattern" but can't find it and says "I need a definition of what a component is and what it means in relation to the pattern." User understands References section. User is a bit confused by "Publishing Actions" section and says, "I'm assuming I can come back to this at anytime...this is a draft submit."

Once uploaded a large image and it didn't show up with no error given. Another time, received an error message after attempting to upload a large JPG (9MB)
"warning: Invalid argument supplied for foreach() in /usr/webserver/content/osdpl/patterns/drupal-5.12/sites/default/modules/imagefield/imagefield.module on line 850.
Solution Image is required. Please upload an image." which she found confusing--didn't know what it meant (wonders if it was too big, it had to have certain dimensions or if there needed to be a certain naming convention). Tried one more upload and got the same message again. After talking about it, user does realize that solution image source isn't a link to an outside image, but a caption for the uploaded image.

User suggests that perhaps she could have used keywords instead of categories.

(single page)
Without prompting, user says "I want to see if the pattern I was trying to describe is in there" and searches on "audio" but gets no results. Thinks maybe it's not there because she forgot to use audio in pattern text and just keywords.

Goes to advanced search and thinks it's "cool" but that tags/keywords isn't something she can search on. Enters audio in "any of the words" and then sees there are no results on "audio audio."

User 3

User thinks about an issue he has with flashcards on a smartphone. Would like a "2 column thing" to move them back and forth. 
Looks at Browse Patterns and comments, "I don't know how much is there." "Content Management & Forms are different so I'll guess content management." User is unsure what Information Organization is.

User was very focused in this task about what the design pattern library told him about components -- seemed at first to think the design patterns were the same as the components. However, at the end of this task when asked about the relationship between design patterns and components (which is a somewhat leading question) he says, "I think it's close, but wouldn't assume there is a connection."

He looks at the Uploader pattern to see if he could use the Fluid Uploader, and when he sees the drag and drop patterns he says, "So this is the Reorderer stuff." User fairly quickly realized List Ordering & Layout Preview were children of Drag & Drop. Wonders if a reorderer may help him move things between two bins. User comments that "Use When" (especially showing there is a downside to using a pattern) is helpful in determining whether he would need LO or LP D&D.

- not sure what each category is and what content is in each.
- After finding a pattern by category, remarks that it doesn't feel there necessarily be a coupling between Fluid components and the patterns on the osdpl.

(tabbed)
Generally:
- Error messages and required fields confused the user right from the start.
- Fills everything in rather than risking hitting the submit button again. This indicated a lack of confidence in system or perhaps lack of clarity on the error?
- keeps scrolling back to the top to check error messages. Unclear whether or not user thinks they are new errors or old ones.
- User remarks on the tantalizing submit button on every page. But even if the submit button isn't there, you would worry about when you can submit.

Begin Task:
- Went to "How to Write a design pattern" FAQ first.
- then after reading it, decided to "dive right in" and clicked the "Create" pattern link in the center of the front page.
- got Access Denied error and was thrown off by the title of the error.
- Initially wasn't sure how to correct this, but slowly read the error message and decided to sign up for an account. (not sure how many users will actually do this).
- not sure if he will need to check his email as part of the registration process: but then said: "I guess we'll cross that bridge if we need to"

"Info Tab"
- When seeing the Contributor field, said: "naughty naughty. you should know who I am...I shouldn't have to say!"
- Button scope problem: "those buttons only look like they're referring to the current information, but they actually refer to the whole smash."
- Says, "What would preview mean...not sure so I'll Submit."
- "Filter cache cleared" status message appeared after clicking submit for the first time.
- Sees Submit button again and says, "Oh, I don't want to do that again!" because he knows that despite the appearance of the button on the tab, it's really not time to submit yet.
- Remarks: once making the Submit error once, he knew not to click Submit again until errors are fixed.

"Solution Tab"
- Browsed for an image and selected "OK". Then lingered over the "upload" button, as if pondering. Did not click it.
- Then read "Image Sources" and understood it for attribution (unlike User #2 who interpreted it as a synonym to a hyperlink).
- then reads instructions and clicks Upload.
- Went to the next field and started typing: "why is it showing up in red?"
-- doesn't realize it's one of the errors from a previous submit.
- Remarked that it felt like an awkward transition from getting hung up on uploading images and then going back to top level fields.
-- could we combine Solution image and Example image upload fields and attribution etc into a single page?
-User thinks the description of "How" below "How" applies to "Rationale," partially because the input fields are so big and the text above appears visually to go with the box below. He puts "Pros" in "Rationale" and seems unsure what it means.
- User realizes at this point that the red boxes show fields that are required.
"Examples Tab"
- doesn't realize that Examples are optional, but now entering them.
- Skips over the "Upload" button initially. Doesn't understand why you would want to Upload an image.
-- this implies that user is assuming that Submit should send it?
- Uploads an image. Now understands why you would want to upload an image.
- Remarks that it is pretty common that Alt text and title to be the same. Possibly combine it.
- "Alt text isn't required tsk tsk!" Wonders whether alt text and title should be the same, and says it seems common for them to be the same.
- When to submit is confusing: "don't know if I can submit yet, have to come back to the top...maybe Submit isn't there if I haven't filled in the right stuff."
"Accessiblity Tab"
- User isn't sure how the instructions around Accessibility Concerns relate.
"Relationships Tab"
- User looks for related design pattern in the Yahoo DP library
- User has to go back to the top of the page to see where he is in the process. Comments, it still says "Submit UI Design Pattern...I'd say maybe it's Create."
- "Since I was saving it as a draft, maybe I'd like to save it without all the required fields."

(tabbed)

(think this task was skipped)

User 1

(single page)
Pattern entered: new Layout Preview.

Creates a Layout Preview Pattern. Enters required fields only again. This time she reads "Solution Image Sources" text more closely and enters "igoogle.com."

User wonders where the tabs are and we explain they are not part of this form. After submitting pattern she appears confused by solution image, thinking it's an actual rich text editor and tries clicking on it.

Other remarks:
- did not click the "upload" button when specifying images.

(single page)

This task was skipped.

Does this task before #4 & #5. Goes to View/Edit My Patterns. Wonders if this is the only way to see her pattern, so she searches on "My Profile" and gets no results.

Comments that she just saw top-rated patterns and wants to see what others have done. Looks at Drag & Drop List Ordering and then to Drag & Drop Layout Preview as it was linked within the text of D&D-LO.

• wasn't sure that "Browse patterns" would give a list of examples.
• "Contribute" isn't clear. Maybe just "create a pattern" instead.
• "Browse pattern" - not sure if the position is good. When hovering over Browse pattern she was expecting to get a tooltip.
• Suggests to indicate on the main page to indicate clearly that the Browse pattern menu is the collection of patterns on the site.
• Category name "Form" is ambiguous.
• "Editing and making a design pattern is easier than I thought because the process is simple and I can find the ways to find the design patterns easily.
  

User 2

(Tabbed)
User thought she should be the only contributor at first, then realizes it's anyone who contributed to the pattern. Wonders, "should I be forced? What if I didn't select myself, would I be listed as a contributor just because I authored it?"

About the form: "one thing I find surprising is that I have to scroll up (to move to the next tab). Maybe there should be a "continue" button or tabs at the bottom?"

User realizes that red fields are required. User wants to be able to submit after she's filled in just the required fields, saying, "I could have tried to submit after solution because I think that was the last red one (required item), but I wasn't really sure I could submit at that point because the first two tabs had set me up to think I couldn't." "I'd expect that I could start and then if I put in enough info so that I could find it at a later date that would be enough. It depends on how long of a stretch of time it was. I need to get a sense of everything that is required I think." "I don't think I'd write this up in one sitting. I'd put it up here and if I was working with a colleague they could come in and edit it somewhat." "I kind of like the fact that it's broken up into tabs in some ways." To make when she can submit more clear, she says, "initially I thought perhaps you could put all the required fields in one tab, but that breaks down the problem & solution tabs."

(Tabbed)
User is discouraged from earlier problems with search and says, "hopefully I can f