Allison's Evaluation
Sakai User Experience Walkthrough ReportÂ
(Accessibility and usability heuristic evaluations combined with cognitive walkthroughs)
Evaluation Completed by:Â Â Â Allison Bloodworth
URL:Â http://sak-qa.berkeley.edu
Date: 7/24/07
Heuristic reviews and cognitive walkthroughs were performed on a MacBook Pro at 140 x 900 resolution (most common resolution) and Color LCD colors (most common novice resolution; if graphics work at this resolution, they'll work in higher resolutions) using Firefox 2.0.0.5 on a somewhat slow wireless connection.
See UX Walkthrough Heuristics for heuristics used in this evaluation.
Scope of Walkthrough
•   See UX Walkthrough Work Breakdown
User profile(s) and context of use:
Profiles will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, users may be set up with a certain role in the system (e.g. student, instructor) for the heuristic evaluation.
•   See Sakai Personas
Scenarios
Scenarios will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, it is fine to use these scenarios for the heuristic evaluation if it is helpful to evaluators in figuring out how to walk through the application. See Content Management Scenarios for a list of all possible scenarios. Evaluators will likely only cover some of the scenarios, in one of the roles, in their individual evaluations.
•   General Overview (heuristic evaluation)
•   Scenario 2
•   Scenario 3
•   Scenario 4
•   Scenario 5
Assumptions for this evaluation:
•   [List any additional assumptions used in evaluation]
 Positive Findings
All positive findings, from both the heuristic evaluation and the cognitive walk-through, should be placed in this section.Â
Usability Positives |
Tool |
Evaluator |
---|---|---|
On the "Upload Files" page, the blinking "Processing" instruction gives users immediate and clear feedback that the system is working. |
Resources |
AB |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Accessibility Positives |
Tool |
Evaluator |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Summary of Usability & Accessibility Issues FoundÂ
All usability & accessibility issues found, in both the heuristic evaluation and the cognitive walk-through, should be placed in this section.Â
Priority Legend:
High = Task cannot be completed
Medium = Task completed with significant effort and failed attempts
Low = Task completed with minor complications and/or annoyanceÂ
Usability Issues |
Principle |
Link to screen shots |
Priority |
Suggestions for solution |
Tool |
Component Identified? |
---|---|---|---|---|---|---|
It is not clear to users how the different Resources tools in different sites relate to each other. Specifically, it is not clear whether Resources in My Workspace is its own separate instance of the tool, which doesn't include Resources from all the users' sites, or whether it is an aggregation of all the resources in the different Resources tools. The "Show/Hide Other Sites" link is styled strangely, making it hard to tell it is a link. |
Visibility of System Status, Consistency & Standards |
 |
Medium |
Rename Worksite Setup's Resources tool to make it clear it is a global tool which shows the Resources for all the users' sites. Remove the underline from the arrow in front of the "Show/Hide Other Sites" link to make it clear that it is a standard link, and that the arrow (which is not part of the link) shows that there is something under it. |
Resources |
Responsive Disclosure "Show/Hide" widget |
When you just have one folder is not clear why there are two separate "Add" and "Actions" drop-downs. Especially since the "Actions" drop-down only has two options (Edit Details & Reorder) for a folder, is it really necessary to give the users extra information to parse? |
Aesthetic & Minimalist Design, Recognition Rather than recall. |
 |
Low |
Consolidate the drop-downs as appropriate, or make them look clearly different (and perhaps locate them differently) to indicate their different functions. |
Resources |
 |
"X" button on the "Upload Files" page allowing you to remove a file from the files to be uploaded is an unusual affordance, and not clearly associated with the file it affects. This is evidenced by the fact what it does must be explained in directions at the top of the page. It also doesn't work unless you have more than one file (e.g. it has no effect on a single file. |
Consistency & Standards |
 |
Medium-Low |
Reorganize the page and use visual design to better indicate the association between related components. |
Resources |
 |
The "Display Name" title on the "Upload Files" page for an uploaded file is not clear. Where will this name be displayed? |
Match between the system and the real world. |
 |
Medium-Low |
Give instructions for entering this data and explain where it will be used/displayed, perhaps in pop-up help. |
Resources |
(Pop-up) Help Component |
The "Add Another file" button on the "Upload Files" page isn't clearly associated with the uploaded files. |
Consistency & Standards |
 |
Low |
Re-organize the page to group related components and show relationships. |
Resources |
Add Another of the Same |
When multiple files are uploaded, it isn't clear that the "Email Notification" is associated with all the uploaded files. |
Consistency & Standards |
 |
Low |
Re-organize the page to group related components and show relationships. |
Resources |
|
When a file is being uploaded on the "Upload Files" page, it seems that the "Cancel" button should work to cancel the action, but it doesn't. Even the "Stop" button on my browser didn't work to stop an overly long upload (over 10 minutes) and I had to log out. This was only an issue with a slow wireless connection. |
Consistency & Standards, Help users recognize, diagnose & recover from errors |
 |
Medium-Low |
Allow the Cancel button to work when files are in the process of being uploaded, or don't display it when it doesn't have an effect. |
Resources |
 |
The directions on the "Upload Files" page are incorrect. They reference an "Upload" button, but the button is really called "Upload Files Now." It is an indication of a somewhat confusing interface that these directions have to be given at all. |
Error Prevention, Consistency & Standards |
 |
Medium-Low |
Re-organize the page to group related components. Put buttons which save changes in clear, standard places (consider having them always on the very bottom -- and maybe the top -- of the page, without directions underneath then as they are here). |
Resources |
 |
The "Access" label by itself is unclear. It is also not clear that you should be able to get to it from the "Edit Details" page. Inside "Edit Details" it is called "Availability & Access." More confusingly, there is a section on this page called "File Details," and an unlabeled section at the top. This makes it seem like the unlabeled section at the top and "Availability & Access" *are not* covered under the "Edit Details" page title. |
Match between the system and the real world, Consistency & Standards |
 |
Low |
Consider a new label for Access on the main page, or pop-up help. Use the same label consistently in the tool. Consider a new organizational scheme for the information items in the Edit Details screen, and possibly re-naming it to better convey the different types of information it contains. |
Resources |
(Pop-up) Help Component |
In the "Optional Properties" section of "Edit Details," how does "Creator" and "Date Created" related to the uneditable "Created by" and "Created" fields at the top of the page? Where are these fields displayed? It seems that they are only displayed in this "Optional Properties" section and don't affect anything else, but this is not at all clear. |
Match between the system and the real world, Consistency & Standards |
 |
Medium |
Remove redundant fields, or rename them to clearly show how they are different. |
Resources |
 |
In the "Optional Properties" section of "Edit Details," how should "Subject & Keywords" be entered? A comma separated list? Where are these fields displayed? |
Error prevention |
 |
Medium-Low |
Give instructions for entering this data and explain where it will be used/displayed, perhaps in pop-up help. |
Resources |
(Pop-up) Help Component |
In the "Optional Properties" section of "Edit Details," how should the date-related fields be entered? Where are these fields displayed? |
Error prevention |
 |
Low |
Give instructions for entering this data and explain where it will be used/displayed, perhaps in pop-up help. |
Resources |
(Pop-up) Help Component |
Users see different options when from the "Upload Files" screen they "Show Details for this item" vs. when they "Edit Details." How do you change the dates the files can be viewed in "Edit Details?" - bSpace bug ONLY |
Consistency & Standards |
 |
High |
Add missing options to "Edit Details." |
Resources |
 |
Used the Back button in Resources and got a blank screen. Wouldn't have known how to recover from this if I didn't have prior domain knowledge. After I go to the "Edit Details" screen, where I really just want to "View Details," I shouldn't have to feel like I'm canceling an action just because the page is misnamed. It would make more sense to be able to choose the back button. |
Consistency & Standards |
 |
High |
Users expect the back button to work based on their experiences with other web sites and web applications. Make it work. Perhaps also add Breadcrumbs to encourage users to navigate to higher level pages instead of using the back buttons. |
Resources |
Breadcrumbs |
On the "Edit Details" page, the user is allowed to "Change file type." This is almost never done in a traditional app; if the user wants to change the file extension, they access this by changing the name |
Consistency & Standards |
 |
Low |
Allow users to change the name & the file type at the same time |
Resources |
 |
"Copy," "Move" & "Remove" links are presented in a completely not-standard format. It is very confusing as it is in essence a second somewhat hidden nav that appears below the global action links. These are normally treated as buttons in the rest of Sakai. |
Consistency & Standards |
 |
Medium |
If you need to operate on multiple items at once, in the current scheme for remove it is a column at the end with checkboxes. Then there is an update link on the bottom. I'm guessing Copy could have a single link on each line as announcements has an "Edit" link I tried to click a box an choose Copy but it didn't seem to do anything at all except turn the lines orange. |
Resources |
 |
I wasn't really sure what the color scheme in this tool was about--if it selects items, that (and what to do next) should be made clearer. |
Error Prevention, Consistency & Standards |
 |
Medium |
 |
Resources |
 |
Upload Download Multiple Resources link is very midleading for users looking for this function, as you can up/download multiple resources in the normal (Site Resources) screen and the only thing you can do at this link is use Web-DAV. It seems this name is a bit redundant, since the title above it is "Resources." |
Consistency & Standards |
 |
Low |
Come up with better naming conventions. E.g. "Up/Downlowad Multiple Resources with WEB-DAV." Resources should be called "Site Resources" on most sites; in My Workspace it should be called "All Resources." The main page could be called "Resource Viewer." |
Resources |
 |
Resources was in a strange state for me, with lots of the rows colored. I am not sure how it got that way, but may have had to do with copying or reordering files. I clicked on a clipboard icon to the left of the "Add" drop-down for the top folder, and the page revered back to it's normal color and apparently state. I have no idea what occurred there. Aha! It duplicated some files. I had no idea how to finish that duplication step, and clicking on that clipboard icon was not at all a logical way to complete that action. I guess the idea was that I was pasting copied items here, but that is not a logical workflow; it would seem from the context that the copy link would also create copies of the files. |
Prevent Errors |
 |
High |
 |
Resources |
 |
I keep wanting to click the items in Resources to edit them instead of download them. |
Prevent Errors |
 |
Medium |
Perhaps we need separate links for download and other functions, and nothing should be linked from the resource title itself. I have to think about it almost every time to get it right when I'm editing. |
Resources |
 |
Somehow I created a strange red file which makes Resources blank when I click on it. It disappeared when I logged out and logged back in. - Can't reproduce. It appears this was the icon for a binary file, which was uploaded when I encountered an error somehow. |
Prevent Errors |
 |
High |
Don't use a red icon for this. Make it clear what has happened when there is an error instead of placing a red file icon there. |
Resources |
 |
It is confusing that both the folder the user is in as well as the contents of the folder are shown. Though the containing folder is indented slightly, this isn't prominent and the user may not realize there is a distinction between the two. This is very different from the metaphors in most file systems where each folder in the hierarchy can be opened to view its contents, while maintaining the overall view of all the folders in the system. The metaphor to navigate to the containing folder for the current selection is different from the metaphor to navigate to its containing folder (there you use breadcrumbs), which can be confusing. Especially if the user navigates away from Resources and then comes back, when state is saved, it can be especially unclear where they are in the hierarchy. |
Consistency & Standards, Recognition rather than Recall |
 |
Medium |
Use a more standard presentation of the breadcrumbs which a user will recognize at a glance. It may be that the folder icons and "Location: " header throw people off. |
Resources |
Where am I/Breadcrumbs |
When a URL is "uploaded", if the user enters http://, the // is changed to __ which breaks most links (though not Google, strangely enough). Users also cannot edit the URL after it's initially entered |
Prevent Errors, Help users recognize, diagnose, and recover from errors, Consistency & Standards |
 |
High |
 |
Resources |
 |
Accessibility Issue |
Principle |
Link to screen shot |
Priority |
Suggestions for solution |
Tool |
Component identified? |
---|---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Cognitive Walkthrough Worksheet
This section should be used by evaluators to keep track of the steps & screens they followed through the application in the cognitive walk-through. Any positive results or issues found should be included in the two sections above.
Scenario 1:Â Brief description
# |
Step |
Screen |
Comments/issues |
Principle |
Suggestions for solution |
---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Scenario 2:Â Brief description
# |
Step |
Screen |
Comments/issues |
Principle |
Suggestions for solution |
---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Scenario 3:Â Brief description
# |
Step |
Screen |
Comments/issues |
Principle |
Suggestions for solution |
---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Scenario 4:Â Brief description
# |
Step |
Screen |
Comments/issues |
Principle |
Suggestions for solution |
---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |