Allison's Evaluation

Sakai User Experience Walkthrough Report 

(Accessibility and usability heuristic evaluations combined with cognitive walkthroughs)

Evaluation Completed by:    Allison Bloodworth

URL:  http://sak-qa.berkeley.edu

Date: 7/24/07

Heuristic reviews and cognitive walkthroughs were performed on a MacBook Pro at 140 x 900 resolution (most common resolution) and Color LCD colors (most common novice resolution; if graphics work at this resolution, they'll work in higher resolutions) using Firefox 2.0.0.5 on a somewhat slow wireless connection.

See UX Walkthrough Heuristics for heuristics used in this evaluation.

Scope of Walkthrough

•    See UX Walkthrough Work Breakdown

User profile(s) and context of use:

Profiles will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, users may be set up with a certain role in the system (e.g. student, instructor) for the heuristic evaluation.

•    See Sakai Personas

Scenarios

Scenarios will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, it is fine to use these scenarios for the heuristic evaluation if it is helpful to evaluators in figuring out how to walk through the application. See Content Management Scenarios for a list of all possible scenarios. Evaluators will likely only cover some of the scenarios, in one of the roles, in their individual evaluations.

•    General Overview (heuristic evaluation)
•    Scenario 2
•    Scenario 3
•    Scenario 4
•    Scenario 5

Assumptions for this evaluation:

•    [List any additional assumptions used in evaluation]

 Positive Findings

All positive findings, from both the heuristic evaluation and the cognitive walk-through, should be placed in this section. 

Usability Positives

Tool

Evaluator

On the "Upload Files" page, the blinking "Processing" instruction gives users immediate and clear feedback that the system is working.

Resources

AB

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Accessibility Positives

Tool

Evaluator

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Summary of Usability & Accessibility Issues Found 

All usability & accessibility issues found, in both the heuristic evaluation and the cognitive walk-through, should be placed in this section. 

Priority Legend:

High = Task cannot be completed

Medium = Task completed with significant effort and failed attempts

Low = Task completed with minor complications and/or annoyance 

Usability Issues

Principle

Link to screen shots

Priority

Suggestions for solution

Tool

Component Identified?

It is not clear to users how the different Resources tools in different sites relate to each other. Specifically, it is not clear whether Resources in My Workspace is its own separate instance of the tool, which doesn't include Resources from all the users' sites, or whether it is an aggregation of all the resources in the different Resources tools. The "Show/Hide Other Sites" link is styled strangely, making it hard to tell it is a link.

Visibility of System Status, Consistency & Standards

 

Medium

Rename Worksite Setup's Resources tool to make it clear it is a global tool which shows the Resources for all the users' sites. Remove the underline from the arrow in front of the "Show/Hide Other Sites" link to make it clear that it is a standard link, and that the arrow (which is not part of the link) shows that there is something under it.

Resources

Responsive Disclosure "Show/Hide" widget

When you just have one folder is not clear why there are two separate "Add" and "Actions" drop-downs. Especially since the "Actions" drop-down only has two options (Edit Details & Reorder) for a folder, is it really necessary to give the users extra information to parse?

Aesthetic & Minimalist Design, Recognition Rather than recall.

 

Low

Consolidate the drop-downs as appropriate, or make them look clearly different (and perhaps locate them differently) to indicate their different functions.

Resources

 

"X" button on the "Upload Files" page allowing you to remove a file from the files to be uploaded is an unusual affordance, and not clearly associated with the file it affects. This is evidenced by the fact what it does must be explained in directions at the top of the page. It also doesn't work unless you have more than one file (e.g. it has no effect on a single file.

Consistency & Standards

 

Medium-Low

Reorganize the page and use visual design to better indicate the association between related components.

Resources

 

The "Display Name" title on the "Upload Files" page for an uploaded file is not clear. Where will this name be displayed?

Match between the system and the real world.

 

Medium-Low

Give instructions for entering this data and explain where it will be used/displayed, perhaps in pop-up help.

Resources

(Pop-up) Help Component

The "Add Another file" button on the "Upload Files" page isn't clearly associated with the uploaded files.

Consistency & Standards

 

Low

Re-organize the page to group related components and show relationships.

Resources

Add Another of the Same

When multiple files are uploaded, it isn't clear that the "Email Notification" is associated with all the uploaded files.

Consistency & Standards

 

Low

Re-organize the page to group related components and show relationships.

Resources


When a file is being uploaded on the "Upload Files" page, it seems that the "Cancel" button should work to cancel the action, but it doesn't. Even the "Stop" button on my browser didn't work to stop an overly long upload (over 10 minutes) and I had to log out. This was only an issue with a slow wireless connection.

Consistency & Standards, Help users recognize, diagnose & recover from errors

 

Medium-Low

Allow the Cancel button to work when files are in the process of being uploaded, or don't display it when it doesn't have an effect.

Resources

 

The directions on the "Upload Files" page are incorrect. They reference an "Upload" button, but the button is really called "Upload Files Now." It is an indication of a somewhat confusing interface that these directions have to be given at all.

Error Prevention, Consistency & Standards

 

Medium-Low

Re-organize the page to group related components. Put buttons which save changes in clear, standard places (consider having them always on the very bottom -- and maybe the top -- of the page, without directions underneath then as they are here).

Resources

 

The "Access" label by itself is unclear. It is also not clear that you should be able to get to it from the "Edit Details" page. Inside "Edit Details" it is called "Availability & Access." More confusingly, there is a section on this page called "File Details," and an unlabeled section at the top. This makes it seem like the unlabeled section at the top and "Availability & Access" *are not* covered under the "Edit Details" page title.

Match between the system and the real world, Consistency & Standards

 

Low

Consider a new label for Access on the main page, or pop-up help. Use the same label consistently in the tool. Consider a new organizational scheme for the information items in the Edit Details screen, and possibly re-naming it to better convey the different types of information it contains.

Resources

(Pop-up) Help Component

In the "Optional Properties" section of "Edit Details," how does "Creator" and "Date Created" related to the uneditable "Created by" and "Created" fields at the top of the page? Where are these fields displayed? It seems that they are only displayed in this "Optional Properties" section and don't affect anything else, but this is not at all clear.

Match between the system and the real world, Consistency & Standards

 

Medium

Remove redundant fields, or rename them to clearly show how they are different.

Resources

 

In the "Optional Properties" section of "Edit Details," how should "Subject & Keywords" be entered? A comma separated list? Where are these fields displayed?

Error prevention

 

Medium-Low

Give instructions for entering this data and explain where it will be used/displayed, perhaps in pop-up help.

Resources

(Pop-up) Help Component

In the "Optional Properties" section of "Edit Details," how should the date-related fields be entered? Where are these fields displayed?

Error prevention

 

Low

Give instructions for entering this data and explain where it will be used/displayed, perhaps in pop-up help.

Resources

(Pop-up) Help Component

Users see different options when from the "Upload Files" screen they "Show Details for this item" vs. when they "Edit Details." How do you change the dates the files can be viewed in "Edit Details?" - bSpace bug ONLY

Consistency & Standards

 

High

Add missing options to "Edit Details."

Resources

 

Used the Back button in Resources and got a blank screen. Wouldn't have known how to recover from this if I didn't have prior domain knowledge. After I go to the "Edit Details" screen, where I really just want to "View Details," I shouldn't have to feel like I'm canceling an action just because the page is misnamed. It would make more sense to be able to choose the back button.

Consistency & Standards

 

High

Users expect the back button to work based on their experiences with other web sites and web applications. Make it work. Perhaps also add Breadcrumbs to encourage users to navigate to higher level pages instead of using the back buttons.

Resources

Breadcrumbs

On the "Edit Details" page, the user is allowed to "Change file type." This is almost never done in a traditional app; if the user wants to change the file extension, they access this by changing the name

Consistency & Standards

 

Low

Allow users to change the name & the file type at the same time

Resources

 

"Copy," "Move" & "Remove" links are presented in a completely not-standard format. It is very confusing as it is in essence a second somewhat hidden nav that appears below the global action links. These are normally treated as buttons in the rest of Sakai.

Consistency & Standards

 

Medium

If you need to operate on multiple items at once, in the current scheme for remove it is a column at the end with checkboxes. Then there is an update link on the bottom. I'm guessing Copy could have a single link on each line as announcements has an "Edit" link I tried to click a box an choose Copy but it didn't seem to do anything at all except turn the lines orange.

Resources

 

I wasn't really sure what the color scheme in this tool was about--if it selects items, that  (and what to do next) should be made clearer.

Error Prevention, Consistency & Standards

 

Medium

 

Resources

 

Upload Download Multiple Resources link is very midleading for users looking for this function, as you can up/download multiple resources in the normal (Site Resources) screen and the only thing you can do at this link is use Web-DAV. It seems this name is a bit redundant, since the title above it is "Resources."

Consistency & Standards

 

Low

Come up with better naming conventions. E.g. "Up/Downlowad Multiple Resources with WEB-DAV." Resources should be called "Site Resources" on most sites; in My Workspace it should be called "All Resources." The main page could be called "Resource Viewer."

Resources

 

Resources was in a strange state for me, with lots of the rows colored. I am not sure how it got that way, but may have had to do with copying or reordering files. I clicked on a clipboard icon to the left of the "Add" drop-down for the top folder, and the page revered back to it's normal color and apparently state. I have no idea what occurred there. Aha! It duplicated some files. I had no idea how to finish that duplication step, and clicking on that clipboard icon was not at all a logical way to complete that action. I guess the idea was that I was pasting copied items here, but that is not a logical workflow; it would seem from the context that the copy link would also create copies of the files.

Prevent Errors

 

High

 

Resources

 

I keep wanting to click the items in Resources to edit them instead of download them.

Prevent Errors

 

Medium

Perhaps we need separate links for download and other functions, and nothing should be linked from the resource title itself. I have to think about it almost every time to get it right when I'm editing.

Resources

 

Somehow I created a strange red file which makes Resources blank when I click on it. It disappeared when I logged out and logged back in. - Can't reproduce. It appears this was the icon for a binary file, which was uploaded when I encountered an error somehow.

Prevent Errors

 

High

Don't use a red icon for this. Make it clear what has happened when there is an error instead of placing a red file icon there.

Resources

 

It is confusing that both the folder the user is in as well as the contents of the folder are shown. Though the containing folder is indented slightly, this isn't prominent and the user may not realize there is a distinction between the two. This is very different from the metaphors in most file systems where each folder in the hierarchy can be opened to view its contents, while maintaining the overall view of all the folders in the system. The metaphor to navigate to the containing folder for the current selection is different from the metaphor to navigate to its containing folder (there you use breadcrumbs), which can be confusing. Especially if the user navigates away from Resources and then comes back, when state is saved, it can be especially unclear where they are in the hierarchy.

Consistency & Standards, Recognition rather than Recall

 

Medium

Use a more standard presentation of the breadcrumbs which a user will recognize at a glance. It may be that the folder icons  and "Location: " header throw people off.

Resources

Where am I/Breadcrumbs

When a URL is "uploaded", if the user enters http://, the // is changed to __ which breaks most links (though not Google, strangely enough). Users also cannot edit the URL after it's initially entered

Prevent Errors, Help users recognize, diagnose, and recover from errors, Consistency & Standards

 

High

 

Resources

 


Accessibility Issue

Principle

Link to screen shot

Priority

Suggestions for solution

Tool

Component identified?

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Cognitive Walkthrough Worksheet

This section should be used by evaluators to keep track of the steps & screens they followed through the application in the cognitive walk-through. Any positive results or issues found should be included in the two sections above.

Scenario 1:  Brief description

#

Step

Screen

Comments/issues

Principle

Suggestions for solution

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


Scenario 2:  Brief description

#

Step

Screen

Comments/issues

Principle

Suggestions for solution

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


Scenario 3:  Brief description

#

Step

Screen

Comments/issues

Principle

Suggestions for solution

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


Scenario 4:  Brief description

#

Step

Screen

Comments/issues

Principle

Suggestions for solution