Justin's Walkthrough
Sakai User Experience Walkthrough ReportÂ
(Accessibility and usability heuristic evaluations combined with cognitive walkthroughs)
Evaluation Completed by:Â Â Â [Insert evaluators name]
URL:Â http://bspace.berkeley.edu/portal/site/24cc0997-e57a-4f55-00ca-02eb0b35a6e7
Date: 22 Oct 07, 29 Oct 07
Heuristic reviews and cognitive walkthroughs were performed on a [Dell] at [screen resolution] resolution (most common resolution) and [color setting] colors (most common novice resolution; if graphics work at this resolution, they'll work in higher resolutions) using [Firefox and Win XP Pro SP2] at a [broadband] connection rate.
See UX Walkthrough Heuristics for heuristics used in this evaluation.
Scope of Walkthrough
•   See UX Walkthrough Work Breakdown
User profile(s) and context of use:
Profiles will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, users may be set up with a certain role in the system (e.g. student, instructor) for the heuristic evaluation.
•   See Sakai Personas
Scenarios
Scenarios will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, it is fine to use these scenarios for the heuristic evaluation if it is helpful to evaluators in figuring out how to walk through the application. See Content Management Scenariosfor a list of all possible scenarios. Evaluators will likely only cover some of the scenarios, in one of the roles, in their individual evaluations.
•   General Overview (heuristic evaluation)
•   Scenario 2
•   Scenario 3
•   Scenario 4
•   Scenario 5
Assumptions for this evaluation:
•   [Performed based on an instructor]
 Positive Findings
All positive findings, from both the heuristic evaluation and the cognitive walk-through, should be placed in this section.Â
Usability Positives |
Tool |
Evaluator |
---|---|---|
Clicking the help icon takes you directly to the help section for the tool it was called from. |
all |
JO |
Sortable tables show an arrow icon beside a heading. This shows both the method of sort (ascending/descending) and the heading which for which the sort is based on. |
all sortable tables (i.e. announcements and resources) |
JO |
Users are able to include only the tools they want for their site |
 |
JO |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Accessibility Positives |
Tool |
Evaluator |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Summary of Usability & Accessibility Issues FoundÂ
All usability & accessibility issues found, in both the heuristic evaluation and the cognitive walk-through, should be placed in this section.Â
Priority Legend:
High = Task cannot be completed
Medium = Task completed with significant effort and failed attempts
Low = Task completed with minor complications and/or annoyanceÂ
Usability Issues |
Principle |
Link to screen shots |
Priority |
Suggestions for solution |
Tool |
Component Identified? |
---|---|---|---|---|---|---|
 General |
 |
 |
 |
 |
 |
 |
Use of the term "Reset" for the mechanism to take the user back to the first page of the tool. Reset is often used to refer to clearing the information entered and not just returning to a previous location. |
Consistency and standards |
 |
 |
A term such as "Return" may be more appropriate |
All |
 |
There are three methods to organize content. Method 1 uses up/down arrows seen in the Syllabus, Method 2 uses drag and drop seen in the Site Info, and Method 3 uses numbers in a combo box seen in the Forums |
Consistency and standards |
 |
 |
Choose one or at most two methods of organization. |
Syllabus, Site Info, Forums |
Syllabus Items, Tool Order, Organize Forums and Topics |
Should the instructor be able to set permissions. Is this more of an admin function? |
Aesthetic and minimalist design |
 |
 |
Remove from the instructor's view |
Announcements, |
Permissions |
Both the Question Pools and Resources pages are used to manage a hierarchical system of information, but are designed with a different look and feel. |
Consistency and standards |
 |
 |
Choose one method to be used in all necessary areas |
Resources, Quiz |
Site Resources, Question Pools |
The horizontal navigation bar used for navigation within a tool is different across the tools, some user lighter text to show selection, others use darker, some are have vertical bars to separate items, others do not, some have no underline for the selected item, and etc. |
Consistency and standards |
 |
 |
Choose one method to be used in all necessary areas |
All |
 |
Syllabus |
 |
 |
 |
 |
 |
 |
Complicated work flow if the instructor just wants to add a file as the Syllabus. (Syllabus > create/edit > add > add attachment > upload file |
Flexibility and efficiency of use, Match between system and the real world |
 |
 |
Add an option along side create/edit to add a file. (System should probably also be able to display the contents of the file and not just a link) |
Syllabus |
|
The Add Syllabus form and editor do not display the same as the actual output. The style of bullets changes from the editor to the output. Also the preview shows the Title in a different font size and colour than is actually outputed. |
Consistency and standards |
 |
 |
The preview should be identitcal in appearance to the output on the site. The user may also like to be able to control the style of the Title. |
Syllabus |
Add Syllabus |
In the create/edit screen of the Syllabus tool, the "Preview" and "Reset" link perform the same actions. |
Consistency and standards, Aesthetic and minimalistic design |
 |
 |
If there are no other functions for the "Preview" link to that what is performed by the "Reset" link, "Preview" should be removed. |
Syllabus |
create/edit |
Redirecting a syllabus prevents other posted syllabus from being displayed. This is the case even after adding a new syllabus and verifying it's output with preview. |
Error prevention, visibility of system status |
 |
 |
The system should inform the user that a redirected website will block all other syllabus, or display all of them sequentially. |
Syllabus |
Redirect |
To remove a redirected syllabus (web site) the user has to erase the url and push the save button. For all other syllabus the user can remove after it by going to create/edit, which isn't even viewable if a redirect has been used. |
User control and freedom, consistency and standards, error prevention |
 |
 |
Provide an item in the syllabus list (create/edit) so that all of the syllabus can be managed in a like manor. |
Syllabus |
Redirect, create/edit |
When re-ordering syllabus in the Syllabus Item list, the user does not need to click the update button for the changes to take effect. Update is only needed for removal. |
Error prevention, User control and freedom, Visibility of system status |
 |
 |
Require that the user perform an update before the reordering takes effect, or rename the button to "Remove". Also disablling the button until an update/removal is needed will signal to the user when it is needed. |
Syllabus |
Syllabus |
Home |
|
 |
|
|
|
|
On the "Home" page, clicking "Options" under Worksite Information allows you to add text. However, clicking on "Options" under Recent Announcements provides no ability to add text. |
Consistency and standards |
 |
 |
Provide a link to the announcements editing page. |
Home |
Recent Announcements |
The "Title" and "URL" text fields are both smaller in width than the "Description" text area |
Consistency and standards |
 |
 |
The text fields do not limit the input to their size, so they should be equal in width to the Description text area, allowing for maximum visibility of information. |
Home |
Worksite Information > Options |
Site Info |
 |
 |
 |
 |
 |
 |
Should an instructor be able to add/remove users and change their role? Is this more of an admin function? |
Aesthetic and minimalist design |
 |
 |
Remove these features from the instructor, or put them under a subheading |
Site Info |
 |
The Edit Tool and Tool Order links have the same function but different implementations of it (with the exception of the ordering option for tool order) |
Consistency and standards, Aesthetic and minimalist design |
 |
 |
They should probably be combined or have the titles changed so that one indicates for keyboard use and the other for mouse. |
Site Info |
Edit Tools, Tool Order |
Slow to add/remove tools |
Visibility of system status |
 |
 |
There is a message displayed after the tool has been added/removed. There should also be a message indicating that the process is in progress. |
Site Info |
Tool Order |
The user can drag to add tools but can't drag to remove tools |
Consistency and standards |
 |
 |
The user should be able to drag the tool off of the column and drop it either onto the available tools column or drop anywhere off the current column to remove. |
Site Info |
Tool Order |
Removed tools do not show up in available tools until after a save is performed. |
Visibility of system status |
 |
 |
Tools should show up in the available tools list as soon as they are removed from the list of current tools. |
Site Info |
Tool Order  |
The "Reset" button is used to put the order of the tools to the default ordering. The term "Reset" is also used at the top as a means to return the tool to it's start page. |
Consistency and standards |
 |
 |
One or both of these items should be renamed. The "Reset" button could be renamed to "Default Order". |
Site Info |
Tool Order |
Assignments |
 |
 |
 |
 |
 |
 |
The descriptive text is unclear as to whether is is describing the combo boxes above or below it. |
Error prevention |
 |
 |
Information in the Assignment form needs to be borken up by white space to better seperate the sections. |
Assignments |
Add |
Polls |
 |
 |
 |
 |
 |
 |
When adding a poll, the user has to add the question and all possible answers on different pages. The only item that persists across pages is the question. If the user forgets what answers were already entered, they have to save, leaven and exit. |
Recognition rather than recall |
 |
 |
The user should be able to add the question and all possible answers on the same page. |
Polls |
Add |
Schedule |
 |
 |
 |
 |
 |
 |
When setting the time for an event, the time zone is displayed after the time combo boxes. However, the user is unable to change the time zone. |
Aesthetic and minimalist design |
 |
 |
The time zone seems to be set by the server, if the user has no control over it, it should not be displayed here. |
Schedule |
New Event |
The legend below the Calendar is quite large and contains many items that may never appear on any one calendar all at a time. |
Aesthetic and minimalist design |
 |
 |
It would be better to only include those items which are currently visible in the calendar. Another solution would be to have a tooltip for the icon rather than include the icon in the event's title. |
Schedule |
Calendar |
To scroll through the hours on the calendar, the user has to click on the links for earlier and later. |
Consistency and standards |
 |
 |
Users are often more familiar with using a scrollbar. |
Schedule |
Calendar |
Announcements |
 |
 |
 |
 |
 |
 |
The update button in the Announcements section is only used for removing announcements. This can cause confusion as the user can sort the list based on the table headings; however these changes don't affect the announcements posted on the site. |
Error prevention, User control and freedom, Visibility of system status |
 |
 |
Rename the button to "Remove". Also disabling the button until an announcement has been checked for removal. Another option would be to have a removal button next to each announcement instead of a check box. |
Announcements |
 |
Resources |
 |
 |
 |
 |
 |
 |
It is not clear what the resources are for. Items such as Quiz and Syllabus allow the user to upload the resource directly and don't seem to link to the resources for the user to select an item from there. |
Consistency and standards, Aesthetic and minimalist design, Match between system and the real world |
 |
 |
 |
Resources |
 |
When viewing other site's resources the user is given the same access level to the files but does not have access to the sort methods that they would on resources from their own site. Also, the items are listed with all of the information that the items in the user's table but the headings aren't easily matched. |
Consistency and standards |
 |
 |
When selecting a folder from another site it should be broken off to show the table headings for the columns that are displayed. This will allow the user to sort, understand what the information is, and separate it from the other resource folders available. |
Resources |
Site Resources |
When creating a new folder, uploading a file, or adding a web link there is an ' X ' at the far right. The tooltip for the ' X ' says "Remove this item". Clicking the ' X ' does nothing and there is already a cancel button at the bottom. |
Consistency and standards, Aesthetic and minimalist design |
 |
 |
remove the ' X ' |
Resources |
Upload files, Create folders, Add web links (urls) |
When adding a new folder there is an option to add another folder. This can be done when the first folder isn't created. Also it is unclear whether this other folder is meant to be a sub folder to the original or the one you are creating. |
Error prevention |
 |
 |
Only show the add another folder link once the original folder has been specified, or have an add multiple folders option. Also allow the user to add subfolders for the newly created folder, thereby distinguishing the levels of the folders created. |
Resources |
Create Folders |
The text document and the HTML page result in similar documents. The exception being that the HTML page has more formating options |
Consistency and standards |
 |
 |
Rename the html page to something like rich text document and/or remove the simple text document. |
Resources |
Create HTML page, Create text document |
The options section says " Select or unselect resources types to indicate whether resources of that type can be created in this site." Currently it is only a list of checked items which cannot be edited, except for Form item. Unchecking Form item and updating does not seem to have any effect. |
Consistency and standards, Aesthetic and minimalist design |
 |
 |
If these items are set by another user (i.e. an admin) this section shouldn't be called options, as all other instances of options contain editable options. It may be better to include it as a list somewhere on the main resources page or to omit it completely; as the user can easily determine the types of resources by hovering over add. |
Resources |
Options |
When hovering over add or actions when they are close to the bottom of the browser window, the context sensitive menus are cut off. |
Visibility of system status, Error prevention |
 |
 |
The menus should open upwards when at the bottom of the screen instead of always opening downwards. |
Resources |
Site Resources |
Can't collapse items into the top level folder. |
Consistency and standards, Aesthetic and minimalist design |
 |
 |
Allow the user to collapse contents of the top level folder. |
Resources, Drop Box |
 |
Roster |
 |
 |
 |
 |
 |
 |
There are two views of the roster, Overview and Pictures. The information provided for the picture view is cut off. |
Aesthetic and minimalist design, Recognition rather than recall |
 |
 |
Rather than having two separate pages, there should be one page with an option (e.g. a check box) to show pictures. When show pictures is selected the image should appear in a new Pictures column next to the name. |
Roster |
 |
The term "Roster" may not be immediately relevant to all users |
Consistency and standards, Match between system and the real world |
 |
 |
A term such as course list, course users, or course participants may be more applicable. |
Roster |
 |
Chat Room |
 |
 |
 |
 |
 |
 |
Chat room does not show current user in the "users in chat" list, to the current user. This prevents the user from knowing if others can see they are there. |
Visibility of system status |
 |
 |
The users should see everyone available in the chat, even themselves |
Chat Room |
 |
Email Archive |
 |
 |
 |
 |
 |
 |
There are two separate tools, one for sending e-mail (Mailtool) and one for receiving e-mail (Email Archive). |
Match between system and the real world, Consistency and standards, Aesthetic and minimalist design |
 |
 |
The MailTool and Email Archive should be rolled into one Email tool. If the instructor is the only one who needs to send e-mails this portion should be blocked from the students. Users are used to having their email clients able to both send and receive e-mail. |
Email Archive, Mailtool |
 |
Forums |
 |
 |
 |
 |
 |
 |
When adding a new Topic the user selects "New Topic", when the page opens it is called "Topic Settings" |
Consistency and standards |
 |
 |
Keep the name the same, the page should also be called New Topic |
Forums |
Topic Settings |
The buttons at the bottom of the Topic Settings page include "Save Settings" and "Save Settings & Add Topic". The first implies that only the settings are being saved and the second that the settings and the topic are being saved. In actuality "Save Settings" saves the topic and the other saves and opens a new page to add another topic. |
Match between system and the real world, Consistency and standards |
 |
 |
Change the names to something like "Save Topic" and "Save & Add Another Topic" |
Forums |
Topic Settings |
Gradebook |
 |
 |
 |
 |
 |
 |
When a category at the top is selected the text colour becomes a lighter shade to indicate that it is the currently selected category. However, the Upload/Import link is always a lighter colour |
Visibility of system status, Consistency and standards |
 |
 |
Make the text colour for Upload/Import follow the same conventions of the other links, being a darker shade when not selected. |
Gradebook |
 |
News |
 |
 |
 |
 |
 |
 |
Descriptive text between Page Title and URL text fields is not clear as to whether it refers to the field above or below. |
Error prevention |
 |
 |
Use whitespace to separate it from the lower field. |
News |
News Options |
Not clear that the URL field requires the URL for an RSS feed |
Error prevention |
 |
 |
Indicate that an RSS feed is required. Change "URL" to "RSS feed URL" |
News |
News Options |
Web Content |
 |
 |
 |
 |
 |
 |
The web content does not show the browsers icons when opened in a new page. Many users may not know the keyboard shortcuts for those actions. |
Recognition rather than recall |
 |
 |
Allow the page to load with a normal browser view, or provided icons within the browser to fulfill those functions. |
Web Content |
 |
Wiki |
 |
 |
 |
 |
 |
 |
When editting a wiki, clicking on the modifier icons for bold, italic, and etc, repeatedly doesn't remove a modification but adds it again instead. |
Consistency and standards, Error prevention |
 |
 |
A second click should remove the modification |
Wiki |
Wiki Editor |
Home and View appear to perform the same function. Both bring the user back to the home page of the wiki. |
Aesthetic and minimalist design, Consistency and standards |
 |
 |
If it performs the same function remove "view" |
Wiki |
 |
Mailtool |
 |
 |
 |
 |
 |
 |
There are two separate tools, one for sending e-mail (Mailtool) and one for receiving e-mail (Email Archive). |
Match between system and the real world, Consistency and standards, Aesthetic and minimalist design |
 |
 |
The MailTool and Email Archive should be rolled into one Email tool. If the instructor is the only one who needs to send e-mails this portion should be blocked from the students. Users are used to having thier email clients able to both send and recieve e-mail. |
Email Archive, Mailtool |
 |
Drop Box |
 |
 |
 |
 |
 |
 |
Can't collapse items into the top level folder. |
Consistency and standards, Aesthetic and minimalist design |
 |
 |
Allow the user to collapse contents of the top level folder. |
Resources, Drop Box |
 |
Accessibility Issue |
Principle |
Link to screen shot |
Priority |
Suggestions for solution |
Tool |
Component identified? |
---|---|---|---|---|---|---|
General |
 |
 |
 |
 |
 |
 |
It isn't always clear which item is selected when tabbing through a page. The selection bounding box is not always visible. |
 |
 |
 |
 |
 |
 |
After selection of a tool from the left navigation bar, the tab traversal resets and cycles back from the start. |
 |
 |
 |
It would be more helpful to the user if the tab traversal starts inside of the tool's frame |
all |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |