UX Walkthrough pilots - Daphne, Clayton & Colin

Original reporting template

(Accesibility and usability heuristic evalutions combined with cognitive walkthroughs)

Evaluation Completed by: Daphne Ogle (DO), Clayton Lewis (CL), Colin Clark (CC)

URLs:

Tools Heuristic Overview: https://sak-qa.berkeley.edu/portal/; using the '107 Plant Morphology' site as our target since it is suggested as a "typical" course site
Scenario Walkthroughs: http://bspace.berkeley.edu;
Each evaluator used a unique instructor login created for the evaluation and created individual course sites.

Date: August 15, 21, & 22, 2007

See UX walkthrough guidelines for heuristics and cognitive walkthrough questions used in this evaluation.

Scope of Walkthrough

Functional areas (Sakai content-management related Tools) focused on for review:

Announcements
Home
Assignments
Drop Box
Email Archive
Forums
Mail Tool
Resources
Syllabus
Web Content

User profile(s) and context of use:

Persona: Sarah Windsor - Overwhelemed Faculty

Scenarios:

General Overview of content management tools (traditional heuristics-style)
Create course site
Add syllabus to site

Assumptions for this evaluation:

Through working in groups, usability and accessibility perspectives will both be represented

bSpace (Berkeley's instance of Sakai) is a decent representation of Sakai instances. Evaluators will look for issues related directly to the skin which is much different than the Sakai out-of-the-box skin.

Summary of Positives found

Usability Positives	Tool	Evaluator
Permissions matrix format is mostly consistent across tools. Users can leverage learning in one tool for usage in others.	All tools > Permissions	CL
When adding participants to a site via their email address, the system checks for valid email address. Protection against users entering the wrong address and not realizing it.	Worksite Setup & Site Info > Add participants	CL







Accessibility Positives	Tool	Evaluator
Tool & content distinction pretty clear
While creating course, default of current semester is good - in most cases user won't have to move down in drop down.
WYSIWYG editor listens to common keyboard shortcuts

Priority Legend:

High = Task cannot be completed

Medium = Task completed with significant effort and failed attempts

Low = Task completed with minor complications and/or annoyance

Summary of Issues Found

Usability Issues	Principle	Link to screen shots	Priority	Suggestions for solution	Tool	Component Identified?
*General*
On initial view of system it is not clear why I see what I see (what role am I) and how that compares to what others see. This is particularly important because of the hierarchical structure of a class and the need to protect students confidential information.	Visibility of system status		High	Display users role on every page. Allow instructors to "see what students" see. Rough designs were started on this for the gap work during Sakai's grant days. Recommendation for next step: Component creation	All	"Role identifier" and/or "Student View"
Permissions settings are not human readable ( ex. "all.groups", "new")	Match between system and real world - System should speak user's language	Permissions	Medium	Use terminology that matches user's mental model of the action. The label should be in the "verb + noun" format. For example, "new" in the Assignments Tool could be replaced with "create assignments". Tool tips on the permission settings could add additional information if needed. Recommendation for next step: Analysis of all permission matrix for consistency in language. Customize terminology to each particular tool.	All
Permissions table is not clear. It is not immediately clear is the row or column is the 'role'	Minimalist design	Permissions	Medium	If it's meant to be spreadsheet like then make it look more like a spreadsheet with gridlines and the such. Recommendation for next step: Various visual and interaction design models should be reviewed to determine the best representation. The redesign could become a component. IMPORTANT: The fact that the matrix is consistent is a positive so any changes should be made across the board.	All > Permissions	Permissions
If the user plays around with permissions, there is no way to get back to the default unless they remember.	Recognize rather than recall		High	Add a "revert to default" button (better terminology than default should be used) Recommendation for next step: Add above functionality to Sakai out-of-the box and/or include it in a permissions component	All > Permissions	Permissions
*Gradebook*
"Release assignment" is jargon -- not likely a term familiar to instructors.	Match between system and real world - System should speak user's language		Medium	New GB designs for 2.5 have a definition link built into several terms like this. Term is always displayed clickable link that on click displays the definition. Need to confirm this was implemented with IU. Not sure this is accessible. Recommendation for next steps: Use GB behavior as starting point to design and develop a "term definer" component. Perhaps this could be more like a 'tool tip on demand'.	Gradebook	Term Definer
*Home*
"Worksite info" is not a familiar term to users. It doesn't align with the idea that they are working in their Course site.	Match between system and real world - System should speak user's language	Home	Medium	Rename to "Course site information". The appropriate course type should be inserted for "course". Recommendation for next steps: Sakai community should change the language in the out-of-box version.	Home	Home
Arrangement of tools is inflexible and doesn't always match user needs. For instance, new announements may be most important as deadlines approach yet they are buried below the fold.	User control & freedom		Low	Allow users to easily reorganize this page as appropriate for current situation	Page layout/organizer
*Syllabus*
If user has uploaded a pdf of their syllabus it displays as a link to the pdf rather than the contents displaying in-line on the page. The users viewing the pdf are required to manage multiple windows since the pdf opens seperately. This particularly cumbersome for keyboard and screen reader users.	Minimalist design	Syllabus w/pdf link	Medium	Display contents of the pdf on the page (as if an image). The root of the issue is the metaphor the tool currently uses -- syllabus are created within the tool so each document uploaded is treated as if an item in the syllabus. The more typical user model is to have an existing syllabus document that they can either display in this tool or copy and paste into the tool. Recommendation for next steps: - Redesign of syllabus tool to allow for it's current model plus the 2 mentioned above. The tool could ask users what they want to do and direct them to the right workflow. For example, "do you have an existing document you want to display?" - Coordinate with Cambridge on the Resources viewer to see if it can be leveraged.	Syllabus	Resources Viewer (look at what Cambridge is doing here)
*Schedule*
Intra-tool navigation links at the top of the page are not clear from their labels -- they are not explicit and some use language that will be unfamiliar to users in this context.	Match between system and real world - System should speak user's language	Schedule - intra-tool navigation	Low	Use explicit terminology in the form of "verb + noun". Use terms that match users model of the action (i.e. "Display more schedules" rather than "Merge"). Recommendation for next steps: Card sorting will help create better language. Then change the labels in Sakai out-of-box.	Schedule
If user has no other schedules to merge (they don't own other site for instance), when they arrive at the "Merge" page, there is nothing listed. This is further confused by the unclear label of the page itself.	Match between system and real world - System should speak user's language	Schedule > Merge	Medium	Rather than a blank list, display a message: "You do not have any other schedules to display. You may only display schedule items from other sites you own." (needs better language). Recommendation for next steps: - See above terminology change for nav and page label - Display message above in Sakai out-of-the box	Schedule > Merge
Title of Merge page and instructions refer to "calendar" but "schedule" is used everywhere else	Consistency & Standards	Schedule > Merge	Low	Use either calendar or schedule everywhere when referring to the same thing. Recommendation for next steps: Change calendar to schedule in those places (see next issue for caveat)	Schedule > Merge
Calendar is more common terminology for what that schedule tool is (ical, Oracle calendar, Outlook calendar, etc.)	Consistency & standards System should speak user's language		Low	Change tool name and all related references to Calendar. Recommendation for next steps: Sakai community should change this language. It is a bit tricky since tool name changes are not backward compatible (at least they have not been yet). Any existing sites would keep the tool title Schedule. Some analysis should go into the trade-offs in user experience.	Schedule
On the Fields page, it's not clear what "fields" are (same goes for intra-tool nav label). Am I choosing new fields, do I need to choose from this list? Also, what kind of interaction will these have for users creating new items: date for instance?	Recognition rather than recall - User should not have to remember information from another part of the system.	Schedule > Fields	Medium (edge case functionality not used often)	Rename this functionality to something that better matches the user's mental model. Even something like "Schedule item attributes" would be better (although attribute is pretty geeky). OR create a 'wysiwyg interface' for this. The fields can be edited while looking at a view of the form used to create schedule items Recommendation for next steps: - Card sorting to find out how user's semantically think about this functionality. Then change the terminology in Sakai out-of-the box. - OR redesign entire page as described above	Schedule > Fields
On the Fields page, the remove checkboxes do not look connected to the fields. Also the initial inclination is that checked items will appear and unchecked won't since this is a common interaction in Sakai.	Consistency and Standards	Schedule > Fields	Medium (edge case functionality not used often)	At a minimum make the left table column less wide so there is a visual connection between the item in the list and it's checkbox. Even better change the interaction so there is no doubt what the user is doing. Some ideas are a "wysiwyg interface" (described in issue above) or a different layout where the fields are checked if they're included. Recommendation for next steps: - Adjust column width in Sakai out-of-the box (short term) - Redesign page in wysiwyg style (long term)	Schedule > fields	Form editor (perhaps far reaching but could somehow be related to the form creation in OSP assuming they start with a good default)
User expects to be able to click on date to enter a new activity.	Consistency and Standards		Medium	Clicking on date should open form to add a new date with known information filled in (i.e. date & time)		Calendar, Add calendar item
*Assignments*
List navigator widget defaults to "show 200 items per page" even though there is only 1 item in the list. Would there ever be 200 assignments?	Visibility of system status		Low	Default to more reasonable number. The style guide says the default is 25. Since the number of assignments varies by teaching style it would be even better if this widget was "smart" and remembered the users last setting. The few times they set it differently than their norm they simply change it back (better than changing it every time). Recommendation for next steps: Create Fluid component: List Navigator. Make it a smart component as described above.	Assignments	List Navigator
In permissions, the site ID displays at the top of the page which means nothing to users. In fact it looks pretty scary.	Minimalist design - every extra unit of information competes with the relevant information	Schedule > Permissions	Low	Hide the site ID. It adds no value to users. Recommendation for next steps: Sakai community should hide the site ID on this page.	Schedule > permissions
*Email Archive*
Site ID as email address means nothing to users. There is NO WAY they will remember that address. It is not needed since the user creates an address.	Minimalist design	Email Archive	Low	Hide the site ID email. In fact, there is no need to create one since users are forced to create one for the site. Recommendation for next steps: Hide the site ID email in Sakai out-of-the-box	Email Archive
*News*
Choosing "graphic version" links opens a new window which displays the graphic version forcing the user to manage multiple windows (additionally difficult for keyboard and screen reader users)	Visibility of system status & Accessibility?		Medium	At minimum add text letting the the user know the link will open in a new window so it is not as jarring. Even better display the graphic version inside the tool -- replacing the text version. For the latter a "text only" link would also be needed that toggles with the "graphic version". Recommendation for next steps: Apply one of the above solutions to Sakai out-of-the box.	News
Only 1 RSS feed is allowed which is very similar to the web content functionality aside from the URL.	Consistency and Standards -- because of other RSS readers users expect an aggregator		Low	Treat this tool as an aggregator and allow several RSS feeds to be added. The page will need some redesign to allow for multiple to display. Recommendation for next steps: Create Fluid component	News	RSS Aggregator
News requires the user connect to an RSS feed to display in the window. However, the form asks only for a URL with no indication that it must be an RSS format.	Error prevention	News > Options	Medium	Add example and/or text to explain this is for an RSS feed. A bonus would be to list possible RSS's users in higher ed might be interested in. This would need to be customizable by institution. Recommendation for next steps: - Add example and/or text to existing RSS feeder (short term) - Create a Fluid Component (long term)	News	RSS Aggregator
*Web Content*
If the user does not rename their web content tool and includes more than 1 in a site, it is impossible to tell the difference between the 2 without opening each tool (left nav labels are identical)	Recognition rather than recall	Left Nav w/ 2 web content tools	Medium	User's shouldn't have to remember what is behind "web content". Given the function of the tool, to display another website within the site. Make renaming the tool required rather than optional. Recommendation for next steps: Make renaming the tool required in Sakai out-of-the-box	Web Content (left Nav)
The options form asks for both 'tool title' and 'page title'. It's not clear what the difference is or which one will change the name in the left navigation.	Match between system and real world. Consistency and Standards Recognition rather than recall	Web Content > Options	High (edge case functionality?)	Use labels that clearly state what these labels point to -- in the user's language. Users may have the concept of a tool but likely not a page. Recommendation for next steps: Card sorting could help define the right labels. Also look at what uPortal does. Are there some synergies here?	Web Content > Options
*Site Info*
"Site Info" is not a term familiar to most faculty. For instructors the tool is more than "info" it is an administrative tool.	Match between system and real world - speak the users language.		Low	For roles that are allowed to edit the site, such as instructors, rename to "Course Site Administration" (where "course" is replaced with the appropriate site type). For roles without site edit ability rename to "Course site Info" since it is just a read-only tool for them. Adding the site type helps with the overall terminology confusion of "site", "worksite", "course site" being used interchangeably throughout Sakai. Recommendation for next steps: Rename tool as suggested above in Sakai out-of -the-box. Currently tool name changes are not backward compatible. From a usability perspective, it is very important for this tool to be named the same across sites. Perhaps the name change should only take place for newly adopting institutions until backward compatibility is possible.	Site Info
Various terminology on the site info page is confusing. "Site", "worksite", "course site" are used interchangeably throughout Sakai. We should be explicit so user's don't need to guess what is meant.	Match between system and real world - speak the users language.	Site Info - terminology	Low (but easy fix)	Rename title "107 Plant Morphology" to "Course site for 107 Plant Morphology". Rename "site description" to "course site description" Recommendation for next steps: Change terminology as specified above in Sakai out-of-the-box.	Site Info
Intra tool navigation links include "edit tools" and "tool order". As an instructor it is unclear the difference between these 2.	Recognition rather than recall. Group like functionality together when it enhances understanding.	Site Info - Intra-nav bar	Medium	At a minimum, move the links closer together so it's clear they are 2 different options. The more usable fix is to integrate the 2 into one area of functionality. Both of these functions scream for a wysiwyg interface -- user sees their current tool bar on the page and they work directly on it to add tools and change their order. Recommendation for next steps: - Move the 2 links next to one another in Sakai out-of-the-box (short term) - Integrate the funtionality into one page using a wysiwyg design metaphor (long term)	Site Info / Worksite Setup
The "Import from site" intra-tool nav link doesn't clearly express what will happen when clicked. The page itself doesn't clearly express how this functionality works either. As a user I'm left asking, which material? how & where does it get imported? What about dated materials? Etc.	Recognition rather recall - Instruction for use of the system should be visable or easily retrievable whenever appropriate.	Site Info - Import from site	Medium (High)	This functionality conceptually "Imports material into this site from other sites the user owns". How to say this in a few words will take some more thinking. At a minimum: 1) add a tool tip to the intra-tool nav link on the main Site Info page to describe what it does, 2) add instructions to the page itself to explain what happens on import. Explaining dates, etc. could get long so perhaps a short set of instructions with an "extras on demand" design pattern to see more would be best. Recommendation for next steps: - Design & implement above in Sakai out-of-the-box. (short term) - Combine functionality of "Import from Site" and "Import from File" under one intra-tool nav link as described below in "import from file" issue. (long term)	Site Info > Import from Site
The "Import from file" intra-tool nav link doesn't clearly express what will happen when clicked. The page itself doesn't clearly express how this functionality works either. As a user I'm left asking, which material? how & where does it get imported? What about dated materials? Etc.	Recognition rather recall - Instruction for use of the system should be visable or easily retrievable whenever appropriate.