Kathy's Evaluation

Sakai User Experience Walkthrough Report 

(Accessibility and usability heuristic evaluations combined with cognitive walkthroughs)

*Evaluation Completed by:Kathy Moore

http://wiki.fluidproject.org/pages/editpage.action?pageId=1705336

9/10/07
Heuristic reviews and cognitive walkthroughs were performed on a Dell pc using an 1152px resolution and millions of colors using Win xp Firefox 2.0.0.6 on a T1 connection.

See Heuristic Evaluation for heuristics used in this evaluation.

Scope of Walkthrough

Announcements, Home, Assignment, Drop box, Email Archive, Forums, Mail tool, News, Resources, Syllabus, Web content,

*User profile(s) and context of use: Ed, student

Profiles will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, users may be set up with a certain role in the system (e.g. student, instructor) for the heuristic evaluation.

•    See Sakai Personas

Scenarios

Scenarios will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, it is fine to use these scenarios for the heuristic evaluation if it is helpful to evaluators in figuring out how to walk through the application. See Content Management Scenarios for a list of all possible scenarios. Evaluators will likely only cover some of the scenarios, in one of the roles, in their individual evaluations.

  • General Overview (heuristic evaluation)
  • Scenario: Your instructor has made next week's assignment available.  You are to write a 10 page paper reflecting on what you've learned half way through the term.  Complete and turn in the assignment (since this is a paper you likely do this over a number of sittings).  NOTE:  You must have created this assignment as an instructor first.
  • Scenario: Each week you need to post a topic and comment on one other student's post.  Do your homework for this week's topic.  You can respond to your own comment.
  • Scenario: Figure out what the instructor expects you to do this week for class.

Assumptions for this evaluation:

•    [List any additional assumptions used in evaluation]

 Positive Findings

All positive findings, from both the heuristic evaluation and the cognitive walk-through, should be placed in this section. 

Usability Positives

Tool

Evaluator

Viewing an individual announcement, student sees helpful navigation buttons (Previous, Return to list, Next. Good navigation.

Announcements

Kathy

Viewing an individual email, student sees helpful navigation buttons (Previous, Return to list, Next. Good navigation.

Email Archive

 

After opening an assignment, the student sees buttons above and below the assignment leading back to the full listing. Good navigation.

Assignments

 

Could that pattern be usefully applied to other Sakai tools? Could it possibly work in Resources? Here's a design pattern that Sakai uses well in a number of places; are there more where it could be used?

 

 

Clear indication of status: After beginning the assignment and saving a draft, the student who returns and opens an assignment sees 'Assignment - In progress
Fill out form. Make sure you click "Submit". '

Assignments

 

Feedback that an assignment has been submitted is very clear, in the assignments listing and because the user can send himself a confirmation.

Assignments

 

 

 

 

 

 

 

 

 

 

Accessibility Positives

Tool

Evaluator

Page is structured into logical sections

Home

Kathy

Important navigational elements are high on the page

Home

 

Tab order is logical and prioritizes major navigational  elements: Login, sites, tools

Home

 

Page and type resize in response to user actions

Home

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Summary of Usability & Accessibility Issues Found 

All usability & accessibility issues found, in both the heuristic evaluation and the cognitive walk-through, should be placed in this section. 

Priority Legend:

High = Task cannot be completed

Medium = Task completed with significant effort and failed attempts

Low = Task completed with minor complications and/or annoyance 

Usability Issues

Principle

Link to screen shots

Priority

Suggestions for solution

Tool

Component Identified?

Tool menu difficult to scan, no obvious organizational principle

Flexibility and efficiency of use

Is the order of information natural and logical?

Medium

-   Alphabetize,
-   Provide alphabetize button
-   Let user customize

Home

Drag and drop utility?

In Syllabus tool, document attachment doesn't offer immediate view of content

Flexibility and efficiency of use

Low

Is there a way to show word or pdf doc in own i-frame?

Syllabus

 

In Syllabus tool, print button appears when only an attachment is visible; it has no meaning since it vanishes when document opens and can be printed.

Error prevention Match between system and real world

Medium

Do not display print button if the only content is an attachment

Syllabus

 

In forums, page is blank if there's been no instructor input

Visibility of system status; Help and documentation

 

Medium

Display default text for students (and instructors?)

Forums

Automatic notification of last changes/ next step

In Worksite setup, By default, student does not have ability to do anything but edit/delete "My Workspace"
Should student be allowed to delete "My Workspace?"What has to happen for student to create project site?

Visibility of system status; Help and documentation
 Error prevention

 

Medium


Worksite setup

Default text for students
(and instructors?)
Automatic notification of last changes/ next step, customized for role

Default name for "Web Content" tells user nothing.

Does a page title accurately describe its purpose?

High

Do not allow user to add tool without custom name; eliminate default in "add tool" process for this tool.


Web Content

 

Initial News page and links come up in Sakai but  clicked links open in new tab

Visibility of system status;
Consistency (and standards)

 

Medium

Continue to show content within Sakai?(Discuss accessibility of alternatives.)

News

Provide portal-like display widgets to give user control of link display. (uPortal/LMS convergence?)

In absence of clear direction from Instructor, student must look in many places to see what is new in a class. (Syllabus, Schedule, Announcements, Assignments, Gradebook, Forums, Resources

Flexibility and efficiency of use

 

High: not  insurmountable but a constant annoyance likely to impact success of pedagogy

A "What's New" tool /utility might be a good thing This could be useful in uPortal as well.

Home

A "What's New" tool /utility might be a good thing This could be useful in uPortal as well.

Permissions section in Forums is forbidding and intrudes itself into student attempt to post Topic

Aesthetic and minimalist design

Medium

Allow faculty to set defaults, subject to change only upon request;
Design pattern: Details on request

Forums

New permissions utility

Does "Gradebook Assignment" belong under the "Permissions" heading?

Are headings and titles clear and distinct?

Medium

Put this elsewhere and label it meaningfully

Forums

 

Submit buttons in Forums bury "Go" among too many choices

Is navigation consistent and intuitive? Does the application follow platform and usability conventions?

Medium

Rethink integration of settings here Break buttons into two rows; one shows standard Save/Cancel
(Post Topic/Cancel)

Forums

 

No confirmation after posting

System visibility:
Does the system always keep users informed about what's going on?

Medium

 

Forums

 

There's more
-- but is this too much detail on one tool, in the absence of component ideas?

 


 

 

Forums

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


Accessibility Issue

Principle

Link to screen shot

Priority

Suggestions for solution

Tool

Component identified?

False? User is forced to tab through sometimes-lengthy tool menu to reach tool controls within page  (Is this true?)

Are shortcuts or skip links provided?

 

Medium

Are there shortcuts that aren't obvious to me?

All

Shortcut utility/coding

I-frames constrain type awkwardly when user requires small screen/big type

Is text still visible? Does it overlap?

Medium

 

Home

 

WSYWYG editor doesn't seem keyboard-accessible;

Can't tab to formatting choices

Can move cursor to text entry box, but typing does nothing

Are there any areas where you get stuck and have to use a mouse?

 

High

 

Schedule, others

 

Action buttons in Resources (Add, Actions) do not seem accessible by tab

 

 

High

 

Resources

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Cognitive Walkthrough Worksheet

This section should be used by evaluators to keep track of the steps & screens they followed through the application in the cognitive walk-through. Any positive results or issues found should be included in the two sections above.

  • Scenario: Your instructor has made next week's assignment available.  You are to write a 10 page paper reflecting on what you've learned half way through the term.  Complete and turn in the assignment (since this is a paper you likely do this over a number of sittings).  NOTE:  You must have created this assignment as an instructor first.

Tool: Assignments

#

Step

Screen

Comments/issues

Principle

Suggestions for solution

 

1

 

After opening an assignment, the student sees buttons above and below the assignment leading back to the full listing. Good navigation. (plus)

Flexibility and effieciency of use

 

 

2

 

After beginning the assignment and saving a draft, the user who opens an assignment sees 'Assignment - In progress
Fill out form. Make sure you click "Submit". (plus)

Clear indication of status

 

 

3

 

Feedback that the assignment has been submitted is very clear, in the assignments listing and because the user can send himself a confirmation. (plus)

Clear indication of status

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Scenario: Each week you need to post a topic and comment on one other student's post.  Do your homework for this week's topic.  You can respond to your own comment.

#

Step

Screen

Comments/issues

Principle

Suggestions for solution

1

After filling in other fields, scroll down to find submit button; whoops, there's more...

The permissions section is more than I need. Should students even see this?I don't know much about most of the custom roles, do I? Can't I do a blanket setting?

Aesthetic and minimalist design

Make permissions a section to be opened on demand rather than displaying all this by default

1

After filling in other fields, scroll down to find submit button; whoops, there's more...

Does "Gradebook Assignment" belong under the "Permissions" heading?

Are headings and titles clear and distinct?

This might make more sense somewhere else on the screen.

1

Still looking to submit new topic

"Post" button is buried in middle of four. Finding the right ("post message") button requires a larger mental load than necessary.

Is navigation consistent and intuitive? Does the application follow platform and usability conventions?

Rethink integration of settings here.
Break buttons into two rows; one shows standard save/cancel

2

Add Topic (with third button above)

No confirmation

Visibility of system status

Show confirmation or list of topics

3

Try to "reply" (thinking that topic has begun the thread)

No obvious way to reply to post
("You can respond to your own comment." ??)

System Visibility

 

3

Try to reply, post new message

There's a lot of room for confusion between "forum." "topic," and "thread."

Match between system and real world;Is plain language used?

 

 

 

 

 

 

 

 

 

 

 

 


  1.  
  2.  
  3. Scenario: Figure out what the instructor expects you to do this week for class.

#

Step

Screen

Comments/issues

Principle

Suggestions for solution

1-8

Check
Syllabus
Schedule
Announcements
Assignments
Gradebook
Section Info
Forums
Resources

 

This is a lot of places to check. The instructor has probably made it clear what the procedure is, but...

Flexibility and efficiency of use

A "What's New" tool /utility might be a good thing This could be useful in uPortal as well.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


Scenario 4:  Left in place just in case...

#

Step

Screen

Comments/issues

Principle

Suggestions for solution