User Testing Results - CMHR Keypad

Trends

icon used for screen reader on/off key is not clear
- Visually identified as a captions button
- Tactile cue is unclear - should use a more accepted tactile convention (possibly a speaker with sound waves)
- Volume controls could be up/down arrows or +/- symbols grouped together with the screen reader on/off key
buttons could be a little bigger and more spaced out
symbols/icons on the keys should be bigger and/or better defined/easier to identify
labelling keys with braille and raised print would help identify the key functions
function of the single zoom button is unclear - suggest having a pair of keys: zoom in key and zoom out key.
back button is visually and tactilely unclear - use conventional straight back arrow instead of circular arrow (confused with refresh key)
home key symbol feels like an arrow
navigation keys and select key functions were clearly understood
up/down navigation keys were expected to navigate up and down when a visual grid was presented

User 1

Button design: Size, spacing, labels, etc.

user found the raised symbols on the keys difficult to identify
- recognised a circle shape on the zoom key and an "S-like" shape on the info key (question mark)
- in general did not find the shapes of the keys or the symbols useful in identifying key function
pressed the "?" button because it had an intriguing S shape.
found the home and back keys confusing
finds the positioning of the keys more useful in understanding their function (is familiar with a numerical keypad layout).
thought that the key sizes and spacing were good
when asked if she would add any keys to the keypad she stated she would add the 4 and 6 keys
suggested that we add a key that would allow the screen reader to read back letters or words, as well as to adjust voice rate.
longer keys with braille on the surface would be helpful. Braille labels adjacent to the keys would not be as effective as braille directly on keys.
user wishes for better audio feedback when pressing the keys.

Tactile Identification

user guessed that the keypad was based on a numerical keypad, such that the top row of help/home/undo keys were 1/2/3, and wondered why the 4 and 6 key were missing
user guessed that the column of keys on the left side of the keypad must be function keys
after trial and error in using the keypad the user stated that she felt the shape of the up/down/left/right navigation keys made their function clear
after an initial encounter, understands that the middle button is for select.
user referred to the select key as the "8" key

Screen Reader

When asked to turn on the screen reader:
- pressed all the keys until she found the screen reader on/off key.
- then wanted to turn up the volume but couldn't identify a key that would allow her to do that
- stated that she is feeling for a key with a "V" letter (for volume) or braille
- stated that key textures matter a lot

Zoom

Was not tested

General

when asked to describe overall experience and comfort level with the keypad, the user stated that it was pretty good, and better than other keypad studies she'd been involved with
user rated comfort level with technology a 2 (where 1=very comfortable, 2=comfortable and 3=uncomfortable)
uses screen reader when on computer at home but no other custom devices

User 2

Button design: Size, spacing, labels, etc.

user likes simplicity of the keypad
key spacing is good - no changes needed there.
user would change keypad to have two zoom keys, in and out, and would place them on the right-hand side of the keypad.
- this would have the added benefit of aesthetic balance.
- having a single zoom button by itself in the corner makes it seem more like a utility (i.e. search).
user would change the undo/back key symbol to a back pointing arrow and would put it on a square key in order to avoid confusion with back navigation key
make the buttons a little bigger, especially the navigation buttons.

Visual Identification

visually identified zoom key as a search key
visually identified undo key as a refresh key
thought audio key might be a captions key, but wasn't sure
visually identified the intended functions of the home key, help key, volume controls and navigation keys
user would have selected the undo/back key to go back to last page if he had recognised it as such (used navigation and select keys instead to navigate back to last page)
when asked to turn on the screen reader user selected the intended key, but said it was only through process of elimination

Software Experience

lack of initial focus confusing - didn't know where to start
expected up and down arrows to move focus up and down

Zoom

when asked to zoom, used the "refresh" key because thought the arrow was for zoom.
eventually tried the magnifying glass to zoom in and out

Screen Reader

when asked to turn on the screen reader, found the key by process of elimination - did not expect the "captions" button to be used for this purpose.

Alternate keyboard

when shown alternate keyboard design #1, thought that round speech bubble still indicated captions.
when shown alternate keypad design #1 (which includes a back/undo key and a skip forward key), suggested that if you provide the user with a back/undo key, there should also be a forward key (like a web browser back and forward)
when asked about an alternate symbol/icon for screen reader on/off key, suggested a speaker symbol, with up/down arrow keys below it for volume up/down
does not think forward/back arrow keys would be confused with navigation keys, especially if the back/forward keys are square
prefers layout of alternate keypad #1
thinks that alternate keypad #2 (with zoom and volume modes) would be too confusing

General

when asked to describe overall experience and comfort level with the keypad, the user stated that it was pretty good, but that there were 3 keys which he found difficult to identify
user rated comfort level with technology a 1 (where 1=very comfortable, 2=comfortable and 3=uncomfortable)
uses keyboard-only input with personal computer on a regular basis

User 3

Button design: Size, spacing etc.

would put a little more space between navigation key group and upper row of keys
noticed later in the test that there was a larger gap between the zoom key and the audio key group
- thought that the space between the zoom key and the audio on/off key should be the minimum key spacing everywhere on the keypad
would make the keys a little larger in general, except for the navigation keys which are a good size
felt in general that the symbols on the keys are too small and too detailed to be identifiable
likes the texture and concavity of the navigation arrow keys - texture helps prevent slippage
- flat, smooth keys (like the Apple keyboard) are hard to use because they are slippery
thought the key groupings were logical, but might put the home key first in the top row, followed by the back/undo key, then the help key.
- would put the help button last because it won't be used right away, home key first because it is the beginning
the shapes of the keys didn't help the user personally, but she appreciated the fact they were different shapes. Said that it may help others even if it didn't help her.

Labels and Braille

suggested that text and Braille labels beside the keys would be good, in addition to larger raised symbols on the keys
- user remarked that braille grade 2 is more common
- raised print is also good since not all non-sighted users are braille literate.
- navigation keys do not need labels because their function is clear; could maybe label the whole cluster as "Navigation"
could have audio instructions telling you where each key is located

Tactile Identification

thought the zoom key was a Q at first but eventually identified it as a magnifying glass and thought it would have a zooming function
- wasn't sure at first if pressing the zoom key again would zoom it back out
identified the question mark and understood that it would be some kind of help key
back/undo key was not clear at first (said it felt like a water droplet)
- later, when asked to navigate to a previous page, the user stated that "there must be a back key" and eventually identified the back key as a "circular arrow"
home key was not clear (thought it might be a left arrow)
- when asked to go back to the first page of the interface, used the back button to go back step by step
screen reader on/off key was not clear (said it feels like a fish)
volume key functions were not clear
immediately identified the 4 triangular keys as navigation keys and the center key as an "OK" button (identifed the central circle as "O for OK")
- understood that these navigational keys were for "moving around on the page"
in general, when exploring the keypad, user is looking for audio and tactile clues to help give direction

Screen Reader

user was able to enable the screen reader after some trial and error and process of elimination
- user already understood that some of the keys were navigation, other keys felt like arrows, so that left the keys with unknown shapes.
suggested that "voice" or "speech" would be good text labels for the screen reader on/off button
suggested that a "bucket on its side with brackets" might be used as a symbol for the screen reader on/off button
- otherwise could not suggest an alternate symbol for screen reader on/off button

Zoom

thought the zoom button might bring up a menu with zoom levels to choose from
wasn't sure if the zoom key was pressed a second time if it would zoom back out or not
user knew that this was the zoom button because it was the only button left to press
thought "Mag" could be used as text lable; later decided that "zoom" would be better
thought having two zoom keys (in and out) would be better, especially because it would match the up/down volume key pairing

Volume

found the volume control buttons through process of elimination - knew roughly what the other keys were, so the remaining keys were likely volume.
- thought that the up volume key felt like some kind of arrow
suggested up and down arrows for volume key labels, with a text label ("VOL") between them or beside them
- user suggests that volume arrows would not be confused with navigation keys since the navigation cluster has well defined function and is spatially different
- suggested that if the up and down volume keys were square this would also distinguish them from the triangular navigational keys
when asked, thought that + and - might be OK for up and down volume symbols too
the speaker symbols on the volume control keys were hard to decipher due to their small size and lack of context.

Navigation

when asked about why she used mainly the left/right navigation keys and not the up/down navigation keys user stated that she expected the links would go across the page, and that either the left/right or up/down keys would do the same thing.
user did start using the up/down keys when the left/right keys didn't seem to be working.
"when I surf the web at home I use either the tab key to go from link to link or the up and down arrow keys if I want to listen to a line of text/links"

Back

thought that the back/undo key should be labelled with a back arrow

Home

when it was pointed out that there was one key left to identify, user identified the home key symbol (understood it was a house) and thought that it would function like a home key to take you back to first page.
stated that "usually when there is a back key there is also a home key, like on a web browser"
reiterated that the home symbol is unclear, and likely needs to be bigger

Voting Machine

user described her experience with accessible voting machines
- these have an audio prompt asking which language to choose, as well a simple keypad.
- when asked, user wasn't clear on how the interaction was initiated - suggested that maybe there was a sensor that detected her presence (which started the audio), or if the attendent started the session for her.

General

when asked to describe overall experience and comfort level with the keypad, the user stated that it was a little hard to use, but she appreciated the tactile features, found the screen reader helpful, and the navigation keys were easy to identify; it could be simpler.
user rated comfort with technology a 2 (where 1=very comfortable, 2=comfortable and 3=uncomfortable)
uses ergonomic keyboard (with concave keys) and screen reader when on computer at home
noted that self-check-outs in grocery stores sense your presence and just start talking to you
felt that it would be necessary to have someone in the museum tell you what is available at the kiosks/how they work in order to have a good experience

User 4

Button design: Size, spacing etc.

did not find that the key groupings helped to identify the key functions
found the square keys more helpful in identifying the functions
user stated that symbols were not helpful.
user found volume, zoom, and screen reader keys easier to push
- Observer hypothesis: the square keys are easier to push than the arrow keys - so by comparison the left side keys are easier.
- Observer hypothesis: Also due to their shape, size, and proximity to the user, they are easier than the smaller triangle keys and the square keys which are further away (thus requiring more force to push with stick).
Sizes:
- found the size of the keys to be OK, but thought that they could be a little bigger
- user would generally prefer a larger keyboard
- test administrator observation: arrow keys are oddly shaped and hard to press using a stick.
Spacing:
- thought that the distance between the keys could be a bit bigger for both visual identification as well as ease of use
Labels & Cues:
- user had difficulty seeing the small labels on the keys and asked to have the keypad held closer
- stated that the symbols on the keys should be more defined
- when asked if he would prefer text labels he said yes
- user would prefer to have an audible/tactile click when the keys are pressed

Visual Identification

identified the volume down key as a number sign
thought the volume up key might be a memory key
thought the home key might be an up arrow
identified the question mark and stated that he would press this key to get information
stated that he expected that the triangular navigational keys would "work a cursor"
stated that the central select key would be used "to click on"
stated that the zoom key symbol "looks like an eyeglass" and he expected it would be used to enlarge text (and added that this feature would be useful to him)
when asked what he thought the key on the upper right of the keypad might be for (back/undo key) he stated that he had no idea

Functional Identification

through trial and error, user identified the screen reader on/off key as well as the zoom key
when asked to zoom back out, user tried the screen reader key, volume keys, select key, then the zoom key
when asked to turn the screen reader back on the user found it again through trial and error
when asked to turn the volume up, user tried pressing the home key twice (earlier identified as an up arrow), then tried pressing the up cursor keys.
- user then proceeded to tap randomly on keys until the volume was increased
- stated that they did not expect the speaker button to do what it did.

General

when asked to describe overall experience and comfort level with the keypad, the user gave it a 6/10, where 10 would be very good/comfortable
user rated comfort with technology a 2 (where 1=very comfortable, 2=comfortable and 3=uncomfortable)
uses sticky keys, screen reader and mechanical keyboard when on computer at home

IDRC User 1

Button design: Size, spacing, labels, etc.

a square undo button caused user to think that it could be related to the other square keys on the keypad (i.e. zoom, audio and volume controls), could this be a good thing?
did not use the back button during testing
confused as to why "search" button is grouped with audio buttons.

Function Identification

identified the zoom key as a search key, the audio on/off key as a Captions key, undo key as a refresh key
undo key should go back in hierarchy rather than in history
undo key not that useful for sighted users
suggested to put undo key on the interface (focusable) rather than on keypad

Software Experience

navigating through horizontal time bar would be better achieved with L/R navigation keys rather than up/down
would recommend voice commands or joy stick as alternative inputs to the keypad
suggested to have audio descriptions of all selectable options when landing on a page (e.g. "press the back button to return to X page, press the Home key to return to first page, press the Info key to get more information about this exhibit)
focus state on a whole box indicates you might get more info by selecting it (rather than just focusing on Close button)
Original zoom functionality (press zoom key once, then use up and down navigation keys to zoom in and out) was very confusing - no visual feedback. No idea what is happening.

IDRC User 2

Stories/Geography buttons could be more of a toggle and should only appear on each menu page (i.e. on Stories menu page you can switch to Geography menu and vice versa)
home/reset button could be placed in a separate space from the keypad, because all users will want access to a Reset key if they arrive at the kiosk and it is not at the start
zoom button could be a toggle that stays depressed while zoom is active
having modes (zoom mode and/or volume mode) would be confusing
activating zoom could open a pop-up instructional window; this box could be focusable such that you would just navigate your way out of it and it would disappear

IDRC User 3

Button design: Size, spacing, labels, etc.

volume controls could be one rocking toggle key (+/- on each end)
would like to see text labels on the keypad itself; this would help especially for the screen reader on/off key

Function Identification

initially thinks the select key is for centering, but after some thinking, user states that he thinks it is to play and pause media if focused.
also initially thinks the arrow keys control media, and select key is to stop/play, then thinks that the cursor keys can be used for many different things
shapes and sizes of keypad are good
suggests to put volume up and down on a teeter-totter / rocker switch.

Software Experience

pairing of up/right and down/left navigation keys to forward/back navigation functions is confusing
- user expects to be able to navigate up and down through the items on the page as well as left and right and finds it frustrating that he can't
lack of focus on initial page is confusing - was looking for some indication on where to start, but no cursor or hint.

Zoom

looking at the button itself, user guesses that the zoom button would be stepped and cycle back to initial setting; pressing zoom will zoom in repeatedly and the cycle back to default.
looking at the keys, it's not immediately clear how the zoom button will work.
when asked to zoom in, user thinks zoom key will make the video full screen.
- also thinks the on screen graphics +/- indicated whether the zoom function will work or not.
- wonders if zoom will be a window that zooms and then you navigate using cursor keys like Google maps.
- when asked to return magnification to default, user thinks pressing "Home" will accomplish this.

Screen Reader Button

suggested that the screen reader key icon could be a play/pause button - this would allow pausing of media and screen-reader narration
the same toggle key could be used for play/pause as well as zoom in/zoom out
when identifying the keys, user thought that the screen reader on/off key would provide extra dialog options
it isn't clear what this key does
the icon looks more like a tooltip / information note button - not for sound.