PhET John Travoltage Sonification Prototype Analysis and Recommendations

Prepared by: Jonathan Hung and Justin Obara

Additional input from Colin Clark.

Prototypes Analyzed

Recommendations

With all the possibile sounds playing, the experience can get confusing. The user may be caught trying to interpret meanings of the audio cues, all the while trying to figure out the learning goals.

It’s our recommendation that the first step is to make a simplified sonified version of the sim, with additional layers which can be added, or even enabled via user options in future iterations.

We should aim to simplify the experience by:

Removing the hand-doorknob proximity sonification.
- The sonification could be confusing - is the sound indicating something being emitted - like an energy field or a beam?
- The sonification information is redundant - the position of the hand is already reported by the scene description, as well as read back when interacting with the hand position slider.
- This sonification can be brought back as an option in later iterations if it’s pedagogically interesting.
Change the “charge quantity” sonification to be more “ambient” and “rounder”
- Have a tone play continuously in the background like ambiance if charges are present.
- Timber to be “rounder” in effect - easier to listen to, not hard on the ears.
- Inspiration (videos): Osmos iPad game, Brian Eno’s Bloom app.
- Perhaps start with something like the sound in sonification=4.
Remove the bonk sound at either end of the foot range.
- This could be misinterpreted as an event occurring in the sim (i.e. the foot has kicked the door knob).
- For sighted users it’s clear the foot doesn’t move anymore, and for screen reader users the screen reader will report they have reached either end.

Future research and design areas:

Create a context aware channel mixer which will help fade (“duck”) sounds prior to important events, and fade the sound back in after the event.

For example: During the discharge, you can duck the sounds of the electrons so you can hear the “zap” more clearly.

Give users options to customize their sound experience:

Enable additional audio disabled by default (i.e. option to enable Stereo sound)
Choose different audio schemes depending on their setup (i.e. an slider that chooses between headphones, laptop, and desktop speakers).

Other notes / questions:

Sound on headphones is noticeably louder than on computer speakers. Perhaps audio should be designed for headphone usage first since headphones generally have less control for volume.
The foot brushing sound could be a bit sharper and more defined. Perhaps the sound of a broom brush? Example: https://www.youtube.com/watch?v=rdi3Rz_uc4Q
The charge acquired sound has a little “tick” at the end. On some speakers (i.e. laptop speakers) it comes across loudly like someone is unplugging / jiggling the speaker connector.
The “ZAP” sound during discharge is hidden amoung other sounds of the sim. This should be more prominent by making it louder or by fading other sounds.
The brushing sound could be an issue for small movements (i.e. a keyboard user or small mouse movement).

Should we play just a short sound for small changes? If so how would it sound if several small changes were chained together (i.e. continuous increment key presses or small mouse movements).

John Travoltage Sonification Notes

1.3.0-dev.2 v1

Leg movement

Electron rattling sound can be ambiguous:

Could sound like something moving (ie. objects on a conveyor belt), or something with mass.
Sound keeps rattling even when the electrons aren’t visually moving

Volume of rattling sound on headphones is loud, but seem acceptable with stand-alone speakers tested.
Extreme end beep

sounds like a confirmation for an action. Mixed message.

If you move the leg quickly it sounds like you’re getting points for kicking the door knob.
If you are using keyboard shortcut to move from one end of the other, you would get bonking noises - could this be confusing?

Arm movement

Continuous tone increasing in frequency with warble sounds like the finger is emitting something.
Perhaps instead of a continuous range, use discrete ticks with increasing pitch to convey proximity. For each movement, you will hear a tick consisting of two notes: the position you are currently at, and the position you are going to.

There might be an issue if the user moves quickly - will it sound okay, or will the system start clipping sounds?

Suggest to remove this sonification all together. It doesn’t add much benefit, could be confusing, and, if interpreted correctly, gives away too much information about the door knob being important.

Electron discharge

Couldn’t hear the decreasing pitch of electrons leaving the body.

1.3.0-dev.2 v2

Charge Level

60 Hz sound is a bit intense, especially when charges are at maximum
When the 60 Hz volume increases, it gets increasingly hard to hear the sounds of the electrons and the foot rub
If you are non-visual, Increasing volume of 60Hz could be interpreted as getting closer to a sound source
60 Hz may also be interpreted as static from the speaker.

Foot brushing sound

With keyboard movement of the foot, the position change can be quite slight what should be done in this case?

It doesn’t make sense to play a full brushing sound if the position changed by 1 increment.
If you play a short sound for each increment, how does it sound if the user presses the keyboard quickly?

The foot brushing sound could be a bit more distinct and light. Example: consider this sound of a broom brushing (https://www.youtube.com/watch?v=rdi3Rz_uc4Q)

Arm movement

When arm is moving with charges at maximum, the sound is intense.

1.3.0-dev.2 v3

Arm movement

Prefer the this lack of warble effect (compared to Sonification=2). Gives less of an impression of “emitting something”
Can be hard to hear the arm movement when there is a larger buildup of electrons

1.3.0-dev.2 v4

Like the puslating quality when charges are present. Use a similar sound for indicating presence of chagres in the redesign from “charge quantity” to “charge presence”.

Colin Clark feedback

There’s a clear activity / pressure building as you “kick the carpet”.
Hand position - liked the electricity beating sounds.
The experience of moving the foot and the arm are very distinct interactions
Liked 3 or 4.
Sonification may require exploration and context. Sonification alone may not be enough.
Lack of research on the effects of sonification in classroom environments.
Context aware channel mixer

Prioritize pings and pads
I.e. the zap is currently getting lost with the other sounds. Prioritize it higher.

Electrons could be a rounder timber that’s “just there”.
Can figure out that pressure is building up, but couldn’t figure out the quantity.
Laptop speaker mode - shift the Hz

Choose between devices: i.e. headphones, laptop, stand-alone speakers