Definition

Sonification refers to the audio representation of data, this should not be confused with sound effects which is an audio representation of an event.

Current Development Version

The a development version of John Travoltage sim with sonification and accessibility can be found on the PhET development website.

What can be sonified?

Hand proximity to the doorknob:

• For example: an audio signal that changes in quality (frequency, pitch etc) the closer or further the hand moves away from the doorknob.
• To test: This may lead a user too much?

Charge presence:

• A subtle / low pitched hum for the presence of charges.
  
  • None = very quiet hum
  • Little amount of charges
  • Moderate amount of charges
  • A lot of charges
  • Maximum charges = loudest hum
• Note: it is important that even if there are no charges on John Travoltage there should some audible sound. One user remarked that the absense of sound made them think something broke with their audio.

Discharge of electrons: quantity and rate

• Descending notes indicating the decreasing quantity of charges on JT
• The zapping sound should play for the duration of the discharge.
• The zapping sound should also convey the distance from the hand to the doorknob somehow? A louder zap for closer to the doorknob, and a quieter zap for further away

Charge Acquired

• A tone is played when a charge is acquired.
• The tone increases in pitch indicating the quantity of charges currently on JT.

Foot rubbing on carpet

• A brushing sound plays whenever John's foot is on the carpet.
• A longer brushing sound is heard the longer the foot is moved on the carpet.

Maximum charges reached

• The carpet rub sound effect with the absence of charge acquired sound may be sufficient to imply the maximum level has been reached.
• Alternatively if maximum level reached, and the user attempts to gather more charges, the charge acquired tone plays, but doesn't change in pitch implying that the level isn't changing.

Maximum range reached for hand and foot

• When the arm or foot reaches the extreme in their range in motion, a sound should play. Something similar to a "bloop" or "bonk".
• Note: There are no limits to motion on the hand for a mouse user. For a keyboard user, the hand stop at around the 10 o-clock location.

Other Questions

• Consider "intelligent" audio manager to handle what sound gets played back?
  
  • Limits and prioritizes what audio to be played back if multiple events occur
  • Adjusts volume of certain sounds depending on other events (i.e. sounds of colliding electrons fades if the sound of foot rubbing on carpet occurs).
• How much audio information can be presented before the information becomes not useful?
• How accurate does the audio representation need to be? Precise or can it be abstracted / estimated.
• For example, if we use volume to indicate the rate of discharge, and use volume to indicate quantity of charges - would this be conceptually confusing?

Not going to implement

Quantity of charge

• Do we want a sonification to continuously play while there are charges on John? If so what is that sound?
• Is there enough information from other sources that this does not need to be sonified.
• How do you convey quantity?
  • Volume?
  • Pitch?
  • Frequency of sound?
• Is it enough to represent quantity by the frequency of collision sounds? Answer: probably not enough.
  • If there is only 1 electron, you won't hear anything.
  • Depending on quantity of electrons, they will stabilize and stop moving - if there are collision sounds it will eventually stop. This may imply the wrong model to a non-sighted interaction - absence of sound may be confused with charges disappearing. For example, generate 2 charges. After the initial collision, there won't be any other sound.

Collision of charges

• What possible sounds would we want to use? a clinking / tinking sound? sound of marbles tapping together?
• How do you programmatically determine if electrons have collided? (Something John Blanco can help with?)
• Do we want to represent every electron colliding? If more than 1 collide at around the same time, do we want to sonify all of them, or just some?
• What volume should the sound effect be? Keep in mind that the effect could happen frequently when there are many charges.
  
  • use a quieter volume as to not be annoying. Use the sound manager idea to lower volume if other sounds are playing.

Personae

Chloe

Personal Goals:

• Graduate and go to college or university

• Keep up with her Physics class and participate fully

• Making effective use of time

Practical Goals:

• Understand current electricity unit in Physics

Chloe is a busy high school student. Like most of her friends, she’s feeling the pressure of graduating high school, balancing her academics with her social life, and somehow find time to do the things she loves to do like being part of the school’s chess and robotics clubs. For the most part she enjoys everything about high school except for the fact she doesn’t have enough time to do all the things she wants to do. As a result, she’s finding that she has to cut time away from her clubs, or rush her homework. 

Chloe is taking physics this year and thankfully her teacher has done his best to offer the course material in multiple formats and modalities. She finds it useful to hear a podcast or listen to a YouTube video to compliment the text material. However, one of her challenges is following along during applied activities. While her classmates do their best to work collaboratively, Chloe often finds it difficult to follow especially during the excited, frantic discovery process. Chloe wishes there was some way she can do the same (or similar) experiments outside of class and at her own pace.

Use Cases

Chloe (Work in Progress)

In Chloe’s physics class they are covering static electricity and today’s examined static discharge by experimenting with some wool, a balloon, a paperclip, and some tape (reference: see here for how this experiment is done). Chloe had a tough time doing and following this experiment in class. To top it off, Chloe couldn’t tell how close she was to the paperclip and burst the balloon. At the end of class their teacher gave a link to some online resources, one of them was a link to the John Travoltage simulation on PhET.

At home after dinner, Chloe settles down in front of her laptop and loads up the John Travoltage simulation. Doing a quick survey of the simulation with her screen reader, she could tell that there is some descriptive text at the start, then two sliders, and a few other options. Going on a hunch, Chloe selects the “Foot” slider and moves it one spot. Her screen reader reports back that the foot position has changed and is off the carpet. Intrigued, Chloe presses moves the slider a few more times and at some point she hears a different sound. Her screen reader told her that a “charge was acquired” and there was a chirping sound at the same time.

Moving the slider more vigorously now, she hears rapid chirping sounds along with the feedback from her screen reader saying that the foot has moved and a charge has been gained. Suddenly Chloe hears a zapping, buzzy noises followed by what could only be described as “inverted charge acquired chirps”. Her screen reader announces that a discharge occurred and there are “no more charges”.

Not quite sure what happened because her screen reader was also talking the same time these other sounds were playing, Chloe moves the foot slider more slowly and deliberately. With each “charge acquired chirp”, she can hear the screen reader report back the quantity of charges: “31 charges now acquired. 32 charges….”.

Chloe hears something she had not noticed before - she can hear subtle clinking noises now. She wonders if the reason she did not hear them before is because too much was going on earlier?

Slowly Chloe moves the slider to get a few more charges - the screen reader reports back that charges were acquired, and she can also hear that the clinking is more frequent (she thinks “it seems to be related to the number of charges?”).

At the 51 charge mark Chloe hears the zappy, buzzy noises she had heard earlier and the screen reader reporting that there was a discharge.

Satisfied with the outcome, Chloe starts to use the hand slider. The screen reader is reporting that the hand is changing position closer and then further away from the doorknob, but nothing much else is happening. Puzzled, she re-examines the contents of the page, but then recalls her lesson in class.

Moving back to the foot slider, Chloe starts to vigorously gather charges moving the slider rapidly. She hears a noise and a notification from the screen reader that there are 100 charges and that the maximum number of charges had been gained. Half expecting something to have happened by this point, Chloe remembers that she had left the hand far away from the door knob. Tentatively at first, Chloe moves the hand closer to the door knob, but starts moving more quickly. Position 35… 36 ...

Suddenly there was a zapping, buzzing noise - the electrons were being discharged! Chloe shocked that it happened so quickly, reverses the direction of the hand movement. The sound continues for a moment longer and then stops. The screen reader reports that electrons were discharged and that 23 remained. 

Feeling unsure of herself, Chloe thinks that she needs to experiment with this a little more, but there’s other homework to do first. Making a mental note to come back, Chloe sets her computer aside and goes to grab a drink.