Chapter 11: Interactive Sonification

by Andy Hunt and Thomas Hermann


This chapter places a special focus on those situations where there is a tight control loop (a real-time interactive collaboration) between the human user and the system producing the sonification. It explains the background (why humans appear to use interactive sonification as a natural tool for exploring the world) as well as describing the different methods and application domains.

Download Chapter

Download the chapter: TheSonificationHandbook-chapter11 (PDF, 1M)

Media Examples

Example S11.1: Baby crying
All human beings make a noise as soon as they are born. We continue to do so as we grow, but we increasingly monitor the reaction from the world.

media file S11.1
download: SHB-S11.1 (mp3, 88k)
Edited from (submitted by user amm00):

Example S11.2: Child playing
Child playing with wooden bricks, then small stones, then wooden again

media file S11.2
download: SHB-S11.2 (mp3, 291k)
source: Self recorded

Example S11.3: Young children playing with pots and pans
As children grow they explore the world constantly, in a multi-modal fashion, using touch and sound (as here) as well as sight, taste and smell, to intimately learn about the physics and chemistry of the world about them. They are like permanent scientists, running continual experiments.

media file S11.3
download: SHB-S11.3 (mp3, 73k)
Edited from (submitted by user andyt):

Example S11.4: child asking parent for something
Gradually, as children grow, they begin to use communication to get things done (mostly by asking a parent), in addition to directly exploring the world. The direct exploration often takes a back-seat in favour of semantic and syntactic communication. This is reflected in many of our computer interfaces, and in this chapter we speculate on what we have ‘lost’ by concentrating on this mode of relating to the world.

media file S11.4
download: SHB-S11.4 (mp3, 74k)
Edited from (submitted by user andyt):

Example S11.5: someone rapidly changing channels on a tv with others complaining!
This is a mocked-up example of a commonly experienced situation. When you hold the remote control, everything seems fluid and fine. When others take charge it is disrupting and annoying! This leads us to believe that being at the heart of an interface control loop is very important to a human user (and also warns us of the danger of demonstrating software where we are happily ‘channel hopping’ to the annoyance of the audience)!

media file S11.5
download: SHB-S11.5 (mp3, 373k)
source: Self recorded

Example S11.6: Sounds made by stone tools
Rocks being knocked and scraped together rhythmically; like a caveman!

media file S11.6
download: SHB-S11.6 (mp3, 265k)
source: Self recorded

Example S11.7: car engine
Cars are complex machines, and mechanics develop highly-tuned skills of listening to and diagnosing problems.

media file S11.7
download: SHB-S11.7 (mp3, 118k)
Edited from (submitted by user mckinneysound):

Example S11.8: washing machine
As with the above car engine example, washing machine problems are often diagnosed by sound. It seems that in the mechanical world we naturally use sound to inform us of complex, ‘hidden’ behaviours. Our challenge is that this should also be true for the world of digital data in general.

media file S11.8
download: SHB-S11.8 (mp3, 209k)
Edited from (submitted by user Percy Duke):

Example S11.9: Unconscious sounds made while thinking
Drumming fingers and humming while thinking. Helpful to the thinker – really annoying to anyone else who has to share a room with them!

media file S11.9
download: SHB-S11.9 (mp3, 335k)
source: Self recorded

Example S11.10: Sounds of a noisy radio signal
This is a typical set of sounds encountered while trying to tune in to a distant, noisy radio signal. Often the user finds they can hear it clearly (because they operated the controls to suit their own ear-brain systems) while others in the room cannot distinguish the information in the signal.

media file S11.10
download: SHB-S11.10 (mp3, 211k)
Edited blend of two sound sources:

  1. (submitted by user J – Luc):
  2. (submitted by user parabolix):

Example S11.11: Beginner violin player
This is the sort of sound made by someone learning to play an instrument. Usually it is fine for the person learning, and indeed a vital part of the learning process (as they receive continual feedback). However, the same sound is not as pleasant to other listeners because they are not part of the control loop.

media file S11.11
download: SHB-S11.11 (mp3, 601k)
source: Self recorded

Example S11.12: oboe
The oboe is a very hard instrument to learn, and beginners often cannot make a sound, yet it is considered to be a very expressive instrument. This may illustrate a trade-off in accessibilitiy versus long-term control intimacy.

media file S11.12
download: SHB-S11.12 (mp3, 129k)
Edited from (submitted by user Thirsk):

Example S11.13: Guitar (intermediate)
Here is an intermediate guitar-player. Clearly the basics have been accomplished, but there are many years of mastery still left to explore. Interfaces such as the guitar and piano offer beginners are more immediate feedback experience than something like the oboe, but maybe the individual notes (especially long ones) are not as expressive as the oboe.

media file S11.13
download: SHB-S11.13 (mp3, 242k)
source: Self recorded

Example S11.14: Sounds of computer usage
Sounds made while moving the computer mouse on its pad and clicking buttons.

media file S11.14
download: SHB-S11.14 (mp3, 280k)
source: Self recorded

Example S11.15: ‘blinkers’ in a modern car
This is the sound of ‘blinkers’ or ‘indicators’ in a modern car. Old cars had blinkers which made sound as a by-product of their relays. Modern ones do not need to make sound, but it is felt so useful to the user experience that the sound now has to be synthesised.

media file S11.15
download: SHB-S11.15 (mp3, 102k)
Edited from (submitted by user morgantj):

Example S11.16: Typical computer sounds
The sounds of audio beeps etc when using a computer (a montage of ‘startup’, ‘error’ etc).

media file S11.16
download: SHB-S11.16 (mp3, 236k)
source: Self recorded

Example S11.17: A glass being moved around on a table
This is a typical sound from everyday interaction. Someone is idly ‘playing around with’ a glass on a table. The sound forms a continuous stream of feedback about the object’s position, speed, the table surface, whether there is danger of the glass breaking, etc.

media file S11.17
download: SHB-S11.17 (mp3, 188k)
source: Self recorded

Example S11.18: seatbelt warning after starting a car
Modern cars often used sampled or synthesised sounds to give a warning to their users, as in this case where the driver has not yet fastened their safety belt. The following example extends this idea.

media file S11.18
download: SHB-S11.18 (mp3, 176k)
Edited from (submitted by user amliebsch):

Example S11.19: Parameterised seatbelt warning
An edited example showing the effect of simply ‘speeding up’ and playing the seatbelt warning louder, in response to a driver who accelerates without their seatbelt in place.

media file S11.19
download: SHB-S11.19 (mp3, 127k)
source: + editing
Edited from (submitted by user joedeshon):

Example S11.20: Switches
Switches (repeated naturally and repeated by sampling)

media file S11.20
download: SHB-S11.20 (mp3, 266k)
source: Self recorded

Example S11.21: Interactive parameter mapping
The Interaction video shows a simple GUI that allows to interactively adjust mapping parameters while the data sonification is played in a loop with the actual parameters. This allows the user to actively query different ‘sonic views’ that deliver equally valid information about the data. In an ongoing process the user can steadily increase their insight into the data structure.

media file S11.21
download: SHB-S11.21 (mp4, 5.2M)
source: From Thomas Hermann

Example S11.22: Evolutionary Optimization of Parameter Mapping sonifications
The series of parameter-mapping sonifications are selected children (in evolutionary algorithms sometime called mutations) in the evolutionary process where the user provides the fitness function by rating the offered alternatives and thereby directing the process. Here the user aims at discovering the clustering structure as best as possible. The sonification get more and more structured as the mapping evolves towards a projection of data on the time axis so that the groups audibly split in grain clusters.

media file S11.22
download: SHB-S11.22 (mp3, 179k)
source: From Thomas Hermann

Example S11.23: Shoogle demo video
The shoogle system implements a Model-based Sonification that configures the setup of objects in a virtual container from existing text messages in a phone. In consequence, physical interactions allow the user to become aware of, or to actively explore the available messages without looking to the tiny screen.

media file S11.23
download: SHB-S11.23 (mp4, 19.5M)
source: From John Williamson, S. Hughes and Rod Murray-Smith with permission