Chapter 21: Aiding Movement with Sonification in “Exercise, Play and Sport”

by Oliver Höner (Ed.)

Description

This chapter deals with several applications of sonification in the field of sport and movement sciences. A selection of authors from various sciences (e.g., Electronics, Music, Technology and Sport Science) illustrate a wide scope of multidisciplinary applications in aiding human movement using interactive sound. These applications can be allocated to the comprehensive field of “Exercise, Play and Sport” i.e., physical activity in its widest meaning.

Download Chapter

Download the chapter: TheSonificationHandbook-chapter21 (PDF, 5.3M)

Media Examples

Example S21.1: EMG audification
This sound is quite low in level and has a high bass content. This is the sound produced when taking an EMG signal from a patient’s leg as they stand up and sit down again. Although the signal recorded looks like an audio signal, when it is played in real-time as sound (audification) much of the signal is hidden in lower frequencies.


media file S21.1
download: SHB-S21.1 (mp3, 371k)
source: recorded by Andy Hunt

Example S21.2: EMG 2-channel MIDI
This is a 2-channel recording of a 2-channel EMG signal. The stereo panning reflects the position of the sensors on the body. On the left is heard the signal produced when MIDI-fying the data taken from an EMG sensor on the left side of the knee, and the right side from the right of the leg. It can be heard that the signals are out of phase, which indicates an imbalance in the muscle action on either side of the knee. We are listening to the data much slower than real-time (in fact slowed down about 15 times) so that we can hear what would otherwise be ‘bursts’ of data, and make judgements on their phase and other relationships.


media file S21.2
download: SHB-S21.2 (mp3, 895k)
source: recorded by Andy Hunt

Example S21.3: EMG 2-channel MIDI
Here we are listening to the same data as in Example S21.1 above, but via parameter mapping. The pitch of the signal is controlled by the amplitude of a single EMG sensor. This is easier to hear than the audification in Example S21.1, and also allows us to move through the data interactively, backwards and forwards, at any speed, and the sound will always be in the hearing range.


media file S21.3
download: SHB-S21.3 (mp3, 286k)
source: recorded by Andy Hunt

Example S21.4: EMG movie demo
This movie shows one of the researchers testing out the system using standard biomedical test equipment and taking part in a standardised test procedure. The EMG sensors on his leg are also being fed to our interactive sonification system, and the streams of multiple muscle bursts are clearly audible as the patient extends and retracts his leg.


media file S21.4
download: SHB-S21.4 (mp4, 1.8M)
source: recorded by Andy Hunt

Example S21.5: Audiogame “The Matrix”
Matrix Shot was inspired by the first Matrix movie and the artistic movements of certain characters to evade enemy bullets. Therefore, the goal of Matrix Shot is to detect the approach of virtual acoustic bullets and to evade them in the most acrobatic style possible. The bullets itself are moving 3D sound sources that approach the player from the front, while an additional USB camera is employed to determine whether the player was hit or not. Using head rotations, the player detects the direction and distance of the virtual bullets and can evade them.


media file S21.5
download: SHB-S21.5 (mp4, 5.3M)
source: created by Niklas Röber, first published in Niklas Röber, “Interaction with Sound”, PhD thesis, University of Mageburg, Dr.Hut publishing house, 2009

Example S21.6: Audiogame “The hidden Secret”
The story of the The hidden Secret evolves around a tourist visiting the city of Magdeburg and his adventures in the city’s main cathedral. Thereby the player discovers and unveils several mysteries and puzzles to gain the long lost cathedral’s treasure. The story is loosely constructed around several real myths and sagas found in an old book about the Cathedral of Magdeburg. The game is realized as a plain 3D auditory adventure and uses spatialized sound sources and 3D head-tracking to enhance the orientation and navigation in the virtual game world.


media file S21.6
download: SHB-S21.6 (mp4, 19.2M)
source: created by Niklas Röber, first published in Niklas Röber, “Interaction with Sound”, PhD thesis, University of Mageburg, Dr.Hut publishing house, 2009

Example S21.7: Augmented Audiogame “The hidden Secret”
This example of The hidden Secret is realized as an augmented audio game and is also set in the Cathedral of Magdeburg. It unfolds a storyline similar to the other implementation of the game. The augmented audio reality system employed in this game uses bone-conducting headphones, a digital compass for head-tracking, and a WiFi network for user positioning. The video shows a replay of a recorded game session.


media file S21.7
download: SHB-S21.7 (mp4, 15.1M)
source: created by Niklas Röber, first published in Niklas Röber, “Interaction with Sound”, PhD thesis, University of Mageburg, Dr.Hut publishing house, 2009

Example S21.8: Audiogame “Frogger”
AudioFrogger is an acoustic adaptation of the classic Frogger game developed by Sierra Online in 1983. Similar to the original implementation, the player has to cross several streets without getting involved in a traffic accident. The difficulty increases with the number of lanes and the moving objects to keep track off. The avatar is controlled with a gamepad interface and 3D head-tracking is used to improve the perception of the auditory scene.


media file S21.8
download: SHB-S21.8 (mp4, 5.2M)
source: created by Niklas Röber, first published in Niklas Röber, “Interaction with Sound”, PhD thesis, University of Mageburg, Dr.Hut publishing house, 2009

Example S21.9: Audiogame “Mosquito”
Mosquito is played in real 3D space and with a possible 360° radius of interaction. The scope of this game is to repel up to three attacking mosquitos and to avoid being stung. The virtual mosquitos circle the player, who can focus on individual mosquitos using 3D head-tracking and kill them by using a virtual fly swatter. The game can be played by either using a gamepad interface, or an additional 3D sensor from the Polhemus tracking system.


media file S21.9
download: SHB-S21.9 (mp4, 10.1M)
source: created by Niklas Röber, first published in Niklas Röber, “Interaction with Sound”, PhD thesis, University of Mageburg, Dr.Hut publishing house, 2009

Example S21.10: Blindminton
This video shows some interactions with the blindminton system. The user is capable to return the ball from listening to the sound alone. The sound conveys both vertical, horizontal displacement and distance between ball and racket in realtime. Instead of a second player as in real Blindminton, here the ball is reflected by virtual walls


media file S21.10
download: SHB-S21.10 (mp4, 4M)
source: created by Thomas Hermann and Oliver Höner

Example S21.11: Goalball (slow)
The video shows a goallball player of the German National paralympic team trying our system as a virtual ball throwing machine. The sound approximates the rolling sounds of a real goalball. The actual performance can be precisely measured by motion tracking of the racket.


media file S21.11
download: SHB-S21.11 (mp4, 25.5M)
source: created by Oliver Höner and Thomas Hermann

Example S21.12: Goalball (fast)
This is a video similar to the above, but the ball is thrown at much higher velocity. The sportsmen perform astonishingly well, regular (non-expert) users won’t catch a single ball, apart from chance level.


media file S21.12
download: SHB-S21.12 (mp4, 7.6M)
source: created by Thomas Hermann and Oliver Höner

Example S21.13: Goalball sound recording
This is a recorded sound of a real goalball as it is used in the paralympic discipline.


media file S21.13
download: SHB-S21.13 (mp3, 176k)
source: recorded by Oliver Höner and Thomas Hermann

Example S21.14: Breaststroke sonification – arm-leg-cycle
Movement-sonification based on kinematics of relative wrist and ankle motion – strokes only. Modulation of sound frequency and amplitude.


media file S21.14
download: SHB-S21.14 (mp4, 791k)
source: http://www.sportwiss.uni-hannover.de/fileadmin/sport/pdf/onlinepublikationen/effenberg/de_armzug-beinstoss.mpg

Example S21.15: Breaststroke sonification – arm-leg-stroke
Movement-sonification based on kinematics of relative wrist and ankle motion – complete cycles. Modulation of sound frequency and amplitude.


media file S21.15
download: SHB-S21.15 (mp4, 786k)
source: http://www.sportwiss.uni-hannover.de/fileadmin/sport/pdf/onlinepublikationen/
effenberg/de_arm-beinzyklus.mpg

Example S21.16: Breaststroke sonification – arm-leg-torso-cycle
Movement-sonification based on kinematics of relative wrist, ankle and neck point motion – complete cycles. Modulation of sound frequency and amplitude.


media file S21.16
download: SHB-S21.16 (mp4, 799k)
source: http://www.sportwiss.uni-hannover.de/fileadmin/sport/pdf/onlinepublikationen/
effenberg/de_arm-bein-rumpfzyklus.mpg