Chapter 15: Parameter Mapping Sonification

by Florian Grond, Jonathan Berger

Description

This chapter introduces the benefits and challenges of parameter mapping sonification. The chapter explores features of data and their preparation for sonification; synthesis parameters, both in the signal and perceptual domains, and methods of creating mapping functions to bridge between data and sound synthesis.

Download Chapter

Download the chapter: TheSonificationHandbook-chapter15 (PDF, 2.4M)

Media Examples

Example S15.1: Sonifying arrival at a target data point in a time-series
A pre-recorded sound sample sounds when water in a teakettle reaches the boiling point.


media file S15.1
download: SHB-S15.1 (mp3, 278k)
source: Paul McCartney, Uncle Albert/Admiral Halsey, from the album ‘Ram’, 1971.

Example S15.2: Sonifying both process of a time-series and arrival at a target data point
The continuous change of rising water temperature with a separate sound indicating the boiling point as water is boiled in a tea-kettle.


media file S15.2
download: SHB-S15.2 (mp3, 482k)
source: Programmed and rendered by Jonathan Berger.

Example S15.3: Sonifying specific data points in a sequence along with the entire process
Continuous change of rising water temperature with five data points (in the original sonification, each respectively mapped to each of the five stages of heating water in traditional Chinese tea preparation (simulated here).


media file S15.3
download: SHB-S15.3 (mp3, 450k)
source: Programmed and rendered by Jonathan Berger.

Example S15.4: Adding a reference tone
Same as S15.3 but with the boiling point sounding throughout. The progressive reduction in frequency beating to a pure unison clarifies and anticipates the point of arrival.


media file S15.4
download: SHB-S15.4 (mp3, 451k)
source: Programmed and rendered by Jonathan Berger.

Example S15.5: Mapping to musical mode or scale degrees
In sound example S15.5 the tea water temperature is mapped to quantized frequencies corresponding to a pentatonic scale with pitch classes [C, D, E, G and A] on the musical scale. The lowest pitch sounded is C2, 212 F is set to C5, and all other sounded pitches are scaled to their closest match within this range.


media file S15.5
download: SHB-S15.5 (mp3, 482k)
source: Programmed and rendered by Jonathan Berger

Example S15.6: one-to-many mapping
This example shows a sonification using a one to many mapping strategy which corresponds to spectrogram in Figure 3. The transition between unvoiced and voiced and the transition between vowels are both clearly audible and visually identifiable in the spectrogram.


media file S15.6
download: SHB-S15.6 (mp3, 170k)
source: F. Grond, T. Bovermann, and T. Hermann, “A supercollider class for vowel synthesis and its use for sonification,” in Proceedings of the 17th International Conference on Auditory Display (ICAD-2011) (D. Worall, ed.), (Budapest, Hungary), OPAKFI, June 20 – 24 2011.

Example S15.7: singing function
This is an example of an auditory graph using vowel synthesis. The function value f(x) is mapped to pitch, the slope f’(x) is mapped to the vowel transition [a:]-[i:] and the curvature f’’(x) to vowel brightness. In the first repetition only f(x), in the second f(x) and f’(x), and in the third f(x), f’(x) and f’’(x) are mapped to the respective sound features. The movement in the stereo panorama corresponds with the progression along the x- axis.


media file S15.7
download: SHB-S15.7 (mp3, 736k)
source: F. Grond and T. Hermann, “Singing function, exploring auditory graphs with a vowel based sonification,” Journal on Multimodal User Interfaces, online since Oct. 2011. DOI: 10.1007/s12193-011-0068-2

Example S15.8: Probing
Probing is typically necessary in diagnostic situations. In this example a high dimensional hyperspectral image of a colon cell is probed at various points by holding down the mouse button.


media file S15.8
download: SHB-S15.8 (mp3, 3M)
source: R. J. Cassidy, J. Berger, K. Lee, M. Maggioni, and R. R. Coifman, “Auditory display of hyperspectral colon tissue images using vocal synthesis models,” in Proceedings of the 10th International Conference on Auditory Display (ICAD2004) S. Barrass and P. Vickers, eds., (Sydney, Australia), 2004. Sound generated by software designed by Woon Seung Yeo.

Example S15.9: Linear scanning
In this example probes are taken in the same hyperspectral colon tissue images to gain a general auditory representation of local regions along a predefined linear scan line by dragging the cursor across the image.


media file S15.9
download: SHB-S15.9 (mp3, 911k)
source: R. J. Cassidy, J. Berger, K. Lee, M. Maggioni, and R. R. Coifman, “Auditory display of hyperspectral colon tissue images using vocal synthesis models,” in Proceedings of the 10th International Conference on Auditory Display (ICAD2004) S. Barrass and P. Vickers, eds., (Sydney, Australia), 2004. Sound generated by software designed by Woon Seung Yeo.

Example S15.10: Spiral scanning
In this example probes are taken in the same hyperspectral colon tissue images along a pre-set spiral generating from the originating pixel of the mouse position.


media file S15.10
download: SHB-S15.10 (mp3, 911k)
source: R. J. Cassidy, J. Berger, K. Lee, M. Maggioni, and R. R. Coifman, “Auditory display of hyperspectral colon tissue images using vocal synthesis models,” in Proceedings of the 10th International Conference on Auditory Display (ICAD2004) S. Barrass and P. Vickers, eds., (Sydney, Australia), 2004. Sound generated by software designed by Woon Seung Yeo.

Example S15.11: Listening for a particular phoneme
This example demonstrates a means of anchoring data by scaling to a particular phoneme. In this example, a particular trend in stock market data is detectable by mapping the sound to a frequency modulation synthesis algorithm that can produce vocal formants. This mapping allowed the sonification task to be described as simply `you are seeking for a condition displayed when you hear the ‘rounder’ phoneme.


media file S15.11
download: SHB-S15.11 (mp3, 243k)
source: O. Ben-Tal and J. Berger, “Creative aspects of sonification,” Leonardo, vol. 37, no. 3, pp. 229–232, 2004. Sound generated in Common Lisp Music by Oded.Ben-Tal.

Example S15.12: additive a
This example displays a sequence of two 12-dimensional vectors which are mapped to twelve partials of a harmonic complex such that the data values dictate the relative amplitudes of each partial. Although it is impossible to interpret the timbral result of the complex in terms of the specific mappings, patterns of relationships based, for instance, on the degree of spectral brightness are easily distinguished.


media file S15.12
download: SHB-S15.12 (mp3, 386k)
source: Programmed and rendered by Jonathan Berger.

Example S15.13: additive b
In this example the same vector elements from the previous example are mapped to the rise time of each partial’s amplitude envelope.


media file S15.13
download: SHB-S15.13 (mp3, 74k)
source: Programmed and rendered by Jonathan Berger.

Example S15.14: additive c
In this example the same vectors are mapped to frequency ratios. The relationships are scaled such that there is a linear relationship between the matrix values and the mapped frequency. The mapping produces a complex sound that is primarily inharmonic.


media file S15.14
download: SHB-S15.14 (mp3, 75k)
source: Programmed and rendered by Jonathan Berger.

Example S15.15: additive d
In this example the vectors are also mapped to frequency ratios the values are scaled such that the tendency gravitates toward harmonic complexes. Note that data points sharing harmonic relationships fuse while elements that dissonate with the fundamental are noticeable.


media file S15.15
download: SHB-S15.15 (mp3, 228k)
source: Programmed and rendered by Jonathan Berger.

Example S15.16: stock a
In this example (same as Example S15.11:) subtractive synthesis is used to sonify stock market data from two respective years (2000 and 2001) such that the resonances express formant regions producing phoneme like sounds that identify particular trends.


media file S15.16
download: SHB-S15.16 (mp3, 243k)
source: O. Ben-Tal and J. Berger, “Creative aspects of sonification,” Leonardo, vol. 37, no. 3, pp. 229–232, 2004. Sound programmed and generated by Common Lisp Music by Oded Ben-Tal.

Example S15.17: stock b
In this sound examples subtractive synthesis is used to sonify stock market data from two respective years (2000 and 2001) such that the resonances express formant regions producing vowel like sounds that identify particular trends.


media file S15.17
download: SHB-S15.17 (mp3, 262k)
source: O. Ben-Tal and J. Berger, “Creative aspects of sonification,” Leonardo, vol. 37, no. 3, pp. 229–232, 2004. Sound programmed and generated by Common Lisp Music by Oded Ben-Tal.

Example S15.18: mesh hit
In this example phyisical models based sonification of two dimensional waveguide mesh is used to sonify proximity to data clusters. In these examples, coded by Greg Sell, the popular game Battleship is simulated such that the ‘ship’ coordinates are mapped to the scatter functions on junctions along the two dimensional mesh. The proximity of a probe to target data clusters, as well as characteristics of the cluster are audible. This example displays a mesh hit.


media file S15.18
download: SHB-S15.18 (mp3, 42k)
source: example coded in MatLab by Gregory Sell. Lee, K. Sell, G. Berger, J. 2005 Brazil, in Proceedings of the 11th International Conference on Auditory Display (ICAD2005) Limerick, Ireland Department of Computer Science and Information Systems, University of Limerick 140-145

Example S15.19: mesh miss
From the same application of physical models as above, this example displays a mesh miss.


media file S15.19
download: SHB-S15.19 (mp3, 42k)
source: example coded in MatLab by Gregory Sell. Lee, K. Sell, G. Berger, J. 2005 Brazil, in Proceedings of the 11th International Conference on Auditory Display (ICAD2005) Limerick, Ireland Department of Computer Science and Information Systems, University of Limerick 140-145