by Thomas Hermann
Description
This chapter introduces the sonification technique of Model-Based Sonification. Guidelines for model design are given, and several different example sonification models are explained and shown with sound examples. The advantages and differences to other sonification methods and in particular Parameter-Mapping Sonification are discussed.
Download Chapter
Download the chapter: TheSonificationHandbook-chapter16 (PDF, 1.4M)
Media Examples
Example S16.1: Listening Experiment
This is a helpful experiment to understand how the human auditory system works. Please listen to the sound several times and describe in as much detail as you can what you have heard. Perhaps take notes. Then read on in the main text.
media file S16.1
download: SHB-S16.1 (mp3, 30k)
source: Thomas Hermann, recorded sound
Example S16.2: Data Sonogram of the Iris data set
These sonifications represent the acoustic response from initiating a shock wave in data space from 3 different positions. The data set itself consists of 150 data points, each representing an Iris flower by 4 geometric measurements. These cluster in three groups and a biological classification is available. Here the class label determines the spring stiffness. In consequence the different Iris sorts sound differently. The different auditory ‘views’ on one and the same data set allow to perceive that the different types group in homogeneous groups. The initial noisy sound indicates the shock wave start, so that the listener can judge how far away from that position the data points are located.
media file S16.2 (a)
download: SHB-S16.2a (mp3, 19k)
source: programmed and rendered by Thomas Hermann
media file S16.2 (b)
download: SHB-S16.2b (mp3, 19k)
source: programmed and rendered by Thomas Hermann
media file S16.2 (c)
download: SHB-S16.2c (mp3, 19k)
source: programmed and rendered by Thomas Hermann
Example S16.3: Tangible Data Scanning
This video shows Tangible Data Scanning, an interaction with data sonograms by using an interaction object in the hand of the data analyst. The video shows a scanning of the space using a clustered data set (Iris data, as described in S16.1 above)
media file S16.3
download: SHB-S16.3 (mp4, 1.6M)
source: supplementary material for T. Bovermann, T. Hermann, and H. Ritter. Tangible data scanning sonification model. Proc. ICAD 2006, pp. 77–82, London, UK.
Example S16.4: Principal Curve Sonification of a noisy spiral data set
The sound example presents a PCS of a data set where the data are distributed along a noisy spiral: density modulations along the spiral become more easily audible than they can be perceived visually.
media file S16.4 (a)
download: SHB-S16.4a (mp3, 52k)
source: rendered by the author. Also used as supplementary material to T.Hermann, P.Meinicke, and H.Ritter. (2000). Principal curve sonification. In Proc. ICAD 2000, pp. 81–86.
media file S16.4 (b)
download: SHB-S16.4b (mp3, 46k)
source: rendered by the author. Also used as supplementary material to T.Hermann, P.Meinicke, and H.Ritter. (2000). Principal curve sonification. In Proc. ICAD 2000, pp. 81–86.
Example S16.5: Data Crystalization Sonification
The sound examples demonstrate interactive exploration of data with the DCS sonification model. The test data set consists of three clusters in a row with variance (intrinsic dimensionality, id) in 2, 4, and 8 dimensions.
In the first example excitation starts in the leftmost cluster, so the sonification starts with the soft timbre and becomes more brilliant with inclusion of the second (id=4) and third (id=8) cluster.
media file S16.5 (a): DCS of mixture of 3 gaussians started in the id=2 cluster
download: SHB-S16.5a (mp3, 16k)
source: sound rendered by the author, published in T. Hermann and H. Ritter. Crystallization sonification of high-dimensional datasets. ACM Trans. Applied Perception, 2(4):550–558, 2005.
In the second example excitation starts in the middle cluster (id=4), so the sonification starts at higher brightness due to the higher covariance structure and shows only a second inclusion sound for the two equally neighboring neighbor clusters with (id=2) and (id=8).
media file S16.5 (b): DCS of mixture of 3 gaussians started in the id=4 cluster
download: SHB-S16.5b (mp3, 16k)
source: sound rendered by the author, published in T. Hermann and H. Ritter. Crystallization sonification of high-dimensional datasets. ACM Trans. Applied Perception, 2(4):550–558, 2005.
In the third example excitation starts in the rightmost cluster (id=8), so the initial timbre is very bright due to the highest covariance and decays in brightness with the subsequent inclusion of data points from lower dimensional clusters, first with (id=4) and finally with the (id=2) cluster.
media file S16.5 (c): DCS of mixture of 3 gaussians started in the id=8 cluster
download: SHB-S16.5c (mp3, 16k)
source: sound rendered by the author, published in T. Hermann and H. Ritter. Crystallization sonification of high-dimensional datasets. ACM Trans. Applied Perception, 2(4):550–558, 2005.
Example S16.6: Particle Trajectory Sonification for 1 particle
The sound examples first demonstrate how the trajectory sounds when a single particle is injected into the data set potential function under different conditions as explaned below. The data set is for all examples a sample drawn from a 5D gaussian distribution.
media file S16.6 (a) PTS for 1 particle at low damping constant
download: SHB-S16.6a (mp3, 9k)
source: created by Thomas Hermann
media file S16.6 (b) PTS for 1 particle at higher damping constant
download: SHB-S16.6b (mp3, 1k)
source: created by Thomas Hermann
media file S16.6 (c) PTS for 1 particle at lower particle mass
download: SHB-S16.6c (mp3, 1k)
source: created by Thomas Hermann
media file S16.6 (d) PTS for several particles simultaneously injected while reducing for each injection the bandwidth parameter sigma
download: SHB-S16.6d (mp3, 20k)
source: created by Thomas Hermann
Example S16.7: Particle Trajectory Sonification for N particles in 3 Clusters
Sound examples (a) and (b) are sweeps while decreasing σ. The first example is for a data set consisting of three clusters. Stable pitches occur during decay at middle values of σ corresponding to well-shaped potential troughs at clusters. The second data set is only a single Gaussian distribution without further substructure, and in turn this pitch structure is absent in the sonification.
media file S16.7 (a)
download: SHB-S16.7a (mp3, 23k)
source: by the author. see T. Hermann and H. Ritter. Sound and meaning in auditory data display. Proceedings of the IEEE (Special Issue on Engineering and Music – Supervisory Control and Auditory Communication), 92(4):730–741, 2004.
media file S16.7 (b)
download: SHB-S16.7b (mp3, 23k)
source: by the author. see T. Hermann and H. Ritter. Sound and meaning in auditory data display. Proceedings of the IEEE (Special Issue on Engineering and Music – Supervisory Control and Auditory Communication), 92(4):730–741, 2004.
Example S16.8: Growing Neural Gas Sonification
Sonification examples (a-c) show that clusters of different intrinsic dimension sound differently when energy is injected into one of their neurons: note that higher-dimensional distributions automatically sound more brilliant without this feature having been mapped or computed explicitly during any part of the model construction.
media file S16.8 (a) GNGS excited in the subgraph within the id=1 cluster
download: SHB-S16.8a (mp3, 17k)
source: by the author. T. Hermann, H.Ritter. (2004). Neural gas sonification – growing adaptive interfaces for interacting with data. In E. Banissi and K. Börner, (eds.), IV ’04: Proc. of the Information Visualisation, IV’04, pp. 871–878, Washington, DC, USA, 2004. IEEE CNF, IEEE Computer Society.
media file S16.8 (b) GNGS excited in the subgraph within the id=3 cluster
download: SHB-S16.8b (mp3, 30k)
source: same as S16.8 (a)
media file S16.8 (c) GNGS excited in the subgraph within the id=5 cluster
download: SHB-S16.8c (mp3, 30k)
source: same as S16.8 (a)
media file S16.8 (d) GNGS excited in the subgraph within the id=7 cluster
download: SHB-S16.8d (mp3, 30k)
source: same as S16.8 (a)
Example S16.9: GNG Sonification of the growth process
Sonifications of a growth process, that means that all neurons create sound during the GNGS learning process. In consequence the changes of neuron interconnectivity lead to changing timbres. This allows the user to hear when a network has reached a stable structure, when the GNGS starts to overfit the data and adapt to the noise. The video depicts a simple 2D example of a noisy spiral data where the sonification can be heard while observing how the interconnectivity changes.
media file S16.9 (a) mixture of a 2d and 5d gaussians centered at origin in data space
download: SHB-S16.9a (mp3, 30k)
source: by Thomas Hermann
media file S16.9 (b) GNG growth sonification in a pure 5d gaussian
download: SHB-S16.9b (mp3, 30k)
source: by Thomas Hermann
media file S16.9 (c) video of a GNG growth for a 2d spiral data set
download: SHB-S16.9c (mp3, 2.1M)
source: by Thomas Hermann. see also: T. Hermann, H.Ritter. (2004). Neural gas sonification – growing adaptive interfaces for interacting with data. In E. Banissi and K. Börner, (eds.), IV ’04: Proc. of the Information Visualisation, IV’04, pp. 871–878, Washington, DC, USA, 2004. IEEE CNF, IEEE Computer Society.