Proposing the methods of measuring the relationships between hearing and vision

The following article constitutes the theoretical basis for preparing the multimedia questionnaire verifying how we react to audiovisual transmission. The subject of the hearing studies is loudness, oscillation frequency and timbre of the sound, while location, size, brightness, shape and colour of the luminous surface belong to the area of vision studies. The purpose of our research is to build a successive model of synesthetic perception and to find the reasons for similarities occurring between associations.

Previous achievements

Currently we can distinguish two types of statistical studies related to synesthesia. The first study includes selecting a group of people responding to stimuli that normally affect one sense by many senses¹. More detailed data present various connections between the senses. The second study is connected with the questionnaires determining the polysensory associations that appear in our imagination. The differences between the methods of these studies result mainly from the differences in understanding the phenomenon of synesthesia. The study I would like to suggest is convergent with the second type described above.

The similar study was conducted in 1968 and 2002 by the researchers in the Institute of Experimental Aesthetics "Prometheus" in Kazan². From my subjective point of view, the most significant conclusion drawn by the scientists was the fact that the knowledge and experience of the respondents strongly influenced the results of the research³. What is more, it was shown that the majority of musicians perceive brightness, size and shape of the sound's edge in a similar way.

The questionnaire was directed to the Russian composers, who could easily imagine the sounds described in the questions. However, the differences concerning musical imagination of the individual composers could cause informational noise, especially when answering more difficult questions. For this reason, in my opinion, the need arises to prepare a multimedia questionnaire. Additionally, thanks to the application of the Internet, it will be possible to extend the group of respondents.

The main assumptions of the questionnaire

The object of our research will be these features of sound that can be easily presented with the help of different kinds of images. One sound and different moving images, from which the respondents will choose the one that (in their opinion) suits the played sound best, will constitute the continually recurring scheme. Due to the fact that the majority of our society is characterized by the relative pitch, the relationships between the sounds and images will be examined and shown in a continuous way as a modification of particular sound and image features. This solution aims at simplification and standardization of sounds and images.

As for the additional information helpful in determining the reasons for the similar answers appearing in the questionnaires, the respondents will be asked to provide their age, sex, nationality, right- or left-handedness, as well as to briefly describe their musical and artistic skills. Furthermore, the response time (spent on thinking about the answer) and the number of the image displays will be recorded.

The statistical data gathered during the research will be able to be used as the guidelines for planning the multimedia communication. They can be useful also in light-music art during the conscious act of searching for the surprising devices of expression.

Relationships

Four different sounds will serve as the tools to examine the particular relationships between hearing and vision. The first one is a sinusoid that steadily increases in loudness (with constant frequency of 440 Hz⁴). This sound will appear only in one test concerning the loudness of sound. The other two sounds are the sinusoid that is constant in loudness (with steadily increasing oscillation frequency from 440 to 880Hz⁵), together with steadily changing intensity level of harmonics of instruments (from the most bass to the most treble ones). I would like to use these two sounds inseparably on purpose, because they reflect the changes of two very important sound features (pitch and timbre) in a similar way. The last sound will be the fluent transition from the clear sinusoid with frequency of 440Hz to noise. This sound will be used in a test concerning the saturation of the sound colour. Let us now take a look at the particular vision-hearing analogies.

The brightness of sound resulting from its loudness

The amount of light emitted by the image is strictly connected with the size and brightness of the luminous surface. The crucial factor in this case is the brightness of the background, on which the luminous surface reflecting the sound appears. If we want to present the sound by means of the white circle on the black background, the amount of light will be proportional to the size and brightness of the circle. The situation becomes complicated when the background is white. If the reduction of the black circle on the white background is associated with the enhancement of the sound's loudness, this will mean that the amount of light corresponds with the loudness of the sound.

The location of sound on the plane

The notion of sound position functions in musical terminology and is equally understood even among the people that do not have musical skills. The sounds that are characterized by large oscillation frequency are usually called "high", whereas those of small frequency are called "low". However, it is possible that there are people who experience the vertical axis movement differently. The guitar strings which produce low sounds are located farther from the floor, which may influence the dissimilar idea of the sound position. This test will serve as the source of information about how much the location of sound is related to its tone and oscillation frequency.

I suggest to conduct another similar research on the basis of the one described above, concerning the horizontal axis movement as well. Thanks to these two studies we will obtain information about the location of sound on the plane.

The location of sound in the depth axis

Leonardo da Vinci claimed that there is a relationship between the linear perspective and the sound location⁶. The majority of musical forms are organized in such a way that low sounds appear more rarely, setting a scheme according to which the higher sounds emerge. Transferring this to the imaginary artistic space automatically causes the appearance of high sounds before the low ones. The difficulty we have to face in this case is the easiest possible way to present the movement along the depth axis.

The size of sound

Imagining low sounds as the large colour spots is bound with the knowledge concerning the fact that the size of an item producing sound influences the oscillation frequency. However, we have to remember that in the case of an artistic piece enhancing the element in the foreground means reducing the background area. If the background is bright, and the spot that visualizes the sound is dark, the enhancement of the spot will darken the whole image. Yet, low sounds are associated with darker colours. Hence, the need arises to verify the associations between the sound size with the graphic forms presented on different types of background brightness.

The shape of the sound's edge

The research conducted in 2002 by the scientists of the "Prometheus" Institute showed that circle is often associated with low sounds, whereas triangle with the high ones. However, so far more attention has been paid to the relationship between the shapes and tones of different phones⁷. Also Wassily Kandinsky noticed that the sounds of the instruments may be either oval or sharp⁸. The justification of this analogy is visible primarily on the sound wave graphs illustrating the change of acoustic pressure in time. The increase of oscillation frequency is connected with the densification of the graph and the sharpening of the wave's crests. Following this track, some musicians try to create music by artistic means of drawing the graphs of waves, then transformed to sounds⁹.

The brightness of sound resulting from the oscillation frequency

Associating high sounds with bright points may be related with the increased easiness of locating them in space. Just like in the case of the previous test and the one connected with the loudness of sound, the brightness of the background may influence the study's results, hence the need arises to conduct the two tests.

The colour of sound

Among the commonly used synesthetic definitions of sound features, position and colour are the most popular ones. These two notions interweave with each other, which resulted in attempts to assign certain sounds to the colours of the rainbow. One of the scientists that followed this line of thinking was Isaac Newton¹⁰. Although the theory concerning the relationship of successive sounds of the piano keys with specific colours was effectively refuted¹¹, it is beyond doubt that Newton's considerations are inspiring. On the basis of the psychophysiological experiment, he claimed that the ability to distinguish between colours is convergent with the ability to distinguish between sounds. In the case of the study that I proposed this is a signal that it is worth to take into account the ability to associate the change of sound pitch with the change of colour. However, the colour discussed by the musicians is more related to the tone, often appearing as the result of the indirect association with the subjective sensation of the temperature of sound. Therefore, I suggest that we base on this indirect association in our research.

Creating the moving image presenting the changes of sound colour is connected with two difficulties. Firstly, apart from the fact that the colours differ in terms of temperature, they also differ in terms of brightness. The solution that enables to avoid the change of colour brightness is limiting this change to such an extent so that the compromise between the greatest possible difference of temperatures and the smallest possible difference of brightness is reached. An additional difficulty is that practically every colour has its cold and warm equivalent. Hence, a question arises: which colour has to be displayed?

The saturation of sound

Graphic designers have probably noticed that only one colour parameter is missing to create the perceptive model of colours. This component is saturation. How can the saturation of colour be linked to sound? I think that this is connected with the structure of the instrument's timbre. The sounds characterized by more ordered structure (e.g. sinusoid) can be perceived as clearer colours, whereas the noises are closer to shades of gray.

Conclusions

Conducting the questionnaire will be more objective, if more respondents take part in the research. This means lengthening the research time and putting more effort in preparing the study. Thus, it is necessary that this questionnaire is thought out as carefully as possible already at the preparatory stage. This is why the present lecture is not a closed set of information about the topic provided in the title. It is definitely more of a question that can be formulated in the following way: where am I wrong? It is an invitation for you to cooperate mainly in preparing the assumptions for the questionnaire. In the very beginning the questionnaire will be prepared in Polish, Russian and English, so the help in translating the questions would be highly appreciated. Another form of participating in this study would be also popularizing the questionnaire.

Probably the present conclusions would not appear, if it weren't for the long-term studies conducted by B. Galeyev and his wife I.Vanechkina, since their contribution to the development of knowledge about synesthesia is priceless. Taking the opportunity, I would like to thank doctor Kazimierz Jałowczyk and Maciej Zdanowicz for our valuable conversations about synesthesia. Above all, however, I express my thanks to God, who created sound and who Himself is the Light.

Bibliography

1. Thomas Birch, The history of the Royal Society of London for improving of natural knowledge, A. Millar in the Strand, 1757
2. Ванечкина И.Л., Некоторые итоги анкетного опроса по выявлению закономерностей цветного слуха среди членов союза композиторов СССР, 1968
3. Ванечкина И.Л., Галеев Б.М., „Цветной слух” музыкантов: мифы и реальность, 2003
4. Ванечкина И.Л., Галеев Б.М., Кандинский и Скрябин: мифы и реальность, 1998
5. Ванечкина И.Л., Галеев Б.М., Овсянников А.А., Анкетный опрос „цветного слуха” композиторов России, 2002
6. Wassily Kandinsky, Concerning the spiritual in art, New York 1977
7. Bartosz Mozyrko, Nazywanie kształtów, kształtowanie nazw. O znaczeniu asemantycznych skojarzeń nazw dla projektowania logo
8. Овсянников А.А., Анкетное исследование „цветного слуха” и „цветового мышления", 2000
9. Oliver Sacks, Musicophilia: Tales of Music and the Brain, Vintage Books, 2008
10. Maciej H. Zdanowicz, Obrazowanie muzyki - badanie nowych możliwości wizualnej transkrypcji dźwięków, Łódź 2010