[Das Erkennen von emotionaler Prosodie durch Cochlea-Implantat-Nutzer und die Selbsteinschätzung ihrer Wahrnehmung]

Friederike Issing ¹
Markus Meis ²
 Beate Stadler <sup>3

1</sup> University of Würzburg, Germany
² Hörzentrum Oldenburg gGmbH, Oldenburg, Germany
³ University Clinic of Otolaryngology, Head and Neck Surgery, Magdeburg, Germany

Zusammenfassung

In den zurückliegenden Jahren wurde eine signifikante Weiterentwicklung von Cochlea-Implantaten (CI) beobachtet. Stand bisher vor allem das Sprachverstehen im Zentrum der Versorgungsabsicht, nimmt jetzt darüber hinaus auch immer mehr die Wahrnehmung von Hörqualitäten einen zentralen Punkt ein. Ein mögliches Modell für das Gelingen der Wahrnehmung von Klangqualitäten ist die emotionale Prosodie. Bislang ist nur wenig über die Möglichkeiten der emotionalen Prosodiewahrnehmung bei CI-Nutzern bekannt. Deswegen wollten wir wissen, inwieweit CI-Nutzende generell in der Lage sind, emotionale Prosodie wahrzunehmen und einem emotionalen Ausdruck zuzuordnen.

Wir haben zwei Querschnittsstudien mit CI-Nutzenden durchgeführt. In der ersten Studie wurde ein „Test zur emotionalen Prosodie-Perzeption“ (TEPP) mit emotional gesprochenen Stimuli durchgeführt. In der zweiten Studie wurde die Wahrnehmung emotional gesprochener Sprache im Alltag durch CI-Nutzende mit der deutschen Variante des Fragebogens „Emotional Communication in Hearing Questionnaire“ (EMO-CHeQ) abgefragt. Sowohl TEPP als auch EMO-CHeQ werden bei uns seit Jahren in der klinischen Routine durchgeführt.

An der TEPP-Studie nahmen bisher 181 postlingual ertaubte CI-Nutzende teil. Dabei mussten akustische Prosodie-Stimuli von den CI-Nutzenden über einen Touch-Screen-Monitor hinsichtlich des emotional prosodischen Ausdrucks dem intendierten Ausdruck zugeordnet werden. Die Anzahl der korrekten Zuordnungen beim TEPP lag im Mittel über dem Zufallsniveau, jedoch mit einer großen Streuung.

In der EMO-CHeQ Studie wurden bisher 161 CI-Nutzende befragt. Beim EMO-CHeQ konnte mittels explorativer Analysen die vier-faktorielle Struktur der englischsprachigen Version des EMO-CHeQ bestätigt werden. Weitere Analysen zeigten, dass SSD-Patienten geringere Beeinträchtigungen berichteten als Patienten mit Hörgerät und CI.

Zukünftig sind Studien geplant, bei denen eine Korrelation des EMO-CHeQ mit der TEPP-Testbatterie erfolgen soll und bei einem Vorher-Nachher-Vergleich die Effekte einer CI-Versorgung bezüglich der emotionalen Prosodie geprüft werden.

---

---

Introduction

To facilitate effective communication, it is not only important to comprehend the content of what is being expressed, but also the way in which it is conveyed. Prosody, defined as the linguistic and vocal expression contained in the speech signal that works alongside the semantic statement, is a pivotal component in successful communication (e.g. [1], [2], [3], [4]). It has been demonstrated that prosody can reveal underlying emotions, thereby facilitating comprehensive interpersonal interaction (e.g. [5], [6], [7], [8]). It plays a central role in interpersonal communication. However, cochlear implant (CI) users encounter significant challenges in understanding speech due to their limited access to auditory cues (e.g. [9], [10], [11], [12]). This poses a major challenge in ascribing meaning to prosodic patterns, with the potential of misinterpretation and misunderstanding.

This issue served as the focus of our cross-sectional studies, in which we employed two approaches:

1. a recognition test using acoustic stimuli as a clinical test under ideal conditions, the “Test of Emotional Prosody Perception” (TEPP) [13], and
2. a questionnaire for CI users to assess their ability to recognise emotional prosody in everyday life. This was measured using the “Emotional Communication in Hearing Questionnaire” (EMO-CHeQ) [14].

The EMO-CHeQ is a reliable and ecologically valid measure for rapidly assessing experiences of hearing and handicap when listening to signals containing vocal emotion cues [14]. It consists of four sub-scales and has been evaluated in the context of hearing aid provision in English-speaking countries [14] and Germany [15].

The first aim of these cross-sectional studies was to determine to which extent CI users can correctly match the perceived emotional prosody to the speaker’s intended expression. This phenomenon was evaluated using through the utilisation of a prosody test, a standardised auditory assessment tool known as TEPP [13]. The second aim addressed by the second was to investigate whether CI users divided into sub-groups depending on the mode of support perceive their limited perception as a handicap in their everyday life. This was examined with reported outcome measurement (PROM) using the EMO-CHeQ questionnaire [14] in the domain of CI provision.

Study 1 – “Test of Emotional Prosody Perception” (TEPP)

Participants

So far, n=181 experienced CI users (period of use >1 y 3 m), aged 20–92 years, have taken part in the study. Participants had to be native German speakers and be able to complete the TEPP independently. The study population comprised 90 women and 91 men, with 93 left ears and 98 right ears (n=191 ears, so far 10 BiCI tested in both ears) measured. At the time of the measurement, the CI users differed regarding the fitting of the contralateral ear. Most of them had a bimodal fitting (hearing aid in the opposite ear) (n=91), followed by individuals with single-sided deafness with CI support in the opposite ear (SSD; n=46), and bilateral CI users (BiCI, n=29). Some participants used a Vibrant Sound Bridge (VSB, n=2), bone-anchored hearing aid (BAHA, n=1), electric-acoustic stimulation (EAS, n=1), or were not fitted with a device in the contralateral ear (monolateral CI, n=11).

The CI users used sound processors from the companies Cochlear (84) and Med-El (97). The participants were measured on one ear with their own processors.

Material – Test of Emotional Prosody Perception (TEPP)

The TEPP was used to assess the perception of the emotional vocal cues within speech. The test used two-syllable pseudo-words from the Corpus WaSeP [16]. The WaSeP speech material was recorded during utterances of two professional actors (male/female) which should transfer the prosodic emotional expressions of ‘joy’, ‘sadness’, ‘anger’, ‘fear’, ‘disgust’ and ‘neutral’.

A former TEPP-study in 79 normal hearing subjects evaluated the most frequently correctly assigned stimuli per prosodic emotional expression from 384 stimuli (32 per speaker, twelve per prosodic emotional expression) [13]. The mean accuracy rate for normal hearing subjects on the TEPP was over 70% for all prosodic expressions. The short form of the TEPP, the TEPP-CI, contains 60 stimuli (30 per speaker, five per prosodic emotional expression). TEPP-CI is already used in cochlear implant recipients [17], [18].

Experimental procedure

Since 2011, the TEPP-CI has regularly been conducted on CI users during routine appointments at our ENT clinic. This took place in a sound insulated listening booth.

If both ears were tested (n=10 BiCI-users), the measurements were taken in separate sessions.

With the OPUS2, Rondo, Freedom, CP810 processors the stimuli were presented in free-field via a Behringer 1C monitor (1 m distance from the front). Opposite ear masking with 65 dB of noise was administered, if indicated. With the CP910 and Sonnet and subsequent processors, the stimuli were presented directly via the processor (with cable connection, MiniMic, AudioLink, T-coil) at ‘comfortable’ loudness. Stimulus presentation changed from a loudspeaker to direct presentation as more and more patients with substantial hearing in one ear, e.g. SSD participated. Consequently, the volume of the masking signal had to be increased according to the residual hearing in the opposite ear. This may lead to changes in auditory perception, whereby louder masking noise leads to worse comprehension [19]. The “comfortable” volume for each CI user (such that even quieter, softly spoken stimuli could be easily heard) was determined individually during the training phase in which the sample stimuli were presented.

The stimuli were presented in a randomized order using PsychoPy^® software (psychophysics software written in Python) [20], [21]. Initially, the test subjects were presented with one example for each speaker and each prosodic expression. Using a touchscreen monitor, the participants were presented with a six-alternative forced-choice paradigm and were asked to assign each stimulus to the perceived emotional prosodic expression cue. Each response was recorded and the correct assignments were noted as hit rates.

Results

Descriptives

For the TEPP-CI of 191 CI ears, the descriptive statistics showed an average hit rate of 37% for expected emotional prosody cue, which is above chance (16.67%), albeit with a large spread of 83.3% maximum and 8.3% minimum. Results of 11 tested CI ears were below the chance level.

As illustrated in Figure 1 [Fig. 1], the prosodic expressions such as ‘neutral’, ‘anger’ and ‘sadness’ were categorized with a high degree of accuracy. Expressions such as ‘disgust’, ‘joy’ and ‘fear’ were more challenging to categorise accurately. On average, it was found that ‘disgust’ and ‘joy’ were categorized at higher rates (approximately 30%) than ‘fear’ (approximately 22%).

The hit-rate varied both within and across prosodic emotional cues (Table 1 [Tab. 1]).

The matrix of confusion demonstrated that the expression ‘joy’ was most likely to be confused with the expression ‘neutral’. The emotional cue ‘fear’ was highly confused with both ‘anger’ and ‘neutral’. In this case, more attributions were made to an emotional cue other than the intended one.

Conclusions study 1

CI users have limited ability to distinguish emotional prosodic expressions, with considerable variability both among them and across different prosodic expressions.

Study 2 – the EMO-CHeQ Questionnaire

Participants

For the EMO-CHeQ analyses, we have so far included at 163 CI users who completed the EMO-CHeQ questionnaire. The mean age was M=61.5 years (SD=13.9) and 50.3% of participants were female. Subject cohorts were subdivided according to the mode of provision: bimodal patients (n=83); SSD (n=45); and BiCI (n=22). In the remaining cases, some individuals were fitted with a VSB (n=2); EAS (n=1); and BAHA (n=1). Nine individuals were not fitted with any device on the contralateral side (monolateral CI).

Material – German version of the EMO-CHeQ Questionnaire

The survey of CI users was conducted using “The Emotional Communication in Hearing Questionnaire” (EMO-CHeQ – Singh et al. [14]). This questionnaire was developed to examine the prevalence of challenges in recognizing the vocal emotion information in listeners with hearing impairment, and the subsequent impact of misinterpretations and misunderstandings on the severity of the subjectively perceived handicap [14].

The German version of the EMO-CHeQ [15] consists of 16 statements with the response categories from ‘strongly disagree’, ‘slightly disagree’, ‘neither agree nor disagree, ‘slightly agree’, and ‘strongly agree’. Participants with hearing aids provided answers relating to the aided situation. Higher scores on the EMO-CHeQ indicate greater impairment.

The original 16 EMO-CHeQ questions were translated into German by means of an intensive forward-backward translation procedure with several steps [15] including native German and English speakers. The German version of the EMO-CHeQ questionnaire can be obtained from Attachment 1 [Att. 1].

To our knowledge, the EMO-CHeQ has not yet been used in CI care. As the factor structure for hearing aid users and CI users due to the difference between “acoustic” and “electrical” hearing, we have conducted an Exploratory Factor Analysis (EFA) on a group of CI users.

Experimental procedure

Most CI users completed the EMO-CHeQ questionnaire without assistance of the instructor to avoid a response bias. The questionnaire was issued in paper form and completed independently by the participants. A cohort of n=163/181 individuals, participants of the first study (TEPP-CI), answered the questionnaire.

Statistics

The data were analysed utilising SPSS (version 29), analysing descriptives, analyses of variance with covariates (ANCOVAs), and Exploratory Factor Analyses (EFA). The significance level for the ANCOVAs was set at alpha=5%.

Results

Explanatory factor analyses (EFA)

An EFA with n=163 CI users was conducted. Following several iterations, a four-factor solution was found that explained the greatest amount of variance. The 4-factor structure and the item allocation to the four factors (see Table 2 [Tab. 2]) were identical to results presented by Singh et al. [14].

The first factor with the items 12–16 refers to the factor ‘Socio-emotional well-being’ with 57.0% explained variance, items 1–5 refer to the factor ‘Talker’ with an explained variance of 10.2%, items 6–9 refer to the factor ‘Production’ with 5.8%, and the items 10 and 11 refer to the factor ‘Situation’ with 5.7% explained variance; 78.7% explained variance in total.

The Cronbach’s α values for the four factors are as follows: ‘Socio-emotional well-being’=0.93, ‘Talker’=0.89, ‘Production’=0.90, and ‘Situation’=0.90. The Kaiser-Meyer-Olkin analysis shows a KMO value of 0.916, indicating that the sampling is adequate.

Descriptives

In the EMO-CHeQ, descriptive statistics showed that, on average, CI users scored at 2.7 (min. 1, max. 4.45, median 2.58) for their handicap, on a scale ranging from 1 (‘strongly disagree’) to 5 (‘strongly agree’).

The CI users rated the individual factor ‘Situation’ as the worst (Figure 2 [Fig. 2]). This relates to questions about understanding in noisy environments.

ANCOVAs in subgroups depending on the mode of provision

We conducted an ANCOVA in three subgroups bimodal (n=83), SSD (n=45), and BiCI (n=22).

Control factors were gender and the covariates age and duration of experience. Regarding the total score on the EMO-CHeQ, there was a significant effect for the subgroup factor with F(2/144)=4.13, p=0.018. The subgroup SSD showed the lowest level of impairment with M=2.26 (SD=0.91) compared to the bimodal subgroup with M=2.79 (SD=0.78) and BiCI patients with M=2.52 (SD=0.76). Bonferonni corrected post hoc tests showed significant differences between the SSD and bimodal subgroup. Age was a significant covariate (F(1/144)=7.74, p=0.006) – the older the individual, the higher their handicap. The significant effect of the mode of provision has been confirmed using non-parametric Kruskal-Wallis (Chi²=10.37, 2/150, p=0.006) as well as paired posthoc tests. However, it was not possible to control for confounding variables. Significant effects at the subscale level were only found for the factor ‘Situation’, which comprised the items “Identify emotions in a noisy environment” and “Identify emotions of others in competing speech” (F(2/144)=9.58, p=0.000). The handicap was found to be approximately one scale unit (M=3.78; SD=1.00) greater in bimodal patients than in SSD patients (M=2.88; SD=1.24).

Discussion and conclusions

These two studies demonstrated that TEPP-CI and EMO-CHeQ are suitable tools for assessing the efficacy of CIs in laboratory conditions and their effectiveness in everyday life among CI patients in terms of emotional prosody.

The EMO-CHeQ also suggests that this test is sensitive to different types of CI provision. SSD patients showed a significantly lower impairment regarding emotional prosody, and previous studies in the field of hearing aid provision have also been demonstrated differences between different types of hearing aid provision [14]. The initial exploratory findings for SSD and bimodal subgroups suggests for those who could use acoustic cues via their unimpaired ear consistently reported the least impairment. However, this data set does not allow us to determine the extent to which different signal processing delays between ears [22] are responsible for the greater handicaps of the bimodal subgroup.

Sensitivity to different modes of provision is still pending for the TEPP-CI test battery because the measurements were taken separately for each CI ear. Despite noise masking, the other ear may have access to the stimuli. In follow-up studies, we will also conduct both tests bilaterally in a free field in the lab and administer the questionnaire in everyday life. This will enable us to calculate correlations and to compare different provision subgroups (SSD, bimodal and BiCI) in a prospective pre-post study.

Whether the variability of the TEPP-CI can be explained by correlations with other factors remains to be ascertained. Relevant influencing factors could include, for instance, speech perception and age, as well as different generations of audio processors and coding strategies. This information could be used in order to formulate recommendations regarding appropriate auditory training techniques.

A qualitative analysis of the confusion stimuli (Table 1 [Tab. 1]) is yet to be conducted. This should clarify which acoustic parameters cause certain prosodic expressions to be categorised better or worse by CI users.

Moreover, the TEPP-CI and the EMO-CHeQ test procedures can be used to visualise learning effects. To this end, CI users should be tested at the beginning of CI treatment and after follow-up therapy has ended. If both measurement methods are suitable, it would also be possible to visualise the development of the perception of sound quality with a CI, at both the level of auditory perception and subjective assessment.

Notes

Conference presentation

This contribution was presented at the 27^th Annual Conference of the German Society of Audiology and published as an abstract [23].

Acknowledgements

We thank the CI users for their participation!

Positive ethics committee vote

Otto-von-Guericke University Magdeburg (61/10)

Competing interests

The authors declare that they have no competing interests.

---

References

[1] Stock E. Emotionserkennung und Stimme. Sprache und Sprechen. 1991;25:173-82.
[2] Neuber B. Prosodische Formen in Funktion. Leistungen der Suprasegmentalia für das Verstehen, Behalten und die Bedeutungs(re)konstruktion. Laussane: Peter Lang Verlag; 2002. (Hallesche Schriften zur Sprechwissenschaft und Phonetik; 7).
[3] Kehrein R. Prosodie und Emotionen. Tübingen: Niemeyer; 2002.
[4] Izard CE. Die Emotionen des Menschen: Eine Einführung in die Grundlagen der Emotionspsychologie. 4th ed. Weinheim: Belz; 1999.
[5] Scherer KR. Vocal correlates of emotional arousal and affective disturbance. In: Wagner H, Manstead A, editors. Handbook of social psychophysiology. Hoboken: John Wiley & Sons; 1989. p. 165-97.
[6] Scherer KR. Vocal communication of emotion: a review of research paradigms. Speech Communication. 2003;40:227-56. DOI: 10.1016/S0167-6393(02)00084-5
[7] Paeschke A. Prosodische Analysen emotionaler Sprechweise. Berlin: Logos; 2003.
[8] Kienast M. Phonetische Veränderungen in emotionaler Sprechweise. Düren: Shaker Verlag; 2002.
[9] Saß AC. Dekodierung emotionaler Sprechweisen durch Patienten mit Cochlear Implant. Zs f Angewandte Linguistik. 2002;36:81-91.
[10] Meister H, Tepeli D, Wagner P, Hess W, Walger M, von Wedel H, Lang-Roth R. Experimente zur Perzeption prosodischer Merkmale mit Kochleaimplantaten. HNO. 2006; 55:264-70. DOI: 10.1007/s00106-008-1692-3
[11] Meister H. Verarbeitung prosodischer Merkmale mit Cochlea-Implantaten. Sprache-Stimme-Gehör. 2011;35(03):e99-104. DOI: 10.1055/s-0031-1284405
[12] Mühlhaus J, Bartel-Friedrich S. Konzeption und Realisierung einer Prosodie-Testbatterie: CI-Träger im Fokus der Prosodieforschung. HNO. 2008; 56(3):258-61. DOI: 10.1007/s00106-008-1692-3
[13] Wendt B. Analysen emotionaler Prosodie. Frankfurt/M.: Peter Lang; 2007. (Hallesche Schriften zur Sprechwissenschaft und Phonetik; 20).
[14] Singh G, Liskovoi L, Launer S, Russo F. The Emotional Communication in Hearing Questionnaire (EMO-CHeQ): Development and Evaluation. Ear Hear. 2019;40(2):260-71. DOI: 10.1097/AUD.0000000000000611
[15] Meis M, Krueger M, Besser J, Wietoska L, Wagner B, Launer S, Singh G. Validierung des «Emotional Communication in Hearing Questionnaire» (EMO-CHeQ) im Rahmen einer feldexperimentellen Interventionsstudie zur Versorgung mit Hörgeräten. In: Deutsche Gesellschaft für Audiologie e.V., editor. 22. Jahrestagung der Deutschen Gesellschaft für Audiologie. Heidelberg, 06-09.03.2019. Düsseldorf: German Medical Science GMS Publishing House; 2019. Doc115. DOI: 10.3205/19dga115
[16] Wendt B. The Research Infrastructure for Language as Social and Cultural Data [https://www.clarin.eu/]: Gesprochenes Wortkorpus für Untersuchungen zur auditiven Verarbeitung von Sprache und emotionaler Prosodie. [updated 2017, cited 2025 Jun 29] Available from: https://clarin.phonetik.uni-muenchen.de/BASRepository/index.php?target=Public/Corpora/WaSeP/WaSeP.2.php
[17] Wendt B, Braun A, Hessel H, Stadler J, Vorwerk W. Unterscheiden sich prä- und postlingual ertaubte CI-Träger hinsichtlich ihrer emotionalen Prosodieperzeption? In: Deutsche Gesellschaft für Audiologie e.V., editor. 16. Jahrestagung der Deutschen Gesellschaft für Audiologie. Rostock, 27.02.-02.03.2013. 2013.
[18] Wendt B, Braun A, Stadler J, Vorwerk W. Emotionale Prosodieperzeption bei CI-Trägern. Sprachtherapie aktuell. 2020;7(1):e2020-25.
[19] Wendt B, Stadler J, Verhey JL, Hessel H, Angenstein N. Effect of Contralateral Noise on Speech Intelligibility. Neuroscience. 2021 Apr 1;459:59-69. DOI: 10.1016/j.neuroscience.2021.01.034
[20] Peirce JW. Generating Stimuli for Neuroscience Using PsychoPy. Front Neuroinform. 2009 Jan 15;2:10. DOI: 10.3389/neuro.11.010.2008
[21] Peirce JW. PsychoPy--Psychophysics software in Python. J Neurosci Methods. 2007 May 15;162(1-2):8-13. DOI: 10.1016/j.jneumeth.2006.11.017
[22] Zirn S, Angermeier J, Arndt S, Aschendorff A, Wesarg T. Reducing the Device Delay Mismatch Can Improve Sound Localization in Bimodal Cochlear Implant/Hearing-Aid Users. Trends Hear. 2019 Jan-Dec;23:2331216519843876. DOI: 10.1177/2331216519843876
[23] Issing F, Meis M, Stadler B. Erkennung emotionaler Prosodie bei CI-Tragenden und Selbsteinschätzung ihrer Wahrnehmung. In: Deutsche Gesellschaft für Audiologie e. V.; ADANO, editors. 27. Jahrestagung der Deutschen Gesellschaft für Audiologie und Arbeitstagung der Arbeitsgemeinschaft Deutschsprachiger Audiologen, Neurootologen und Otologen. Göttingen, 19.-21.03.2025. Düsseldorf: German Medical Science GMS Publishing House; 2025. Doc046. DOI: 10.3205/25dga046

---

Attachments

Attachment 1	EMO-CHeQ German version (zaud000089_Attachment1.pdf, application/pdf, 81.97 KBytes)