A comparison of perception and everyday evaluation of emotional prosody at the beginning of cochlear implant treatment and after completion of follow-up therapy

Beate Stadler; Jan Hots; Tereza Vosiková; Martin Durisin; Nicole Angenstein; Markus Meis

doi:10.3205/zaud000097

Short Report

A comparison of perception and everyday evaluation of emotional prosody at the beginning of cochlear implant treatment and after completion of follow-up therapy

Beate Stadler ¹
Jan Hots ²
Tereza Vosiková ¹
Martin Durisin ¹
Nicole Angenstein ¹
Markus Meis ^3,4

¹ Otto von Guericke University Magdeburg, University Clinic for Otorhinolaryngology and Head and Neck Surgery, Magdeburg, Germany
² Otto von Guericke University Magdeburg, Department of Experimental Audiology, Germany
³ Hörzentrum Oldenburg gGmbH, Oldenburg, Germany
⁴ Cluster of Excellence “Hearing4all”, Carl von Ossietzky University Oldenburg, Germany

Abstract

Prosody is the vocal and speech-related expression contained in the speech signal that, in addition to the semantic meaning conveyed by word choice, conveys, for example, the emotional content of the speaker’s voice. However, cochlear implant (CI) users have only limited access to speech sound characteristics for auditory comprehension. This makes it difficult to attribute meaning to prosodic patterns. This can lead to misinterpretations, a reduced intelligibility, and misunderstandings in communication and social interactions. As a result, it is to be expected that people with severe hearing impairments will feel excluded in social situations. Both aspects – the reduced ability to recognize emotions and the psychosocial consequences – are the subject of this article.

The aim of this study was to undertake a retrospective comparison of two methods for measuring emotional prosody perception: firstly, the “Test of Emotional Prosody Perception” (TEPP-CI); and secondly, the “Questionnaire on emotional communication in hearing situations” (EMO-CHeQ). The EMO-CHeQ has already demonstrated its efficacy in cross-sectional studies involving users of hearing aids, CI as well as and normal-hearing (NH) listeners. The present study focuses on longitudinal measurements in CI care, specifically at the beginning of CI care and after the end of follow-up therapy.

The stimuli used in the TEPP-CI were drawn from an evaluated prosodic speech corpus. These consist of 2-syllable nonsense pseudowords. These were spoken by two actors (one female, one male using the prosodic expressions of joy, sadness, anger, fear, disgust, as well as in a neutral tone. The stimuli were presented to postlingually deafened CI users directly through the speech processor at a comfortable level. The EMO-CHeQ measures subjective emotional prosody recognition in everyday life using four subscales and an overall scale.

In the TEPP-CI, correct classifications exceeded the chance level. The CI users improved in emotional prosody perception, though not to the same extent as in the speech tests. The EMO-CHeQ showed an improvement in subjective emotional prosody recognition. The results of the follow-up assessments will be used in the future to improve the perception of sound qualities in CI users through targeted auditory training.

Keywords

CI users, emotional prosody, TEPP-CI, EMO-CHeQ, cochlear implant, questionnaire

Introduction

Newly fitted CI users undergo an extensive process of learning auditory comprehension after they begin using their CI. During this phase, the brain relearns how to process and interpret the information that is reduced and transmitted via the CI, and integrates it with existing knowledge in memory. The goal here is the ability to communicate auditorily [1]. Although CI fitting primarily focuses on speech comprehension, in recent years, CI users have come to expect that their CI will also allow them to experience better sound qualities. Due to the shifting of indication criteria in the guidelines for CI treatment and advancements in CI technology, CI users are now achieving remarkably good results. Some professional musicians, for example, report that after appropriate training, they are able to practice their profession even with a CI (see testimonials on the MED-EL and Cochlear websites).

One possible model for investigation of processing sound qualities involves the perception of emotional prosody [2]. This is highly relevant in everyday life, as emotional prosody plays a crucial role in interpersonal communication. Since CI users have only limited access to speech sound characteristics for auditory comprehension, attributing meaning to prosodic patterns can be difficult in conversational contexts (e.g. [3], [4], [5], [6]. If this fails, it leads to misinterpretations and misunderstandings. We use emotional prosody here as a model to illustrate auditory development in the perception of sound qualities following CI implantation [5], [7], [6].

At various points in time during the CI fitting process, participants complete both a prosody test (TEPP-CI) and a questionnaire (EMO-CHeQ). The EMO-CHeQ was first completed at the start of baseline therapy (initial fitting – EMO-CHeQ_1) and the second time after the end of follow-up therapy, i.e., after approximately 2 years of CI use (EMO-CHeQ_2). The TEPP-CI was first administered after 1 month of CI use (untrained – TEPP-CI_1) and a second time after the end of follow-up therapy, i.e., after approximately 2 years of CI use (trained – TEPP-CI_2). All measurements were collected in parallel with routine clinical examinations.

Study 1 – TEPP-CI

The criteria for inclusion were that the CI users were receiving a CI for the first time in that ear, that they were native German speakers and of legal age, and that they were able to complete the “Test of Emotional Prosody Perception” (TEPP-CI) independently.

Participants

Data from 92 CI users were analysed (51 female, 41 male, aged 19 to 90). Measurements were performed with 48 left ears and 50 right ears (a total of 98 ears, as six participants had bilateral cochlear implants). All participants were tested using their own speech processors. These were divided into 51 speech processors from the MED-EL company (Sonnet, Sonnet2, Rondo2, Rondo3) and 47 speech processors from the Cochlear company (CP810, CP910, CP1000, CP1110, Kanso, Kanso 2). The participants gave informed consent to participate in the measurements.

Material – TEPP-CI

The “Test of Emotional Prosody Perception” (TEPP) contains stimuli from an evaluated speech corpus, namely the WaSeP [8]. The TEPP uses two-syllable pseudowords as stimuli, which were spoken by an actress and an actor in the prosodic expressions of joy, sadness, anger, fear, disgust, and neutral tone.

A total of 89 NH participants, aged between 18 to 82 years, participated in the validation study for the stimuli. They assigned their perceived expression to 384 stimuli (32 per speaker and prosodic expression). On average, the NH were able to correctly assign the stimuli to the intended expression in over 70% of cases [9].

From the TEPP validation study for each prosodic expression and speaker, the five stimuli were selected that NHs most frequently assigned to the intended prosodic expression (30 stimuli each for female and male actors). These were presented to the CI users (TEPP-CI).

Experimental procedure

The TEPP-CI contains 60 stimuli, which are presented in random order using the Psychopy software (v 1.71.01) [10], and takes approximately 10–15 minutes to complete. CI users were asked to assign one of six possible intended emotional prosodic expressions to each presented stimulus (six-alternative forced-choice paradigm). Participants entered their responses via a touchscreen monitor. The responses were recorded and saved as an Excel spreadsheet at the end. The stimuli were presented in a soundproof booth at the ENT clinic directly via the speech processor at a comfortable level. For direct transmission from the stimulus computer to the speech processor, devices provided to patients by the CI companies were used (MiniMic, AudioLink, T-coil, and cables).

After each measurement, participants were asked to rate on an 11-point scale how difficult and how exhausting they experienced the measurement (0=not at all, 10=extremely).

Results

The average accuracy rate exceeded the chance level of 16.67% at both times of measurement. In the first measurement, the median was 24%, and in the second measurement, it was 32%. There was a positive correlation between the first and second measurement (Spearman’s ρ=0.586, p<0.001). In the first measurement, 15 ears were below the chance level, and in the second measurement, 9 ears were below the random level.

The hit rate improved significantly between the first and second measurements (Wilcoxon, p<0.05), as shown in Figure 1 [Fig. 1] left.

Figure 1: Significant improvement in the hit rate in the TEPP-CI (left panel) and in monosyllabic words test (right panel) from the first to the second time of measurement (median (horizontal line), mean (x), interquartile range (box), range (whisker) and outliers (dots) are shown; the light bar shows TEPP-CI_1, the dark bar shows TEPP-CI_2). The solid line in the left panel shows the chance level (16.67%).

Although prosody recognition is above the random level, it does not achieve the level of speech recognition (Figure 1 [Fig. 1] right panel). The speech recognition also significantly improved from the first time of measurement to the second time (Wilcoxon, p<0.001).

In terms of accuracy, the individual prosodic expressions can be broadly divided into two groups (see Figure 2 [Fig. 2]), the better-recognized expressions of neutral, sadness, and anger, and the less well-recognized expressions of joy, fear, and disgust. The expression of fear is not well recognized in the first measurement, with the median accuracy falling below the chance level. For all prosodic expressions, the accuracy rate improved from the first to the second measurement. This improvement is significant for the prosodic expressions of anger and sadness (Wilcoxon, p<0.05).

**Figure 2: Hit rate for individual prosodic expressions at the first and second time of measurement. Data presentation is the same as in the previous figures.**

Figure 3 [Fig. 3] shows how difficult and how exhausting the participants experienced the prosody test. In general, the TEPP-CI was perceived as both difficult and exhausting. However, the second survey revealed a decrease in perceived difficulty and effort. This decrease was significant for difficulty (Wilcoxon, p<0.05).

Figure 3: Survey of participants regarding how difficult (left panel) and exhausting (right panel) the TEPP-CI was perceived to be at the first and second time of measurement. Data presentation is the same as in the previous figures.

Conclusion study 1

The median hit rates in the TEPP-CI were above the chance level, except for the prosodic expression fear at the first measurement point, though with considerable variation. Although the hit rates of the CI users were above chance level, they were clearly below those of NH listeners [9]. On average, the prosodic expressions of neutral, anger, and sadness were recognized better than those of disgust, joy, and fear. The correct classifications for the individual prosodic expressions varied strongly, both within a single prosodic expression and across the different emotional expressions. For all prosodic expressions, there was an improvement in the hit rate, which was significant for anger and sadness.

Study 2 – EMO-CHeQ

The EMO-CHeQ questionnaire is used to assess the perception of emotions in spoken language [11]. The German version has been validated by Meis et al. among normal hearing participants and hearing-aid users [12]; for the validation with CI users see Issing et al. [7].