Developmental Trajectory of McGurk Effect Susceptibility in Children and Adults With Amblyopia

Abstract

Purpose.: The McGurk effect is an audiovisual illusion that involves the concurrent presentation of a phoneme (auditory syllable) and an incongruent viseme (visual syllable). Adults with amblyopia show less susceptibility to this illusion than visually normal controls, even when viewing binocularly. The present study investigated the developmental trajectory of McGurk effect susceptibility in adults, older children (10–17 years), and younger children (4–9 years) with amblyopia.

Methods.: A total of 62 participants with amblyopia (22 adults, 12 older children, 28 younger children) and 66 visually normal controls (25 adults, 17 older children, 24 younger children) viewed videos that combined phonemes and visemes, and were asked to report what they heard. Videos with congruent (auditory and visual matching) and incongruent (auditory and visual not matching) stimuli were presented. Incorrect responses on incongruent trials correspond to high McGurk effect susceptibility, indicating that the viseme influenced the phoneme.

Results.: Participants with amblyopia (28.0% ± 3.3%) demonstrated a less consistent McGurk effect than visually normal controls (15.2% ± 2.3%) across all age groups (P = 0.0024). Effect susceptibility increased with age (P = 0.0003) for amblyopic participants and controls. Both groups showed a similar response pattern to different speakers and syllables, but amblyopic participants invariably demonstrated a less consistent effect.

Conclusions.: Amblyopia is associated with reduced McGurk effect susceptibility in children and adults. Our findings indicate that the differences do not simply indicate delayed development in children with amblyopia; rather, they represent permanent alterations that persist into adulthood.

Amblyopia is a neurodevelopmental disorder affecting 3% to 5% of the population, and is the leading cause of monocular blindness worldwide.^1,2 It is defined clinically as a unilateral or bilateral loss of visual acuity caused by abnormal visual experience early in life.³ In addition to impaired visual acuity,⁴ it is well documented that patients with amblyopia exhibit deficits in spatiotemporal tasks, including attention^5,6 and motion perception,^7,8 as well as higher order cognitive tasks, such as number processing,⁹ reading,¹⁰ and real world scene perception.¹¹ Furthermore, there is recent evidence that impairments associated with amblyopia extend beyond vision, and that audiovisual perception may also be affected.¹² 

Audiovisual integration in adults with amblyopia has been investigated previously by our group using the McGurk effect, an audiovisual illusion that involves the concurrent presentation of a phoneme (auditory syllable) with an incongruent viseme (visual syllable).¹³ Visually normal individuals perceive a phoneme that does not match the actual auditory stimulus, due to the dominant influence of the viseme. Even when participants have prior awareness of the effect, the visual signal is sufficiently dominant that the illusion still is effective.¹⁴ When compared to visually normal controls, adults with amblyopia show a less consistent McGurk effect, indicating that they are less susceptible to the illusion, implying altered audiovisual integration.¹² 

Several studies have examined the developmental trajectory of McGurk effect susceptibility in visually normal children. There is a general trend toward less susceptibility to the illusion at younger ages, with the effect becoming more consistent as the age range of participants moves toward adulthood. In the original study in which the McGurk effect was introduced, preschool aged children (ages 3–4 years) showed a less consistent effect than school-aged children (ages 7–8 years), who in turn showed a less consistent effect than adults.¹³ In a more recent study, 4- to 5-year-old children were compared to adults on McGurk performance, and also showed a less consistent effect.¹⁵ However, differences in susceptibility seem to be more pronounced at younger ages and begin to plateau later in development. Visually normal children aged 8 to 16 years showed no relation between age and McGurk effect within that age group.¹⁶ In another study, visually normal children aged 5 to 9 years showed a less consistent McGurk effect when compared to children aged 10 to 14 and 15 to 19 years, but the two older age groups showed no difference from one another.¹⁷ Visually normal adolescents showed a more consistent McGurk effect when compared not only to younger children, but also to adults, implying that older children are less likely to show deficits in task performance.¹⁸ Given the pattern of McGurk susceptibility in these studies, differences in effect consistency may be limited to younger rather than older children. 

A recent study showed reduced McGurk effect in children with amblyopia, but it did not examine the variations in performance across age ranges, and it also did not include an adult group.¹⁹ To our knowledge, the developmental trajectory of susceptibility to the McGurk effect has not been investigated previously in amblyopia. In this study, we investigated whether impaired performance is manifest at an early age, or whether it is a consequence of altered visual development, such that children with amblyopia “fall behind” their typically developing peers later in childhood. We examined two separate age groups of pediatric participants on McGurk effect performance to look for differences between younger and older children, and compared their performance to that of our previously reported adult cohort.¹² 

Methods

Participants

All participants were fluent English speakers with no known auditory or neurological disorders, and no eye pathology other than amblyopia, strabismus, or ametropia. Participants were assessed by a certified orthoptist for visual acuity (Early Treatment of Diabetes Retinopathy Study [ETDRS] test), stereoacuity (Randot), and eye alignment (cover–uncover test and alternate prism cover test). Amblyopia was defined as a visual acuity of 0.18 logMAR (20/30) or worse in the amblyopic eye, as well as an intraocular difference (IOD) greater than or equal to 0.2 logMAR (2 chart lines difference). Anisometropia was defined as a difference of 1 diopter (D) in either the spheric or astigmatic correction between the two eyes. Strabismic amblyopia refers to amblyopia in the presence of a manifest deviation (heterotropia) on cover test. Mixed amblyopia refers to the presence of anisometropia and manifest strabismus. 

Participants were divided into three age groups: younger children (ages 4–9), older children (ages 10–17), and adults (ages 18+). The age groups were selected based on a previous study examining development of audiovisual illusions in children.¹⁷ Data from adult participants were obtained from a previous study by our group¹² and are included here for comparison purposes; no additional adult data have been added to the current study. We tested 62 participants with amblyopia: 28 younger children (12 female; mean age, 6.3 ± 1.3; age range, 4–9 years), 12 older children (3 female; mean age, 11.5 ± 2.0; age range, 10–16 years), and 22 adults (19 female; mean age, 32.8 ± 10.9 years; age range, 20–56 years). Clinical details for the participants with amblyopia are shown in Tables 1 to 3. We tested 66 visually normal controls: 24 younger children (11 female; mean age, 7.5 ± 1.4; age range, 5–9 years), 17 older children (10 female; mean age, 11.8 ± 2.0; age range, 10–16 years), and 25 adults (16 female; mean age, 31.8 ± 9.4 years; age range, 19–56 years). Control participants had normal or corrected-to-normal visual acuity of 0.1 logMAR (20/25) or better and stereoacuity of 40 seconds of arc or better. Informed consent was obtained from all adult participants and from parents/guardians of all child participants. Assent was obtained from all child participants. The study was approved by the Research Ethics Board at The Hospital for Sick Children. All protocols adhered to the guidelines of the Declaration of Helsinki. 

Table 1

 

Clinical Characteristics of Younger Children With Amblyopia

 

Table 2

 

Clinical Characteristics of Older Children With Amblyopia

 

Table 3

 

Clinical Characteristics of Adults With Amblyopia

 

Stimuli

Stimuli consisted of video and audio of two female speakers, using five syllables: “ba,” “da,” “ga,” “tha,” and “va.” Videos displayed the head and shoulders of the speakers, and were approximately 5 seconds long. Congruent trials in which the video and audio were matched served as control stimuli. Incongruent McGurk trial videos were made by using video of “da,” “ga,” “tha,” and “va,” combined with audio of “ba.” Further details of the video and audio stimuli have been described in our previous study.¹² 

Each control and McGurk trial from each speaker was repeated twice per experimental session, for a total of 36 trials ([5 control videos + 4 McGurk videos] × 2 speakers × 2 repetitions). Each trial consisted of a single presentation of a control or McGurk video, containing the three repeats of a single syllable. Control and McGurk trials were interleaved within a single block. Trial order was randomized within each experimental session. 

Apparatus and Procedure

Stimuli were viewed on a 15.6-inch laptop screen at a viewing distance of 60 cm. Further details of the apparatus are reported in our previous study.¹² Before beginning the experiment, participants were shown a sheet of paper displaying the five syllables and the phrase “something else” in large, high-contrast letters to familiarize them with the response options. Child participants were asked to speak the five syllables aloud to verify that they understood the sounds. In the case that a child was unable to pronounce a syllable, they were tested only on the syllables they were able to pronounce correctly. During the experiment, participants were instructed to maintain their gaze on the speaker in the video for the entire duration of the trial before responding. All child and adult participants were tested binocularly. After each video was displayed, a prompt was presented onscreen instructing the participant to make their response. Adult participants responded by pressing one of the six labeled keys on the laptop keyboard, corresponding to the five syllables or the “something else” option if they heard something that did not match any of the syllables listed. Child participants responded by pointing to one of the six options on the sheet of paper, or by making a verbal response in cases where the child was unable to read. The experimenter then made the key presses on the participant's behalf. No time limit was imposed for responses. Reaction time was not recorded due to the variation in response methods. After the participant made a response, the next trial was triggered. An optional break was offered to each participant halfway through the experiment. 

Analysis

Response accuracy was analyzed for congruent and incongruent trials. For analysis purposes, any non-“ba” responses to an incongruent trial were considered incorrect and grouped together. Response accuracy was the outcome measure that indicated the susceptibility to the McGurk illusion. Lower accuracy indicates a strong McGurk effect, implying that auditory input was influenced strongly by visual input, causing incorrect responses. Response accuracy is reported as mean and standard error in the text and Figures. Greenhouse-Geisser correction was used for any statistical test where the assumption of sphericity was violated. Post hoc tests were corrected for multiple comparisons using the Tukey-Kramer test. 

Response accuracy was analyzed using a 2 × 3 ANOVA, with Group (two levels: amblyopia and visually normal) and Age (three levels: younger children, older children, adults) as between-subjects factors. Congruent trials and incongruent trials were analyzed separately. Effects of visual acuity and stereoacuity on response accuracy in participants with amblyopia were analyzed with separate Spearman correlation analyses for incongruent trials. 

Speaker and syllable effects on response accuracy were analyzed separately, using repeated measures ANOVAs, with Group and Age as between-subjects factors, and Speaker (Speaker 1, Speaker 2) or Syllable (da, ga, tha, va) as within-subjects factors. Four participants performed the task with a reduced syllable set because they were unable to pronounce all five syllables correctly, and were excluded from syllable analysis. 

Results

Congruent Trials

There was a significant main effect of Age (F_[2,122] = 27.5, P < 0.0001), with younger children (90.5% ± 1.1%) showing lower accuracy than older children (96.6% ± 0.9%), and with both groups showing lower accuracy than adults (98.9% ± 0.4%). There was no significant effect of Group (F_[1,122] = 0.8, P = 0.37), and no interaction between Group and Age (F_[2,122] = 2.6, P = 0.08). No further analysis was performed on congruent trials, as all groups performed at a level close to ceiling. Further details of congruent trial performance are shown in Table 4. 

Table 4

 

Accuracy by Group and Age (Mean ± SE)

 

Incongruent Trials

There were significant main effects of both Group (F_[1,122] = 9.61, P = 0.0024) and Age (F_[2,122] = 8.56, P = 0.0003), with no interaction between Group and Age (F_[2,122] = 0.18, P = 0.83). Participants with amblyopia (28.0% ± 3.3%) showed a less consistent McGurk effect (i.e., higher response accuracy) than visually normal controls (15.2% ± 2.3%). Younger children (30.2% ± 3.8%) showed a less consistent McGurk effect than older children (21.1% ± 3.6%), who in turn showed a less consistent McGurk effect than adults (11.7% ± 2.2%). McGurk effect susceptibility increased with age for amblyopic participants and controls (Fig. 1). For amblyopic participants, mean accuracy was 36.6% ± 6.0% for younger children, 29.2% ± 5.2% for older children, and 16.5% ± 3.4% for adults. For control participants, mean accuracy was 22.9% ± 4.2% for younger children, 15.4% ± 4.6% for older children, and 7.5% ± 2.7% for adults. Further details of incongruent trial performance can be found in Table 4. 

Figure 1

 

Accuracy of incongruent trials by group and age group. Lower accuracy indicates a more consistent McGurk effect. Normal and amblyopic groups display a similar developmental trajectory of effect susceptibility. Error bars: represent ±1 SE.

No significant correlations were found between accuracy and visual acuity (r_[60] = −0.08, P = 0.54), or between accuracy and stereoacuity (r_[60] = −0.14, P = 0.27) in participants with amblyopia. 

There was a significant main effect of Speaker (F_[1,122] = 13.74, P = 0.0003), but no interaction between either Speaker and Group (F_[1,122] = 1.63, P = 0.20) or Speaker and Age (F_[2,122]) = 1.25, P = 0.29). Speaker 1 (24.4% ± 2.3%) elicited a less consistent McGurk effect than Speaker 2 (18.4% ± 2.2%), with amblyopic participants exhibiting a less consistent McGurk effect than controls for both Speaker 1 and 2 (Fig. 2). Further details of speaker results can be found in Table 5. 

Figure 2

 

Accuracy of incongruent trials by group and speaker, across all age groups. Lower accuracy indicates a more consistent McGurk effect. Normal and amblyopic groups show a more consistent effect with Speaker 2. Error bars: represent ±1 SE.

Table 5

 

Accuracy of Incongruent Trials for Different Syllables and Speakers, by Group and Age (Mean ± SE)

 

There was a significant main effect of Syllable (F_[2.4,286.5] = 52.06, P < 0.0001). The syllable “tha” (7.5% ± 1.6%) elicited the most consistent McGurk response, followed by “va” (14.9% ± 2.2%), “da” (21.6% ± 2.5%), and “ga” (36.3% ± 3.2%), with amblyopic participants exhibiting a less consistent McGurk effect than controls for all four syllables (Fig. 3). No interaction was found between Syllable and Group (F_[2.4,286.5] = 1.54, P = 0.21); however, there was an interaction between Syllable and Age (F_[4.9,286.5] = 2.32, P = 0.045). Younger children showed a significant difference from adults for syllables “ga” (P < 0.0001), “tha” (P = 0.037), and “va” (P < 0.0001). Older children also showed a significant difference from adults for syllable “ga” only (P = 0.009). All age groups showed a significant difference between syllables “ga” and “tha” (P < 0.0001). In all cases, the effect consistency for specific syllables increased with age. Further details of the syllable results are shown in Table 5. 

Figure 3

 

Accuracy of incongruent trials by group and syllable, across all age groups. Lower accuracy indicates a more consistent McGurk effect. Normal and amblyopic groups show the most consistent effect with syllable “tha,” and least consistent effect with syllable “ga,” with a similar overall response pattern. Error bars: represent ±1 SE.

Discussion

The main findings of this study are that children and adults with amblyopia are less susceptible to the McGurk illusion than visually normal controls, and that their performance is dependent on age. While the effect susceptibility is diminished for all three age groups with amblyopia, the general pattern of performance is consistent with the developmental trend in visually normal participants; namely, an increase in effect susceptibility with increasing age. We also found that while different speakers and syllables elicit different response consistencies, the pattern is analogous between participants with amblyopia and controls. This finding has been demonstrated previously among adult participants,¹² and we now extend a similar finding to a pediatric population. 

What could account for our findings? A few possibilities exist, including unimodal (i.e., vision) and bimodal (i.e., vision and audition as a result of impaired integration versus altered integration from sensory reweighting) explanations. One possibility is that the observed findings are the result of decreased visual acuity or other visual deficits associated with amblyopia. Because we have found previously that adults with amblyopia exhibit reduced susceptibility to the McGurk illusion not only during monocular viewing with the amblyopic eye, but also during monocular viewing with the fellow eye as well as during binocular viewing,¹² a deficit in visual acuity alone could not explain our current results. However, as amblyopia is associated with many visual deficits beyond acuity,^5–11 a unimodal explanation cannot be ruled out. Because the McGurk task uses complex stimuli that not only involve visual and auditory input, but also requires higher level processing (e.g., lip-reading, cognition), another possibility is impaired lip-reading secondary to amblyopic visual deficits. Indeed, patients with bilateral deprivation amblyopia from bilateral congenital cataracts have deficits in lip-reading as well as reduced susceptibility to the McGurk illusion.^20,21 However, when compared to controls with a similar level of lip-reading deficits from other causes, participants with bilateral congenital cataracts demonstrated a further reduced susceptibility to the McGurk illusion, suggesting that lip-reading deficits alone may not fully account for differences between controls and participants with amblyopia. It is not known whether people with unilateral amblyopia have lip-reading deficits, but if such deficits exist, they could be a contributing factor to reduced McGurk effect susceptibility. 

A bimodal explanation based on multisensory interaction is another strong possibility to explain the present findings. Perhaps the simplest bimodal explanation is a failure of integration between auditory and visual signals during early childhood in people with amblyopia. During normal development, repeated exposure to coincident visual-auditory stimuli (e.g., seeing and hearing a sound-making toy) results in the creation of strong relations between the two sensory inputs, allowing for optimal localization of a target of interest. This developmental course of audiovisual integration is particularly relevant in the context of amblyopia, a neurodevelopmental disorder of spatiotemporal vision. During early childhood, people with amblyopia likely develop atypical audiovisual integration based upon abnormal visual afferent information as a result of increased signal variability,^22–26 spatial undersampling,^27,28 and spatial distortions.^29,30 In addition, the deficient audiovisual integration could also be a consequence of temporal differences between the impaired visual system and the presumed normal auditory system during early development. In visually normal people, there is effective integration based on a typical temporal difference between visual and auditory signals reaching their respective primary cortices (on the order of approximately 90 ms to peak response in the primary visual cortex and approximately 30–50 ms to peak response in the primary auditory cortex^31,32). Visual signal processing delays have been shown previously in amblyopia even for very simple visual tasks (e.g., saccade latency is longer, on the order of approximately 30–60 ms, in people with amblyopia^24,25,33). This delay in visual processing in amblyopia may widen the temporal difference between visual and auditory signals reaching their respective processing areas, such that audiovisual integration is altered. 

Alternatively, multisensory integration in amblyopia may be intact, but the sensory modalities may be reweighted to account for deficits in the visual system. Optimization theory proposes that the relative importance of a sensory modality is dependent on the variance of the signal: a more reliable sensory signal is more heavily weighted perceptually.³⁴ In visually normal individuals, vision is usually the dominant sense.³⁵ Because the amblyopic visual system is subject to a high level of internal noise,³⁶ it is reasonable to speculate that visual noise increases the variance of the corresponding visual signal, leading to sensory reweighting with a greater weight given to audition relative to vision. This results in audiovisual integration that is altered (but not impaired per se) to compensate for the limitations of the amblyopic visual system. 

Regardless of the specific mechanism of reduced McGurk effect susceptibility, the results presented here indicate that the underlying changes take place during early development, and represent a permanent deficit rather than a developmental delay because they persist into adulthood.¹² This raises the question of defining a critical period for abnormal visual experience beyond which changes to audiovisual integration become permanent. A recently published study by Burgmeier et al.¹⁹ examined McGurk performance in 24 amblyopic children aged 3 to 9, and reported that the McGurk effect was perceived in only 18.8% of children with amblyopia that remained unresolved by 5 years of age (n = 16). In contrast, they¹⁹ found that the McGurk effect was perceived by 100% of participants whose amblyopia developed at or after 5 years of age (n = 5) or whose amblyopia was resolved by 5 years of age (n = 3). Although their results are limited by the small sample size in their subgroup analysis, and are potentially confounded by the heterogeneity of their sample of late-onset amblyopia, they provided valuable information about the role of age of onset and resolution of amblyopia on the McGurk task. The implications of their findings are that audiovisual integration may not be affected by abnormal visual experience that occurs later in life beyond the critical period, and it can potentially recover if early and effective treatment is initiated before the critical period ends. 

Many questions remain unanswered in our understanding of higher order multisensory integration in amblyopia. What additional conclusions can be drawn regarding the effects of age of onset and resolution of amblyopia on audiovisual integration? Given the frequent disparity between age of onset and age at diagnosis of amblyopia, ascertaining the age of onset could be difficult. Age of clinical resolution, on the other hand, is more straightforward to assess, and raises an interesting question: does audiovisual perception remain affected in individuals despite the recovery of high contrast letter acuity in patients who have been successfully treated? This is relevant because deficits in higher order perceptual tasks have been shown to remain even after successful therapeutic intervention for amblyopia.¹¹ What about unresolved amblyopia? Could the reduced susceptibility to the McGurk effect be explained by visual impairment alone, or is it caused by a failure of audiovisual integration? Is it simply a consequence of altered multisensory processing through sensory reweighting, with a less reliable visual signal leading to greater auditory influence? Studies are presently underway in our lab to examine specifically whether the reduced McGurk effect we observed in amblyopia is the result of defective audiovisual integration versus sensory reweighting in the presence of normal audiovisual integration. 

Acknowledgments

Supported by Grant MOP 106663 from the Canadian Institutes of Health Research (CIHR), Leaders Opportunity Fund from the Canada Foundation for Innovation (CFI), the John and Melinda Thompson Endowment Fund in Vision Neurosciences, and the Department of Ophthalmology and Vision Sciences at The Hospital for Sick Children. 

Disclosure: C. Narinesingh, None; H.C. Goltz, None; R.A. Raashid, None; A.M.F. Wong, None