Abstract
Purpose:
Amblyopia is a neurodevelopmental visual disorder caused by abnormal visual experience in childhood. In addition to known visual deficits, there is evidence for changes in audiovisual integration in amblyopia using explicit tasks. We examined audiovisual integration in amblyopia using an implicit task that is more relevant in a real-world context.
Methods:
A total of 11 participants with amblyopia and 16 controls were tested binocularly and monocularly on the sound-induced flash illusion, in which flashes and beeps are presented concurrently and the perceived number of flashes is influenced by the number of beeps. The task used 1 to 2 rapid peripheral flashes presented with 0 to 2 beeps, at 5 stimulus onset asynchronies, that is, beep (−200 milliseconds, −100 milliseconds) or flash leading (100 milliseconds, 200 milliseconds) or simultaneous (0 milliseconds). Participants reported the number of perceived flashes. Susceptibility was indicated by a “2 flashes” response to “fission” (1 flash, 2 beeps) or “1 flash” to “fusion” (2 flashes, 1 beep).
Results:
For fission with the beep leading during binocular viewing, controls showed an expected decrease in illusion strength as stimulus onset asynchronies increased, whereas the illusion strength remained constant in participants with amblyopia, indicating a wider temporal binding window in amblyopia (P = 0.007). For fusion, participants with amblyopia showed reduced illusion strength during amblyopic eye viewing (P = 0.044) with the flash leading.
Conclusions:
Amblyopia is associated with the widening of the temporal binding window, specifically for fission when viewing binocularly with the beep leading. This suggests a developmental adaptation to delayed amblyopic eye visual processing to optimize audiovisual integration.
Amblyopia is a neurodevelopmental disorder associated with reduced vision, usually in one eye. It is caused by abnormal visual experience in early childhood, and the resulting deficits cannot be fully explained by any ocular structural causes, nor can they be corrected with lenses. Subtypes of unilateral amblyopia include anisometropic (difference in refractive error between the eyes), strabismic (misaligned eyes), and deprivation (e.g., congenital cataract). Amblyopia is associated with a wide array of visual deficits, including reduced visual acuity and stereoacuity,
1,2 reduced contrast sensitivity,
1,2 and difficulties with real-world visual tasks such as reading
3,4 and scene perception.
5 Recent evidence suggests that sensory processing deficits in amblyopia may extend beyond the visual domain, with multisensory integration also being affected.
6–10
Multisensory integration refers to the processing of inputs from different sensory modalities to form a unified percept—a necessity for the accurate processing of complex stimuli in everyday life. The contribution of each modality to the integrated percept varies based on the reliability of the modality and attentional factors.
11,12 In a typically developing system, multisensory integration is thought to be calibrated by the accuracy of unisensory input from real-world sensory experience during childhood, with the unisensory contributions being statistically optimized during adulthood based on their variability.
11,13,14 The resulting integrative behavior is thus dependent on normal sensory development, but sensory development may not be normal in amblyopia. An important aspect of the integrative process is the temporal binding window—the window of time within which separate sensory inputs are merged into a single perceptual event. This window varies in width and symmetry, based on the type of task, type of stimuli, and individual variation.
15,16 It can be quantified by varying the stimulus onset asynchrony (SOA) between two stimuli and examining the perceptual responses. Ideally, the temporal binding window should be wide enough to include all multisensory stimuli that are contextually related and narrow enough to exclude stimuli unrelated to the relevant sensory event. Much like multisensory integration in general, the characteristics of the audiovisual temporal binding window develop during childhood. In typically developing children, the window is wider than that of adults, with narrowing of the window beginning as early as ages 4 to 6
17 and becoming adult-like by age 9.
18
Altered multisensory integration has been demonstrated in amblyopia, specifically for audiovisual integration. People with amblyopia show less susceptibility to the McGurk illusion,
19 a task in which videos of a face speaking a syllable are paired with the audio of a different syllable, and participants are asked to report what they hear.
6,8,9 The visual syllable influences the percept of the auditory syllable. The reduced illusion susceptibility in amblyopia is present even when viewing binocularly or with the fellow eye. Amblyopia is also associated with an altered temporal binding window of audiovisual processing as measured by a simultaneity judgment task, in which a sound and a light are presented at differing SOAs and participants report whether they perceive the stimuli as simultaneous.
20 People with amblyopia have a wider temporal binding window on this task during both binocular and monocular viewing, reporting simultaneity over a larger range of SOAs than visually normal controls.
7,10 Although the McGurk and simultaneity judgment tasks demonstrate differences in audiovisual processing in amblyopia when compared with controls, there are some limitations. The McGurk illusion is a speech processing task involving complex language mechanisms that are difficult to separate from simpler integrative mechanisms. Although a simultaneity judgment is a lower level task, it is also an explicit task by asking, “Do these two stimuli happen at the same time?” However, real-world sensory processing is implicit—humans typically do not make conscious judgments on whether stimuli are simultaneous during everyday behavior but, rather, decide unconsciously whether they are part of the same event. Integrating stimuli requires spatial and temporal propinquity, but not exact conscious simultaneity. Thus, an implicit task is more relevant when investigating audiovisual processing in a real-world context.
A more suitable implicit task that also examines simpler audiovisual integrative mechanisms in a laboratory setting is the sound-induced flash illusion.
21–23 This task pairs a number of rapid flashes with a number of concurrent beeps, with participants reporting the number of flashes they perceive. The percept of the flashes is influenced by the number of beeps. Pairing a single flash with a double beep produces a fission illusion, in which the single flash is perceived as a double flash. Likewise, pairing a double flash with a single beep results in a fusion illusion, with the double flash fused into a percept of a single flash. This task is an implicit task in that the percept of the flashes is based on the neural integration of the stimuli and not on an explicit judgment of their characteristics. Altering the SOAs between the flashes and beeps can quantify the temporal binding window over which the illusion is effective. In visually normal people, the illusion begins to decline in strength at an SOA of approximately 70 milliseconds, but still remains present for a temporal binding window extending beyond ±100 milliseconds and terminating by ±300 milliseconds.
22–25
A preliminary study has examined the sound-induced flash illusion in deprivation amblyopia caused by congenital cataracts.
7 The susceptibility to the fission illusion was reduced in participants with bilateral deprivation, but those with unilateral deprivation showed no susceptibility differences from visually normal controls. Similarly, patients who had monocular enucleation early in life show normal susceptibility to the fission illusion.
26 Previous studies investigating the flash illusion in both deprived and enucleated populations, however, did not report the temporal characteristics of the illusion.
7,26 To date, no studies have examined the fission illusion in anisometropic, strabismic, or mixed-mechanism amblyopia, nor has the fusion illusion or temporal binding window of the illusion been investigated for any type of amblyopia. The current study examines susceptibility to the sound-induced flash illusion in anisometropic, strabismic, and mixed-mechanism amblyopia over a range of SOAs to further investigate the differences in audiovisual integration in amblyopia.
A total of 11 participants with amblyopia (9 women; mean age 29.4 ± 10.5 years; age range 19–48 years) and 16 visually normal controls (10 women; mean age 26.3 ± 7.8 years; age range 18–43 years) were tested. The participants were assessed by a certified orthoptist for visual acuity (Early Treatment Diabetic Retinopathy Study test), stereoacuity (Randot), and eye alignment (cover–uncover test and alternate prism cover test). Amblyopia was defined as a visual acuity of 0.18 logMAR (20/30) or worse in the amblyopic eye as well as an intraocular difference greater than or equal to 0.2 logMAR (two chart lines difference). Anisometropia was defined as a difference of 1 diopter in either the spheric or astigmatic correction between the two eyes. Strabismic amblyopia was defined as amblyopia in the presence of a manifest deviation (heterotropia) on cover test. Mixed-mechanism amblyopia was defined as the presence of both anisometropia and manifest strabismus. Visually normal controls were required to have a normal or corrected-to-normal visual acuity of 0.1 logMAR (20/25) or better and stereoacuity of 40 seconds of arc or better. All participants passed a basic hearing screening using an audiometer (Maico Diagnostics, Eden Prairie, MN, USA) and had no known auditory or neurologic disorders, learning disabilities, developmental delays, or any eye pathology other than amblyopia, strabismus, or ametropia. The clinical characteristics of the participants with amblyopia are presented in the
Table. All participants gave informed consent. The study was approved by the Research Ethics Board of The Hospital for Sick Children, Toronto, Ontario, Canada, and conformed to the guidelines of the Declaration of Helsinki.
Testing was performed in near darkness in a sound-attenuating booth. Participants were seated with the head positioned in a chin rest. The visual stimuli were presented on a 65” light-emitting diode display (NEC E654; NEC, Itasca, IL, USA) with a refresh rate of 60 Hz at a viewing distance of 65 cm. The auditory stimuli were presented from two speakers (HP Compact 2.0 Speakers; HP, Inc., Palo Alto, CA, USA) positioned adjacent to the monitor at eye level and rotated inward by 50°. Participant responses were made using a wireless keyboard (Logitech K520; Logitech International S.A., Romanel-sur-Morges, Switzerland) with photoluminescent stickers indicating the relevant keys.
Participants performed the task under three viewing conditions: binocular, dominant or fellow eye, and nondominant or amblyopic eye. Viewing condition order was randomized across participants and counterbalanced across groups. The nonviewing eye was occluded with an eye patch for monocular viewing conditions. The visual stimuli were presented in white on a black background. The flash stimulus was a circle with radius of 2° of visual angle, presented randomly above or below the fixation cross at a vertical distance of 5° of visual angle between the cross and the center of the circle. Flashes were presented for one frame (∼17 milliseconds) with a separation of three frames (∼50 milliseconds) between double flashes. The beep stimulus was a pure sine wave tone of 3500 Hz, without ramps or windowing, presented at 80 dBA. Beeps were presented for 7 milliseconds, with a separation of 60 milliseconds between double beeps.
Each trial consisted of a fixation cross followed by one of the following combinations of flashes and beeps: baseline (1–2 flashes, no beeps), congruent (1 flash, 1 beep or 2 flashes, 2 beeps), fission (1 flash, 2 beeps), or fusion (2 flashes, 1 beep). Trial order was randomized for each participant, who performed a total of 240 trials for each viewing condition. Participants were instructed to fixate on the central cross and indicate the number of flashes they perceived. Responses were made with a button press on the keyboard. There was no time limit for responses, and the participants were told to respond at their own pace. During each 240 trial session, a break was initiated by the software every 60 trials. Participants determined the length of the break, returning to the experiment when they felt ready. Lights were turned on inside the booth during breaks and between sessions to minimize dark adaptation.
The magnitude of the fission illusion strength was defined as the difference between the accuracy of fission trials and the accuracy of single-flash baseline trials. The magnitude of the fusion illusion strength was defined as the difference between the accuracy of fusion trials and the accuracy of double-flash baseline trials. Data were sorted by SOA into beep leading (−200 milliseconds, −100 milliseconds, 0 milliseconds) and flash leading (0 milliseconds, 100 milliseconds, 200 milliseconds) sets, with each set analyzed separately. Fission and fusion effects were analyzed using separate repeated measured mixed ANOVAs, with group (visually normal, amblyopia) as a between-subjects factor and SOA (−200 milliseconds, −100 milliseconds, 0 milliseconds or 0 milliseconds, 100 milliseconds, 200 milliseconds) as a within-subjects factor. Viewing conditions were analyzed separately; they were not included as an ANOVA factor because they were not considered to be independent from one another, violating the assumption of statistical independence of the ANOVA method. Bonferroni correction was applied for post hoc analyses. Greenhouse-Geisser correction was used for all statistical tests in which the assumption of sphericity was violated according to Mauchly's sphericity test.
The congruent trials primarily acted as catch trials, such that the participants were exposed to both illusory and nonillusory trials during the experiment. Congruent trials were analyzed similarly to fission and fusion trials, and no group differences were found.
With the beep leading (
Fig. 1), there was no significant difference in illusion strength between participants with amblyopia and visually normal controls in all three viewing conditions (binocular:
F1,25 = 0.21,
P = 0.65; dominant/fellow eye:
F1,25 = 0.14,
P = 0.71; and nondominant/amblyopic eye:
F1,25 < 0.1,
P = 0.98). As expected, there were main effects of SOA with decreasing illusion strength as SOA increased for all viewing conditions (binocular:
F2,50 = 15.9,
P < 0.001; dominant/fellow eye:
F2,50 = 24.7,
P < 0.001; and nondominant/amblyopic eye:
F2,50 = 12.7,
P < 0.001). Importantly, there was an interaction of group and SOA for the binocular viewing condition (
F2,50 = 5.49,
P = 0.007). Post hoc testing revealed that the illusion strengths at SOAs of 200 milliseconds and 100 milliseconds were significantly different from that at SOA of 0 milliseconds for the visually normal group (
P < 0.001), but not for the amblyopia group, indicating a widening of the temporal binding window in amblyopia during binocular viewing only. There were no interactions of group and SOA for the dominant/fellow eye (
F2,50 = 1.48,
P = 0.24) or nondominant/amblyopic eye (
F2,50 = 1.97,
P = 0.15) viewing conditions.
Similarly, with the flash leading (
Fig. 2), no significant difference in illusion strength was found between participants with amblyopia and visually normal controls in all three viewing conditions (binocular:
F1,25 < 0.1,
P = 0.99; dominant/fellow eye:
F1,25 < 0.1,
P = 0.79; and nondominant/amblyopic eye:
F1,25 < 0.1,
P = 0.80). There were also main effects of SOA for all viewing conditions (binocular:
F2,50 = 8.09,
P < 0.001; dominant/fellow eye:
F2,50 = 20.1,
P < 0.001; nondominant/amblyopic eye:
F1.59,39.7 = 10.5,
P < 0.001). In contrast to beep leading, there were no interactions of group and SOA for any of the three viewing conditions (binocular:
F2,50 = 1.43,
P = 0.25; dominant/fellow eye:
F2,50 = 0.53,
P = 0.59; and nondominant/amblyopic eye:
F1.59,39.7 = 0.30,
P = 0.74).
With the beep leading (
Fig. 3), there were no main effects of group (binocular:
F1,25 < 0.1,
P = 0.90; dominant/fellow eye:
F1,25 = 0.88,
P = 0.36; and nondominant/amblyopic eye:
F1,25 = 2.69,
P = 0.11) or SOA (binocular:
F2,50 < 0.1,
P = 0.92; dominant/fellow eye:
F2,50 = 1.39,
P = 0.26; and nondominant/amblyopic eye:
F2,50 = 0.72,
P = 0.49), and no interactions between the two for all viewing conditions (binocular:
F2,50 = 0.18,
P = 0.83; dominant/fellow eye:
F2,50 = 1.21,
P = 0.31; and nondominant/amblyopic eye:
F2,50 = 0.17,
P = 0.85).
With the flash leading (
Fig. 4), there was a significant difference in illusion strength between participants with amblyopia and visually normal controls during nondominant/amblyopic eye viewing (
F1,25 = 4.48,
P = 0.044), but not during binocular (
F1,25 < 0.1,
P = 0.98) or dominant/fellow eye (
F1,25 = 0.33,
P = 0.57) viewing. In contrast to beep leading, there were main effects of SOA for all viewing conditions (binocular:
F2,50 = 23.9,
P < 0.001; dominant/fellow eye:
F1.58,39.6 = 15.3,
P < 0.001; nondominant/amblyopic eye:
F2,50 = 18.3,
P < 0.001). There were no interactions of group and SOA for any of the three viewing conditions (binocular:
F2,50 =1.02,
P = 0.37; dominant/fellow eye:
F1.58,39.6 < 0.1,
P = 0.99; and nondominant/amblyopic eye:
F2,50 = 1.01,
P = 0.37).
Spearman correlation analyses were done for visual acuity and stereoacuity for both the fission and fusion illusions at all SOAs and viewing conditions. The only significant correlation for visual acuity was in the fusion condition with flash leading, when viewing with the fellow eye at an SOA of 200 milliseconds (r = −0.63, P = 0.037). No significant correlations for visual acuity were found for any other combination of illusion type (fission or fusion), SOA, or viewing condition. Similarly, no significant correlations for stereoacuity were found for any other combination of illusion type, SOA, or viewing condition.
The main finding of this study is the difference in fission illusion response between participants with amblyopia and visually normal controls during binocular viewing. In visually normal controls, the fission illusion decreases in strength as SOA increases, consistent with previous studies.
22–25 In amblyopia, however, with the beep leading, the strength of the fission illusion remained constant as SOA increased, indicating that participants with amblyopia remain susceptible to the fission illusion at larger SOAs to a greater extent than visually normal controls, but only when viewing binocularly. The specificity of this result suggests a change in the amblyopic audiovisual system shaped by the disorder—namely, the processing differences between the two eyes. It is possible that audiovisual processing in amblyopia is calibrated to accommodate a lagging visual signal from the amblyopic eye when making the decision as to which multisensory inputs should be integrated into a single perceptual event. Temporal delays are known to be associated with the amblyopic eye; for example, saccadic latencies are longer, on the order of 20 to 60 milliseconds.
27–29 Optimizing audiovisual processing would require taking these delays into account.
The temporal window of integration model offers a potential framework for explaining how delayed input might affect multisensory processing.
30 The model proposes a two-stage process for multisensory integration. The first stage is a unisensory competition, in which the temporal binding window is opened by the first stimulus to complete processing. The window remains open to other stimuli in a way that only stimuli that finish processing within the window will be integrated in the second stage. Stimuli that are weaker, with the corresponding slower processing times, are not integrated by design. In this model, if a processing delay is caused by an impaired amblyopic channel rather than the characteristics of the stimulus itself, the result may be suboptimal, with inputs from the same event not being perceptually integrated. A plausible solution is to widen the audiovisual temporal binding window to compensate for the delayed amblyopic eye input. The width of the temporal binding window is known to change during childhood, becoming narrower and thus more precise in typically developing children as they accumulate real-world visual experience.
17,18 The refinement of the window would be expected to correspond to the visual input received, whether typical or abnormal, such that integration is optimized given the available inputs. In the case of amblyopia, during development, a latency difference between the two eyes would guide calibration such that the temporal binding window is widened for binocular visual signals. This widening would be directional, allowing extra time for a visual signal to lag an auditory signal, whereas the reverse case is unnecessary as no auditory delay would be expected in amblyopia. This explanation is consistent with our results showing a widening of the temporal binding window for the fission illusion only when the auditory signal led the visual signal, and only when both eyes were viewing.
The fusion illusion, in contrast, does not show any differences in the width of the temporal binding window between participants with amblyopia and visually normal controls. It is well established that temporal binding windows for an individual can vary in width and symmetry depending on the task.
15,16 At first glance, the fusion and fission illusions appear similar, but both behavioral and imaging data have revealed them to be different types of task.
21,31,32 The main difference is the processing time involved. The fission illusion is processed more rapidly, with an early primary visual cortex response to the combination of flash and sound, followed by a possible feedforward mechanism to higher level multisensory areas such as the superior temporal cortex.
32 The fusion illusion, on the other hand, has a slower processing time with the implication of a feedback mechanism from the superior temporal cortex.
31 The increase in processing time for the fusion illusion may lead to an innately wider temporal binding window and hence negate the need of the visual system to compensate for a delay in signal input from the amblyopic eye.
Our results show one group difference in the fusion illusion—when viewing with the amblyopic eye, participants with amblyopia show less susceptibility to the illusion than visually normal controls with flash leading and trend toward the same with beep leading (P = 0.11). This result may have less to do with illusion susceptibility than with the constraints of the double flash stimulus. Fusion illusion strength was measured based on the difference in accuracy between the fusion condition (two flashes, one beep) and the baseline condition (two flashes, no beeps). The ability to distinguish between two rapid flashes in the baseline condition was extremely variable between individual participants. This was the case both in visually normal controls and in participants with amblyopia and for all viewing conditions, but was most problematic during monocular amblyopic eye viewing. In severe amblyopia, vision could be degraded enough that perceiving the double flash was not possible for the participant. This situation would produce a ceiling effect for that individual's baseline measurement, artificially reducing the magnitude of any fusion effects that might have taken place. Thus, no definite conclusions can be made regarding fusion illusion susceptibility in the amblyopic eye using the current stimuli, and further study will be required.
A curious observation of the fusion illusion that is found in both participants with amblyopia and visually normal controls is the dramatic decrease in the illusion strength at larger SOAs with flash leading, such that the mean illusion strength dips below baseline. There is a potential explanation for this “reverse fusion” effect. A temporal ventriloquism study used sounds to influence the perception of a visual temporal order judgment task, a task in which two stimuli are presented closely in time and the participant judges which one happened first.
33 This variation of the task presented two lights split by various SOAs, with an additional sound preceding or trailing the presentation of the lights. The study
33 showed that presenting the sound 100 milliseconds after the second light pulled the temporal percept of the two lights farther away from one another, increasing accuracy on the temporal order judgment task. In the current study, the visual lead condition of the fusion illusion is similar—two visual stimuli close in time, trailed by an auditory stimulus. The expectation is that the auditory stimulus should pull apart the two flashes, leading to an increase in accuracy and thus a reduction in illusion susceptibility. The “reverse fusion” effect shown here is consistent with the results of the temporal ventriloquism study.
33
The following larger question remains: do the results of this study demonstrate a deficit in audiovisual integration in amblyopia or a beneficial adaptation? An argument can be made that the widening of the temporal binding window may be beneficial in this case. At a SOA of 0 milliseconds, participants with amblyopia exhibit similar illusion susceptibility to visually normal controls, both for the fission and fusion illusions. The SOA 0-millisecond condition roughly approximates simple, close-range, real-world audiovisual stimuli, where the visual and auditory events happen at approximately the same time. The normal illusion susceptibility in this condition indicates that whatever temporal modifications have been made to the amblyopic system result in normal audiovisual integrative behavior. The specificity of the widened temporal binding window—both in terms of viewing conditions and fission versus fusion—offers an additional piece of evidence that the change may be beneficial. A widening of the window happens only when compensation for a monocular visual delay would be necessary—viewing binocularly with the auditory stimulus leading the visual stimulus. It also happens only for the fission illusion, a rapidly processed illusion where small changes in latency are likely to affect integration, and not for the more slowly processed fusion illusion. Answering the “beneficial versus detrimental” question is necessary to analyze the potential clinical implications. It is possible to narrow an individual's temporal binding window on a specific task through training, for example, using a perceptual learning paradigm.
34 However, if the widening of the window in amblyopia is a beneficial adaptation as suggested in this study, narrowing it may actually worsen integration. At this time, there is no evidence to recommend changing the width of the temporal binding window as a treatment for amblyopia.
Supported by Grant MOP 106663 from the Canadian Institutes of Health Research, Leaders Opportunity Fund from the Canada Foundation for Innovation, The Brandan's Eye Research Foundation, the John and Melinda Thompson Endowment Fund in Vision Neurosciences, and the Department of Ophthalmology and Vision Sciences at The Hospital for Sick Children.
Disclosure: C. Narinesingh, None; H.C. Goltz, None; A.M.F. Wong, None