Optimal Audiovisual Integration in the Ventriloquism Effect But Pervasive Deficits in Unisensory Spatial Localization in Amblyopia

Abstract

Purpose: Classically understood as a deficit in spatial vision, amblyopia is increasingly recognized to also impair audiovisual multisensory processing. Studies to date, however, have not determined whether the audiovisual abnormalities reflect a failure of multisensory integration, or an optimal strategy in the face of unisensory impairment. We use the ventriloquism effect and the maximum-likelihood estimation (MLE) model of optimal integration to investigate integration of audiovisual spatial information in amblyopia.

Methods: Participants with unilateral amblyopia (n = 14; mean age 28.8 years; 7 anisometropic, 3 strabismic, 4 mixed mechanism) and visually normal controls (n = 16, mean age 29.2 years) localized brief unimodal auditory, unimodal visual, and bimodal (audiovisual) stimuli during binocular viewing using a location discrimination task. A subset of bimodal trials involved the ventriloquism effect, an illusion in which auditory and visual stimuli originating from different locations are perceived as originating from a single location. Localization precision and bias were determined by psychometric curve fitting, and the observed parameters were compared with predictions from the MLE model.

Results: Spatial localization precision was significantly reduced in the amblyopia group compared with the control group for unimodal visual, unimodal auditory, and bimodal stimuli. Analyses of localization precision and bias for bimodal stimuli showed no significant deviations from the MLE model in either the amblyopia group or the control group.

Conclusions: Despite pervasive deficits in localization precision for visual, auditory, and audiovisual stimuli, audiovisual integration remains intact and optimal in unilateral amblyopia.

Amblyopia is a unilateral, or rarely bilateral, developmental visual disorder that affects 2% to 4% of the population.¹ Its associated visual impairment cannot be directly attributed to a structural eye abnormality, but arises from amblyogenic factors that interfere with normal visual experience during a sensitive period in early life. The most common amblyogenic factors are anisometropia (difference in refractive error between the eyes), strabismus (eye misalignment), or a combination of both.² More rarely, amblyopia may also be caused by visual deprivation from early-onset cataract or ptosis. In addition to the reduction in visual acuity by which it is clinically defined,³ amblyopia is associated with a range of visual deficits including impairments in contrast sensitivity, stereo acuity, recognition of ‘crowded' optotypes, global motion detection, and real-world scene perception.^4–8 The disorder also involves spatial distortions and positional uncertainty that affect vision in both the amblyopic eye and the fellow eye.^9–13 

Emerging research suggests that perceptual abnormalities in amblyopia are not limited to vision, but also involve multisensory integration in the audiovisual realm.^14–16 Multisensory integration refers to the process of unifying overlapping information from different senses to form an accurate and coherent perception of the external world.¹⁷ This ability is not of trivial consequence, but confers numerous adaptive advantages in terms of reaction times,¹⁸ detection sensitivity,¹⁹ discrimination reliability,²⁰ and speech comprehension.^21–24 A well-known paradigm for studying multisensory integration is the McGurk effect—a robust audiovisual illusion in which incongruity between the linguistic content of an auditory speech signal and the accompanying visual speech signal alters the perceived auditory signal.²⁵ For example, if audio of the syllable /ba/ is paired with video of the syllable /ga/, the perceived auditory percept is typically an illusory /da/. This phenomenon generalizes to many syllabic combinations, with the resulting auditory percept being either a fused syllable intermediate to the veridical auditory and visual cues, or dominated by the visual cue.^26,27 Computational modeling within a Bayesian framework suggests that the illusory McGurk percept may represent a statistically optimal strategy to integrate visual and auditory speech cues.²⁸ 

Several recent studies report that susceptibility to the McGurk effect is reduced in people with unilateral amblyopia, even when viewing with both eyes.^14–16 Although this finding may indicate a failure to optimally integrate the available unimodal (i.e., visual and auditory) information, it may alternatively result from deficiencies at the unisensory level propagated through an intact system for optimal multisensory integration. For example, if visual speech information from both eyes is not reliably encoded by the amblyopic brain, then a reduced McGurk effect (i.e., a shift toward veridical auditory perception) may reflect an optimal strategy of sensory re-weighting. However, the studies to date were not designed, nor can their results be modeled, to distinguish integration failure from sensory re-weighting.^14–16 Studies of other multisensory abnormalities in amblyopia have been similarly unable to make this mechanistic distinction. For example, the ability to detect asynchrony between auditory and visual stimuli is diminished in amblyopia.^29,30 While insensitivity to audiovisual asynchrony may indicate abnormal audiovisual integration,³¹ it may also be caused by noisy encoding of visual temporal information that feeds into a normal upstream multisensory processing network.³² Considering these possibilities, it becomes clear that the origin of these multisensory abnormalities in amblyopia remains an unresolved question. Does anomalous audiovisual perception in amblyopia result from failure to integrate the available unisensory information, or does it represent appropriate (i.e., statistically optimal) integration of the available, but deficient, unisensory information? 

Unlike previous investigations of multisensory integration in amblyopia, this study aims to differentiate between deficits at the unisensory and multisensory levels of sensory processing by explicitly modeling audiovisual integration in a ventriloquism effect paradigm.³³ The ventriloquism effect is an audiovisual illusion in which spatially disparate visual and auditory stimuli are perceived as originating from the same location. Typically, the location information of the unimodal visual component dominates in the perceived location of the fused audiovisual percept, a process sometimes termed visual capture.³⁴ In a manipulation of the ventriloquism effect, however, Alais and Burr³⁵ showed that reducing the spatial reliability of (i.e., blurring) the visual stimulus leads to greater dominance by the auditory stimulus. Critically, they also demonstrated that audiovisual spatial perception and the ventriloquism effect obey the maximum likelihood estimation (MLE) model of optimal integration in terms of both localization precision and localization bias (see 5 for details of the MLE model). 

Audiovisual spatial integration and the ventriloquism effect offer several advantages over the McGurk effect as an approach for studying audiovisual integration in amblyopia. First, it involves simple spatial signals that are measured on a continuous scale, as opposed to complex linguistic signals that are measured categorically. Categoric syllabic data are not only more difficult to transcribe faithfully, but also more challenging to model mathematically. Second, the ventriloquism effect is not significantly affected by top-down directed attention or bottom-up automatic attention.^36,37 In contrast, measures of integration using the McGurk effect may be confounded by attentional effects, as it is significantly diminished by increased attentional load.^38,39 Finally, multisensory enhancement in spatial localization precision, if observed in amblyopia, will provide clear evidence of audiovisual integration, rather than nonintegrative unisensory dominance or perceptual blending. 

The MLE model of audiovisual spatial integration offers a powerful methodology to distinguish two possible causes of the multisensory abnormalities observed in amblyopia: optimal integration of deficient unisensory information, or failure to integrate the available unisensory information. If integration is intact, then predictions of the MLE model will be upheld despite deficits at the unisensory level. Specifically, bimodal localization will show precision enhancement over unimodal localization consistent with the MLE model, and spatial bias in the ventriloquism effect will be consistent with MLE-optimal combination of the unimodal position signals. Conversely, if integration is impaired, then spatial precision of audiovisual localization will show less enhancement than predicted by the MLE model, and spatial bias in the ventriloquism effect will not follow the MLE model. 

Methods

Participants

Two groups of participants were recruited: adults with unilateral amblyopia, and adults with normal vision. Participants were excluded if they had manifest or latent nystagmus, or a history of neurodevelopmental or neurological disorder, hearing impairment, high ametropia (hyperopia > +5 diopters [D] or myopia > −6D), or any other ocular pathology or prior intraocular surgery. Each participant underwent an ocular and hearing screen by a certified orthoptist or ophthalmologist. The ocular assessment measured habitual refractive correction (automatic lensmeter), distance visual acuity (Early Treatment Diabetic Retinopathy Study chart with habitual correction), stereo acuity (Randot circles and Titmus fly test), foveal suppression (Worth 4-dot test), and eye alignment (prism cover test). The hearing assessment ensured reliable detection of suprathreshold pure tones (25-dBA sound pressure level) in each ear at four standard frequencies (500, 1000, 2000, and 4000 Hz) using a screening audiometer (model MA 27; MAICO Diagnostics, Eden Prairie, MN, USA) with circumaural headphones (model TDH 39; MAICO Diagnostics).⁴⁰ Amblyopia was defined as visual acuity of 0.18 logMAR (20/30) or poorer in the amblyopic eye, visual acuity of at least 0.1 logMAR (20/25) in the fellow eye, and an interocular acuity difference of 0.2 logMAR or more. Participants were classified as having anisometropic amblyopia if the interocular difference in spherical equivalent or cylindrical error was 1 D or more, as having strabismic amblyopia if there was any manifest deviation on cover testing in the absence of anisometropia, or as having mixed-mechanism amblyopia when both anisometropia and a manifest deviation of 8 prism diopters or more were present. Visually normal was defined as visual acuity of at least 0.1 logMAR (20/25) in each eye, with stereo acuity of 40 seconds of arc or better, and no manifest strabismus. The Research Ethics Board at The Hospital for Sick Children approved the study, and all protocols adhered to the tenets of the Declaration of Helsinki. Written informed consent was obtained from all participants after explanation of the nature and possible consequences of the study. 

Fourteen adults with unilateral amblyopia (mean age ± SD [range]: 28.8 ± 8.9 [19–48] years; subtypes: 7 anisometropic, 3 strabismic, 4 mixed) and 16 visually normal controls (mean age ± SD [range]: 29.2 ±7.1 [23–47] years) participated in the study. Individuals were not age-matched, but the two groups did not differ significantly in age (t₂₈ = 0.137, P = 0.892). Clinical characteristics of the participants with amblyopia are summarized in Table 1. The sample size provided 80% power to detect an effect size of d_z = .81 for within-subjects effects in the amblyopia group (e.g., comparing observed versus MLE-predicted localization precision), and an effect size of d = 1.06 for between-subjects effects (e.g., comparing localization precision between groups), for a two-tailed test at alpha = 0.05 (computed using G*Power 3.1⁴¹). 

Table 1

 

Characteristics of Participants with Amblyopia

 

Apparatus and Stimuli

The entire experiment was conducted in a darkened, double-walled audiometric chamber (internal dimensions 2.0 × 2.1 × 2.2 m). The floor was carpeted, and the walls and ceiling were lined with 5-cm acoustic wedge foam (Foam Factory, Macomb, MI, USA). Head position was constrained by a chinrest fixed to a small table 65 cm from the center of the audiovisual apparatus (see Supplementary Fig. S1). Visual stimuli consisted of medium contrast (39%) Gaussian blobs of five sizes (1 SD = 16°, 20°, 24°, 28°, or 32°), flashed on a large light-emitting diode monitor (model E654; NEC Corporation, Tokyo, Japan) for 32 ms (2 frames at 60 Hz). The monitor subtended 96° × 64° of visual angle (165-cm diagonal) and was overlaid with a three-stop neutral density filter (ND 0.9) to create a Gaussian blob (peak luminance 2.1 cd/m²) with imperceptible steps between gray levels. Auditory stimuli consisted of 32-ms clicks (8 cycles of 2–5 kHz bandpass filtered white noise, 4 ms in duration, enveloped with a 2-ms sigmoid on/off ramp), presented at 62.0 dBA through two speakers mounted on either side of the monitor at the horizontal midline. Apparent click location was controlled by linear amplitude panning of interaural level difference cues.^42,43 The output profile for each speaker was measured across the entire stimulus dynamic range using a sound level meter to ensure their outputs were identical. Auditory and visual stimulus timings were confirmed with an oscilloscope. A wireless gamepad (model F710; Logitech, Newark, CA, USA) was used to initiate trials and enter responses. 

Procedure

All trials were conducted with both eyes open. Participants performed a relative spatial localization task for unimodal stimuli (visual blobs only or auditory clicks only) and bimodal stimuli (blobs and clicks together) similar to that described by Alais and Burr.³⁵ A general trial timeline is illustrated in Figure 1. Upon initiation of each trial, a red fixation dot (0.66°) was presented centrally for 500 ms, followed by a randomized delay of 250 to 400 ms. Two matching stimuli (a test stimulus and probe stimulus, but in random order) were then presented in succession, 500-ms apart, and the participant was asked to “indicate whether the second event occurred left or right relative to the first event.” Participants were instructed to keep their head and eyes aligned centrally. There were 21 test stimulus conditions: six unimodal (1 click-only and 5 blob size variants), five bimodal with spatially congruent clicks and blobs (5 blob size variants paired with a click), and 10 bimodal with spatially conflicting clicks and blobs (5 blob size variants paired with a click, but blob displaced 4° left and click displaced 4° right, or click displaced 4° left and blob displaced 4° right). Bimodal test stimulus conditions with spatial conflict were designed to elicit the ventriloquism effect, and participants were not told of this spatial disparity. The test stimulus was presented centrally (0°) in all trials; for bimodal test stimuli with spatial conflict, the unimodal components were displaced 4° in opposite directions such that that their average location was still 0°. The probe stimulus matched the characteristics of the test stimulus except for horizontal displacement (specified in Table 2), and in some cases, spatial congruency (bimodal test stimuli with spatial conflict were paired with spatially congruent probe stimuli). Data were collected in separate blocks for the unimodal auditory conditions, and for each of the five blob sizes within the unimodal visual and bimodal conditions. Twenty trials were run for each probe stimulus displacement, randomly interleaved within each block. 

Figure 1

 

Illustration of the trial timeline. After trial initiation by the participant, a fixation dot appeared centrally for 500 ms, followed by a dark interval of 250 to 400 ms. Two brief stimuli (test and probe) were displayed in sequence, 500-ms apart, but in random order. The participant judged whether the second stimulus originated left or right relative to the first.

Table 2

 

Probe Stimulus Displacements Used for Each Test Stimulus Condition

 

Data Analysis

Psychometric curve fitting and parameter calculation were performed individually for each participant. For each test stimulus condition, the proportion of trials in which the probe was perceived ‘left' of the test stimulus was plotted as a function of probe displacement. These data were fitted with cumulative normal functions free to vary in horizontal position and width (i.e., unconstrained mean and variance). Fitting was computed by the maximum likelihood method using custom-written scripts in MATLAB version 2011b (Mathworks, Inc., Natick, MA, USA). The mean of the fitted function, or point of subjective equality (PSE), represents the bias in the localization estimate of the test stimulus (Display Formula\(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\unicode[Times]{x1D6C2}}\)\(\def\bupbeta{\unicode[Times]{x1D6C3}}\)\(\def\bupgamma{\unicode[Times]{x1D6C4}}\)\(\def\bupdelta{\unicode[Times]{x1D6C5}}\)\(\def\bupepsilon{\unicode[Times]{x1D6C6}}\)\(\def\bupvarepsilon{\unicode[Times]{x1D6DC}}\)\(\def\bupzeta{\unicode[Times]{x1D6C7}}\)\(\def\bupeta{\unicode[Times]{x1D6C8}}\)\(\def\buptheta{\unicode[Times]{x1D6C9}}\)\(\def\bupiota{\unicode[Times]{x1D6CA}}\)\(\def\bupkappa{\unicode[Times]{x1D6CB}}\)\(\def\buplambda{\unicode[Times]{x1D6CC}}\)\(\def\bupmu{\unicode[Times]{x1D6CD}}\)\(\def\bupnu{\unicode[Times]{x1D6CE}}\)\(\def\bupxi{\unicode[Times]{x1D6CF}}\)\(\def\bupomicron{\unicode[Times]{x1D6D0}}\)\(\def\buppi{\unicode[Times]{x1D6D1}}\)\(\def\buprho{\unicode[Times]{x1D6D2}}\)\(\def\bupsigma{\unicode[Times]{x1D6D4}}\)\(\def\buptau{\unicode[Times]{x1D6D5}}\)\(\def\bupupsilon{\unicode[Times]{x1D6D6}}\)\(\def\bupphi{\unicode[Times]{x1D6D7}}\)\(\def\bupchi{\unicode[Times]{x1D6D8}}\)\(\def\buppsy{\unicode[Times]{x1D6D9}}\)\(\def\bupomega{\unicode[Times]{x1D6DA}}\)\(\def\bupvartheta{\unicode[Times]{x1D6DD}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bUpsilon{\bf{\Upsilon}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\(\def\iGamma{\unicode[Times]{x1D6E4}}\)\(\def\iDelta{\unicode[Times]{x1D6E5}}\)\(\def\iTheta{\unicode[Times]{x1D6E9}}\)\(\def\iLambda{\unicode[Times]{x1D6EC}}\)\(\def\iXi{\unicode[Times]{x1D6EF}}\)\(\def\iPi{\unicode[Times]{x1D6F1}}\)\(\def\iSigma{\unicode[Times]{x1D6F4}}\)\(\def\iUpsilon{\unicode[Times]{x1D6F6}}\)\(\def\iPhi{\unicode[Times]{x1D6F7}}\)\(\def\iPsi{\unicode[Times]{x1D6F9}}\)\(\def\iOmega{\unicode[Times]{x1D6FA}}\)\(\def\biGamma{\unicode[Times]{x1D71E}}\)\(\def\biDelta{\unicode[Times]{x1D71F}}\)\(\def\biTheta{\unicode[Times]{x1D723}}\)\(\def\biLambda{\unicode[Times]{x1D726}}\)\(\def\biXi{\unicode[Times]{x1D729}}\)\(\def\biPi{\unicode[Times]{x1D72B}}\)\(\def\biSigma{\unicode[Times]{x1D72E}}\)\(\def\biUpsilon{\unicode[Times]{x1D730}}\)\(\def\biPhi{\unicode[Times]{x1D731}}\)\(\def\biPsi{\unicode[Times]{x1D733}}\)\(\def\biOmega{\unicode[Times]{x1D734}}\)\({\hat{S}}\), Equation 1 in 5). The maximum slope of the function (β', Equation 5 in 5) represents the precision of the localization estimate, and was computed from the SD of the fitted function.⁴⁴ 

In the 16°/SD unimodal visual condition, undersampling resulted in falsely steep fits for six participants (38%) in the control group and six participants (43%) in the amblyopia group. Data from 16°/SD conditions for all participants were therefore excluded from analysis. 

The MLE-predicted values for bimodal localization precision (i.e., predicted β') were calculated from the variances of the fitted functions in each unimodal condition (using Equations 4 and 5 in 5). The MLE-predicted values for bimodal localization bias (i.e., predicted PSE) in the ventriloquism effect were computed from the variances of the fitted functions in each unimodal condition and the specified locations of the auditory and visual signals (using Equations 1, 2, and 3 in 5). 

All statistical tests were performed using IBM SPSS Statistics version 22 (Armonk, NY, USA). Statistical significance was defined as P < 0.05. Localization bias data from the two bimodal conditions with spatial conflict (i.e., blob 4° left and click 4° right, or blob 4° right and click 4° left) were combined within each blob size because repeated measures ANOVAs (one for each participant group) showed no significant difference in bias between the two conflict conditions across the four blob sizes (control group: F_1,15 = 0.22, P = 0.647, η²_p = .014; amblyopia group: F_1,13 = 1.69, P = 0.216, η²_p = .115). Group membership (i.e., control or amblyopia) was analyzed as a between-subjects factor, while blob size (i.e., 20, 24, 28, 32°/SD) and data source (i.e., observed or model-predicted values) were analyzed as within-subjects factors. The assumption of equality of covariance matrices was satisfied according to Box's M test for all mixed-design ANOVAs. The assumption of sphericity was assessed using Mauchly's test for all repeated measures and mixed-design ANOVAs, and where violated, the conservative Greenhouse-Geisser correction was applied. All post hoc multiple comparisons were computed by the Bonferroni method. 

Results

Representative data and fitted functions for the unimodal and bimodal localization tasks are shown in Figure 2. Subsequent analyses of localization precision, localization bias, and agreement with the MLE model are reported in the sections below. 

Figure 2

 

Unimodal and bimodal localization task performance and fitted cumulative normal functions for a representative participant in the control group (A, C, E, G) and the amblyopia group (B, D, F, H). Open symbols represent the proportion of trials in which the probe stimulus was perceived leftward of the test stimulus. Visual stimuli were Gaussian blobs of specified size (1 SD = 20°, red; 1 SD = 24°, orange; 1 SD = 28°, green; 1 SD = 32°, blue), and auditory stimuli were white noise clicks (black). (A, B) Unimodal conditions with visual or auditory test stimulus centered at 0°. (C, D) Bimodal conditions with unimodal components of test stimulus that in spatial agreement (i.e., blob and click both centered at 0°). (E–H) Bimodal conditions with unimodal components of test stimulus in spatial conflict to elicit the ventriloquism effect (i.e., blob and click symmetrically displaced 4° from center).

Localization Precision

Unimodal Stimuli

Localization precision, defined as the slope of the fitted psychometric function at the midpoint, decreased in both groups for unimodal visual stimuli as the blob size increased from 20°/SD to 32°/SD (Fig. 3A). A two (group) × four (blob size) mixed-design ANOVA for unimodal visual localization precision yielded a significant main effect of group (F_1,28 = 4.692, P = 0.039, η²_p = .144), a significant main effect of blob size (F_2.44,68.32 = 175.45, P < 0.001, η²_p = .862, Greenhouse-Geisser correction), but no significant interaction of group and blob size (F_2.44,68.32 = 2.31, P = 0.096, η²_p = .076, Greenhouse-Geisser correction). A two-tailed t-test conducted to compare unimodal auditory localization precision between groups (Fig. 3B) revealed a significant difference in localization precision between the control and amblyopia groups (t₂₈ = 2.09, P = 0.046, d = .765). 

Figure 3

 

Mean localization precision for (A) visual-only, (B) auditory-only, and (C) spatially congruent bimodal audiovisual stimuli. Control group data are shown in blue and amblyopia group data are shown in red. Error bars: ±1 SEM. Localization precision in the amblyopia group was significantly poorer than in the control group for unimodal visual, unimodal auditory, and bimodal audiovisual stimuli (P < 0.05).

Spearman rank correlations were conducted to assess the relation between localization precision and clinical measures of amblyopia. There were no significant relations between mean localization precision for visual blobs (collapsed across blob sizes) and amblyopic eye visual acuity (R_s = .14, P = 0.62) or stereo acuity (R_s = .45, P = 0.11). Similarly, there were no significant relations between localization precision for auditory clicks and amblyopic eye visual acuity (R_s = .099, P = 0.736) or stereo acuity (R_s = .034, P = 0.91). 

Spatially Congruent Bimodal Stimuli

Localization precision for spatially congruent bimodal stimuli decreased in both the control group and amblyopia group as the blob size increased from 20°/SD to 28°/SD (Fig. 3C). A two (group) × four (blob size) mixed-design ANOVA for bimodal localization precision yielded a significant main effect of group (F_1,28 = 6.724, P = 0.015, η²_p = .194), a significant main effect of blob size (F_1.98,55.39 = 34.42, P < 0.001, η²_p = .551, Greenhouse-Geisser correction), but no significant interaction of group and blob size (F_1.98,55.39 = 0.46, P = 0.650, η²_p = .016, Greenhouse-Geisser correction). 

Spearman rank correlations revealed no significant relations between mean localization precision for spatially congruent bimodal stimuli (collapsed across blob sizes) and amblyopic eye visual acuity (R_s = .42, P = 0.14) or stereo acuity (R_s = .005, P = 0.99). 

Agreement With MLE Model

According to the MLE model, audiovisual integration results in enhanced localization precision for bimodal stimuli by optimal combination of the component unimodal spatial signals. In complete integration failure, however, the best localization precision achievable is that of the more precise unimodal signal. This distinction provides a test for integration in amblyopia. Importantly, the MLE model also predicts that the bimodal enhancement in localization precision is greatest, and therefore most detectable, when the localization precisions of the unimodal components are equal (i.e., β'_V = β'_A) (see Equations 4 and 5 in 5). The bimodal localization precision observed in this study was therefore compared with that expected with intact integration (i.e., MLE-predicted value computed from unimodal component precisions) and with integration failure (i.e., the most precise unimodal component) specifically for the condition in which the unimodal components were most similar for each participant (Fig. 4). For the control group, a one-way repeated measures ANOVA showed a significant difference between the observed precision, MLE-predicted precision, and best unimodal component precision (F_1.04,15.62 = 7.13, P = 0.016, η²_p = .322, Greenhouse-Geisser correction). As expected, Bonferroni post hoc analysis revealed that the observed bimodal precision was significantly better than the best unimodal component precision (P = 0.017, η²_p = .325), but not significantly different from the MLE-predicted precision (P = 0.974, η²_p < .001). For the amblyopia group, the same one-way repeated measures ANOVA showed a significant difference between the observed precision, MLE-predicted precision, and best unimodal component precision (F_1.18,15.39 = 8.83, P = 0.007, η²_p = .404, Greenhouse-Geisser correction). Bonferroni post hoc analysis revealed that the observed bimodal precision was significantly better than the best unimodal component precision (P = 0.011, η²_p = .400), but not significantly different from the MLE-predicted precision (P = 0.727, η²_p = .010). These findings indicate that bimodal localization precision is consistent with the MLE model of optimal integration both the control group and the amblyopia group. 

Figure 4

 

Comparison of observed and predicted bimodal localization precisions. MLE-predicted bimodal precisions were computed solely from unimodal precision measurements, and represent expected performance with intact integration. Best unimodal component precisions represent expected performance in integration failure. Error bars: ±1 SEM. For (A) the control group and (B) the amblyopia group, the observed bimodal precision was significantly better than that of the best unimodal component (P < 0.05), but not significantly different from that predicted by the MLE model.

Localization Bias

Unimodal Stimuli and Spatially Congruent Bimodal Stimuli

Localization bias was defined as the PSE of the fitted psychometric function. For unimodal stimuli, a two (group) × four (blob size) mixed-design ANOVA for PSE yielded no significant main effect of group (F_1,28 = 1.666, P = 0.494, η²_p = .017), no significant main effect of blob size (F_1.89,52.81 = 1.01, P = 0.367, η²_p = .035, Greenhouse-Geisser correction), and no significant interaction of group and blob size (F_1.89,52.81 = 1.36, P = 0.295, η²_p = .010, Greenhouse-Geisser correction). The mean PSE across blob sizes did not differ significantly from the expected value of 0° for the control group (M = 0.17°, 95% confidence interval [CI] = −0.28°, 0.62°) or the amblyopia group (M = −0.07°, 95% CI = −0.66°, 0.52°). For spatially congruent bimodal stimuli, a two (group) × four (blob size) mixed design ANOVA for PSE yielded no significant main effect of group (F_1,28 = 0.24, P = 0.678, η²_p = .006), no significant main effect of blob size (F_2.24,62.58 = 2.22, P = 0.111, η²_p = .074, Greenhouse-Geisser correction), and a significant interaction of group and blob size (F_2.24,62.58 = 3.32, P = 0.038, η²_p = .106, Greenhouse-Geisser correction). Bonferroni post hoc tests failed to detect the source of this interaction (see Supplementary Fig. S2). The mean PSE across blob sizes did not differ significantly from the expected value of 0° for the control group (M = 0.19°, 95% CI = −0.09°, 0.47°) or the amblyopia group (M = 0.10°, 95% CI = −0.27°, 0.47°). 

Bimodal Stimuli With Spatial Conflict

Localization biases for bimodal stimuli with spatial conflict are illustrated in Figure 5 for the control group and the amblyopia group (individual data are shown in Figs. 2E–H). In these trials, the visual and auditory unimodal components were symmetrically displaced about 0°. Both groups showed a monotonic progression in localization bias from vision-dominant (i.e., a classic ventriloquism effect, with PSE biased toward the position of the visual blob) to audition-dominant (i.e., a reverse ventriloquism effect, with PSE biased toward the position of the auditory click) as the blob size increased from 20°/SD to 32°/SD (Fig. 5). A two (group) × four (blob size) mixed-design ANOVA for PSE yielded no significant main effect of group (F_1,28 = 1.04, P = 0.317, η²_p = .036), a significant main effect of blob size (F_3,84 = 106.30, P < 0.001, η²_p = .792), and no significant interaction of group and blob size (F_3,84 = 0.42, P = 0.738, η²_p = .015). This indicates that bimodal localization bias in the ventriloquism effect was not dissimilar between the two groups. 

Figure 5

 

Comparison of observed and predicted bimodal localization bias in the ventriloquism effect. The MLE-predicted values (open circles) were computed solely from the parameters of the unimodal measurements. Error bars: ±1 SEM. For (A) the control group and (B) the amblyopia group, the observed localization bias did not resemble visual capture or auditory capture, but showed a dynamic trend from visual dominance at smaller blob sizes toward auditory dominance at larger blob sizes. Bimodal localization bias showed no significant deviation from the MLE model in either group (P < 0.05).

Agreement With MLE Model

According to the MLE model, the perceived location of a bimodal event in the ventriloquism effect is the weighted average of the unimodal visual and auditory position signals (see Equation 1 in 5). The relative weights of vision and audition, in turn, are determined by their respective unimodal localization precisions (see Equations 2 and 3 in 5). The observed and MLE-predicted bimodal localization biases may therefore be compared as an additional test of the MLE model. For the control group, a two (observed versus MLE-predicted) × four (blob size) repeated measures ANOVA for PSE yielded no significant difference between the observed and MLE-predicted biases (F_1,15 = 1.38, P = 0.259, η²_p = .084), a significant main effect of blob size (F_3,45 = 118.70, P < 0.001, η²_p = .888), and no significant interaction between the two factors (F_3,45 = 1.59, P = 0.206, η²_p = .096). Similarly, for the amblyopia group, a two (observed versus MLE-predicted) × four (blob size) repeated measures ANOVA for PSE yielded no significant difference between the observed and MLE-predicted biases (F_1,13 = 0.05, P = 0.826, η²_p = .004), a significant main effect of blob size (F_3,39 = 120.56, P < 0.001, η²_p = .903), and no significant interaction between the two factors (F_3,39 = 0.96, P = 0.410, η²_p = .070). These findings indicate that bimodal localization bias in the ventriloquism is consistent with the MLE model in both control group (Fig. 5A) and the amblyopia group (Fig. 5B). 

Discussion

We report that under binocular viewing conditions typical of everyday experience, amblyopia is associated with a pervasive impairment in spatial localization precision that involves visual, auditory, and audiovisual (i.e., multisensory) perception. Using the MLE model of the ventriloquism effect,³⁵ however, we show that audiovisual spatial integration is intact in amblyopia, and that the abnormalities in audiovisual spatial perception are consistent with optimal sensory re-weighting in the face of unisensory deficits in both spatial vision and spatial hearing. 

The unimodal visual localization task measured relative localization precision under binocular viewing conditions for diffuse visual blobs of various sizes. The large Gaussian blobs provided no high-contrast edges to aid in the location discrimination task. The negative relation between visual blob size and localization precision observed in both groups signifies that increasingly blurred visual targets elicit greater uncertainty in spatial perception. Compared with the control group, the amblyopia group showed a general reduction in visual localization precision across blob sizes despite normal visual acuity in the fellow eye, and presumably normal visual acuity under binocular viewing conditions.⁴⁵ Several possibilities may account for this finding. Contrary to clinical dogma, vision in the fellow eye is not entirely normal.⁴⁶ Careful psychophysical studies have shown that the fellow eye has reduced optotype^4,47 and Vernier acuity,⁴⁸ and perhaps more importantly, greater spatial uncertainty and distortion affecting both foveal and extra-foveal vision.^9,10 Such deficits in spatial vision, affecting both the amblyopic and fellow eyes, may therefore contribute to the binocular visual localization deficit observed in the present study. Another possible contributing factor may be the temporal interval between the two stimuli whose positions were judged. Previous studies of spatial vision in the fellow eye (mentioned above) used static visual targets whose spatial elements were present simultaneously. In our study, however, the spatial elements (i.e., blobs) were separated by a temporal interval of 500 ms. Time-dependent factors such as reduced visual persistence⁴⁹ or fixation instability^50–52 in amblyopia may have therefore contributed to the observed deficit in localizing temporally separated visual targets. Specifically, reduced visual persistence may speed the rate of decay of the spatial signal, and fixation instability may jitter the mapping of physical space to retinal coordinates. 

This study also revealed a novel amblyopic deficit in auditory spatial localization precision. Two features of the experimental task are particularly notable: (1) trials were conducted in darkness with no visual cues, and (2) localization did not involve pointing of any kind, but was a ‘left' or ‘right' determination entered as a button press on a gamepad. These features mean that the spatial uncertainty in amblyopic vision¹¹ and visuomotor control⁵³ cannot directly account for the observed unisensory auditory effect. Rather, they indicate that the sensory impairment in amblyopia extends beyond vision to the realm of spatial hearing. What might be the basis for the surprising association between amblyopia and impaired sound localization? Gori et al.⁵⁴ have proposed that cross-sensory interactions during development serve a calibrating function such that the more robust and accurate sense informs the other. For example, in healthy children younger than 8 years of age, vision informs touch in spatial orientation discrimination, but touch informs vision in size discrimination.⁵⁴ In early bilateral visual impairment, however, cross-sensory calibration is affected in a predictable way: haptic orientation discrimination (for which vision typically dominates) is impaired, but haptic size discrimination (for which touch typically dominates) is preserved.⁵⁵ Impaired visual input in amblyopia may similarly affect the cross-sensory calibration of auditory spatial abilities in early life. What is striking about our results is that the impairment in cross-sensory calibration of auditory localization occurred despite access to normal visual acuity from the fellow eye, and presumably normal visual acuity under binocular viewing conditions of everyday experience.⁴⁵ We have conducted further experiments to specifically investigate the effects of abnormal vision on the calibration of auditory spatial localization during development, which will be the subject of a separate report. 

The bimodal localization task measured relative localization precision under binocular viewing conditions for diffuse visual blobs of varying sizes paired with a simultaneous auditory click. As with unimodal stimuli, the amblyopia group showed a general impairment in bimodal localization precision across blob sizes. However, comparison with predictions of the MLE model showed that multisensory integration was intact in both the control and amblyopia groups. Indeed, the localization precision for spatially congruent bimodal stimuli (Fig. 4) and the localization bias for spatially incongruent bimodal stimuli in the ventriloquism effect (Fig. 5) represented optimal combination the unimodal spatial features according to the MLE model. Overall, these findings provide independent validation for the MLE model of ventriloquism³⁵ in a larger sample of typically sighted individuals, and suggest that at least some multisensory processing abnormalities reported in amblyopia do not reflect disordered multisensory integration, but rather unisensory deficits that feed into and propagate through an otherwise normal integrative network. 

A common theme in studies of multisensory integration in children is that it develops relatively late⁵⁶ compared with unisensory abilities.^57,58 By some estimates, optimal multisensory integration does not arise until age 8 to 10 years,^54,59 which is beyond the sensitive period for amblyopic visual loss, and after the age at which the greatest recovery may be achieved through occlusion or penalization therapy.¹ This may explain why optimal integration in the ventriloquism effect is spared in amblyopia. If optimal integration typically arises after the amblyopic unisensory deficits are crystallized, then it will naturally be tuned to the stable audiovisual spatial capabilities of late childhood and beyond. This principle also implies that other multisensory perceptual anomalies in amblyopia (e.g., reduced susceptibility to the McGurk effect and poorer audiovisual asynchrony detection) may result from informational deficits at the unisensory level (e.g., increased spatial or temporal uncertainty) rather than deficits in multisensory integration. 

Acknowledgments

Supported by Grant MOP 106663 from the Canadian Institutes of Health Research (CIHR; Ottawa, Ontario, Canada), Leaders Opportunity Fund from the Canada Foundation for Innovation (CFI; Ottawa, Ontario, Canada), the John and Melinda Thompson Endowment Fund in Vision Neurosciences at The Hospital for Sick Children (Toronto, Ontario, Canada), the Department of Ophthalmology and Vision Sciences at The Hospital for Sick Children (Toronto, Ontario, Canada), and the Vision Science Research Program at the University of Toronto (Toronto, Ontario, Canada). 

Disclosure: M.D. Richards, None; H.C. Goltz, None; A.M.F. Wong, None 

Appendix A

The MLE model has been put forward by several groups as a model for multisensory integration of spatial information involving vision.^20,35,60 For the ventriloquism effect, the MLE model predicts that the perceived location of a bimodal event will be the weighted average of the locations of the unimodal events, such that:  

\begin{equation}\tag{1}{ {{{\hat {S}}_{VA}} = {w_V}{{\hat {S}}_V} + {w_A}{{\hat {S}}_A}} } {\rm }\end{equation}

where Display Formula\({{\hat {S}}_V}\) and Display Formula\({\hat {S}}_A\) are the unisensory localization estimates for vision and audition, Display Formula\({w_V}\) and Display Formula\({w_A}\) are the perceptual weights for vision and audition, and Display Formula\({{\hat {S}}_{VA}}\) is the resultant bimodal localization estimate. The perceptual weights, Display Formula\({w_V}\) and Display Formula\({w_A}\), sum to 1, and are inversely proportional the variances of the unisensory localization estimates, Display Formula\({\sigma ^2}_A\) and Display Formula\({\sigma ^2}_V\), such that:  

\begin{equation}\tag{2}{ {{w_V} = {{{\sigma ^2}_A} \over {{\sigma ^2}_V + {\sigma ^2}_A}}} } {\rm }\end{equation}

and  

\begin{equation}\tag{3}{ {{w_A} = {{{\sigma ^2}_V} \over {{\sigma ^2}_A + {\sigma ^2}_V}}} } {\rm }\end{equation}

 

Experimentally, localization variance can be estimated from the psychometric curve fit to the unimodal localization data. The combination of unisensory localization estimates in the MLE model is mathematically optimal in that it results in a bimodal localization estimate with the lowest possible variance (i.e., highest possible precision):  

\begin{equation}\tag{4}{ {{\sigma ^2}_{VA} = {{{\sigma ^2}_V{\sigma ^2}_A} \over {{\sigma ^2}_A + {\sigma ^2}_V}} \le \min \left( {{\sigma ^2}_V,{\sigma ^2}_A} \right)} } {\rm }\end{equation}

 

If the psychometric response is represented by a cumulative normal function, the variance of the function is inversely related to maximum slope, Display Formula\({\beta ^{\prime} }\), at the inflection point of the curve:  

\begin{equation}\tag{5}{ {{\beta ^{\prime} } = \left( {{1 \over {\sqrt {2\pi } }}} \right)\cdot\left( {{1 \over {\sqrt {{\sigma ^2}} }}} \right)} } {\rm }\end{equation}

 

Therefore, following from Equations 4 and 5, the MLE model predicts that spatial localization precision (represented by Display Formula\({\beta ^{\prime} }\)) for the bimodal event is greater than or equal to that of its unimodal components, and that the bimodal advantage in spatial localization precision is greatest when the precisions of the unisensory components are equal.