Abstract
Purpose.:
Vision loss due to retinitis pigmentosa affects an estimated 15 million people worldwide. Through collaboration between Second Sight Medical Products, Inc., and the Doheny Eye Institute, six blind human subjects underwent implantation with epiretinal 4 × 4 electrode arrays designed to directly stimulate the remaining cells of the retina, with the goal of restoring functional vision by applying spatiotemporal patterns of stimulation. To better understand spatiotemporal interactions between electrodes during synchronous and asynchronous stimulation, the authors investigated how percepts changed as a function of pulse timing across the electrodes.
Methods.:
Pulse trains (20, 40, 80, and 160 Hz) were presented on groups of electrodes with 800, 1600, or 2400 μm center-to-center separation. Stimulation was either synchronous (pulses were presented simultaneously across electrodes) or asynchronous (pulses were phase shifted). Using a same-different discrimination task, the authors were able to evaluate how the perceptual quality of the stimuli changed as a function of phase shifts across multiple electrodes.
Results.:
Even after controlling for electric field interactions, subjects could discriminate between spatiotemporal pulse train patterns based on differences of phase across electrodes as small as 3 ms. These findings suggest that the quality of the percept is affected not only by electric field interactions but also by spatiotemporal interactions at the neural level.
Conclusions.:
During multielectrode stimulation, interactions between electrodes have a significant influence on the quality of the percept. Understanding how these spatiotemporal interactions at the neural level influence percepts during multielectrode stimulation is fundamental to the successful design of a retinal prosthesis.
Vision loss from photoreceptor diseases such as retinitis pigmentosa (RP) and age-related macular degeneration (AMD) affects an estimated 15 million people worldwide.
1 RP begins with photoreceptor degeneration in the periphery, and this degeneration gradually spreads from the periphery to the fovea. In later stages of the disease, the spatial organization of the inner nuclear and ganglion cell layers becomes disorganized, and bipolar, amacrine, and ganglion cells begin to die.
2,3 However, the inner nuclear and ganglion cell layers maintain relatively high cell density,
4–6 and some functional circuitry remains,
7–9 even in later stages of disease. As a result, several groups have developed microelectronic retinal prostheses with the ultimate goal of restoring vision in blind subjects by stimulating remaining retinal cells with spatiotemporal sequences of electrical pulses. Indeed, both semiacute and long-term implanted devices have been demonstrated to be safe and capable of generating visual percepts in human subjects (Humayun MS.
IOVS 2009;50:ARVO E-Abstract 4744; Sachs HG, et al.
IOVS 2009;50:ARVO E-Abstract 4742).
10–16
Although limited data have been reported about how electrodes interact during spatiotemporal stimulation in the retina, it is well known that for cochlear implants the precise timing of stimulation across electrodes has perceptual consequences as a result of both electrical field
17,18 and neuronal interactions.
19 Within cochlear implants, the most common approach to dealing with electrode interactions has been to reduce channel interactions (or cross-talk between electrodes) by phase-shifting stimuli across electrodes, a technique also referred to as continuous interleaved sampling.
20 In the case of cochlear implants, interleaving patterns of electrical pulses reduces electrical and neural nonlinearities generated by channel interactions and makes the resultant electrical fields and percepts easier to computationally model. Besides reducing interactions, phase-shifting stimulation across electrodes provides the technical advantage of allowing multiple electrodes to share the same driver. This is not of great importance for cochlear implants because these devices have a relatively small number of electrodes (<30). Although the retinal devices implanted in the subjects tested here also had a relatively small number of electrodes, the design used shared drivers across pairs of electrodes: there were eight drivers for the 16-electrode implant. The ability to share drivers across multiple electrodes may be more critical in future retinal implants, which are likely to have many hundreds or even many thousands of electrodes.
However, it is also possible that spatiotemporal interactions between electrodes might be a powerful tool for increasing the spatial resolution of retinal implants. Within the cochlear implant literature it has been shown that spatiotemporal interactions between adjacent cochlear electrodes can be used to produce stimuli with pitches that are intermediate between the two stimulated electrodes. Simultaneous
21 or near-simultaneous
22 stimulation of adjacent electrodes produces pitch percepts intermediate to those produced by each electrode separately, thereby increasing the number of place-pitch steps available to cochlear implant listeners (virtual electrodes). Spatiotemporal interactions in cochlear implants have been shown to be capable of creating two to nine virtual electrodes, depending on the observer. Given that the human fovea contains approximately 160,000 cones per square millimeter while current retinal implant technology consists of arrays of 1000 electrodes or fewer, the ability to exploit spatiotemporal interactions may prove critical in improving resolution.
Here we systematically examined a variety of spatiotemporal interactions in two subjects with retinal prostheses. The experiments described here focus on the ability of subjects to discriminate between pulse patterns across groups of electrodes in which the stimulation on any individual electrode was identical across the two pulse patterns, but the temporal relationship (phase-shifting) between electrodes varied. For example, in experiment 1, we tested the ability of subjects to discriminate between stimuli in which 4 electrodes were stimulated simultaneously or were stimulated using pulse trains that were temporally phase-shifted with respect to each other.
It should be noted that in the experiments described here, most results were obtained using pulse patterns that were well above the critical flicker fusion (CFF) limit, the rate at which there is no conscious awareness of flicker. Measured CFF values in our two subjects (75% discrimination thresholds for stimulation on a single electrode) were 60 Hz and 40 Hz (see
Supplementary Material). Consistent with these measured CFF values, our subjects did not report flicker for any stimuli of 40 Hz or greater.
It is known for light stimuli that sensitivity to flicker depends on the intensity and size of stimuli and on their position in the receptive field. Our electrode arrays were positioned fairly centrally (including the fovea). The physical size of individual electrodes corresponded to approximately 1° to 2° of visual angle
23 ; however, the size of the percepts elicited by a given electrode seemed to be approximately twice that according to subject report,
16 possibly because of the spread of current activation. Our stimuli were presented at current levels of 119 to 470 μA (2–3 times threshold), an intensity that seemed bright, but not uncomfortably bright, to the subjects. Our finding of CFF limits of 40 to 60 Hz is, therefore, consistent with data on the light CFF for visually normal observers, which finds CFF limits of approximately 45 Hz for 2° to 4° stimuli presented centrally at asymptotic brightness levels.
24 For comparison, with large, bright peripheral stimuli, subjects have a maximum CFF of approximately 60 Hz. However, it is also worth noting that there is some evidence of cortical sensitivity to rates of flicker above the perceivable limit, as discussed below.
25
Results reported here are based on data from two subjects, S05 and S06, who underwent long-term implantation of 16-electrode retinal prostheses (Second Sight Medical Products, Inc.). The subjects were 59 and 55 years old, respectively, at implantation in 2004. Before surgery, subject S05 had bare light perception (BLP) in the implanted eye and had BLP for 8 years before implantation; subject S06 had no light perception (NLP) for 10 years before implantation. Without stimulation, subjects reported that their visual fields had a grayish background, and this perception of a gray background remained fairly consistent throughout the period of testing.
Collection of data reported here began several months after subjects underwent implantation with the prosthetic devices. These tests were carried out during a period of approximately 90 to 1170 days after implantation for S05 and 30 to 1110 days after implantation for S06.
These two subjects were a subset of six subjects who underwent implantation since February 2002. The other four subjects were excluded for a variety of reasons: one subject was excluded because of geographic location (this subject lived approximately 2600 miles from the testing site), and two subjects were excluded because of unrelated medical conditions. In one subject the array cable became exposed. The combination of a thin conjunctiva and an epithelialized cable meant that repositioning the cable would have required grafting of conjunctiva, sclera, or both over the cable site. Scarring from the previous surgery was likely to have reduced the effectiveness of local anesthetic, and the cardiac status of this subject precluded general anesthesia. As a consequence, the decision was made to cut the multiwire cable connecting the array to the external stimulator and to leave the intraocular portion of the array in place.
In some experiments, data were only collected in subject S06 because a surgical procedure was carried out on S05 in 2008 to adjust the extraocular cable component. This adjustment caused a slight lifting of the array from the retina, which resulted in a substantial increase in perceptual thresholds (in many cases, single-electrode thresholds could not be measured), making it impossible to continue data collection using the suprathreshold paradigms discussed in this article. Some of the brightness-matching experiments (experiment 6) could not be carried out in S06 because his geographic distance (60 miles from the testing center) limited his general availability for testing.
All tests were performed after obtaining informed consent under a protocol approved by the Institutional Review Board at the Keck School of Medicine at the University of Southern California and under the principles of the Declaration of Helsinki.
Stimulation Paradigm.
Same-Different Discrimination.
Experiments 1, 2, 3, and 5 measured performance using a two temporal interval same-different discrimination paradigm. In each trial, subjects were presented with two temporal intervals of stimulation. Each interval contained 1 of 2 pulse trains (A or B). The stimuli in the two intervals could be A and A, B and B, A and B, or B and A. The order of the A and B stimuli across the two intervals was randomized across trials, and each possible combination was presented with equal frequency. Subjects were asked to judge whether the two temporal intervals contained stimuli that were the same or different through a button-press response.
A and B stimuli consisted of suprathreshold pulse train stimulation across groups of four electrodes in a square configuration (
Fig. 2). The temporal properties of the pulse train presented on each electrode were identical in every way (pulse train frequency and pulse width) except for the phase-shift between pulses across electrodes. On any individual electrode, the pulse train presented was identical across synchronous (zero-phase shift across electrodes), pseudosynchronous (0.225-ms phase shifts across electrodes), or asynchronous (1.5- to 12-ms phase shifts across electrodes) stimuli. Stimuli for each experiment are described in further detail below. Subjects were instructed to use any visual cue to discriminate between the two stimulation patterns. Phosphenes on single electrodes were generally reported as round or oval and white or yellow. Shapes were reported as approximately 0.5 to 2 inches in diameter at arm's length, corresponding to roughly 1° to 3° of visual angle. When the percept was reported as oval, the longer axis was generally 2 to 3 times the length of the shorter axis.
When synchronous, pseudosynchronous, or asynchronous stimulation was presented on the 2 × 2 sets of electrodes, the percept was generally of a larger spot of relatively uniform brightness, which was reported to appear to be approximately 2 to 4 inches in diameter at arm's length, corresponding to roughly 3° to 6° of visual angle. The complexity of the stimulus was much greater than with single electrodes: the percept generally consisted of multiple phosphenes but did not generally contain phosphene patterns that aligned with the map of activated electrodes; in other words, the percept elicited by a 2 × 2 array of activated electrodes did not generally map neatly onto a 2 × 2 array of visual percepts in the expected location in space. Synchronous, pseudosynchronous, and asynchronous stimuli were generally perceived as spatially identical and differed only in perceived temporal properties (i.e., flicker) for pulse frequencies of 20 Hz.
Brightness Matching.
Experiment 3: The Effects of Interelectrode Distance—Pseudosynchronous versus Asynchronous Stimulation
As described, one possibility is that discrimination between pseudosynchronous and asynchronous stimulation might be mediated by differences in the response to the two stimulation patterns within individual neurons lying between two electrodes that receive direct stimulation from two (or more) electrodes. It is known that the intensity of electric fields decrease as a function of distance from the electrode.
34,35 For example, suppose electrode 1 was stimulated before electrode 2. A neuron lying closer to electrode 1 would be stimulated by a high-amplitude pulse, followed by a low-amplitude pulse, whereas a neuron lying closer to electrode 2 than to electrode 1 would be stimulated by a low-amplitude pulse followed by a high-amplitude pulse. If low-high versus high-low pulse pairs have a differential effect on driving cell activity, then these different pulse patterns might result in perceptually distinguishable responses even at rates well above the CFF. According to this model, it is only those neurons that happen to receive equal amplitude stimulation from a pair of electrodes (the location of these neurons will depend on the relative amplitudes of current on each electrode and factors such as the height of the electrodes from the retinal surface) that will be insensitive to the timing of stimulation across that electrode pair.
According to this model, differences between low-high versus high-low pulse pairs should also be distinguishable when presented on a single electrode. To test whether this was, in fact, the case, we carried out subjective brightness matching (on a single electrode) between a standard that consisted of pulse pairs of equal amplitude and test stimuli consisting of either low-high versus high-low pulse pairs. The standard pulse pair consisted of a pair of biphasic pulses of equal amplitude (304.8 μA for every electrode). The brightness of this standard was compared to the brightness of two test stimuli, a low- followed by high-amplitude biphasic pulse pair or a high- followed by a low-amplitude biphasic pulse pair. All pulses had 0.075-ms pulse width and a 0.075-delay or interpulse interval. At the start of the experiment, these test pulse pairs were set to have relatively the same total charge as the standard stimulus. The low-amplitude pulse had half the amplitude of the standard stimulus (151.2 μA), and the high-amplitude test pulse was set to have 1.5 times the charge of the standard stimulus (455.1 μA). We used a two-interval, forced-choice procedure, as described, to adjust the charge of the test stimuli based on subject responses. Increases or decreases in test stimuli amplitude were carried out on a logarithmic scale such that a step increase across the pulse pair would lead to an increase of 167.1 μA on the low-amplitude pulse and 503.1 μA on the high-amplitude pulse (an increase of 16 and 48 μA, respectively). Data were collected on three individual electrodes in S06.
As shown in
Figure 5, we found a difference in the charge needed to obtain a brightness match between high-low versus low-high pulse pairs. There was no significant difference in the amount of charge needed to match low-high pulse pairs to the standard containing pulse pairs of equal amplitude (two-tailed
t-test;
P > 0.05). In contrast, high-amplitude followed by low-amplitude pulses required significantly less charge (∼10% less) to appear as bright as the standard (two-tailed
t-test;
P < 0.05).
In a series of previous experiments we evaluated thresholds as a function of the temporal properties of stimulation for a wide variety of pulse trains.
16 Consistent with these previous findings, these results are consistent with the notion of rapid adaptation across pulses that seems to be proportional to charge accumulation. Such a mechanism is consistent with earlier experiments showing rapid adaptation effects in rabbit retina.
36
We show here that changes in spatiotemporal stimulation patterns well above the critical flicker fusion limit do affect perception: subjects can discriminate stimuli that are differentiated by phase shifts of 12 ms or less (corresponding to frequencies of 80 Hz or higher), even when electric field interactions are removed.
Experiment 1 showed that subjects could differentiate between synchronous and asynchronous stimulation patterns with high accuracy. In experiment 2, we found that subjects could still differentiate between pseudosynchronous and asynchronous stimulation, but there was a significant drop in performance. This drop in performance between experiments 1 and 2 suggests that there were significant electrical field interactions under the condition of simultaneous stimulation used in experiment 1. Experiment 6 further confirmed this result by demonstrating that synchronous stimulation results in brighter percepts than asynchronous stimulation. In our experiment, the asynchronous pulse pattern required nearly 20% more charge than the synchronous pattern to appear matched in brightness.
In experiments 2 to 6, we examined spatiotemporal interactions across electrodes once electric field interactions had been eliminated. Experiment 2 demonstrated that subjects could differentiate between pseudosynchronous and asynchronous stimulation; removing electrical field interactions was not sufficient to cause the percept elicited by a given electrode to be independent of stimulation by other electrodes.
One explanation is that these pulse patterns create local differences in brightness mediated by individual neurons that lie intermediate between electrodes. We carried out four experiments to further test this hypothesis. Spatiotemporal interactions decreased with electrode separation (experiment 3). Timing differences analogous to those tested in experiments 1 to 3 were perceptually distinguishable on a single electrode: a high-low pattern of stimulation resulted in a brighter percept than a low-high pattern of stimulation (experiment 4). In experiment 5 we demonstrated that subjects were able to distinguish clockwise from counterclockwise stimulation, and experiment 6 demonstrated that these judgments were unlikely to be based on overall (rather than local) differences in brightness between the two stimuli.
However, it is possible that neuronal lateral connections or cortical sensitivity to precise timing patterns across space may also play a role. Recent evidence suggests very fine temporal sensitivity within lateral connections mediated by wide-field amacrine cells. These connections can span up to many millimeters within the retina.
37,38 These connections, therefore, have many of the qualities required to mediate our subjects' ability to discriminate between patterns differentiated by extremely fine temporal information across relatively wide regions of space. The sensitivity to clockwise versus counterclockwise stimulation demonstrated in experiment 5 is harder to explain in terms of retinal lateral connections. However, it should be noted that though current levels were chosen to roughly brightness-match the percepts across each of the four electrodes in the group, this brightness matching was not perfect. Moreover, electrodes differ in their height from the retinal surface, which presumably means that the extent of current spread on the retinal surface is different across electrodes. Finally, it is likely that there are inhomogeneities in retinal wiring across the 3 mm covered by the electrode array. These inhomogeneities across electrodes and the retinal surface might conceivably produce perceptually distinguishable patterns for clockwise versus counterclockwise stimulation.
It is also possible that the ability to differentiate these patterns is mediated by cortical sensitivity to precise timing information. It is likely that our stimulation patterns created very precise spatiotemporal patterns of spiking activity in the retina. Stimulation using extremely short pulses (∼0.1 ms) results in precise single spikes within ganglion cells that are phase-locked to the pulses with a precision of <0.7 ms,
39,40 and presynaptic-driven spiking is abolished with stimulation frequencies above 10 Hz.
41,42 If precise timing information resulting from direct stimulation of ganglion cells is passed from retina to cortex, it is possible that the sensitivity to pulse timing across electrodes is the result of a cortical mechanism sensitive to spatiotemporal firing patterns originating in the retina.
There is some evidence that the cortex may be sensitive to very high temporal frequencies. For example, it has been suggested that the representation of objects and contours across the visual field is at least partially mediated by synchronous neuronal activity within those neurons representing the contour.
43–51 Synchronous firing at high temporal frequencies has been recorded for contour stimuli within both the retina
37,52–55 and the visual cortex.
50,56–60 However, it is still not known whether these synchronous firing patterns have functional importance.
Psychophysically, there is evidence that grouping of visual stimuli occurs based on temporal structure at frequencies up to approximately 35 Hz,
61,62 but it is possible that such stimuli still contain visible flicker,
63 motion information, or both.
64–70 Although it has not yet been clearly shown that frequencies beyond the CFF mediate grouping performance, there is some evidence of orientation specific (implying a cortical substrate) adaptation to temporal frequencies above the CFF.
25
Modeling percepts in visual prosthetic devices would be computationally simpler if it were possible to create spatiotemporally independent electrodes. However, the interactions described here (both between synchronous and nonsynchronous stimulation and between different patterns of nonsimultaneous stimulation) do offer the potential for significant perceptual flexibility. Simply by altering the order of stimulation, it is possible to create distinct percepts on a given set of electrodes.