Decoding of Temporal Visual Information from Electrically Evoked Retinal Ganglion Cell Activities in Photoreceptor-Degenerated Retinas

Abstract

Purpose.: To restore visual function via the prosthetic stimulation of retina, visual information must be properly represented in the electrically evoked neural activity of retinal ganglion cells (RGCs). In this study, the RGC responses in photoreceptor-degenerated retinas were shown to actually encode temporal information on visual input when they were stimulated by biphasic pulse trains with amplitude modulation.

Methods.: Multiple RGC spike trains were recorded from rd1 mouse retinal patches mounted on planar microelectrode arrays while being stimulated by pulse trains with amplitudes modulated by the intensity variation of a natural scene. To reconstruct the time series of pulse train amplitudes from the evoked RGC activity, spike train decoding was performed. The accuracy of decoding—that is, the similarity between the original and decoded pulse amplitudes—was observed, to evaluate the appropriateness of the stimulation.

Results.: The response strengths of the RGCs could be successfully modulated when the pulse amplitude was varied between 2 and 20 μA. When the amplitude modulation range and pulse rates were determined elaborately, the temporal profile of the intensity could be successfully decoded from RGC spike trains, although abnormal oscillatory background rhythms (∼10 Hz) were consistently present in the rd1 spike activity.

Conclusions.: The results extend previous findings on the possibility of visual information encoding by electrical stimulation of normal retinas to stimulate degenerated retinas, in which neural activity is significantly altered. This supports the feasibility of encoding of temporal information by retinal prostheses.

Alarge percentage of retinal ganglion cells (RGCs), as well as the central visual system, is known to remain structurally intact after the loss of visual function due to degenerative retinal diseases such as retinitis pigmentosa and age-related macular degeneration. ^1,2 This supports active research on visual prosthetic devices, including retinal implants, aimed at the restoration of vision by electrical stimulation. ^3,4 Successful restoration of crude vision has been reported in human trials by several groups. ^{5 –8}  

For the successful restoration of visual function, neural responses evoked by electrical stimulation should accurately represent spatiotemporal information on visual input. We have recently shown that in the normal retina, this is feasible through amplitude modulation of pulse trains when the parameters of stimulation are carefully selected. ^9,10 However, it has been recognized that RGC activity is significantly altered in the degenerated retina due to changes in synaptic properties ^11,12 ; thus, abnormal rhythmic oscillation is present in both spontaneous and stimulated neural activity. Because this rhythmic oscillation does not originate from external visual inputs, it may disturb the representation and transmission of visual information by RGC activity in blind persons. Thus, visual information encoded by prosthetic electrical stimulation should be observed carefully in the degenerated retina. 

Quantitative investigation into information encoding by neuronal networks can be performed by spike train decoding, which consists of estimating (decoding) quantitative information encoded in neuronal activity. Typical applications include the study of fundamental visual information encoding characteristics of neuronal populations ^13,14 and the control of brain-machine interfaces. ¹⁵ In the present study, we sought to determine whether it is possible to decode the temporal information on visual input when RGCs in degenerated retina are electrically stimulated. To verify the feasibility of temporal information encoding by retinal prostheses, we tried to find stimulation strategies that enable successful temporal information transfer under confounding abnormal rhythmic oscillation in degenerated retinas ¹⁶ and to determine the dependency of encoding accuracy on stimulation parameters. Our strategy for the quantitative evaluation of stimulation strategies using spike train decoding is illustrated in Figure 1A. Temporal variation in light intensity in a pixel of visual input is transformed into amplitudes of biphasic current pulse trains. That is, amplitude modulation is used as a potential stimulation strategy for retinal prostheses. Because evoked RGC spikes should convey visual information from the eye to the brain, visual information should be properly represented or encoded in them. By calculating the goodness-of-fit between the original and decoded pulse amplitude time series, we quantitatively evaluated the effectiveness of the stimulation strategy. Various pulse amplitude modulation parameters, such as amplitude range and repetition rate, were explored to determine the accuracy of their representation of temporal information in visual inputs. Some of the experimental data were reported in our recent publication. ¹⁶  

Figure 1.

 

(A) Illustration of how spike train decoding can be adopted for quantitative evaluation of stimulation strategy. Temporal variation in light intensity of visual input is transformed to the amplitudes of biphasic current pulse trains. The RGC spikes evoked by stimulation should convey visual information from the eye to the brain, that is, visual information should be properly represented or encoded in them. By decoding the information from RGC spike trains and calculating the “goodness-of-fit” between the original and decoded pulse amplitude time series, the effectiveness of stimulation can be evaluated. (B) Modulation of pulse amplitude by light intensity derived from a natural scene (“walking man”). Black-and-white movies of natural scenes are recorded by a camcorder for ∼2 minutes at 30 frames/s with 1280 × 720 resolution and then converted to 2 × 2 resolution. The light intensity variation of one pixel is randomly selected, and the amplitude of each pulse is modulated according to the intensity time series within the range of 0 to 20 μA (1-μA resolution).

Materials and Methods

Retinal Tissue Preparation and Electrophysiological Recoding

Animal use protocols adhered to the ARVO Statement for the Use of Animals in Ophthalmic and Vision Research and approved by the institutional animal care committee of Chungbuk National University. The methods for retinal tissue preparation and for the recording of local field potentials and spike trains have been reported ¹⁶ and are briefly described here. Retinal patches from rd1 mice (C3H/HeJ strain) were prepared according to the method of Stett et al. ¹⁷ Retinas were isolated, cut into patches of ∼3 × 3 mm and then mounted onto a planar microelectrode array (MEA; Multichannel Systems GmbH, Rütlingen, Germany) so that the ganglion cell layer faced the MEA. The MEA contained 64 TiN electrodes (circular shape, diameters: 30 μm, interelectrode spacing: 200 μm, impedances: ∼50 kΩ at 1 kHz) on a glass substrate in an 8 × 8 grid layout. The four electrodes at the vertices were inactive. The waveforms from the electrodes were recorded with a sampling rate of 25 kHz/channel (amplification gain, 1200; bandwidth, 10–3000 Hz). Single-unit spiking activities were observed from 1199 RGCs in 25 retinal patches obtained from 24 mice in total. The average number of single-unit RGCs per patch was 47.96 ± 5.99. Of these, 532 RGCs (i.e., 21.28 ± 9.49 per patch) consistently showed evoked spikes and were further analyzed. 

Electrical Stimulation

Amplitude-modulated current pulse trains were generated by a stimulus generator (STG 1004; Multichannel Systems GmbH) and applied to the retina via one channel of the MEA at its center. The other channels were used for recording. The stimuli consisted of symmetric, charge-balanced biphasic pulses (anodic first, with no temporal separation between the anodic and cathodic phases with a 500-μs duration per phase). 

To investigate whether the RGC responses can be made to encode temporal visual information, the pulse amplitudes were modulated according to predetermined temporal patterns, assuming that intensity is converted into pulse amplitudes. Initially, triangular and sawtooth waveforms were used to test whether the RGC responses would follow the pulse amplitude time series as in Ryu et al. ⁹ Gaussian random waveforms and two types of time series of natural scenes (“walking man” [snapshots in Fig. 1B] and “woods” [not shown]) were adopted to test the accuracy of decoding. The procedure for the pulse amplitude modulations based on natural scenes is described below (Fig. 1B). Black-and-white movies of natural scenes were recorded by a camcorder (Xacti HD1010; Sanyo, Tokyo, Japan) for ∼2 minutes at 30 frames/s with a 1280 × 720 resolution. The resolution was reduced to 2 × 2 by downsampling. One pixel of the 2 × 2 resolution scene was randomly selected, and the amplitude of each pulse was modulated according to the time series of the pixel intensity within the range of 0 and 20 μA (1-μA resolution). The pulse repetition rate was varied from 1 to 10 Hz. 

Spike Train Decoding

Figure 2 illustrates the procedure of spike train decoding in detail. RGC spike trains were first transformed to a firing rate time series by counting the number of spikes in 50-ms bins. The delay block denoted a temporal delay of one time bin. The input vector of the input–output mapping block consisted of the past and present firing rates of multiple RGCs. The output was the pulse amplitude to be decoded. Thus, the spike train decoder estimated the pulse amplitude at every time bin from recent RGC firing rates. The extracellular recording from each electrode was first transformed into a single-unit spike train using spike-sorting software (Offline Sorter; Plexon Inc., Dallas, TX). An artifact removal technique based on average subtraction was applied to recover short latency (within ∼5 ms) spikes. ¹⁸  

Figure 2.

 

Block diagram of a spike train decoding procedure. RGC spike trains are first transformed to firing rate time series by counting the number of spikes in 50-ms bins. The input of the input–output mapping block consists of the past and present firing rates of multiple RGCs. The spike train decoder estimates the pulse amplitude at every time bin out of recent inputs; hence, the output is the pulse amplitude to be decoded.

The overall procedure can be regarded as temporal filtering because the spike train decoder calculated its output from the past and present values of the input. ¹⁹ When linear mapping is used, the decoding algorithm becomes an optimal linear finite impulse response (FIR) filter. ²⁰ The output of the decoding filter corresponds to the estimated value of the stimulus at the ith time bin, s(i). The decoding filter output, ŝ(i), can be written as a function of the firing rates of multiple spike trains as follows:  

 

where, ŝ(i) is the estimated value of the input stimulus at the ith time bin, and f_p (j) is the jth coefficient of the linear filter corresponding to the pth RGC; r_p (i) is the firing rate of the pth unit in the ith time bin of 50-ms duration; and φ(·) represents a general nonlinear function. This equation should approximate the inverse mapping of the neuronal encoding process. When linear mapping is used for the decoding, φ(·) is merely a unity function, and the coefficients f_p (j) were obtained by the least-squares method as illustrated by Warland et al. ¹⁴ More generally, nonlinear regression algorithms, such as multilayer perceptron or support vector machine (SVM), were used for the input–output mapping of the decoding algorithm. Here, we adopted the SVM for nonlinear decoding algorithm, because it is known to provide a good method of nonlinear regression in that it readily provides an optimized structure and parameters with excellent generalization performance. ^19,21 The procedure for finding the parameters of an SVM corresponds to a convex optimization problem, and thus, it is guaranteed that the global minimum can be achieved. The specific algorithm of the SVM that we used is described in Hoegaerts et al. ²²  

The recordings were divided into two groups, and one was used for the training of the decoding algorithms; the remaining recordings were used to test the trained decoding algorithms. To quantify the accuracy of the information decoded from RGC activity, similarities between the original and decoded pulse amplitude time series were computed by a coding fraction, which is defined as follows ²³ :  

 

where ε and σ_s denote the mean square error of estimating the original pulse amplitude time series by the decoded waveform and the SD of the pulse amplitude time series, respectively. 

Results

Spontaneous and Evoked RGC Activity

Characteristics of spontaneous and evoked RGC activity are reported in detail in our recent paper ¹⁶ and briefly described herein. As previously confirmed, ^11,12 aberrant oscillatory behavior at ∼10 Hz was persistent in the raw waveform (Fig. 3A). This behavior was manifested in both local field potentials (Fig. 3B, left) and spike activities (Fig. 3C, left). The spike trains consistently showed a temporal structure of rhythmic bursts of spikes, and the interspike interval histogram (ISIH) contained two distinct peaks (Fig. 3C, right). The first and second peaks of the ISIH reflect the interspike interval within a single burst of spikes and the interburst interval, respectively. 

Figure 3.

 

(A) A typical raw waveform of neural activity recorded from degenerated retina. (B) Field potential waveform (left) obtained from low-pass filtering with 20 Hz cutoff frequency and its power spectrum (right). Aberrant rhythmic oscillation at ∼10 Hz is manifested in the waveform and also indicated by an arrow in the power spectrum. (C) The spiking activity obtained by high-pass filtering with 100 Hz cutoff frequency (left) shows a temporal structure of rhythmic bursts of spikes, where the interburst interval of ∼0.1 second (corresponding to ∼10 Hz) is clearly observable in the interspike interval histogram (right).

Because of the persistent rhythm, the temporal structure of the evoked spikes revealed by a poststimulus time histogram (PSTH) was considerably different from that of normal RGCs. As Figure 4A shows, the PSTH included multiple peaks (typically, four or five) with interpeak intervals similar to the interburst interval (∼120 ms). Typically, the response strength, quantified by the number of evoked spikes within 100 ms (first peaks in the PSTHs), monotonically increased as a function of pulse amplitude up to a pulse amplitude of ∼20 μA and was saturated thereafter (Fig. 4B, 4C). We also found that the spikes in the first and second peak could be effectively modulated by pulse amplitudes when the pulse amplitude was varied within a 0- and 20-μA range (see Figs. 4C, 4D in Ryu et al. ¹⁶ for details). 

Figure 4.

 

(A) An example of spiking activity responding to a stimulation pulse (top trace) and corresponding poststimulus time histogram (PSTH, bottom trace). The PSTH showed multiple peaks (typically, four or five) with interpeak intervals similar to the interburst interval. (B) Changes in spiking activities according to the pulse amplitude. Typically, the response strength quantified by the number of evoked spikes within 100 ms (first peaks in the PSTHs) monotonically increased as a function of pulse amplitude up to ∼20 μA and was saturated thereafter. (C) Modulation of RGC spiking by pulse amplitude. RGC1 is the cell whose response characteristic is shown in (B). RGC1, -2, and -3 were from a single retinal patch. Each data point in the plots was obtained from 20 stimulation trials, and the error bar denotes the SE. Overall was obtained from all 15 RGCs in the same retinal patch.

Spike Train Decoding

The pulse amplitude time series could be successfully decoded from multiple RGC spike trains as in Figures 5A–C, which show the original and decoded pulse amplitude time series. (Detailed information on pulse rate and the number of RGCs used for spike train decoding is specified in the figure legend.) As shown in Figure 5D, more accurate decoding was possible when more RGCs were used, but the increase of decoding accuracy was saturated at ∼10 cells. 

Figure 5.

 

Decoding accuracy versus pulse amplitude modulation range. (A) Original and decoded pulse amplitude time series of Gaussian random waveform (top trace, 0–20 μA; bottom trace, 20–40 μA). Pulse rate was 4 Hz. Decoding was performed from simultaneous activities of seven RGCs. (B) Original and decoded pulse amplitude time series of a natural “woods” scene (top trace, 0–20 μA; bottom trace, 20–40 μA). The pulse rate was 8 Hz. Decoding was performed from the simultaneous activities of 20 RGCs. (C) Original and decoded pulse amplitude time series of a natural scene of a “walking man” (top: 0–20 μA, bottom: 20–40 μA). Pulse rate was 4 Hz. Decoding was performed from simultaneous activities of 45 RGCs. (D) Decoding accuracy versus the number of RGCs used for the decoding. (E) Comparison of the decoding accuracies for the two different pulse amplitude ranges, obtained from 14 groups of 10 RGCs. The decoding accuracy was significantly changed with respect to the pulse amplitude range (t-test: linear decoder, P < 1 × 10⁻¹¹; SVM decoder, P < 1 × 10⁻¹⁵).

The decoding accuracy was strongly dependent on the pulse amplitude range. For the band-limited Gaussian random waveform (Fig. 5A, top trace), accurate decoding of the pulse amplitude time series was obtained when it was modulated within a 0- to 20-μA range. For 20 to 40 μA, the decoding accuracy decreased (Fig. 5A, bottom trace). This decrease was also verified from the decoding of two natural scenes (Figs. 5B, 5C). These suggest that 0 to 20 μA is more appropriate for the range of amplitude modulations. The means and standard errors of the decoding accuracies were compared between the two different pulse amplitude ranges in Figure 5E. The difference in the decoding accuracies between the two amplitude modulation ranges was statistically significant (t-test, linear decoder, P < 1 × 10⁻¹¹; SVM decoder, P < 1 × 10⁻¹⁵). 

The difference in the decoding accuracies between different amplitude modulation ranges was substantial under various pulse rates. As shown in Figure 6, successful decoding was possible with various pulse rates, and the decoding accuracy tended to increase according to the pulse rate, remaining high for pulse rates of 6 to 10 Hz. It is also evident that the use of 0 to 20 μA and 8 Hz is optimal. 

Figure 6.

 

Decoding accuracy plotted as a function of pulse rate. Each data point was obtained from decoding of the natural scene “walking man,” from 14 groups of 10 RGCs.

Discussion

In this study, the RGC activity in photoreceptor-degenerated retinas effectively encoded the temporal information of a visual input when stimulated by a biphasic pulse train with amplitude modulation. Previously, we showed by spike train decoding that amplitude modulation can be used for the accurate encoding of temporal information in normal retinas. ¹⁰ Whether this is possible in the case of degenerated retina remains unclear: Several recent studies, including ours, ^11,12,16 have revealed significant alterations of neuronal activity in degenerated retinas due to the modification of synaptic properties, resulting in abnormal rhythmic oscillation. These alterations may compromise visual information representation by RGCs. However, we have shown here that it is still possible in degenerated retinas to successfully encode temporal information. Our results provide direct evidence that the temporal information from natural scenes can be accurately transformed to neural code by means of prosthetic electrical stimulation of photoreceptor-degenerated retinas. 

The decoding accuracy was strongly dependent on the details of stimulation. The range of amplitude modulation was critical for decoding accuracy, as it could also be predicted from the characteristics of the RGC responses under various pulse amplitudes (Fig. 4C in the present study and Ryu et al. ¹⁶ ). Despite the striking differences between behavior in normal and degenerated retina, in the degenerated retina, it was still possible to find a pulse amplitude range in which RGC firing rates showed monotonic and linear increases according to the pulse amplitude. Along with a recent study showing that the perceived brightness of phosphene increased as a function of current amplitude, ²⁴ the present results imply that temporal information about the light intensity of visual input may be successfully encoded in RGC spike trains by amplitude modulation pulses. Here, we did not discriminate between different RGC types such as ON and OFF cells since our study involves direct electrical stimulation of RGCs, not visual stimuli. However, considering subsequent stages of visual system such as visual cortex, differential results may be obtained according to the type of RGCs being stimulated. The decoding accuracy should be investigated while discriminating the RGC types in the future. 

As expected, this range (0–20 μA) was found to be optimal for the pulse amplitude modulation; the decoding accuracy was the highest when the pulse amplitude was modulated within this range. A maximum pulse amplitude of 20 μA is estimated to be within safety limits as the equivalent charge density, 0.014 mC/cm², is an order of magnitude smaller than the values used in human trials. ^5,7  

Based on the observation of RGC firing rate ¹⁶ and decoding results in the present study, we expect poor representation of visual intensity when the current level is increased too high, due to the response saturation. This is also supported by a recent study on a human subject with an implanted retinal prosthesis, ²⁴ which showed that phosphene intensity perception score was decreased when the current level was increased too high. However, we expect that the optimal range should vary significantly according to several factors such as the retinal cell density, material, and dimension of electrodes, degree of retinal degeneration. Accordingly, we do not claim that the current level used in human trials is inappropriate, simply because it is not same as the optimal range in our study. 

The aberrant oscillatory rhythm was consistently present in RGC neuronal activity, and its phase was reset by stimulation. This resulted in altered characteristics of evoked RGC spikes, as was readily observable from the PSTH showing multiple peaks (Fig. 4A). The poststimulus primary response peak was at ∼50 ms, and the spikes with this latency were judged to include the direct consequences of stimulation, along with the effect of phase resetting. ¹⁶ The neuronal activities corresponding to the first peak of PSTH could be efficiently modulated by the pulse amplitude. The secondary and later PSTH peaks decreased in amplitude over time, which is in agreement with the observation that the phase of field potential later dispersed. The secondary and later PSTH peaks seemed to be solely due to phase resetting and subsequent realignment of spontaneous spikes are thus byproducts of stimulation. Nonetheless, the firing rates of the spikes corresponding to secondary and later PSTH peaks could still be modulated by pulse amplitude and thus are expected to contribute to visual information representation. The decoding results demonstrate that, at a minimum, the decoding accuracy does not deteriorate. 

The pulse rate was also crucial for accurate visual information encoding. Considering that the natural scenes used in this study have significant frequency components up to ∼1 Hz, the minimum required sampling rate should be ∼2 Hz according to Nyquist's sampling theorem. The decoding accuracy was highest for the pulse rates of over 6 to 10 Hz (Fig. 6), which seems to be consistent with the reasoning based on the sampling theorem. The slight decrease in decoding accuracy at pulse rates higher than 8 Hz (Fig. 6) may be due to suppression by inhibitory presynaptic activity from the amacrine cell layer, which can last ∼100 ms and impede responses to subsequent pulses at high pulse rates. ^{25 –27} The frequency contents of natural scenes may become notably different from the two that we used in this study, and this may result in different optimal pulse rates, especially in the case of rapidly changing scenes. However, natural scenes are known to have 1/f ²-type power spectra, ²⁸ and thus significant information is contained in low-frequency bands. This causes correct representation of the low-frequency information more crucial for vision, and thus, accurate representation of the high-frequency components may not be urgent for the totally blind patients who would benefit from visual prosthetic devices. 

Here, we focused on the encoding of temporal information. However, meaningful visual perception by visual prostheses requires a certain degree of spatial resolution. Accordingly, the methodology proposed herein should be extended to the case of multichannel stimulation so that multipixel visual information can be translated into neural activity. The amplitude modulation range may be decreased in the case of multichannel stimulation—RGCs may be simultaneously activated by more than one stimulation electrode in close proximity. Different strategies for pulse parameter modulation such as frequency modulation ²⁷ or precise control of short-latency responses ²⁵ may be more adequate for multichannel stimulation. Our method based on spike train decoding can be readily applicable for the purpose of evaluating the effectiveness of different stimulation strategies in future studies. 

In this study, we assumed that reliable encoding at the transmission site (i.e., the retina) is a prerequisite for efficient information retrieval at the receiving site (i.e., the brain) because the retina is at the front end of the visual neural system, and RGC activity is the only source of information on the external visual world available to subsequent stages. Thus, spatiotemporal visual information should be conveyed by the RGC spike trains. If a certain amount of information about stimulus is contained in the decoded output, then we can regard that the spike trains themselves must contain at least as much information. ¹⁴ By decoding the visual intensity time series, we tried to quantify how well incoming visual stimuli are encoded in the RGC spike trains evoked by prosthetic stimulation, so that relevant regions in the brain receive visual information without significant loss and so that subsequent decoding by the brain is achievable. 

Encoding of specific features of a stimulus may be more critical than raw spatiotemporal information for the purpose of prosthetics. ²⁹ The decoding-based evaluation can be applied to the assessment of feature encoding of spatial patterns or movement as well, after defining critical features for restoring visual function by prosthetic stimulation. In this case, it may be necessary to decode the features of neural activity in relevant higher order visual areas in the brain. Insights into the effectiveness of prosthetic stimulation could be obtained by observing how visual information is transformed and preserved along visual pathways. 

Footnotes

 Supported by grants from the National Research Foundation of Korea (NRF) Basic Science Research Program (2011-0004089); and Happy Tech Program (2010-0020852), funded by the Ministry of Education, Science and Technology.

Footnotes

 Disclosure: S.B. Ryu, None; J.H. Ye, None; Y.S. Goo, None; C.H. Kim, None; K.H. Kim, None