Repeatability of Automated Perimetry: A Comparison between Standard Automated Perimetry with Stimulus Size III and V, Matrix, and Motion Perimetry

Abstract

purpose. Standard automated perimetry (SAP) shows a marked increase in variability in damaged areas of the visual field. This study was conducted to test the hypothesis that larger stimuli are associated with more uniform variability, by investigating the retest variability of four perimetry tests: standard automated perimetry size III (SAP III), with the SITA standard strategy; SAP size V (SAP V), with the full-threshold strategy; Matrix (FDT II), and Motion perimetry.

methods. One eye each of 120 patients with glaucoma was examined on the same day with these four perimetric tests and retested 1 to 8 weeks later. The decibel scales were adjusted to make the test’s scales numerically similar. Retest variability was examined by establishing the distributions of retest threshold estimates, for each threshold level observed at the first test. The 5th and 95th percentiles of the retest distribution were used as point-wise limits of retest variability. Regression analyses were performed to quantify the relationship between visual field sensitivity and variability.

results. With SAP III, the retest variability increased substantially with reducing sensitivity. Corresponding increases with SAP V, Matrix, and Motion perimetry were considerably smaller or absent. With SAP III, sensitivity explained 22% of the retest variability (r ²), whereas corresponding data for SAP V, Matrix, and Motion perimetry were 12%, 2%, and 2%, respectively.

conclusions. Variability of Matrix and Motion perimetry does not increase as substantially as that of SAP III in damaged areas of the visual field. Increased sampling with the larger stimuli of these techniques is the likely explanation for this finding. These properties may make these stimuli excellent candidates for early detection of visual field progression.

Perimetric variability occurs on many levels. First, it occurs on a neuronal level. Although action potentials have consistent amplitudes, with disease, the amplitudes decrease and the timing becomes unpredictable. ^{1 2} In a series of experiments, Tolhurst et al., ³ studying single cells in both cat and monkey, compared neural probability functions (multiplicative probability summation) with psychometric functions. They concluded that the slopes of the neural probability functions were shallower than those of the related psychometric function, indicating that psychometric functions were less variable than recordings of single neurons. This conclusion is not surprising, since psychophysical estimates reflect pooling over many receptive fields. 

With regard to the subject, there are several reasons that results of testing the sensory visual system and, in particular, of the visual field vary. Instructions can have a substantial effect. We have shown substantial variations in the results of conventional automated perimetry by instructing subjects in three generally accepted ways. ⁴ Also, visual attention can have a marked affect on thresholds. ⁵ Other cognitive factors are subject motivation and the effects of visual testing fatigue. ⁶ Although in individual subjects these factors can contribute greatly to the retest variability, for most subjects, the effects of these factors can be controlled. ^{7 8 9}  

Perimetric variability is divided into short-term (intratest) and long-term (intertest) variabilities. ^{10 11 12} The short-term variability or “fluctuation” (the variability of repeated measurements of the visual field within one test session) depends primarily on the slope of the frequency (probability) of seeing curve for that test location, and on the frequency of false-positive and -negative responses. 

In a study from our laboratory, the frequency of seeing curves of patients with glaucoma was generated by using a custom test program. The patients were tested with size I, III, and V stimuli in areas of normal sensitivity and areas of 10- to 20-dB loss. The same test locations were used in the same session for both sizes. As shown by a steepening of the slope of the frequency of seeing curve, variability substantially decreased in the patients with glaucoma with 10- to 20-dB loss as the stimulus size was increased. ¹³  

This high variability of standard automated perimetry (SAP) is very apparent in analysis of glaucoma clinical trials. For example, repeatability was a major issue in the Normal-Tension Glaucoma Study. ¹⁴ In another clinical trial, the Ocular Hypertension Treatment Study, almost 90% of the 703 retests failed to confirm defects found on earlier examinations. ¹⁵ The many studies of visual field progression in patients with glaucoma are a testament to the difficulties encountered due to high retest variability. 

Since a major problem of perimetry is this marked increase in variability with decreasing sensitivity, we investigated the retest variability of patients with glaucoma and normal subjects with four perimetry tests: standard automated perimetry size III (SAP III), SAP size V (SAP V), Matrix (FDT II), and Motion perimetry, to test the hypothesis that larger stimuli do not show the marked increase in variability seen in damaged areas of the visual field with SAP III. 

Methods

Subjects

The visual testing protocol was approved by the University of Iowa Institutional Review Board. Sixty ocularly normal subjects and 120 patients with glaucoma were tested at baseline and again at a separate sitting within 1 to 8 weeks. They all gave informed consent to participate in the study. The tenets of the Declaration of Helsinki were followed. The normals were recruits, paid in accordance with the Institutional Review Board, who answered advertisements inviting them to participate in research. The patients with glaucoma were invited from the glaucoma clinic at the University of Iowa Department of Ophthalmology and Visual Sciences if they met the entry criteria. They were enrolled if they had glaucomatous optic disc changes with abnormal standard automated perimetry (glaucomatous visual field defects i.e., three or more adjacent abnormal test locations in a clinically suspicious area at the P < 0.05 level or two adjacent locations abnormal with at least one at the P < 0.01 level. In addition, mean deviation was in the range of 0 to −20 dB on SAP). We included patients with primary, secondary, or normal-tension glaucoma. The patients did not have another disease affecting vision and were capable of performing SAP and returning for follow-up visits. Patients were excluded if they had cataract causing visual acuity worse than 20/30, pupil size less than 2.5 mm, age less than 19 years, or pregnancy. The first 120 consecutive patients who agreed to enter the study constituted the glaucoma cohort. If both eyes qualified for the study, one eye was chosen at random as the study eye. The average age (SD) of the normal subjects was 57.2 ± 7.9 year, with a range of 41 to 78 years. Thirty-eight of the participants were women and 22 were men. The average age of the patients with glaucoma was 64.9 years (SD, 9.5; range, 38–81); their mean deviation was −6.7 dB (SD, 4.4). Patients with glaucoma were free of other diseases that cause visual loss; if a cataract was present, the patient had a visual acuity of 20/30 or better. 

Normal participants were included if they had no history of eye disease, refractive error within ±5 D sphere and ±2 D astigmatism, no history of diabetes mellitus or systemic arterial hypertension, and a normal ophthalmic examination including 20/25 or better Snellen acuity. The subjects either had undergone a complete eye examination within 12 months before this study or were examined by an ophthalmologist on the day, of testing to ensure normal ocular health. One eye of each participant was randomly chosen as the study eye. 

Visual Testing

All subjects underwent automated perimetry using program 24-2 of the Humphrey Field Analyzer (HFA, Carl Zeiss Meditec, Dublin, CA). For SAP, the stimuli of Goldmann size III (0.43° diameter, 4 mm²) were used with the SITA standard 24-2 algorithm. Goldmann size V stimuli (1.72° diameter, 64 mm²) were used along with full-threshold testing; there is no SITA program currently available for size V stimuli. Our pilot data and the work of Artes et al. ¹⁶ show that the differences between estimates of the SITA and full-threshold strategies are minor compared to their test-retest variability. We chose size III SITA so that we would be comparing the test most commonly used in clinical practice. 

We followed the manufacturer’s recommendations for using corrective lenses. Care was taken to prevent lens rim artifacts. The subjects had testing in one eye, chosen at random, but the same eye was used for all tests. All visual field examinations met the following reliability criteria: fixation losses < 20% or normal gaze tracking, false-positive rate < 10%, and false-negative rate < 33%. The four tests were administered in a random order with at least a 5-minute rest break between the testing sessions. 

Motion Perimetry

Motion perimetry uses random dot cinematograms as visual stimuli. The dots are randomly displayed on a gray background with a luminance of 31.5 apostilbs, in a standard VGA video display: 640 × 480 pixels (Fig. 1) . The motion targets were circular random dot cinematograms within which 50% of the dots moved centrifugally and 50% moved in random directions. This stimulus is commonly used stimulus for motion perception experiments because the random placement of stimulus dots reduces the effect of positional cues. ¹⁷ Subjects were asked to respond by touching the monitor with a light pen in the area in which they had perceived the movement. 

The circular target itself was stationary; dots moved within the target. Details of the display can be found in previous publications. ^{12 18 19} The stimuli were of 20 sizes, with a scaling factor of 1.26. Subjects sit 22 cm from the CRT monitor. This 17-inch diagonal monitor gives a 21° test field (42° × 42° total). The angle subtended by the stimuli ranged from 0.25° to 8.46°. The size of the stimulus window varied from trial to trial, and a 2/1 staircase procedure was used to bracket the threshold. The test, therefore, continued until the smallest circle size seen (size threshold) at each test point was bracketed by the staircase procedure. We tested 44 locations which match the 24-2 Humphrey perimetry test points (Carl Zeiss Meditec, Inc.) except for absence of the top and bottom rows (y = 21° and −21°) and the two points along the nasal horizontal (x = 27°). Valid responses were defined by a response time between 110 and 1000 ms and a localization error smaller than 10° from the center of where the target was actually presented. 

Testing was done in a darkened room using an IBM compatible 486 computer with software we have developed. ¹² The patients’ appropriate near correction was again used. Care was taken to prevent lens rim artifact by asking if the subject could see each corner of the video display while looking at the fixation target. 

To facilitate the comparison between the four methods, the threshold values of Motion perimetry were transformed such that their numerical ranges were similar to those of the other tests. This does not imply that we were able to standardize the tests’ dynamic ranges. Motion perimetry measures a size threshold (18 steps) with the smallest size being 1 dB. We transformed the data by the following equation: 18 − observed threshold × 2. 

Matrix

Humphrey Matrix frequency doubling perimetry was performed either before or after conventional perimetry testing with at least a 5 to 10 minute rest period between examinations to reduce the effect of fatigue. Testing was performed in a dim room using the Humphrey Matrix device (Carl Zeiss Meditec, Inc.). The patients were asked to press a response button whenever they saw a small patch of alternating light and dark gray bars at any location within the field of view. Each test lasted approximately 5 to 6 minutes per eye. For this test, patients wore their own prescription glasses and did not use an eye patch to cover the fellow eye. ⁹ Rest breaks were allowed when requested. Details of the testing can be found in publications by Anderson et al. ^{20 21}  

Statistical Analysis

To examine retest variability with the four methods, we established the distributions of threshold estimates at retest, conditional on their value at the first test. Test locations with similar thresholds at the first test were collected together, and the threshold estimates obtained at the second test (retest) were examined. To summarize the retest distributions, we derived the empiric 5th and 95th percentiles (retest intervals) for all threshold levels observed at the first test. These intervals therefore show, for each threshold level, the range within which subsequent estimates from the same location are likely to fall with 90% probability, given that no real change has taken place. Test-retest intervals were established separately for patients with glaucoma and healthy control subjects. To test the hypothesis that retest variability is independent of sensitivity (no increase of variability with decreasing visual field sensitivity), we performed a Friedman ANOVA. For this, the differences between test and retest sessions at each test location were established and related to the mean of both measurements. 

Results

Table 1shows the age and visual field indices of the normal subjects and the patients with glaucoma. Figure 2shows the retest distributions of the patients with glaucoma obtained with the different tests at four levels of visual field sensitivity (0–2, 8–10, 18–22, and 28–32 dB). The first test is not meant to be the true sensitivity, as it has its own variability. With all four tests, there were many more test locations with high than with low sensitivity since most patients with glaucoma had early to moderate visual field damage (Table 1) . SAP V showed the lowest number of test locations with floor effects (0 dB thresholds) by a factor of 3. SAP III, Matrix, and Motion perimetry had similar numbers of absolute defects. Conspicuous in the plots of Matrix perimetry was the more discrete nature of the threshold estimates; the ZEST strategy estimates only 15 different threshold levels whereas 18 levels are obtained with Motion perimetry and up to 45 levels with the SAP tests. 

With all four tests, the shape of the distributions changed considerably throughout the range of stimulus intensities (Fig. 2) . With SAP III and V, the retest distributions were tight at high sensitivities (28–32 dB) but expanded considerably with decreasing visual field sensitivity. No such broadening was seen with Matrix and Motion perimetry. While most test locations with thresholds near 0 dB also had retest thresholds near 0 dB, the distributions of all four tests showed long positive tails. Test locations with initial values near 10 and 20 dB had retest values scattered throughout the entire range. Of interest, with SAP III, the most common retest value for initial thresholds of 10 dB was 0 dB. 

The 5th and 95th percentiles, for each level of sensitivity obtained at the first test, are shown in Figure 3 , for patients with glaucoma as well as healthy control subjects. The 90% range of thresholds seen at test and retest, indicated by heavy lines on the x- and y-axes, respectively, indicate that the stimulus scales were similar though not identical with the four tests. Because Matrix and Motion perimetry both measure fundamentally different psychophysical thresholds, it is problematic to compare the scales with each other. In the patients with glaucoma, the findings from these plots mirrored the large increase in retest variability, with SAP III and V already remarked on earlier. In comparing the retest intervals of SAP III and V, the latter showed a lesser increase in variability with decreasing sensitivity compared with SAP III. While the lower retest limit of SAP III fell to 0 dB for test locations with initial thresholds below approximately 15 dB; with SAP V this did not occur until the thresholds of the first test were reduced beyond 10 dB. It is interesting to note that the retest limits of all four tests were not symmetric around the 1:1 line. When the values of the first test were high, the retest intervals were shifted toward lower values and vice versa. This can be explained by a regression-to-the-mean effect that differs between patients with glaucoma and healthy control subjects. Low values at the first test are statistically more likely to be outliers that are followed by a higher estimate at retest. Similarly, high values are more likely to be followed by a lower value. Apart from this shift, the retest intervals were of similar width in patients with glaucoma and healthy control subjects. 

We examined the relationship between sensitivity and retest variability by Friedmann ANOVA. This nonparametric analysis takes into account only the relative ranking of the differences between test and retest rather than their numerical value, which is dependent on the arbitrary scaling of the tests. This analysis showed that sensitivity accounted for approximately 22% of retest variability with SAP III compared with 12%, 2%, and 2% with SAP V, Matrix, and Motion perimetry. When the analyses were performed excluding threshold estimates below 20 dB for SAP III and below 15 dB with SAP V, the explained variability in these tests decreases and becomes similar to that seen with Matrix and Motion perimetry. 

Discussion

We undertook this study to investigate whether the use of larger sized stimuli might improve the retest variability that interferes with determination of visual field change. The study is unique in that a large group of normal subjects and patients were tested, and each subject took all four tests on the same day. This allows more direct comparison of tests and methods than comparing different samples of subjects. 

Our study confirms many others that with SAP III, variability increases dramatically with a reduction in sensitivity. For example, Heijl et al. ^{22 23} investigated the variability in 51 eyes of 51 perimetrically experienced subjects with glaucoma representing all stages of optic nerve damage. The patients, all clinically stable, were tested four times in a 4-week period. Test locations, initially measured with a 6-dB loss had a 90% prediction interval from −1 to −16 dB. With an 8- to 18-dB loss initially, the 95% prediction interval nearly covered the full measurement range of the instrument (0–40 dB). An important finding of Heijl et al., also observed by others, ^{10 24 25 26 27 28 29} is that point-wise intertest variability increases dramatically with decreasing sensitivity of the test location. This finding has a major ramification: Areas with the most visual loss have the highest variability. Therefore, the most clinically important regions are ones in which determination of change is most difficult. 

Like Heijl et al., ^{22 23} we found this high retest variability for size III stimuli to begin at approximately 25 dB, and by 20 dB the limits extended over almost the entire measurement range of the instrument (Fig. 3) . The data suggest that the large retest variability renders a large part of the measurement range with SAP III ineffective. Inspection of Figures 2 and 3for retest results below 20 dB with size III suggests that testing in the range of 0 to 20 dB may have little practical benefit. We do not mean to imply that 20 dB is an absolute cutoff for the increase in variability. Whether one chooses 22, 20, or 15 dB, the value of single-threshold estimates of SAP III for predicting the values during the next examination is limited, because the retest variability extends over most of the measurement range of the instrument. 

SAP Size V testing gave results similar to SAP III testing except that the 5th percentile peak sensitivities were approximately 2 dB higher, and the retest distributions did not extend over most of the measurement range until thresholds of the first test were below approximately 15 dB. 

Motion perimetry and Matrix (FDT II) testing had considerably different results. With these two tests employing larger stimuli (Goldmann size III: 0.43°, Goldmann size V: 1.72°, Matrix: 4°, and Motion perimetry (range 0.1–8°), there was no clinically meaningful increase in variability with decreasing sensitivity. Artes et al. ³⁰ have shown a similar independence of sensitivity and variability with Matrix perimetry. 

Regression analyses showed that with SAP III, sensitivity explained 22% of the test-retest variability (r ²), whereas corresponding figures for SAP V, Matrix, and Motion perimetry were 12%, 2%, and 2%, respectively. As suggested, if the analyses are performed excluding threshold values below 20 dB with SAP III and below 15 dB with SAP V, the explained variances are similar. 

These findings have important implications for perimetry. As outlined earlier, there are many sources of variability, and only a limited number of approaches are available to reduce it. First, since our data suggest that, with SAP III, threshold estimates below 20 dB have little value for predicting the value at retest, one can make the case that for the purposes of detecting change, examination of such test locations could be eliminated. Whether a Bayesian rule is applied or whether two or three consecutive responses below 20 dB are used, an algorithm could be devised that could conclude with an acceptable level of certainty that the sensitivity is below 20 dB. Testing could cease at this point. Second, if a larger dynamic range is desired, a larger stimulus such as SAP V could be used. Last, since sensitivity accounts for about half the variability for SAP V compared with SAP III and use of FDT II or Motion perimetry is associated with minimal variability associated with sensitivity, it suggests use of these larger stimuli may have important advantages as perimetric stimuli. ³⁰  

Why do stimuli of larger size have lower variability? Three studies (Swanson WH, et al. IOVS 1996;37:ARVO Abstract 4937) ^{31 32} modeled the stimulus-response function of SAP based on the electrophysiological data of Croner et al. ³³ on macaque receptive fields and the anatomic data of Dacey. ³⁴ Garway-Heath et al. ³⁵ have estimated Goldmann stimulus size coverage of receptive fields based on a series of results of human ganglion cell histology. Assuming that the response is mediated primarily by the midget ganglion cells, Garway-Heath et al. ³⁵ estimated that a Goldmann size I stimulus covers two midget cell receptive fields, a size III stimulus covers approximately 20, and a size V stimulus approximately 400 receptive fields; these coverages vary slightly with eccentricity. 

Given these assumptions, a spatially irregular loss of ganglion cell receptive fields produces a poorer signal-to-noise ratio for small stimuli but has a much lesser effect on larger stimuli. In other words, for a stimulus to be seen it must activate a critical number of receptive fields (i.e., a population response is required). On retesting and the variability introduced by small eye movements, ³⁶ if less than this critical number of receptive fields is activated, it is less likely that the stimulus will be seen. This model predicts that in areas of visual field damage, if a larger stimulus is used, this larger stimulus will cover more receptive fields and will increase the margin of error due to small changes in stimulus position at each retest. Small stimuli may in effect fall into “holes” in the remaining “Swiss cheese-like” array of receptive fields on retest. Thus, larger stimuli should provide a more constant and improved signal-to-noise ratio and related decrease in retest variability. Whether this effect is best explained by coverage area or the length of the stimulus edge (since ganglion cell responses are highest around the edge of the stimulus where less of the receptive fields inhibitory surround is being stimulated) can be argued. 

This model is a qualitative one. As yet, there is no comprehensive and universally accepted quantitative model for light sensitivity perimetry. There are many reasons for this. With light sensitivity stimuli, the size of the stimulus is known but determining the effective stimulus size is problematic because of light scatter. The sensitivity to light stimuli is indirectly related to the number of functioning receptive field units (retinal ganglion cell receptive fields and the related cortical receptive fields). Another difficulty with these models is the irregular pattern of visual field damage in optic nerve disease and the resultant irregular receptive field array. Last, cortical mechanisms are important in any model of perception as shown in other studies. ^{31 32}  

A potential confound is the perimetric learning effect, ^{6 37 38 39} However, most of this effect takes place between the first and second test. Since most of our patients were trained perimetry subjects recruited from our glaucoma clinic, we suspect this effect was modest. In a related study, we have evaluated the learning effect of subjects taking the same four tests as in this study, but subjects took the four tests once a week for 5 weeks. We found the majority of learning between test one and test two and modest learning effects of less than one dB for the four tests (Brito CF, et al. IOVS 2007;48:ARVO E-Abstract 1625). Another problem is the possibility of fatigue effects, since the subjects took four perimetry tests in 1 day. ³⁹ To reduce this effect, the subjects were given frequent rest breaks and generous time between tests. No time limits were imposed. Since the test order was randomized, the fatigue effects present should be distributed equally among the tests. Also, this randomization should negate possible effects on the results caused by different preadaptation levels before the commencement of each test. 

Our entry criteria required that persons with glaucoma have a mean deviation on SAP III of between 0 and −20 dB. Subjects with endstage glaucoma were not included. This may have affected the SITA size III results, since it would affect the posterior probability distribution of true thresholds when the threshold estimate is near 0 dB. We believe there were enough end-stage hemifields in our analysis to make this issue minor. 

In summary, our results show that larger sized stimuli show more uniform variability in areas of visual field damage. A moderate reduction of variability and improvement of dynamic range can be accomplished by using size V stimuli. Also, the larger stimuli of Matrix and Motion perimetry provide more uniform variability which may result in earlier detection in visual field change. Studies are ongoing to test this hypothesis. 

 Supported by a VA Merit Review Grant, and by an unrestricted grant to the Department of Ophthalmology from Research to Prevent Blindness, New York, NY.

 Submitted for publication January 24, 2008; revised May 30, August 1, and September 12, 2008; accepted December 4, 2008.

 Disclosure: M. Wall, None; K.R. Woodward, None; C.K. Doyle, None; P.H. Artes, None

 The publication costs of this article were defrayed in part by page charge payment. This article must therefore be marked “advertisement” in accordance with 18 U.S.C. §1734 solely to indicate this fact.

 

Figure 1.

 

Motion perimetry video display. The small circle (left) represents a medium-sized random-dot Motion perimetry target. A magnified view of the target (right) shows that 50% of the dots are moving in random directions (○) and 50% are moving coherently (•). X, fixation target.

Table 1.

 

Age and Descriptive Indices of the Participants

	Age	MD	Pattern SD
Normal subjects
Mean	57.24	0.09	1.60
SD	7.85	0.94	0.39
Median	55.42	0.16	1.55
Minimum	41.05	−2.58	1.15
Maximum	78.19	2.34	3.40
Patients with glaucoma
Mean	64.99	−6.66	7.01
SD	9.48	4.10	3.25
Median	66.09	−5.37	6.63
Minimum	38.50	−19.93	1.87
Maximum	81.11	−0.53	13.85

Figure 2.

 

Distributions of retest thresholds for test values near 0, 10, 20, and 30 dB, with the four tests. Gray bar: the value of the baseline (test) level. Corresponding retest distributions are shown as histograms. The y-axis shows the proportion of retest estimates at each level. The gaps seen with Matrix perimetry are caused by the ZEST strategy, which provides only 15 discrete threshold levels. With Motion perimetry, there were 18 threshold levels; the gaps are related to the transform factor used (×2). The bar along the x-axis shows the 90% range of threshold estimates. Note the following number of 0-dB threshold estimates on the left graph column: size III, 600; size V, 215; Matrix , 593; Motion perimetry, 652, suggesting that that floor effects with the size V stimulus are less common than with the other three tests.

Figure 3.

 

Point-wise limits of test-retest variability established from the empiric 5th and 95th percentiles of the distribution of retest values, stratified by the value of the first test. Numbers and solid points indicate the location of the empiric 5th and 95th percentiles. The numbers give the number of test locations with thresholds at the given level at the first test. Because empiric percentiles are difficult to estimate precisely from small samples, the retest intervals were estimated by Lowess smoothing (shaded areas between the lines connecting the 5th and 95th percentiles). Individual data points are shown as scattered dots with slight jitter to improve visibility. Heavy lines on the x- and y-axis indicate the 90% range of threshold estimates as an approximation to the tests’ dynamic range. Note the lack of increased variability with decreasing sensitivity with Matrix and Motion perimetry.