Assessment of False Positives with the Humphrey Field Analyzer II Perimeter with the SITA Algorithm

Abstract

purpose. To evaluate the effects of false-positive (FP) response errors on mean deviation (MD), pattern standard deviation (PSD), glaucoma hemifield test (GHT), and test duration in the Humphrey Field Analyzer’s (HFA II) Swedish Interactive Threshold Algorithm (SITA; Carl Zeiss Meditec, Inc., Dublin, CA).

methods. Five individuals with glaucoma (ages 52, 63, 69, 77, and 78 years) and five individuals with normal, healthy eyes (ages 25, 34, 43, 45, and 52 years), participated in the study. Each subject was experienced in automated perimetry and performed multiple, monocular baseline SITA-standard (SITA-S) 24-2 visual field tests. In addition, normal subjects completed SITA-S 24-2 field examinations in which known frequencies of FP error were introduced (0%, 5%, 10%, 20%, or 33% frequency). Likewise, the subjects with glaucoma completed visual field examinations with 0%, 20%, and 33% error introduced during the test.

results. Reported FP errors were significantly lower than the introduced frequency of error. The SITA algorithm more accurately identified FP errors when the MD and PSD diverged from normal. Test duration increased as introduced error frequencies increased. The Statpac single-field analyses indicated that two thirds of the tests with introduced errors produced a “low-patient-reliability” determination.

conclusions. HFA II SITA-S underestimates patients’ FP errors, particularly among normal patients. High FP error frequencies can have adverse effects on MD and PSD, leading clinicians and researchers to an inaccurate determination of the amount and severity of visual field loss.

Visual field testing is one of the primary methods used for detection of glaucomatous functional abnormality. Reliability and repeatability of visual field testing are essential for accurate diagnosis and implementation of appropriate disease management regimens. Full-threshold (FT) white-on-white automated static perimetry has been the most common procedure for evaluating glaucomatous visual field loss, but recent advances have produced the development of several new test strategies, such as the Swedish Interactive Threshold Algorithm (SITA), ¹ that decrease test duration and reduce patient fatigue. ^{2 3} It is proposed that a less-fatigued patient should produce more reliable and repeatable visual field test results. 

Reliability affects the validity of an automated perimetric test, and it is therefore important to monitor patient responses during the test procedure. Three reliability indices are available within SITA: false positives, false negatives, and fixation losses. False positive errors occur if patients respond when no stimulus is presented. For the purposes of this study, we define a false positive response as randomly occurring, independent of stimulus presentation, and hence independent of any monitored response window. False-negative errors occur when the patient does not respond to a suprathreshold stimulus in an area where the threshold has already been measured. The interpretation of false-negative errors is not as clear as that of false-positive ones, because they can be produced by a variety of sources. Fixation losses occur when the patient’s eye wanders from the fixation target. 

The method of measuring these error rates has been to present “catch trials” during the visual field test. Catch trials allow the perimeter to estimate the patient’s overall false-positive errors, false-negative errors, and fixation losses during the test. ⁴ The Heijl-Krakau ⁵ method for monitoring the patient’s fixation includes a suprathreshold stimulus presented to the patient’s blind spot. If the patient is properly fixated on the central target, he or she should not be able to see the stimulus. Improper fixation may result in an inappropriate response to a blind-spot check, but false positives may also cause inappropriate responses to blind-spot checks. According to the manufacturer of the Humphrey Field Analyzer (HFA; Carl Zeiss Meditec, Dublin, CA), visual field test results of patients whose fixation losses exceed 20% or whose false-positive or false-negative errors exceed 33% are not considered reliable. ⁶  

Katz and Sommer ⁷ initially reported that results in 30% of normal control subjects and 45% of patients with glaucoma were unreliable by these criteria. They indicated that 41% of unreliable visual fields in patients with glaucoma and 67% of unreliable fields in normal patients were due to high fixation losses. However, if the proper precautions are taken, unreliable test results due to excessive fixation losses can be reduced to 14% by replotting the patients’ blind spots. ⁸ In addition, if the cutoff reliability criteria for fixation losses is modified to ≤33%, only 3% of visual field examinations fall into the “unreliable” category. ⁹  

Catch trials add to the duration of the visual field examination and may be affected by damaged visual field locations. ¹⁰ Swedish Interactive Threshold Algorithms (SITA), designed for the HFA II, were originally developed to improve patient testing reliability by reducing test duration. A portion of the reduction in test duration was due to eliminating false positive catch trials and estimating false positive response errors through the use of “listening windows.” ^{1 4} These windows are intervals between stimulus presentations when no patient response is anticipated. It has been reported that the minimum response time for a perimetric stimulus is approximately 180 ms. ¹¹ A response window is thus defined as the period beginning at the minimum response time, adjusted according to the patient’s individual mean response time. Olsson et al. ⁴ implemented the minimum response interval when designing their new system for evaluating response errors. The test monitors a window of time beginning immediately after the onset of a stimulus, continuing for 180 ms, and directly after a response window, continuing until the onset of the next stimulus. ⁴ An accurate, alert test subject should not respond during these times. Thus, any response during these epochs can be considered a false-positive error. Instead of reporting these errors in ratio form, a percentage of overall error throughout the entire test is reported. 

This new method does not require any additional questions during the testing process, reducing the duration of the examination. With SITA, fixation losses are still determined by using the catch trial method by presenting stimuli in the blind spot. The SITA procedures assume that patients respond at a consistent rate of error during the response windows and listening windows. ⁴ Although the listening window method substantially increases sampling time available for estimating response errors (an average of 15 times greater), the accuracy of the algorithm has not been reported. The purpose of this study was to evaluate the SITA-S testing strategy’s accuracy in reporting patient false positive error rates and to examine the effects of known frequencies of excessive false-positive responses on mean deviation (MD), pattern SD (PSD), glaucoma hemifield test (GHT), test duration, and reliability indices. 

Methods

Five eyes of five normal subjects with a mean age of 39.8 ±10.47 years (SD; range, 25– 52) with no known ocular disease or visual defect and five eyes of five patients with glaucoma with a mean age of 67.8 ±10.75 years (range, 52–78) were selected for this study. In accordance with the tenets of the Declaration of Helsinki, each subject gave voluntary, written consent to participate after being fully informed of the purposes of the study. The study was approved by Legacy Health System’s institutional review board for the protection of human subjects. All participants were experienced in automated perimetry and were required to demonstrate reliable baseline visual field results with few false-positive errors, false-negative errors, or fixation losses. 

Eligibility for normal subjects was based on three criteria (applied to both eyes): (1) normal visual fields, as determined by a within normal limits glaucoma hemifield test (GHT) result and P > 0.05 for PSD and MD; (2) intraocular pressure <21 mm Hg; and (3) normal optic disc appearance (determined by a previous full clinical eye examination). In addition to fulfilling these standard criteria, all the normal subjects had to have no history of systemic diseases or of taking medication known to affect vision. 

Inclusion requirements for patients with glaucoma were based on a previous clinical diagnosis of glaucoma, an outside normal limits result on the GHT, and the presence of glaucomatous optic neuropathy in one or both eyes. Participants with glaucoma were selected on the basis of the severity of the disease. Glaucomatous field loss of patients with primary open-angle glaucoma was used to classify each individual’s disease as mild (n = 2), moderate (n = 1), or severe (n = 2), according to the criteria of Hodapp et al. ¹²  

The Humphrey Field Analyzer II (HFA II) M750 (Carl Zeiss Meditec) was used to conduct 24-2 SITA-S visual field tests in all subjects. SITA strategies are adaptive among perimetry techniques, in that the algorithm constantly uses newly received data to recalculate thresholds throughout the test. 

Five nominal false-positive error frequencies were selected for evaluation in this study: 0%, 5%, 10%, 20%, and 33%. The perimetry test operator generated aperiodic, randomly spaced responses at a predetermined mean frequency by pressing the response button while the patients completed the tests as usual. Calculations for these erratic responses were based on the knowledge that the HFA II Full Threshold testing strategy presents a stimulus to the patient at a mean period of approximately once every 2 seconds. We calculated how often an erroneous response must be made to produce a false-positive reading (i.e., for 33% error frequency, an erroneous response should be introduced, on average, every 6 seconds). We then introduced random fluctuation to the length of this period to determine when each of these responses should be introduced. False-response events were calculated for each nominal error frequency and then randomly introduced during the testing. To reduce bias, patients were not informed of the error frequency chosen by the test operator for each test run and were instructed to respond normally, despite any responses generated by the perimetrist. Subjects and the perimetrist shared the same response button throughout the testing, and the response alarm on the HFA II apparatus remained intact. A total of 25 monocular SITA-S visual field examinations were conducted in each normal patient, five at each of the four predetermined error frequencies and five baselines with no introduced false responses. The latter served to ensure the testing reliability of each patient. Nine SITA-S visual field tests were conducted in the glaucoma subjects (three each at 0%, 20%, and 33% false-positive error frequency). All visual fields were obtained using the appropriate near refractive error correction for each patient. Testing sessions were held within a 1-month period and did not exceed 1 hour. A minimum 10-minute break was given between each test during a session. In addition, 40 SITA-S tests (10 at each of the 5%, 10%, 20%, and 33% error rates) were conducted without a subject present. These tests were conducted to serve as a reference comparison for the internal reporting accuracy of the software and can be used to simulate an eye with complete vision loss. 

Commercial software (Statistical Package for Social Sciences [SPSS] ver. 13.0, SPSS, Chicago, IL; SigmaPlot version 8.0, Systat Software Inc., Point Richmond, CA; and Prism 4, Graphpad Software Inc., San Diego, CA) was used to conduct statistical analyses and construct graphic representations. 

Results

Table 1presents the effect of introducing errors at different rates on the global parameters for the normal and glaucoma groups. 

False-Positive Responses

The mean baseline false-positive responses from the normal and glaucoma groups’ visual field tests were calculated at 0.40% ± 0.957% (SD; range, 0%–4%) and 0.93% ± 1.22% (range, 0%–3%), respectively. Given the low rate of patient-induced false positives, all analyses in this study assume zero patient error. However, all reported false positives reflect both erroneous patient responses and responses introduced by the perimetrist. Consequently, the disparity between induced error and reported error is potentially even greater than we report. 

Figure 1presents the reported false-positive error rate as a function of the nominal introduced false positives for the no-patient test conditions. The slope of the linear regression was 0.893 (i.e., the SITA algorithm tended to report 89.3% of the introduced false positives). This relationship was significantly shallower than a slope of 1 (F = 88.41, df = 39, P < 0.001, r ² = 0.893). 

For normal subjects and patients with glaucoma, the average false-positive rate reported was lower than the introduced error rate. Linear regression of the normal patient test results for false positives revealed slopes that were significantly different from unity (mean slope = 0.41, range, 0.37–0.58, P < 0.001 for all subjects). The R ² values ranged from 0.835 to 0.923 for normal subjects and from 0.887 to 0983 for glaucoma subjects (mean baseline [0% inserted error rate]) MD = −2.13, −2.27, and −4.94 dB). However, in glaucoma subjects 3 (mean baseline MD = −11.18 dB) and 5 (mean baseline MD = −10.79 dB), the slopes were not significantly different from 1. Slopes were significantly different from unity at the 1% error level for the three other patients with glaucoma (P < 0.001 for all). Figures 2 and 3present the relationship between reported false-positive errors and introduced errors for normal subjects and patients with glaucoma, respectively. 

Seventeen of the 75 SITA-S tests with introduced errors (20%, and 33%) conducted on normal patients were determined to have low patient reliability by the Statpac analysis. Two-thirds of the tests (20/30) with the same introduced errors for glaucoma subjects resulted in a low-patient-reliability reading. 

Fixation Losses

All gaze tracks were monitored to ensure that the subjects did not have fixation losses outside the acceptable range. On the baseline examinations, with no introduced false positives, the group mean fixation losses in normal subjects was 2.1% ± 5.19% (SD), whereas in patients with glaucoma the mean was 3.8% ± 4.51% (SD). With a 33% introduced error rate, the mean reported fixation losses in normal subjects was 27% ± 19.51% (SD) in comparison to a mean of 44.3% ± 24.0% (SD) for those with glaucoma. Figure 4depicts the reported percentage of fixation losses as the introduced error rate increased in normal subjects and those with glaucoma. 

Mean Deviation

It was observed that when baseline MD was lower, the slope of the reported versus introduced false-positive regression equations was higher. Figure 5shows that this slope continued to be high when no eye was present, analogous to an eye with complete vision loss. 

Slight change in MD from baseline measurements was noted in all normal subjects at each frequency of introduced error (Table 1) . However, the greatest changes in MD in normal subjects occurred at a 33% error frequency. The mean change in MD from baseline to 33% introduced error for the glaucoma subjects was 2.4 dB (range, 0.05–8.39 dB). High rates of false-positive responses affected some individual field points by as much as 26 dB. 

Pattern Standard Deviation

The PSD increased with introduced false-positive error rate in the normal group, indicating a more irregular field, as expected. In the glaucoma group, introduction of false-positive errors decreased the PSD, because genuine glaucomatous defects were blurred out by the false-positive responses. As with MD, the PSD increased with the false-positive rate when no eye was tested. 

Glaucoma Hemifield Test

With the 33% introduced error rate, only one of the normal patients’ GHT results was outside normal limits on any of the tests. The GHT reading in one of the normal patients indicated abnormally high sensitivity during a baseline test. Given that our patients with glaucoma were selected in part on the basis of their GHT results, all baseline results from these patients were outside normal limits. However, once the response errors were introduced for these subjects, the GHT changed to “borderline” in one instance. 

Test Duration

Mean test duration for normal subjects increased 19.7% (or 54 seconds) from mean baseline tests to 33% introduced error tests. For 5%, 10%, and 20% introduced error rates, the mean test durations increased an average of 1, 13, and 28 seconds, respectively. Similar increases were noted in the glaucoma group as well, with an increase of 31 seconds at 20% introduced error and an increase of 69 seconds (18%) at 33% introduced error. 

The relationship between test duration and introduced false positives appeared to be nonlinear. However, linear regression serves as a reasonable approximation for our data. Linear regression analysis revealed highly variable slopes among all patients for test duration, but they demonstrated reasonably linear relationships with these variables, indicating an increased testing time for patients with more severe visual field defects in the presence of elevated false-positive errors. 

Discussion

Excessive error rates during automated perimetry may indicate decreased reliability of the visual field examination. As demonstrated in this study, sensitivity values are significantly affected by increased false-positive responses. In some patients with severe glaucomatous visual field damage, MD improved from baseline by more than 6 dB when 33% false-positive errors were introduced during the examination. MD differences in normal patients were not as substantial, but did improve an average of 0.4 dB from baseline. Figure 6Ashows a baseline visual field, and Figure 6Bpresents the same patient’s visual field with 33% introduced error. It is important to note that by introducing false positives into a visual field examination, we may also have elevated the reported fixation losses and false negatives. Although a patient may maintain perfect fixation during the test, an introduced false positive during a fixation catch trial would register as a fixation loss. Similarly, introduced false positives increase the estimates of threshold so that when a false-negative check is performed later it may not be as suprathreshold as intended. 

Our findings indicate that there was a difference between the SITA-reported test reliability indices and the underlying amount of error. False-positive estimates may be the only indicator of how reliable a patient’s responses are during their entire clinical examination. The current Humphrey reliability criteria for false-positive and false-negative responses have been established at 33%, ¹³ and the Humphrey Field Analyzer displays an indicator (XX) on the screen and on the test printout when a patient exceeds 33% error at any point during the test. ⁶ Data from our investigation suggest the patient’s actual false-response rate is probably higher than this reported rate. 

A similar study on catch trials by Vingrys and Demirel ¹⁰ also suggested a need to reevaluate the acceptable level of reported error rates. They concluded that a more appropriate cutoff range for reliable visual field testing would be less than 20% false-positive errors. Our findings support their proposal. 

Results of a comparable investigation by Cascairo et al. ¹⁴ based on patient-generated catch trial errors are also consistent with our findings. Their study showed that all global indices and probability maps in normal, reliable patients were significantly altered from their baseline readings when reported false-positive errors reached 33%. 

One limitation of having the operator–introduced errors, as opposed to software–introduced errors, is that the error calculation is not as accurate. As previously stated, aperiodic responses introduced during a fixation catch trial may register as a fixation loss instead of a false-positive error. It has been reported that high fixation losses can result in poor detection of visual field loss as well as exacerbate threshold variability. ¹⁵ Attribution of some false positives to fixation losses in this investigation may account for some of the unreported errors. In addition, our response error frequencies were calculated assuming a stimulus presentation every 2 seconds on average. If a subject’s response time is longer than the average, the interval between stimuli will increase. The worst-case scenario is when the subject has complete vision loss or there is no patient present during the testing. In these situations, the interval between our introduced false positives may be too short to attain the desired false-positive percentage, because the interstimulus interval is automatically increased. Therefore the percentage of introduced false positives would be higher than assumed; the result of this is that the proportion of false-positive errors being reported would actually be worse than in our study. 

Of the three reliability indices, false positives are the least variable in test–retest studies and occur less frequently than either fixation losses or false negatives. ¹⁶ False positives may be the only good indictor of patient alertness and test reliability because high false-negative responses are associated with glaucomatous visual field damage ¹⁷ and fixation losses can frequently be reduced by replotting the patient’s blind spot early during the test procedure. Without an accurate false-positive report, it is difficult to establish which patients’ visual fields are truly indicative of their visual function and which should be retested. Clinicians and vision researchers should be wary of the reliability of a SITA-S 24-2 visual field reading if false-positive or false-negative error reports exceed 20%. In such instances, patients should be asked to retest, and the initial data should not be considered for determining a diagnosis or concluding that progression of field loss has occurred. 

Although the SITA software consistently underreports the actual false-positive error rate, high false-positive estimates in a normal subject are of greater concern than those in a patient with glaucoma. As indicated by the data from glaucoma subjects 3 and 5, our participants with the lowest baseline MD, the algorithm most accurately reports false-positive responses in patients with large field defects. Therefore, a false-positive response rate that borders on the permissible value for error in an otherwise normal eye is, in fact, much higher than the acceptable rate. Such a test should be considered unreliable. Further research, with a larger sample size, is necessary to firmly establish the accuracy of the SITA-S software in deriving and reporting response error estimates. 

 Submitted for publication December 15, 2005; revised May 10, June 5, and June 22, 2006; accepted August 22, 2006.

 Disclosure: M.R. Newkirk, None; S.K. Gardiner, None; S. Demirel, None; C.A. Johnson, None

 The publication costs of this article were defrayed in part by page charge payment. This article must therefore be marked “advertisement” in accordance with 18 U.S.C. §1734 solely to indicate this fact.

 

Table 1.

 

The Effect on Clinical Global Parameters of Introducing False Positives at Different Rates

A. Normal Subjects
Introduced Error	0% n = 5	5% n = 5	10% n = 5	20% n = 5	33% n = 5
Unreliable*	0.0%	0.4%	0.4%	20.0%	48.0%
GHT borderline	0.0%	0.0%	0.8%	0.8%	0.4%
GHT outside normal limits	0.0%	0.0%	0.0%	0.4%	0.4%
False positives	0.4%	2.0%	3.4%	6.2%	15.0%
False negatives	0.0%	0.1%	0.1%	0.4%	3.6%
Fixation losses	2.7%	5.9%	8.6%	18.8%	26.8%
MD	0.86 dB	0.66 dB	0.82 dB	0.91 dB	1.11 dB
PSD	1.37 dB	1.45 dB	1.48 dB	1.56 dB	1.75 dB

B. Glaucoma Subjects
Introduced Error	0% n = 5	20% n = 5	33% n = 5
Unreliable*	0.0%	0.0%	56%
GHT borderline	0.0%	0.4%	0.0%
GHT outside normal limits	100.0%	93.3%	100%
False Positives	0.9%	10.9%	21.6%
False Negatives	1.2%	5.7%	12.5%
Fixation Losses	3.4%	25.4%	44.3%
MD	−6.26 dB	−4.92 dB	−3.89 dB
PSD	6.96 dB	5.91 dB	1.40 dB

 The first three rows show the percentage of fields flagged in each of those categories, out of 25 tests in each column for the normal subjects and 15 tests in each column for the patients. The remaining rows show the global parameters averaged over all fields in the appropriate group at that error rate.

*  Tests results considered unreliable if the printout indicated “low patient reliability.”

Figure 1.

 

SITA-reported false-positive responses as a function of frequency of introduced false positive responses when no eye is present. Dotted line: best-fitting linear regression slope; dashed lines: 95% CI; solid line: slope = 1.0 (i.e., 100% reporting of introduced false positives).

Figure 2.

 

Normal subjects’ false-positive responses as a function of introduced error. Each line represents the linear regression results for individual patients, based on introduced error rates of 0%, 5%, 10%, 20%, and 33%. Solid line: slope of 1.0 (i.e., 100% reporting of introduced false positives).

Figure 3.

 

Glaucoma subjects’ false-positive responses as a function of introduced error. Each line represents the linear regression results for individual patients, based on introduced error rates of 0%, 20% and 33%. Solid line: slope of 1.0 (i.e., 100% reporting of introduced false positives).

Figure 4.

 

Fixation losses for given error frequencies of all normal subjects and patients with glaucoma. (•) Outliers.

Figure 5.

 

Regression slope of false-positive response versus frequency of introduced error (Figs. 2 3)as a function of baseline MD. (•) Patients with glaucoma; (○) normal subjects.

Figure 6.

 

(A) SITA-Standard 24-2 baseline visual field of a glaucoma participant. (B) Same participant’s visual field with 33% introduced error.