Abstract
purpose. To evaluate the impact of converting from Humphrey 24-2 full-threshold (FT) visual field (VF) testing to SITA-Standard (SS) VF testing during the follow-up phase of a clinical trial.
methods. VF data were obtained from 243 patients in the Collaborative Initial Glaucoma Treatment Study (CIGTS) who had follow-up visits in 2004. FT and SS VF tests were performed in random order on the same day.
results. The average duration of the SS test (6.3 minutes) was shorter (P < 0.0001, paired t-test) than the FT test (11.8 minutes). The mean deviation did not differ between SS and FT testing. A small difference was found in the pattern SD (PSD) (P = 0.02). The mean CIGTS score from the FT test (4.5) was significantly lower (P < 0.0001) than the mean CIGTS score from the SS test (6.0). Although the two tests yielded identical Glaucoma Hemifield Test (GHT) results in 179 patients (76%), 16 patients had a normal GHT result on FT testing and an SS test result that was outside normal limits. Six patients had the reverse finding. The most significant factor associated with an increased (positive) difference between the CIGTS VF score generated from SS and FT testing was conducting the FT test first (P < 0.0001).
conclusions. Although SS and FT testing yielded very similar mean deviation results, the CIGTS VF score and GHT differed between SS and FT tests. Changing the approach used to measuring a study’s primary VF outcome should be accompanied by a critical evaluation of the change’s impact.
In the course of conducting long-term clinical trials, key methods used to assess outcomes may become obsolete, be difficult to maintain, or be updated over time. In the Collaborative Initial Glaucoma Treatment Study (CIGTS),
1 visual field (VF) change over time is the primary outcome. Its measurement at the study onset (in the fall of 1993) was made with the Humphrey 24-2 full-threshold (FT) test (Carl Zeiss Meditec, Dublin, CA)
2 and that testing approach was uniformly used at the study’s 14 clinical centers through December 2003. At that time, study investigators approved converting to VF testing by the Swedish Interactive Threshold Algorithm (SITA) Standard test.
3 This decision was based on evidence that the SITA strategy produces results similar to those of the FT test,
4 is sensitive and specific for detecting glaucomatous VF defects,
5 6 7 and reduced testing time substantially.
4 5 6 7 8 9 10 To evaluate the impact of this conversion for the study’s primary outcome, a protocol was instituted in the CIGTS to determine whether the SITA-Standard results yielded similar or different scores from those produced by the FT test.
A total of 245 CIGTS patients attended study follow-up visits between January 2004 and January 2005, the time frame in which the VF conversion protocol was under way. A center-specific randomization schedule was used to determine the order of VF testing of the patient’s study eye. The FT test and the SITA-Standard test were then obtained in the specified order. Recovery time was allowed between testing, with the amount dependent on the patient’s assessment of readiness. The CIGTS protocol required reliable tests, based on scoring of fixation losses (criterion of <33%), false-positive and -negative errors (criterion of <33%), and the short-term fluctuation (SF) value (criterion, ≤4.0 dB). Because the SITA-Standard test does not produce an SF value, its reliability was based on the first three parameters. In two patients, pupil dilation was induced between the two VF tests, as indicated by the intertest pupil diameter being 4 mm disparate. These two patients’ data are not included in this report, yielding n = 243 with comparable VF test results.
Both VF test results were scored according to the CIGTS VF scoring algorithm,
1 11 which assigns weights to points on the VF test’s total deviation probability plot according to the extent of departure from normal values, as expressed by point-specific probabilities, which are empirically derived percentiles from the distributions of values at each of the 52 points from age-specific sets of normal subjects collected by the manufacturer.
2 The proprietary distributions are built into the VF test software and are not available for inspection. The probability at each of the 52 points is reported as no defect or
P ≤ 0.05, ≤ 0.02, ≤ 0.01, or ≤ 0.005, meaning that the measured value at that point was at or below the respective percentile of the age-specific empiric distribution at that position in the field for normal subjects. A point is called defective if its probability is 0.05 or less and it has at least two neighboring points with probabilities of 0.05 or less in the same vertical hemifield (superior or inferior). A weight is assigned depending on the minimum depth of the defect at the given point and the two most defective neighboring points. A minimum defect of 0.05, 0.02, 0.01, or 0.005 is given a weight of 1, 2, 3, or 4, respectively. A point without two neighboring points all depressed to at least
P ≤ 0.05 is given a weight of 0. For example, a point at
P ≤ 0.01 with only two neighboring points of defect, both at
P ≤ 0.05, would receive a weight of 1. The weights for all 52 points in the field are summed, resulting in a value between 0 and 208 (52 × 4). The sum is then scaled to a range of 0 to 20 (by dividing by 10.4), resulting in a score that is a nearly continuous measure of VF loss. Other Humphrey VF test parameters that are common to both testing procedures—test duration, pupil diameter, mean deviation (MD), pattern SD (PSD), and Glaucoma Hemifield Test (GHT) result
12 —were recorded.
Comparisons between test results were made with paired Student’s
t-tests and scatterplots for continuous variables. For categorical variables, we used the McNemar test for dichotomous variables and the Bowker test for symmetry
13 for more than two categories. Factors predictive of differences in test results (SITA minus FT) were evaluated by linear regression. Data analyses were performed on computer (SAS, ver. 9.1; SAS, Cary, NC).
14
This research adhered to the tenets of the Declaration of Helsinki. All CIGTS patients gave written informed consent to participate, and the institutional review boards at the CIGTS clinical centers approved the study.
The 243 patients ranged in age at testing from 35 to 84 years, with a mean age of 66.2 ± 10.2 years (SD). Of those, 133 (55%) were men, and 88 (36%) were African American. Descriptive and comparative information on the patients’ VF test results are shown on
Table 1 . As expected, the SITA test’s average duration (6.3 minutes) was significantly shorter (
P < 0.0001) than the FT test’s (11.8 minutes). The MD did not significantly differ between SITA and FT testing (
P = 0.29), whereas a small difference (0.1 units) was seen between PSD results (
P = 0.02). The CIGTS VF score derived from these tests differed substantially (
P < 0.0001), with the mean CIGTS VF score from the FT test (4.5) 1.5 units lower (indicating a better VF) than the mean CIGTS VF score computed from the SITA-Standard test (6.0).
Results of the GHT are shown in
Table 2 . Although the two tests yielded identical GHT results in 179 (76%) patients, inspection of the off-diagonals shows 16 (7%) patients in whom the FT test’s GHT results were normal and the SITA-Standard test’s results were outside normal limits (ONL) and 6 (3%) in whom the opposite was found. These results were not related to test order. Of the 16, 8 took the FT test first, and of the 6, 2 took the FT test first. The SITA-Standard results yield glaucomatous VF findings in more patients; however, the extent of imbalance was not significant (
P = 0.14, Bowker test for symmetry). Even so, the extent of intertest agreement was at best moderate (weighted κ = 0.61), reflecting the 22 (9%) patients for whom a normal versus ONL result was found, and the additional 36 patients (15%) whose intertest GHT results were off by one category.
The scattergrams displayed in
Figures 1 2 and 3show the SITA data plotted against the FT data for MD, PSD, and CIGTS VF score, respectively. Close similarities and few large disparities were found for comparisons of SITA and FT data for the MD
(Fig. 1)and PSD
(Fig. 2)measures, whereas
Figure 3shows the striking disparity of the CIGTS VF score on the FT test versus the SITA-Standard test. The area defined within dotted lines in
Figure 3displays results that are within two CIGTS VF score units. Most of the results that differed by more than 2 units resulted from higher CIGTS VF scores based on the SITA test rather than on the FT test. As can be seen from the plot, the largest differences between the test scores occurred when the FT test was given before the SITA-Standard test. Bland-Altman plots
15 (results not shown) of the difference between tests (
y-axis) to the mean of the two tests (
x-axis) for the three outcomes showed no pattern in the variation of the intertest difference across the mean result for the MD or PSD measure, whereas floor effects were shown for the CIGTS VF score.
Factors associated with differences in testing results were evaluated by modeling the dependent variable—the difference in results between tests (SITA minus FT)—on subject factors (e.g., age, sex, race, or smoking status) and eye-specific factors (pupil diameter, test sequence, or severity of VF loss). No subject factors were associated with observed differences in test results.
Table 3shows the results of multiple linear regression models.
For the MD, two factors were associated with the SITA test’s showing more loss than the FT test: conducting the FT test first (
P < 0.0001) and the presence of less visual field loss (
P = 0.05). When the SITA test was performed first, the resultant MD was 0.44 units better than that of the FT test. If the FT test was performed first, the resultant MD from the SITA test was 0.47 units worse than that of the FT test—that is, no matter which test was conducted first, the MD resulting from the first test was better than that from the second test. The difference between these MDs, −0.91, is equal to the regression coefficient in
Table 3 . Patients with increasingly worse average MDs had SITA MDs that showed more loss than the FT MDs. This effect was more pronounced when the FT test was performed first versus when the SITA test was first.
For the PSD outcome, modeling the differences between FT and SITA testing showed significant associations with two factors: the mean PSD (P = 0.03) and the mean pupil diameter (P = 0.05). Higher PSDs and smaller pupils yielded larger positive PSD differences between SITA and FT VF tests.
For the CIGTS VF score, the most predictive factor associated with the SITA test’s showing more loss than the FT test was the order in which the tests were conducted (
P < 0.0001). Regardless of which VF test was first, the score was higher (worse) from the SITA than the FT test. When the SITA VF test was first, the resultant CIGTS VF score was 0.87 units higher than the FT test result; when the FT test was first, the resultant CIGTS VF score from the SITA test was 2.22 units higher than the FT test result. The difference between these two results, 1.35 score units, is equal to the regression coefficient in
Table 3 . Other significant factors that were associated with a SITA test’s showing more loss than the FT test (in CIGTS VF scores) included more VF loss (
P = 0.003) and a smaller mean pupil diameter (
P = 0.02).
The impact of observed differences in CIGTS VF scores from the two tests can be displayed in the frequency of triggering the intervention failure protocol
(Table 4) , which is based on detecting a 3-unit or greater increase (worsening) in the CIGTS VF score from the patient’s current reference CIGTS VF score (based on previous FT testing). If the SITA-derived CIGTS VF score was used, it would have triggered initiation of the intervention failure protocol (i.e., additional VF testing for verification) in 30 (12%) patients for whom the FT test would not have, whereas using the FT-derived score would trigger this protocol in only three patients (1%) for whom the SITA test would not have done so (
P < 0.0001, McNemar test).
In this randomized contrast of the FT and SITA approaches to measuring VFs in 243 patients with glaucoma, we found several results that are consistent with prior studies. One expected finding was the reduced test time for the SITA-Standard (6.3 minutes) versus the FT test (11.8 minutes). This 47% reduction in mean test time is essentially the same as that observed (48% reduction) by Sharma et al.
6 on 24-2 VF testing of both strategies, and similar reductions in test time have been observed when comparing the two tests by using 30-2 VF testing.
3 5 8 9 10 16 17
The fact that these two testing strategies yielded quite similar MDs and PSDs is noteworthy. Our average intertest differences in MDs and PSDs of 0.12 and 0.18 dB, respectively, indicate close agreement from a clinical perspective, as illustrated in
Figures 1 and 2 . In most prior comparative studies, MDs from SITA testing were slightly better than those from FT testing. Bengtsson and Heijl
18 conducted a study of 330 normal subjects that indicated a 1.6-dB higher (better) MD from SITA-Standard testing than from FT testing. Their evaluation of 44 patients with glaucoma and 21 normal subjects
19 found almost identical average MD results from SITA standard and FT testing of the patients, but noted that the number of significantly depressed points was higher in SITA testing than in FT testing. Heijl et al.
4 tested 31 patients with glaucoma and found that MDs with SITA were on average approximately 1 dB less severe than the 30-2 FT values. They concluded that the SITA test yields results similar to those of the FT test. Sharma et al.
6 reported that the MDs and PSDs from SITA and FT 24-2 testing correlated highly (
r = 0.92 and 0.93, respectively). In 82 patients with glaucoma, Budenz et al.
7 found better MDs derived from SITA-Standard testing than from FT testing, by 0.7 dB. Our lack of difference in MDs from the two testing approaches, although not substantially disparate from that found by others, may have been caused by differences in our patients’ distribution of VF loss, their relatively extensive experience with VF testing, or other factors.
There was a substantial difference between CIGTS VF scores produced from the FT test and from the SITA-Standard test. On average, SITA scores were higher (indicating more VF loss) than FT scores by 1.5 CIGTS VF score units. Test order came into play in this finding. Conducting the SITA test first and the FT test second resulted in less disagreement between CIGTS VF scores. Even so, when this test order was used, CIGTS VF scores were still higher when based on the SITA tests, and the amount of difference was still statistically significant. Characteristics that are unique to the computation of the CIGTS VF score may have contributed to the intertest difference. The CIGTS VF score is derived from weighting the probabilities in the total deviation probability plot. A specific point contributes to the score only if at least two neighboring points show some defect. We speculate that higher CIGTS VF scores from SITA testing result from a lower threshold for points in SITA testing to be declared depressed relative to the threshold needed to be passed for FT significance. Support for this possibility comes from the report of Bengtsson and Heijl
18 of the establishment of normal threshold limits for the SITA strategies. They found smaller intersubject variability in SITA-Standard testing (31% less) than in FT testing among 330 normal subjects who were tested, which resulted in normal limits for SITA that were “tightened” between 9% and 29%. Thereby, the statistical significance of a depressed point on SITA testing would be achieved more readily than on FT testing. Of course, this speculation can be critically evaluated only with knowledge of the distributions of normal values used by the SITA-Standard and FT testing software, which are proprietary.
The role of test order was important for both CIGTS VF scores and MDs. Conducting the longer FT test first may result in a fatigued patient taking the SITA test. A patient who is fatigued after taking the FT test may well fail to respond to less-intense test stimuli, thereby producing a SITA VF result that indicates more loss and a higher (worse) CIGTS VF score or a more negative MD result. The finding that more visual field loss was associated with greater differences between SITA and FT tests may relate to reported differences between SITA and FT test results at sensitivity losses of approximately 15 dB, wherein SITA results yield higher sensitivity estimates than do FT results.
20 As pupil diameter increased, two trends were observed in the VF results. First, regardless of the order of testing, the average CIGTS VF score was smaller (indicating less VF loss) with increasing pupil diameter. Second, differences between SITA and FT outcomes for CIGTS VF scores lessened with increasing pupil diameter, although SITA outcomes were consistently greater than FT outcomes across the range of pupil diameters. These results probably relate to a more reliable VF assessment when the pupil diameter is sufficiently wide to allow for optimal light exposure.
GHT results from the two tests yielded 22 instances wherein one test’s GHT was within normal limits (WNL) and the other test’s GHT results was ONL. An inspection of these VFs by two glaucoma specialists (PRL, RPM) found factors such as better defect characterization on SITA testing, probable artifact, questionable reliability, possible patient fatigue, and early defects at the edge of detection underlying these GHT differences. Such variation in GHT results has been reported previously by Sharma et al.,
6 who found variation to be more likely in normal control subjects and patients with suspected or mild glaucoma.
The decision to change VF testing strategies in our ongoing follow-up of patients with glaucoma compelled us to study its impact on the CIGTS VF score. This score is used in the CIGTS to monitor a patient’s VF status, and VF change is the primary outcome of the study. A 3-unit increase in the CIGTS VF score, if shown to be consistent on two repeated VF tests, requires advancement to the next treatment step (e.g., from topical medicine to argon laser trabeculoplasty). As
Table 4shows, reliance on the SITA result would have caused VF follow-up testing of 30 patients for whom FT test results would not have indicated such a follow-up. This far exceeds the three patients whom FT flagged for further VF testing and SITA did not. These SITA-versus-FT VF findings do not permit us to view the CIGTS VF score resulting from FT testing to be directly comparable to that obtained from SITA testing. Rather, we are establishing a new baseline for each patient on converting to SITA-Standard testing.
Our results show that comparability of SITA and FT testing for one VF outcome (MD) does not imply comparability on another VF outcome (CIGTS VF score). To be safe, then, a critical evaluation of the comparability of old and new test strategies should accompany change in the approach used to measuring a study’s primary VF outcome.