Interobserver Agreement for the Detection of Optical Coherence Tomography Features of Neovascular Age-Related Macular Degeneration

Abstract

Purpose.: To investigate the interobserver agreement for the detection of optical coherence tomography (OCT) features of disease activity in patients with neovascular age-related macular degeneration (nAMD).

Methods.: This was a cross-sectional agreement study in which grading of OCT line scans from patients with nAMD was conducted by two retinal specialists before the patients received treatment. Scans were graded for the presence of features of nAMD disease activity (intraretinal cysts [IRC], subretinal fluid [SRF], diffuse retinal edema [DRE], retinal pigment epithelial detachment [PED], and subretinal tissue [SRT]).

Results.: Although scans from 78 patients were available for analysis, five patients were excluded because of a mean signal strength of <7. Two hundred seventy-eight line scans were analyzed from 73 patients (40 with cross-hair scan sets and 33 with radial line scan sets). Agreement for per line scan analysis was 77% for IRC (κ = 0.41), 81% for SRF (κ = 0.62), 91% for macular fluid (κ = 0.28), 79% for DRE (κ = 0.10), 90% for PED (κ = 0.78), and 79% for SRT (κ = 0.53). Both observers disagreed regarding the presence of macular fluid in one patient (with a cross-hair scan).

Conclusions.: Interpretation of OCT line scans from patients with nAMD is subject to interobserver variability. However, when all line scans acquired are examined for the presence of fluid (IRC or SRF), there is a high level of agreement for the detection of macular fluid on a per patient basis.

The introduction of antiangiogenic agents into clinical practice has transformed the prognosis for patients with neovascular age-related macular degeneration (nAMD). The treatment paradigm used in the pivotal phase III trials with ranibizumab (Lucentis; Genentech, South San Francisco, CA) involved continuous monthly intravitreal injections for 24 months in the ANCHOR ¹ and MARINA ² trials. To reduce the burden of intravitreal injections and to optimize the risk/benefit profile, investigators subsequently explored an alternative pro re nata (prn) dosing strategy with three loading monthly doses and further injections, depending on the detection of features of disease activity. ³ Results from this open-label study suggest that this prn treatment paradigm can achieve similar efficacy outcomes if standardized retreatment criteria are used by investigators to guide further retreatment. These retreatment criteria include features of nAMD activity determined by optical coherence tomography (OCT) imaging, change in visual acuity, and appearance of new classic choroidal neovascularization (CNV) on fundus fluorescein angiography. More recent evidence from the HORIZON trial, ⁴ an open-label extension of the ANCHOR and MARINA trials using a prn retreatment strategy, suggests investigator-guided retreatment may not achieve outcomes comparable to those seen with continuous retreatment strategy in the third year of ranibizumab therapy. Although the clinician or trial investigator applies a complex Bayesian analysis of structural and functional information when establishing disease activity and, therefore, the need for retreatment in patients with nAMD, information from OCT imaging (detection of macular fluid and change in retinal thickness) forms a central part of the retreatment decision. Indeed, during clinical trials evaluating prn retreatment, in every case in which non-OCT-based observations (new subretinal hemorrhage or new classic CNV on fluorescein angiography) led to retreatment, there were qualitative features of CNV activity on OCT imaging. ³ Knowledge of the interobserver agreement for the detection of OCT features of CNV activity in a clinical setting is, hence, important in confirming the validity and reliability of retreatment decisions for patients with nAMD undergoing therapy in clinical trials, during which investigator-determined retreatment is used, and in clinical practice. Work by others ⁵ has evaluated the agreement for OCT features detected by trained OCT graders in a reading center setting. Previous work from our group has established the repeatability of OCT (StratusOCT; Carl Zeiss Meditec Inc., Dublin, CA)-based retinal thickness measurements in patients with nAMD. ⁶ To the best of our knowledge, this study is the first to investigate the interobserver agreement for OCT features of CNV activity in nAMD made by investigators in a clinical setting. In addition, results from this work may help assess the validity of OCT-based retreatment decisions and will be of use when developing clinically robust prn retreatment algorithms. 

Methods

All OCT imaging was performed using a commercially available OCT scanner (StratusOCT; Carl Zeiss Meditec Inc.) with software version 4.0. This is the third generation of OCT scanner and provides an axial resolution of <10 μm. The OCT scanner is serviced regularly in line with manufacturer recommendations by authorized technicians from Carl Zeiss Meditec Inc. to ensure that the scanner is calibrated and operating correctly. 

Patients

OCT scans of 78 eyes of 78 consecutive patients with subfoveal choroidal neovascularization (CNV) caused by AMD about to undergo treatment with anti-VEGF agents as part of a clinical trial evaluating the safety and efficacy of bevacizumab ⁷ (Avastin; Genentech, South San Francisco, CA) were analyzed by searching the OCT database at the Clinical Trials Unit at Moorfields Eye Hospital. In total, 130 patients were due to be enrolled in the double-masked randomized controlled trial; however, scans from only the first 78 patients were included in this retrospective analysis. Approval for the collection and analysis of OCT images was obtained from the Research Governance Committee of Moorfields Eye Hospital and by the Steering Committee of the clinical trial. The research followed the tenets of the Declaration of Helsinki. 

All patients in this study had subfoveal CNV caused by AMD in the study eye and had not received previous treatment for nAMD in the study eye. For each patient, only images from the eye about to undergo treatment were used in the analysis. 

The OCT scans analyzed in this study had been acquired between August 11, 2006, and July 11, 2007, using a single OCT (StratusOCT; Carl Zeiss Meditec Inc.) scanner. All patients underwent imaging performed after visual acuity measurement and pupil dilation with one drop of 2.5% phenylephrine hydrochloride and 1% tropicamide. All patients had given consent to OCT imaging as part of clinical trial involvement. 

Image Acquisition and Scanning Protocol

All OCT scanning was performed by a single technician certified by image reading centers for OCT scanning in pharmaceutical company-sponsored AMD clinical trials. For the purposes of this retrospective analysis, only scans with mean scan signal strength of 7 were included in the analysis. In the first 40 eyes included in the analysis, a cross-hair scanning protocol was used. This provides vertical and horizontal 6-mm line scans centered on the fovea. The radial line protocol was used to assess retinal morphology in the rest of the cohort because of a change in the OCT scanning protocol. The radial line scanning protocol uses six high-speed, 6-mm radial lines (oriented 60° apart) delineating macular anatomy and pathology. Both the radial line and cross-hair scanning protocols sample 512 points in the transverse axis. This provides better lateral resolution than the fast macular thickness map protocol, which only samples 128 points in the transverse plane and is more commonly used for retinal thickness measurement. At each of these locations, the signal is sampled axially at 1024 equal intervals over a depth of 2 mm. The advantage of the radial line scan protocol is that it provides six lines per scan set rather than the two lines acquired with cross-hair scans, effectively scanning more of the retina. 

Assessment of Interobserver Agreement in Determining Features of Disease Activity

Two experienced retinal specialists certified as investigators in pharmaceutical-sponsored studies involving OCT-based retreatment decisions (PJP and ACB) independently graded each line scan. Observers were asked to comment on the presence of intraretinal cysts (IRC), diffuse retinal edema (DRE), subretinal fluid (SRF), retinal pigment epithelial detachment (PED), and subretinal tissue (SRT) using standardized definitions (Table 1) from descriptions outlined by others. ⁸ These definitions were applied in a qualitative manner without reference to standard images. This was a deliberate effort to minimize standardization between the two observers, ensuring the results would be more translatable to clinical practice and investigator-determined retreatment decisions in AMD clinical trials. 

Table 1.

 

Definition of OCT Features of Neovascular Age-Related Macular Degeneration

OCT Feature	Description
IRC	Areas of low reflectivity in the intraretinal space
SRF	Area of low reflectivity in the subretinal space
MF	Presence of either IRC or SRF
DRE	Sponge-like thickening resulting in increased retinal thickness with areas of reduced retinal reflectivity compared with retina without thickening
PED	Areas of elevation or detachment of the retinal pigment epithelium
SRT	Thickening of the outer high-reflectance band

 DRE, diffuse retinal edema; IRC, intraretinal cysts; MF, macular fluid; PED, pigment epithelial detachment; SRF, subretinal fluid; SRT, subretinal tissue.

If a feature was present, the observer further commented on whether it was present in the central 1-mm zone of the scan, outside this zone, or in both zones, giving 4 categories of grading in total, including absence of the feature. The position of the A-scan and caliper measurement was used to define the limits of the central 1-mm zone in difficult cases (the total length of scan was 6 mm; calipers were used to define the midpoint of the scan line, and limits of the central 1-mm zone were taken as 0.5 mm either side of the midpoint). Scans were analyzed by viewing images on the OCT (Stratus; Carl Zeiss Meditec Inc.) station with no image processing. The prevalence of each feature (presence of the feature by both observers) was also reported. Paired gradings from the two observers were compared using cross-tabulations, percentages of agreement/disagreement, and the kappa statistic (κ, a measure of concordance adjusting for chance agreement). The κ statistic was interpreted in line with the ranges suggested by Landis and Koch ⁹ : <0, poor agreement; 0 to 0.20, slight agreement; 0.21 to 0.40, fair agreement; 0.41 to 0.60, moderate agreement; 0.61 to 0.80, substantial agreement; and >0.80, almost perfect agreement. For features with extremely low or high prevalence, the κ statistic is unstable and difficult to interpret. Analysis was performed using the original scale with four categories: feature absent, feature present outside central 1-mm zone, feature present inside 1-mm zone, feature present both inside and outside central 1-mm zone. In addition, these categories were recoded into feature absent or feature present for each line scan. Agreement was calculated for each of the five OCT features of CNV activity and for the feature of macular fluid (MF) by combining the IRC and SRF data fields. Analysis was performed on both a per line scan and a per patient basis by combining observations made on line scans from each patient. 

Results

Of the initial 78 patients with scans included in this analysis, scans from five were excluded because of a mean signal strength of <7. Paired gradings by two observers were performed on 278 line scans from 73 eyes of 73 patients with nAMD attending for treatment. The mean age (±SD) was 79 years (±6 years), and the range was 64 to 92 years. There were 43 women, 30 men, and 39 left eyes. Seventy-one patients were Caucasian. Forty patients had cross-hair scans, and 33 patients had radial line scans. All line scans had one or more feature of CNV activity detected by both graders. The prevalence of features of CNV activity detected by both graders is shown in Table 2. Figure 1 shows examples of features of CNV activity for which there was agreement between the two observers. Figure 2 provides examples of more problematic cases, and Figure 3 shows an example in which there was disagreement regarding the presence of MF on a cross-hair OCT scan for a single patient. 

Table 2.

 

Prevalence of OCT Features of CNV Activity

Feature	Line Scans in which Feature Detected by Both Graders	Patient Scans in which Feature Detected by Both Graders
		Cross-Hair Scans (N = 40)	Radial Line Scans (N = 33)	Total (N = 73)
IRC	172 (62)	30 (75)	31 (94)	61 (84)
SRF	136 (49)	23 (58)	26 (45)	49 (67)
MF	247 (89)	37 (93)	33 (100)	70 (96)
DRE	214 (77)	34 (85)	30 (91)	64 (88)
PED	91 (33)	18 (45)	15 (45)	33 (45)
SRT	158 (57)	28 (70)	26 (79)	54 (74)

 Data are presented as n (%). DRE, diffuse retinal edema; IRC, intraretinal cysts; MF, macular fluid; PED, pigment epithelial detachment; SRF, subretinal fluid; SRT, subretinal tissue.

Figure 1.

 

Examples of OCT features of neovascular AMD with good interobserver agreement. (A) RPE detachment (arrow) with an area of subretinal fluid (arrowheads). (B) IRCs (arrowheads) with a small area of subretinal fluid (arrow). (C) Diffuse retinal edema with a RPE detachment (arrow). (D) Subretinal tissue (arrow) with an adjacent area of diffuse retinal edema (arrowheads).

Figure 2.

 

Examples of problematic cases: (A–C) line scans with IRCs but with areas of possible subretinal fluid (arrows). (D) Small area of intraretinal cysts (arrow).

Figure 3.

 

Cross-hair line scans. (A) Horizontal line scan and (B) vertical line scan in which there was disagreement regarding the presence of macular fluid. One observer detected IRCs in line scan (B) (see insets and arrow). No fluid was detected by either grader in the horizontal line scan

Per Line Scan Analysis

Table 3 illustrates the percentage agreement and κ statistic for each feature of CNV activity. This is expressed for the raw data (using the four categories based on the presence or absence of the feature in the central 1-mm zone of the line scan) and after reclassifying the grading to a dichotomous scale (presence or absence of each feature). 

Table 3.

 

Percentage Agreement and Kappa (κ) Statistic for Each Feature of CNV Activity Expressed for Raw Data and after Reclassifying the Grading to a Dichotomous Scale with Analysis on a Per Line Scan Basis

Feature	Agreement for Grading Based on 4 Categories* (N = 278)		Agreement for Grading Based on Presence or Absence of Feature on Line Scan (N = 278)
	%	κ	%	κ
IRC	62	0.43	77	0.41
SRF	71	0.56	81	0.62
MF	72	0.50	91	0.28
DRE	61	0.05	79	0.10
PED	88	0.76	90	0.78
SRT	72	0.50	79	0.53

 Abbreviations are as defined in Table 2.

 *Feature absent or present outside central 1 mm, inside central 1 mm, or present in both zones.

Per Patient Analysis

Table 4 illustrates the percentage agreement for each feature of CNV activity when the grading data from the two observers are recoded to determine the presence of absence of each feature of CNV feature on OCT scan sets (all six radial line scans considered together or two line scans for cross-hair scans) for each of the 73 patients. 

Table 4.

 

Percentage Agreement for Each Feature of CNV Activity when Data from All Lines Scans Are Pooled for Each Eye (Per Patient Analysis)

Feature	Percentage Agreement for Grading Based on Presence or Absence of Feature in Scan Set
	Cross-Hair Scans (N = 40)	Radial Line Scans (N = 33)	Total (N = 73)
IRC	80	94	86
SRF	88	91	89
MF	98	100	99
DRE	85	91	88
PED	83	91	86
SRT	90	79	85

 Abbreviations are as defined in Table 2.

Discussion

To our knowledge, this study is the first to estimate interobserver agreement in the detection of OCT features of CNV activity in a clinical setting. The introduction of new antiangiogenic therapies for nAMD has led to a vastly improved prognosis for patients with nAMD. The initial phase III clinical trials applied a continuous therapy paradigm, ^1,2 but, more recently, investigators have been attempting to attain the efficacy achieved in these trials using fewer treatments to improve the risk-benefit profile of therapy. ³ This has been done using prn retreatment strategies, which rely heavily on OCT-based retreatment criteria. The appearance of macular fluid (either IRC or SRF) is one of the important triggers for retreatment; however, the interpretation of OCT line scans may be subject to interobserver variability. Variability in OCT image interpretation may lead to interobserver variability in ranibizumab retreatment decisions and may compromise outcomes achieved using this type of retreatment strategy. The aim of this work was to investigate the interobserver variability in the detection of OCT features of disease activity in a clinical setting. 

Previous work has concentrated on establishing interobserver agreement for observations made by reading center-based OCT readers. ⁵ This has relevance for interventional nAMD clinical trials, which use reading center-based OCT outcomes but is not directly translatable to observations made by clinicians when determining retreatment in clinical practice or in investigator-determined retreatment in clinical trials. The study by Zhang et al. ⁵ investigated the interreader and intrareader agreement for OCT features in scans read at a reading center. This study used experienced reading center-based graders who read OCT images taken as part of an interventional nAMD clinical trial. They used the fast macular thickness map scan (with six low-resolution scan lines sampling 128 transverse points over 6 mm) and a 7-mm horizontal line scan offset by 5° starting at the optic disc and bisecting the fovea to detect features of CNV activity. The 7-mm line scan is used for reading center analysis but is not the commonly used scanning protocol in clinical practice or for investigator-determined retreatment in clinical trials. In both these clinical settings, radial line or cross-hair scanning protocols are used to assess qualitative features of CNV activity. Scan protocols using a single line scan are exquisitely sensitive to correct positioning of the line scan and will detect only abnormalities that lie on the path of the single scan line. A more common approach in clinical practice with OCT (StratusOCT; Carl Zeiss Meditec Inc.) is to use the cross-hair or radial line scan protocols, which sample between two and six lines per scan. This provides better opportunity to image abnormalities lying in the macular area but in an off-center location not affecting the center of the fovea. Indeed, for the detection of disease recurrence in patients with treated quiescent CNV, the first sign may be SRF in a juxtafoveal or an extrafoveal location that is lying on a meridian not detectable by a single 7-mm, 5° off-set line scan but that would be detected by a cross-hair or radial line scan. Although the study by Zhang et al., included intergrader agreement values for the six line scans which constitute the fast macular thickness map scan mode, this scan protocol is more commonly used to measure retinal thickness rather than to detect qualitative features of macular pathology because only 128 points are sampled per 6-mm line scan compared with 512 points per line scan for the radial line or cross-hair scan protocols. 

The prevalence of abnormalities in line scans analyzed in the study by Zhang et al. was similar to our study allowing a comparison of the two studies. However, it is important to remember that the study by Zhang et al. was in a reading center setting, whereas our study was in a clinical setting using retinal specialists accredited as investigators in pharmaceutical company-sponsored trials for nAMD but with no reading center experience. In addition, the precise definition and classification of OCT abnormalities might have varied between the two studies, as did the scanning protocols. 

Zhang et al. found a high rate of agreement for the detection of features of AMD with percentage agreement ranging from 85% to 96% for the single 7-mm line scan. The results from our study suggest more variability when experienced observers are used to interpret OCT images in a clinical setting, with concordance ranging from 62% to 91% in a per line scan analysis. However, when the findings from all line scans (two for cross-hair scans and six for radial line scans) are pooled, agreement is very high (concordance range, 86%–99%). The better agreement for the per patient analysis when compared to the per line scan analysis suggests that in a clinical setting, all line scans acquired from an eye undergoing therapy must be analyzed for the presence of features of CNV activity to minimize the impact of interobserver variability when making retreatment decisions. Although this recommendation agrees with the findings in this study, it may not be clinically feasible for clinicians to carefully analyze all six scans for every patient, especially in a busy clinical setting. A refinement to our gold standard practice recommendation of examination of each line scan would be to review all lines scans only when the initial printout appears to be normal and the retinal specialist is considering dose-withholding ranibizumab. Of note, an estimate of the time taken in this study to analyze a six-line scan set was 6 minutes (1 minute per line scan). 

Agreement for IRC was 92% in the study by Zhang et al. but only 77% (κ = 0.41; fair agreement) for the per line scan analysis to 86% for the per patient analysis in our study. For the important feature of SRF, interobserver percentage agreement was 91% in the study by Zhang et al. and ranged from 81% (κ = 0.62; substantial agreement) for the per line scan analysis to 89% for the per patient analysis in our study. There was excellent agreement for the detection of MF in our study, with interobserver agreement ranging from 91% (κ = 0.28) for the per line scan analysis to 99% for the per patient analysis. This suggests that when fluid was detected on a line scan, there was interobserver variability in classification as IRC or SRF. This potential for interobserver variability may reflect the low resolving power of OCT (Stratus; Carl Zeiss Meditec Inc.) and, in some cases, the hyporeflectivity of fluid is difficult to define with certainty. 

We encountered several examples of discrepancies between high percentage agreement values with low κ values because of the dependence of the κ statistic on the prevalence of the attribute being measured. A very high (or very low) prevalence results in a very high level of expected agreement and the κ statistic becomes very unstable. ^10,11 In the case of MF, there was a very high prevalence of this feature because all scans were taken from patients with active nAMD, leading to instability in the κ statistic. 

The results we present are important for clinical practice and for retreatment decisions made as part of clinical trials involving investigator-determined retreatment. The results suggest that there is excellent interobserver agreement for the detection of macular fluid on a set of OCT line scans acquired from one eye of a patient with nAMD about to undergo therapy (99% concordance in this study). However, there is more variability when determining whether the fluid is IRC or SRF. There is also variability in determining whether the fluid occurs in the central 1-mm zone of the scan. This could occur because observers differ in judgment about whether an identical feature is present in the central 1-mm zone or it could be attributed to observers identifying two different areas on a single line scan as constituting pathology. When the findings from individual line scans are collated, there is good agreement for the detection of the features studied (concordance ranging from 85% for SRT to 99% for macular fluid). One had a cross-hair scan, and the observers disagreed regarding the detection of macular fluid (see Fig. 3). This suggests that it is desirable to acquire more than two line scans per eye when scanning patients with nAMD to minimize the effect of interobserver variability in the detection of macular fluid on retreatment with antiangiogenic agents; further studies are needed to investigate this. Furthermore, improvement in the interobserver agreement when findings from each line scan are pooled for eachpatient and each eye shows the value of examining all line scans acquired (two lines for cross-hair scans and six for the radial line scan protocol) and not relying on the analysis of a single line scan. It is unclear whether the new technology of spectral-domain OCT with faster scan acquisition ¹² will lead to improved interobserver agreement. It includes as many as 128 individual line scans per scan set available for analysis, though the improved resolving power over time-domain imaging may lead to better interobserver agreement when distinguishing between IRC and SRF. 

The high level of interobserver agreement in the detection of macular fluid on a per patient basis in this study suggests that interobserver variability in OCT interpretation is unlikely to account for the compromised efficacy outcomes seen in the latest prn ranibizumab retreatment trial. ⁴  

The observations made in this study were undertaken by two experienced retinal specialists who are accredited as investigators for pharmaceutical company-sponsored AMD clinical trials but without formal teaching before this study. This is a strength of the study because previous training might have improved interobserver agreement but would not have reflected real clinical practice, or indeed investigator-dependent retreatment decisions, in clinical trials. In view of the potential for disagreement between observers, it may be prudent to provide investigators with illustrative examples of OCT-based retreatment decisions or to use online training to ensure standardization of decisions for borderline cases in clinical trials. 

In summary, this study investigates the interobserver agreement for the detection of OCT features of disease activity in patients with nAMD undergoing anti-VEGF therapy. The findings confirm that interpretation of OCT line scans from patients with nAMD undergoing therapy is subject to interobserver variability in a clinical setting. However, in this cohort, when all line scans acquired are examined for the presence of fluid (either IRC or SRF), there is a high level of agreement for the detection of macular fluid and other features of disease activity on a per patient basis. 

Footnotes

 Disclosure: P.J. Patel, None; A.C. Browning, None; F.K. Chen, None; L. Da Cruz, None; A. Tufail, None

Footnotes

 The publication costs of this article were defrayed in part by page charge payment. This article must therefore be marked “advertisement” in accordance with 18 U.S.C. §1734 solely to indicate this fact.