Gaze Behavior among Experts and Trainees during Optic Disc Examination: Does How We Look Affect What We See?

Abstract

Purpose.: The authors compared the visual gaze behaviors of glaucoma subspecialists with those of ophthalmology trainees during optic disc and retinal nerve fiber layer (RNFL) examination.

Methods.: Seven glaucoma subspecialists and 23 ophthalmology trainees participated in the project. Participants were shown eight glaucomatous optic disc images with varied morphology. Eye movements during examination of the optic disc photographs were tracked. For each disc image, graders were asked to assign a presumptive diagnosis for probability of glaucoma. There was no time restriction.

Results.: Overall, trainees spent more time looking at disc images than glaucoma subspecialists (21.3 [13.9–37.7] vs. 16.6 [12.7–19.7]) seconds; median [interquartile range (IQR)], respectively; P < 0.01) and had no systematic patterns of gaze behavior, and gaze behavior was unaltered by disc morphology or topographic cues of pathology. Experienced viewers demonstrated more systematic and ordered gaze behavior patterns and spent longer times observing areas with the greatest likelihood of pathology (superior and inferior poles of the optic nerve head and adjacent RNFL) compared with the trainees. For discs with focal pathology, the proportion of total time spent examining definite areas of pathology was 28.9% (22.4%–33.6%) for glaucoma subspecialists and 13.5% (12.2%–19.2%) for trainees (median [IQR]; P < 0.05). Furthermore, experts adapted their viewing habits according to disc morphology.

Conclusions.: Glaucoma subspecialists adopt systematic gaze behavior when examining the optic nerve and RNFL, whereas trainees do not. It remains to be elucidated whether incorporating systematic viewing behavior of the optic disc and RNFL into teaching programs for trainees may expedite their acquisition of accurate and efficient glaucoma diagnosis skills.

Glaucoma remains undiagnosed in at least 50% of patients with it in Europe, the United States, and Australia, despite many of these patients having recently visited an eye care professional; these rates approach 90% in developing countries. ^{1 –3} Optic nerve head changes typically occur before functional loss is evident on conventional white-on-white perimetry. ^{4 –11} Therefore, accurate and reproducible evaluation of the optic nerve head is essential for early diagnosis and long-term management of this disease. ^{12 –16} Novel imaging techniques to evaluate the optic nerve head have permitted quantitative imaging of it and the retinal nerve fiber layer (RNFL). ^17,18 However, despite their relatively high sensitivity and specificity in detecting glaucomatous structural change, ^{19 –21} these techniques do not negate the need for accurate assessment of the optic nerve by clinicians. ^{22 –24} Clinical examination and fundus photography remain the gold standard. 

The phenotypic diversity of both normal and diseased discs ²⁵ and the often subtle nature of early glaucomatous change create an important challenge in evaluating the optic nerve head and implementing teaching modules on disc assessment technique. ^7,26 It is well established that optic disc examination, in terms of glaucoma diagnostic accuracy, is a skill that improves with experience. ^27,28 It is not known whether experts observe the optic nerve head and RNFL in ways that substantially differ from trainees. 

Previously, through the glaucomatous optic neuropathy evaluation (GONE) system, we showed that glaucoma experts have significantly higher consistency and accuracy in examining optic disc characteristics and glaucoma likelihood than do ophthalmology trainees. ²⁹ We also found that ophthalmology trainees tend to underestimate glaucoma likelihood and that inaccurate assessment of key morphologic features of the disc lead to an increased likelihood of misdiagnosis (unpublished data, 2010). The use of eye tracking provides a potential means of gaining greater understanding of the method of optic disc examination and may help explain the discrepancy occurring between experts and trainees when assessing the optic nerve head. The potential to link gaze behavior with ophthalmic diagnosis and decision-making processes has not been explored. 

Studies on eye movements have been adopted in other medical fields to assess clinical skill acquisition. ³⁰ Eye tracking technology has been used in radiology interpretation to analyze and understand patterns of image evaluation, providing information on the nature of errors in reading radiographs, including gaze behavior, scan patterns, and accuracy among experienced observers and trainees. ^{30 –35} It has also been applied to general surgery, whereby eye tracking has been used to investigate orientation and gaze patterns of surgeons to improve technique when performing both minimally invasive and robotic-assisted surgery. ^36,37  

To date, the assessment of gaze behavior among ophthalmic clinicians in examining ocular disease and, thus, the possibility to impact on current teaching methods remains unexplored. The aim of the present study was to examine the eye movement and gaze pattern behaviors of clinicians with varying levels of expertise when examining optic disc images and to identify potential differences in visual search and gaze behaviors. 

Methods

Optic Disc Images

Eight optic disc images, four with diffuse rim thinning and RNFL loss and four with focal rim and RNFL defects (Figs. 1A, 1B) were selected from a set of 42 planimetric monoscopic disc images, used in the GONE project (www.gone-project.com) ²⁹ and representing a range of physiologic and glaucomatous disc characteristics with high demonstrated interobserver agreement for glaucoma likelihood among glaucoma subspecialists (kappa [κ_w] = 0.63; 95% confidence interval [CI], 0.60–0.67). 

Figure 1.

 

Disc images presented to participants. (A) Discs with diffuse rim and retinal nerve fiber layer loss. (B) Discs with focal rim and/or retinal nerve fiber layer loss. (C) Areas of focal pathology defined as AOIs using eye-tracker analysis software.

Images were stored as high-quality (low-compression) jpg images and were standardized in size and magnification to fit to a screen resolution of 1280 × 960 pixels, at 32-bit color. Participants were asked to examine each disc image and to provide a subjective grade for clinical impression of glaucoma likelihood based on optic disc appearance using a four-point ordinal scale as unlikely, possible, probable, or certain. All answers were recorded verbatim. 

Institutional review board approval was obtained for this project, and the research was conducted in accordance with the Declaration of Helsinki guidelines. 

Eye Tracking

To assess gaze behaviors during optic disc examination, eye movements were recorded with an eye tracker (T120; Tobii Technology, Stockholm, Sweden), a remote eye-tracking device using near infrared and both bright and dark pupil-centered corneal reflection to track eye movements. This has an accuracy of 0.5° and a sampling rate of 120 Hz, integrated with a 17-inch TFT monitor with a screen resolution of 1280 × 1024 pixels. It can tolerate moderate head movement within a 30 × 22 × 70-cm volume at 50 to 80 cm in front of the device, thus providing a relatively natural environment for assessing optic disc images. All experiments were carried out in ambient room lighting with minimal noise. 

Each participant was positioned 60 cm ± 10 cm in front of the screen. Standardized five-point calibration on the Tobii eye tracker was performed. Before the assessment, participants were given identical verbal instructions and written on-screen instructions. The same researcher conducted all trials for all participants. A sample disc was first shown to familiarize participants with the process for disc examination before the test series. The eight test images were shown sequentially. Optic disc image was shown alone without query on diagnosis. When participants were satisfied they had examined the image, they exited the image page and entered the diagnosis page. Participants were given no details of patient medical history, ancillary ophthalmic investigations, or other test results. There was no time limit on the evaluation, and eye movements were tracked throughout. 

Statistical Analysis

Gaze data for each participant were analyzed. Total time taken to interpret the optic disc image, the gaze behavior, and the fixation points during disc image assessment and their correlation with diagnostic accuracy was noted. The mode of the expert group was adopted as the correct diagnosis, and an incorrect diagnosis (either overestimation or underestimation) was defined as a glaucoma likelihood rating at least one or more scale point away from the reference diagnosis (Table 1). Areas of interest (AOI) were defined by one expert examiner on disc images as areas with focal pathology using Tobii analysis software (Fig. 1C). Total dwell time and dwell time on AOI were examined. Statistically significant differences between group data were analyzed with the Mann-Whitney U test, and significance was confirmed using the two-tailed Student's t-test; α < 0.05 was taken to indicate statistical significance. All statistical analyses were performed with statistical analysis software (SPSS, v15.00; SPSS Inc., Chicago, IL). 

Table 1.

 

Incorrect Diagnosis of Glaucoma Likelihood

Error	Glaucoma Subspecialist Diagnosis	Trainee Diagnosis
Underestimation	Certain	Possible or unlikely
Underestimation	Probable	Unlikely
Overestimation	Possible	Certain
Overestimation	Unlikely	Probable or certain

 Incorrect diagnosis based on a trainee rating glaucoma likelihood scale ≥1 point different from the correct reference diagnosis.

Results

Thirty participants in a single tertiary referral ophthalmic center were recruited for this project. They included 7 glaucoma subspecialists invited for their expertise in optic nerve examination and 23 trainees, among them 13 junior trainees (first-year ophthalmology residents with 6 months' experience and no previous glaucoma unit experience) and 10 more experienced senior trainees (third-year ophthalmology residents). 

Analysis of total dwell time showed ophthalmology residents spent significantly longer total time examining each disc image than glaucoma subspecialists (median [IQR] 21.3 [13.9–37.7] and 16.6 [12.7–19.7] seconds, respectively; P < 0.01). First-year residents had a median total dwell time of 23.0 (15.9–42.3) seconds, which was significantly longer than both third-year residents and glaucoma subspecialists (16.3 [10.7–27.0] seconds [P < 0.01] and 16.6 [12.7–19.7] seconds [P < 0.01], respectively; Fig. 2). 

Figure 2.

 

Total time examining disc images. Box plot of the distribution of total times examining disc images for glaucoma. Subspecialists compared with ophthalmology senior trainees (third-year residents) and junior trainees (first-year residents). Trainees spent more time examining discs than glaucoma subspecialists (P < 0.05). (A) Box plot distribution of total times examining disc images for all discs among the three groups. (B) Box plot distribution of total examination times for each disc and group. Boxes: IQRs; whiskers: nonoutlier ranges; open circles: outliers; solid lines: group medians.

When examining the four discs with focal pathology, glaucoma subspecialists spent a larger proportion of total time examining AOIs than trainees (median [IQR] 28.9 [22.4%–33.6%] and 13.5 [12.2%–19.2%], respectively; P < 0.05; Table 2, Fig. 3). The differences were statistically significant for 3 of 4 discs when compared with first-year residents. Participants who made the correct diagnosis for glaucoma likelihood spent a greater proportion of total time examining AOIs than those who chose the incorrect diagnosis, irrespective of training level. This was statistically significant in 3 of the 4 discs with focal pathology (Table 3, Fig. 4). 

Table 2.

 

Percentages of Time Examining AOIs by Group

	Disc 4	Disc 6	Disc 7	Disc 8
Glaucoma subspecialists	30.4	33.6	23.8	28.7
Ophthalmology residents, year 3	19.1	8.1*	11.6	17.5
Ophthalmology residents, year 1	12.8*	5.0*	12.3	9.7*

 Values are medians.

*   P < 0.05 using Mann-Whitney U test compared with glaucoma subspecialist assessment.

Figure 3.

 

Proportion of time spent examining AOIs by group. Box plot of the distribution of proportional time examining AOIs for glaucoma subspecialists compared with ophthalmology senior trainees (third-year residents) and junior trainees (first-year residents). Boxes: IQRs; whiskers: nonoutlier ranges; open circles: outliers; solid lines: group medians. *P < 0.05 when compared with glaucoma subspecialist assessment.

Table 3.

 

Percentages of Time Examining AOIs by Diagnostic Accuracy

	Disc 4*	Disc 6*	Disc 7	Disc 8*
Correct diagnosis	25.2	13.7	16.7	23.8
Incorrect diagnosis	15.0	0.8	16.0	12.2

 Values are medians.

*   P < 0.05 using Mann-Whitney U test.

Figure 4.

 

Proportion of time spent examining AOIs by diagnosis. Box plot of the distribution of proportional time examining AOIs for correct versus incorrect diagnosis. Boxes: IQRs; whiskers: nonoutlier ranges; open circles: outliers; solid lines: group medians. *P < 0.05 when compared with correct diagnosis group.

Overall, experts exhibited adaptation of gaze behavior depending on the type of glaucomatous damage present and the extent of disease. Trainees did not exhibit the same adaptive behavior. Figure 5 illustrates the fixation pattern and gaze behavior of one glaucoma subspecialist for all eight discs. Gaze behavior was systematic in all cases, with examination of the rim and the RNFL in areas of likely pathology. 

Figure 5.

 

Gaze patterns of representative glaucoma subspecialists throughout the eight disc images assessed (numbers indicate the order in which the discs were presented to the participants).

Figure 6 illustrates the patterns of disc assessment of four glaucoma subspecialists compared with four trainees for a disc image with diffuse loss. Glaucoma subspecialists showed a systematic and consistent approach to examining the optic disc rim, superior and inferior poles of the optic nerve head, and adjacent RNFL. Ophthalmology trainees' examinations focused predominantly on the optic cup. 

Figure 6.

 

Gaze pattern behaviors of four glaucoma subspecialists compared with those of four ophthalmology trainees when assessing disc 2 to illustrate gaze behavior when assessing a disc with diffuse rim and nerve fiber loss. Overall, only a small area of the entire image was examined. Trainees did not examine the nerve fiber layer, and experts' viewing habits were more systematic.

Figure 7 illustrates the fixation patterns and gaze behaviors for an image with focal loss (large superotemporal notch and RNFL wedge defect). The glaucoma subspecialists examined the image, identifying and assessing the focal pathology, the remaining disc rim, and the RNFL and completed their disc examination. Ophthalmology residents focused their examination centrally on the optic cup and did not examine areas of focal loss. 

Figure 7.

 

Gaze pattern behaviors of four glaucoma subspecialists compared with those of four ophthalmology trainees when assessing disc 6 to illustrate gaze behavior when assessing a disc with focal rim and nerve fiber layer loss, with a superotemporal notch and superotemporal RNFL wedge defect present.

Discussion

This study demonstrates fundamental differences in visual search and gaze behavior between trainees and experts. Using novel eye-tracking technology we have shown qualitatively and quantitatively that glaucoma subspecialists develop a more systematic efficient and logical approach to examining disc images than trainees. Experts spend more time observing the likely areas of disc pathology (superior and inferior poles of the optic nerve head and adjacent RNFL) than trainees (Supplementary Movies S1 and S2). A more systematic approach to examining discs for disease by glaucoma subspecialists is associated with higher consistency in glaucoma risk assessment by subspecialists and higher tendency for trainees to underestimate the likelihood of glaucoma (unpublished data, 2010). The differences in gaze behavior could be attributed to inexperience regarding the characteristic features of glaucomatous optic nerve damage and the more subtle indicators of disease and may account for the difference in performance when assessing glaucoma risk between these groups. 

Similarly, it has been shown that experienced observers have more systematic scanning patterns than inexperienced observers when examining different types of radiographs (mammograms, chest x-rays, bony surveys). ^{31 –35} Among radiologists, eye-tracking behavior appears to evolve from a localized central pattern by the untrained observer to the more circumferential pattern by the experienced radiologist. ³⁸ This is presumed secondary to the development of experience about the location and characteristic appearance of normal versus pathologic features. ³⁸ Interestingly, in our study, visual search and gaze pattern among trainees was also localized and central, focusing on the optic cup without examining other potential areas of pathologic change. However, glaucoma subspecialists, displayed consistent and logical visual search and gaze behavior, systematically examining the entire optic disc and the superior and inferior poles of the optic nerve head and adjacent RNFL, the most common areas of focal pathology in this disease. They also exhibited adaptation of gaze behavior, depending on the type of glaucomatous damage present (focal loss versus diffuse loss) and the extent of disease. For disc images showing obvious gross advanced diffuse loss of both the rim and the RNFL and advanced cupping, expert total examination time was shorter and RNFL layer assessment was less extensive. Trainees do not exhibit the same adaptive behavior, with unchanged examination time and gaze remaining central and on the optic cup. Perhaps this is attributable to trainees appearing to prioritize assessment of the cup/disc ratio (CDR) to the detriment of what might be perceived as more subtle indicators of glaucoma likelihood, including RNFL defects. 

The nature of errors when examining radiographs has also been explored. In detecting pulmonary nodules on chest X-rays, Kundel et al. ³² found that at least 30% of false-negatives were due to incomplete scanning, with novices particularly prone to search-related errors. In a study of bone X-rays, Leong et al. ³⁴ found subspecialists were more accurate and consistent and spent less time fixating on fracture sites than trainees. In our study, trainees spent more total time examining disc images, with junior trainees spending significantly more time than either senior trainees or glaucoma subspecialists, suggesting that greater experience shortens total examination time. Among discs with focal pathology, glaucoma subspecialists spent a greater proportion of total time examining areas of pathology than trainees (median, 28.9% vs. 13.5% respectively; P < 0.05). Several trainees failed to examine focal areas of pathology, possibly because of their inexperience regarding the characteristic features and common areas of pathologic change found at the optic nerve head in glaucoma. 

Optic disc examination is a skill that improves with training, and it is well established that glaucoma subspecialists perform better in terms of diagnostic accuracy than residents in training. ³⁹ In this study, we found a difference in the fixation and visual search patterns between glaucoma subspecialists and trainees and a significant difference in the proportion of total time spent examining focal areas of pathology among those who chose the correct diagnosis compared with those who chose the incorrect diagnosis, irrespective of training level. 

Recently, it has been suggested that viewing the eye movement behavior of another observer completing the same task may help with the development of visual scanning and recognition skills through the demonstration of an effective search process. Litchfield et al. ⁴⁰ found that both novice and experienced radiographers demonstrated improved diagnostic performance when shown the eye movements of an expert radiologist, and the diagnostic performance of novice radiographers consistently improved when shown the expert's search behavior. Thus, viewing the eye movement behavior of expert observers may enable novices to improve their clinical skills and scaffold their decisions based on the search behavior of others, ⁴⁰ with potential acceleration of clinical skills acquisition among trainees. 

There are some potential limitations to this study. First, we used a small sample of optic disc images—four with diffuse pattern loss and four with focal pattern loss—which may not fully reflect the optic disc examination skills of clinicians. Second, images were not stereoscopic, and participants had to rely on monocular clues for rim assessment. However, monoscopic images are used widely in clinical practice. Additionally, the need for a stereoviewer would preclude eye movement tracking with this noninvasive technology. Third, participants in the subspecialist group were selected because of their expertise, but trainees were not. This might have led to some degree of bias because of greater homogeneity of clinical experience in the subspecialist group; the impact of specific previous training received by residents was not addressed in this study. Further studies, with increased numbers of participants and a greater number and variety of disc images are required. The effectiveness of including observation of expert gaze behavior in teaching trainees optic disc evaluation for glaucoma will be addressed. 

In conclusion, we have investigated for the first time how fixation pattern and gaze behavior may affect diagnostic accuracy in an ophthalmic setting, specifically disc examination for glaucoma. Overall, glaucoma subspecialists demonstrate a systematic and logical approach to examination of the disc images, but trainees do not. Future studies will assess whether gaze behavior modification might increase the diagnostic accuracy of trainees in assessing commonly observed ophthalmic conditions, analogous to what has been observed previously in the field of radiology. 

Supplementary Materials

Movie sm01, WMV - Movie sm01, WMV 

Movie sm02, WMV - Movie sm02, WMV 

Footnotes

 Supported by Operational Infrastructure Support from the Victorian Government (Centre for Eye Research Australia [CERA]); an unrestricted grant and an educational grant (CERA) from Allergan, Inc.; the Helen McPherson Trust Grant; and the Cranbourne Trust.

Footnotes

 Disclosure: E.C. O'Neill, None; Y.X.G. Kong, None; P.P. Connell, None; D.N. Ong, None; S.A. Haymes, None; M.A. Coote, None; J.G. Crowston, None