Abstract
Purpose :
The early COVID-19 pandemic posed unique challenges in the healthcare system including limiting clinic volumes. This necessitated a robust triage process of consults. The aim of this study was to determine inter-rater agreement in the prioritization of ophthalmology consults in a VA hospital.
Methods :
Three comprehensive ophthalmology attendings at Tennessee Valley Healthcare System Veterans' Affairs Hospital independently rated 100 ophthalmology consult scenarios. The ratings were given a priority rating 1 through 4 with a 5th category of denial of consult for insufficient information. Priorities 1 through 4 were as follows: 1: needs urgent evaluation < 1 week, 2: within 1 month, 3: within 3 months and 4: within 6 months.
The attendings were trained and knowledgeable in the consult priority levels. Agreement between the raters was evaluated by Fleiss' Kappa analysis by priority level and overall. Percentage of agreement was also assessed between raters and overall. A second analysis was done with two priority levels, grouping priorities 1&2 compared to priorities 3&4.
Results :
The overall inter-rater agreement was fair to moderate, kappa (k) = 0.49. The highest agreement was for priority 1 consults having excellent agreement, k = 0.87. Priorities 2 through 4 and 5 had similar agreements (k = 0.43, 0.31, 0.37 and 0.44 respectively). The percentage of inter-rater agreement was similar between raters (60.39% for each inter-rater) and as a group overall (61%). When grouped as two categories, priority 1&2 compared to 3&4, agreement increased to moderate to substantial for each inter-rater and overall (k = 0.65 for each). The percentage of agreement also increased to 83% overall and 87.34% for raters 1&2, 84.81% for raters 1&3 and 79.75% for raters 2&3. For all kappa values the p-value was < 0.005 with the exception of priority 4, p = 0.52.
Conclusions :
These findings show that even with trained attending ophthalmologists, there may be significant variability in consult triage. However, the agreement for more urgent consult scenarios (priority 1) is excellent. This suggests that these consults would likely be seen expeditiously during times of clinic volume limits, such as in a pandemic, independent of the trained rater.
This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.