Abstract
Purpose::
To measure agreement of plus disease diagnosis among retinopathy of prematurity (ROP) experts.
Methods::
A set of 34 wide-angle retinal photographs from infants with ROP was compiled on a secure website, and interpreted independently by 22 recognized ROP experts. Diagnostic agreement was analyzed using 3-level ("plus," "pre-plus," "neither") and 2-level ("plus," "not plus") categorizations.
Results::
In the 3-level categorization, all experts agreed on the same diagnosis in 4 (11.8%) of the 34 images, and the mean weighted kappa for each expert compared to all others was between 0.21-0.40 (fair agreement) in 7 (31.8%) experts, and 0.41-0.60 (moderate agreement) in 15 (68.2%) experts. In the 2-level categorization, all experts who provided a diagnosis agreed in 7 (20.6%) of the 34 images, and the mean kappa for each expert compared to all others was between 0-0.20 (slight agreement) in 1 (4.5%) expert, 0.21-0.40 (fair agreement) in 3 (13.6%) experts, 0.41-0.60 (moderate agreement) in 12 (54.5%) experts, and 0.61-0.80 (substantial agreement) in 6 (27.3%) experts.
Conclusions::
Inter-expert agreement of plus disease diagnosis is imperfect. This may have important implications for clinical ROP management, continued refinement of the international ROP classification system, development of computer-based diagnostic algorithms, and implementation of ROP telemedicine systems.
Keywords: retinopathy of prematurity • imaging/image analysis: clinical • clinical (human) or epidemiologic studies: health care delivery/economics/manpower