Evaluation of the accuracy of AI-generated clinical summaries from Glaucoma outpatient visits.

Abstract

Purpose : To evaluate the accuracy of the large language model (LLM) chatbot Llama 2 (Meta AI, NYC, USA) to summarize glaucoma clinic notes in patient-friendly language.

Methods : A random sample of deidentified clinic notes from unique patients who visited the Glaucoma Service at Massachusetts Eye and Ear from 5/1/2007-9/15/2023 was presented to Llama 2. Llama 2 was asked to provide paragraph and bullet point note summaries in patient-friendly language and answer the following patient-relevant clinical questions: 1. does the patient have glaucoma, and if so what type; 2. is the patient’s glaucoma progressing; 3. summarize the treatment plan; 4. identify changes to the treatment plan; 5. identify any laser or surgical intervention offered, and if so, generate relevant educational material. Llama 2 responses were evaluated against original clinic notes for accuracy and completeness by an ophthalmologist reviewer. Each response was evaluated as “correct” if it is accurate and complete, “partially correct” if it is accurate but incomplete or with equivocal clinical interpretation by AI, or “incorrect” if it contains irrelevant or false data.

Results : A total of 139 clinic notes were presented. 72 notes (52%) were correctly summarized, and 64 notes (46%) were partially correctly summarized in paragraph format. In bullet-point format, 81 notes (58%) were correctly summarized, and 57 notes (41%) were partially correctly summarized. Llama 2 correctly identified the glaucoma diagnosis 80% of the time (111 notes), correctly identified whether the patient’s glaucoma was progressing 84% of the time (117 notes), correctly summarized the treatment plan 83% of the time (115 notes), correctly identified treatment changes 88% of the time (122 notes), and correctly identified the surgical or laser interventions offered and generated accurate educational material 84% of the time (117 notes). The rate of incorrect answers for specific clinical questions ranged between 3-7% (Table 1). Hallucinations were not observed.

Conclusions : While Llama 2’s performance in this study was not accurate enough for clinical adoption, it shows potential for LLMs to accurately summarize clinical information with more focused training. With increasing adoption of open notes, using LLMs to help organize clinical notes in standardized patient-friendly language could someday improve patient experience and promote health literacy and understanding.

This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.