Abstract
Purpose :
Can AI chatbots be used for patient education on retinal floaters? Retinal floaters affect 7 out of 10 people of all ages and ChatGPT™ (OpenAI, San Francisco), a new chatbot, produces AI mediated natural language responses. 54% of U.S. adults have a literacy below 6th grade level, thus new AI chatbots may be useful.
Methods :
5 retinal floater questions were created and typed into ChatGPT™ and Google Assistant™ .
Qns:
1. What is an eye floater?
2. What are the risks of eye floaters?
3. Why do I have eye floaters?
4. What is the treatment for eye floaters?
5. Who should I see if I have eye floaters?
Three metrics assessed ChatGPT™ and Google Assistant™ responses: Dale-Chall, word count, and a scoring system devised by a retinal specialist (GW). The Dale-Chall readability score, used by the U.S. Department of Education, evaluated the grade level of the written answer.
For medical accuracy, we looked for keywords in the chatbot answers: vitreous, vitreous traction, retinal hole, retinal detachment, vitreous bleeding, systemic disease, DM, HTN, trauma to face or eye/orbit, weightlifting, valsalva, no immediate action, vitrectomy, YAG laser, complications, cataracts, retinal specialist, Eye MD, optometrist, internal medicine. Each term was assigned a score correlating to its importance to the answer (see Figure 2.). We asked ChatGPT™ and Google Assistant™ the questions and scored the response. Each question was 20 points, thus 5 questions = 100 points.
Results :
ChatGPT™ Word Count Avg: 179.8
Google Assistant™ Word Count Avg: 52.2
p-value = 0.00089; ChatGPT™ had longer responses vs Google Assistant™.
ChatGPT™ Dale-Chall Adjusted Grade Level: 9 to 10
Google Assistant™ Dale-Chall Adjusted Grade Level: 11 to 12
ChatGPT™ Retinal Specialist Rubric Score: 41/100
Google Assistant™ Retinal Specialist Rubric Score: 42/100
Conclusions :
ChatGPT™ and Google Assistant™ are not fully accurate for the questions related to floaters, from a retinal specialist point of view. While the AI systems are not perfect, they are understandable for patients with a high school education and beyond. We look forward to the future improvements of AI Chatbots and its role in patient education.
This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.