Abstract
Purpose :
Our research explores the effectiveness of AI-generated content in ophthalmic patient education, focusing on its readability compared to the established standard of the American Academy of Ophthalmology (AAO) materials published on the AAO website. It further investigates the potential of custom-built GPT models, taking advantage of OpenAI’s Custom GPT Builder to creat a custom tailored chatbot nammed "Eye Health Educator," in an attempt to help create a more tailored and patient-friendly educational experience.
Methods :
We analyzed patient education materials (PEMs) on ten common ophthalmic diagnoses, comparing the readability of content from the AAO Website, ChatGPT 4.0, Google Bard, and the custom "Eye Health Educator" GPT model. Readability was assessed using industry standard metrics such as Word Count, Flesch Reading Ease, Gunning Fog Index, Flesch-Kincaid Grade Level, Coleman-Liau Index, SMOG Index, Automated Readability Index, and Linsear Write. Chatbots were prompted to maintain similar word counts as the AAO material. Statistical significance was evaluated through T-Tests comparing the averages of AI-generated readability materials with that of the AAO.
Results :
The analysis, presented in Table 1, revealed that while ChatGPT 4.0 often fell short in readability compared to AAO materials, the custom "Eye Health Educator" GPT model demonstrated a closer alignment in readability metrics to that of the AAO. Table 2's T-Test outcomes showed significant variances in readability between general AI models and AAO materials, highlighting the superior performance of custom-tailored GPTs like "Eye Health Educator" in matching the readability of AAO content. GPT 4.0 often had lengthier responses and was difficult to read for most readability metrics.
Conclusions :
The study emphasizes the potential of custom GPT models in generating patient education materials that align closely with the readability standards of established medical resources like the AAO. While general-purpose AI models like ChatGPT 4.0 show limitations in readability for patient education, custom GPTs can be fine-tuned to deliver more accessible and patient-centric content. This tailored approach in AI content generation could significantly enhance the effectiveness of patient education in ophthalmology and other medical fields.
This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.