Large Language Models: A New Frontier in Pediatric Cataract Patient Education

Abstract

Purpose : Patient education materials (PEMs) available online for pediatric cataract have been shown to be composed at a reading level which exceeds the average American's comprehension skills. Large language models (LLMs) can rewrite text materials, and generate novel text. Our cross-sectional comparative study evaluated the ability of 3 LLMs (ChatGPT-3.5, ChatGPT-4, and Google Bard) to generate novel PEMs, and improve the readability of existing PEMs on pediatric cataract.

Methods : We compared LLMs’ responses to 3 prompts. Prompt A (control) requested they write an "educational handout on pediatric cataract that was easily understandable by the average American." Prompt B contained a modifier statement that requested the handout be written "at a 6^th grade reading level (as recommended by the American Medical Association), using the SMOG (Simple Measure of Gobbledygook) readability formula." Prompt C rewrote existing PEMs on pediatric cataract "to a 6^th grade reading level using the SMOG readability formula." Reponses were then compared on their readability (SMOG, Flesch-Kincaid Grade Level (FKGL)), quality (DISCERN tool, Patient Education Materials Assessment Tool (PEMAT)), and accuracy (Likert misinformation scale).

Results : All generated PEMs were of high quality (median DISCERN score ≥4, PEMAT understandability ≥70%) and accuracy (Likert score=1). Generated PEMs were not actionable (PEMAT actionability <70%). PEMs generated in response to Prompt B were significantly more readable than Prompt A responses (p<0.001). ChatGPT-4 generated PEMs more readable (lower SMOG and FGKL scores; 5.6 ± 0.5 and 4.3 ± 0.7, respectively) than the other two LLMs (p<0.001). Although all LLMs improved readability of existing PEMs (Prompt C), only ChatGPT-4 consistently rewrote them to or below the specified 6th grade reading level (SMOG: 5.1 ± 0.3, FKGL 3.8 ± 0.6).

Conclusions : This study underscores the value of LLMs, particularly ChatGPT-4, as strong supplementary tools for generating high-quality, readable, and accurate PEMs on pediatric cataract. Through effectively improving the readability of existing PEMs, they demonstrate potential to make information on pediatric cataract more accessible to a greater number of patients.

This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.

 

A) Prompt A vs B: Generated patient education materials’ readability
B) Differences in Prompt B readability scores

 

A) Readability: Online materials vs Prompt C rewrites
B) Prompt C rewrites: head-to-head comparison