Abstract
Purpose :
As cataract surgery numbers rise, follow up demands place further strains on hospital services. This study uses data collected using an automated telephone platform Dora, which asks patients symptom-based questions to elicit postoperative concerns. We compare the ability of different machine learning techniques to understand patients’ descriptions of their symptoms during a cataract surgery follow up with Dora.
Methods :
The training and test datasets were collected from two non-overlapping patient populations who used Dora as part of ethically approved research studies across 3 diverse UK hospitals. The training dataset was augmented with members of the public using the Dora platform. All participants consented to their data being used and the data was fully anonymised. The datasets consist of transcribed utterances of patients describing their symptoms in response to questions like “is your eye red?”. Each utterance was labelled with an intent, for example, “gritty” or “ red eye” from a total of 24 different intents. Two ophthalmologists independently labelled the dataset and resolved conflicts to establish ground truth labels.
We compared 5 different deep learning and traditional machine learning models; the optimal hyperparameters for each were determined using a grid search and 4-fold cross-validation on the training set. The models were then trained on the entire training dataset and their performance evaluated on the test set.
Results :
The training dataset included 1558 unique utterances from 191 patients and 254 members of the public. The test dataset had 255 unique utterances from 142 patients. The best performing model was the Dual Intent and Entity Transformer (DIET) classifier using word embeddings from BERT as presented in Table 1.
Conclusions :
Deep learning models have the potential to accurately classify patient intents as part of a natural language conversation, with the DIET classifier performing best. Particularly in this low-data domain, the models benefit greatly from pretrained language models such as BERT. These models can thus be used to deliver autonomous clinical conversations for cataract surgery follow up.
This abstract was presented at the 2022 ARVO Annual Meeting, held in Denver, CO, May 1-4, 2022, and virtually.