Abstract
Purpose :
Create a generative AI (GenAI) tool powered by LLMs that automatically uses pre-validated answers or suggests new ones to provide standardized guidance for Port Delivery System (PDS) clinical trials.
Methods :
To enhance performance and address knowledge cutoff and hallucination limitations, we have implemented a solution that utilizes a Language Model (LLM) grounded by a database of frequently asked questions (FAQ) and supporting documents for the PDS (Fig. 1). This approach builds upon the Retrieval Augmented Generation (RAG) method, extending it to a Retrieval Retrieval Augmented Generation (RRAG) by incorporating semantic search through the FAQ as a first step.
In our RRAG approach, we initially employ the LLM to map all FAQ and supporting documents into an embedding space, storing the representations in a vector database. Subsequently, we embed the user's question in the same space and compute semantic similarities.
If a match is found within the FAQ, the corresponding answer is provided as a suggestion. For new questions that do not resemble those in the FAQ, the RRAG method generates draft answers (including the source) based on the supporting documents.
To ensure the confidentiality of the data, we have opted to use open-source language models within self-hosted environments, avoiding the reliance on externally hosted API options like OpenAI's GPT or Google's PaLM.
Lastly, we have deployed the solution in a web-based interface designed to be user-friendly for the FAQ moderator.
Results :
We evaluate our RRAG approach against a ground truth dataset of expert-curated questions and answers for the accuracy and relevance of retrieval (top-k match) and the correctness of generated answers (semantic similarity).
After tuning our RRAG approach, the performance of the FAQ retrieval achieved a top-k accuracy of 82%, and the correctness of the generated answers achieved an embedding cosine similarity of 71% against our evaluation dataset.
Conclusions :
We demonstrated how LLM technologies based on an enhanced RAG approach provided accurate and relevant retrieval and generation of answers. This shows the potential to facilitate and standardize response in clinical development potentially making more timely and efficient clinical suggestions for patient care in clinical trials.
This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.