Abstract Body (Do not enter title and authors here): Introduction: Large language models (LLMs) show proficiency in clinical capabilities such as responding to medical inquiries, and summarizing information, but their application in patient education requires evaluation. This study optimizes LLMs' interpretation of dietary guidelines for nutrition education using a framework that integrates American Heart Association (AHA) guidelines with Retrieval Augmented Generation (RAG), outperforming the baseline models.
Methods: Analyzing 105 dietitian-authored visit notes (623,214 words) with Latent Dirichlet Allocation revealed two themes (coherence score 0.9). We posed 30 questions on these themes, along with the prompt, “Answer the following question as if you were a dietitian and cite sources from AHA,” three times to the baseline Perplexity (perplexity.ai), GPT-4o (OpenAI) models, and a customized LLM framework using RAG with formatted guidelines and citations applied to the Llama 3 70B model (Meta). Three expert reviewers evaluated the responses, scoring 0 or 1 for reliability, appropriateness, guideline adherence, and harm (Figure 1). Mean scores across groups were compared using analysis of variance.
Results: Our LLM framework scored higher than baseline Perplexity and GPT-4o models on reliability (0.57±0.45 vs. 0.24±0.42, 0.11±0.25; p<.001, 95% CI [.07, .25]), appropriateness (0.83±0.28 vs. 0.62±0.34, 0.57±0.37; p<.001, 95% CI [.06, .23]), guideline adherence (100% vs. 47.5%, 80%; p<.001, 95% CI [.21, .41]), readability using the Flesch-Kincaid grade level (11.1±2.4 vs. 9±2.1, 9.4±1.8; p<.001, 95% CI [.07, .25]), and showed no harm (0 vs. 0.23±0.43, 0.26±0.44; p<.001, 95% CI [.02, .17]). Cohen’s kappa coefficient (k>70%, p<.001) indicates high reviewer agreement.
Conclusion: The baseline models scored lower across all measures and included Type 2 errors. While the optimized model shows promise, improvements in reliability, readability, and inclusivity of dietary restrictions, preferences, and cultural considerations are needed for ethical and equitable clinical use. Future steps include refining model capabilities, building a comprehensive knowledge base, and field testing to assess impact of these improvements on clinicians' educational and documentation tasks.