Electronic Theses and Dissertations Archive

Date

2026

Document Type

Thesis

Degree Name

Master of Science

Department

Computer Science

Committee Chair

Vasile Rus

Committee Member

Deepak Venugopal

Committee Member

Keishana Barnes

Abstract

In recent years, there has been an explosive increase in the use of AI-powered tools in a variety of fields nationwide. Recent studies on AI usage by Elon University and the American Medical Association highlight that 52% of adults and 72% of physicians in the United States indicated AI-based tool usage. The most common uses in healthcare settings of these tools were medical research summaries, question answering, and documentation in practice. But with widespread AI chatbot usage comes expanded responsibility to ensure that the model is producing factually accurate, culturally competent, and safe responses. Retrieval Augmented Generation is a technique used to increase model response safety and accuracy and has been introduced into chatbot design used for health care and educational settings. There is limited literature covering the differences in performance between retrieval augmented local models and commercially available models when paired with a custom database to encourage factual and socially aware responses to specialized healthcare or education questions. This study aims to develop and evaluate an AI-powered educational assistant for parents of children with attention deficit hyperactivity disorder and/or autism spectrum disorder. The chatbot responses were evaluated with custom scoring criteria to address the anticipated concerns relevant to ADHD/ASD, with LLM-As-Judge systems rating the responses. The LLM-As-Judge conditions showed the impact of RAG varied by metric grouping, where the local LLM with RAG showed comparable factuality scores to the commercial LLM with RAG and its base local LLM without RAG. There was a significant difference across metrics in the response quality, hallucination, and safety metrics between the local LLM and the commercial LLM, which suggests a need for future work expanding the RAG component. The results were high across all groups for neurodiversity-affirming and culturally responsive language, which may point to the importance of system prompt engineering to include social aspects. This study highlights the potential of a custom local RAG system to be used as a more sustainable AI tool with comparable accuracy and social metrics to commercial options.

Comments

Data is provided by the student.

Library Comment

Dissertation or thesis originally submitted to ProQuest/Clarivate.

Notes

Open Access

Download

Share

COinS