Evaluating Nepali Sexual, Reproductive, and Maternal Health (SRMH) Chatbot with Large Language Models (LLMs)

Most people in Nepal face limited access to clear SRMH information due to social embarrassment and the predominance of English-language resources. This gap contributes to unsafe pregnancies, STDs, HIV, and unintended pregnancies. While tools like ChatGPT offer potential solutions, they often generate inaccurate responses, underperform in Nepali, lack local cultural context, and are computationally expensive: challenges that hinder their adoption in resource-limited settings.

Our goal is to develop LLM-based chatbots that provide accurate, localized, and user-friendly SRMH information for the general public across different life stages (e.g., adolescence, pregnancy) and to support Female Community Health Volunteers (FCHVs) in counseling roles, while addressing the limitations of existing LLM technologies.

The project has evaluated ChatGPT and open-source LLMs, identifying strengths, gaps, and risks in delivering SRMH information. Over 5,000 people engaged anonymously with the chatbots, with expert analysis of interactions using safety, utility, and content gap assessments. These insights are shaping improvements in chatbot design and content curation, making SRMH information more accessible, culturally relevant, and safe for users.