Real-World AI Chatbot Use in Mental Illness using Health Records

AI chatbots are now widely used by people experiencing mental illness, including young adults and adolescents. Data released by OpenAI indicates that hundreds of thousands of users discuss psychotic or suicidal content with ChatGPT every week, and widespread media reports have documented cases of AI chatbot use associated with acute psychiatric episodes. Despite this, no study has directly evaluated these interactions in a clinical population with serial ground-truth assessments from behavioral health providers. As a result, low-quality anecdotal "evidence" (news reports, fragmented logs, etc.) have been used to develop concerns about phenomena termed "AI psychosis" and "AI suicidality" by the media. Though these concerns merit investigation, understanding the bidirectional relationship between people with mental illness and LLM-based chatbots requires rigorous data collection, analysis, and in-depth interpretability efforts in order to produce actionable insights.

Therefore, in this study, we plan to directly measure the relationship between measured clinical symptoms and chatbot usage patterns, using clinical records and raw AI chatbot log data, to enable the development of effective chatbot safety guardrails and the safe use of these tools for therapeutic benefit. This study will be the first to link real-world AI chatbot interaction logs with clinical ground truth from electronic health record (EHR) data in a behavioral health population. This study has already recieved IRB approval under the UCSF IRB, and we plan to expand the study to Stanford.

We propose two integrated research components:

Component A - Clinical Characterization of AI Chatbot Use. We will recruit children, adolescents, and adults currently in active behavioral health treatment at UCSF across inpatient and outpatient settings. Using a screening instrument developed by our group, we will collect measures of chatbot usage, dependency, and epistemic vulnerability. We will collect chatbot interaction logs digitally and link these to EHR clinical data, including diagnoses, validated psychometrics (PHQ-9, GAD-7, and others), psychiatric assessments, and treatment records. We will then analyze the bidirectional relationships between mental health symptoms and chatbot engagement patterns, with a focus on usage patterns that correlate with illness progression, exacerbation, or remission.

Component B — Digital Biomarker Discovery. Building on the clinical characterization, we will analyze linguistic and semantic features of the collected chat logs using both natural language processing methods, including embedding models and latent pattern analysis, and direct language model pattern analysis. We will identify usage patterns that temporally precede clinically documented escalation of care (e.g., emergency visits, hospitalizations, symptom exacerbations) and develop preliminary chatbot-based digital biomarkers predictive of mental health symptoms and crises. These biomarkers will lay the groundwork for future early-warning systems that could alert clinicians, patients, and AI model developers to emerging risk and enable evidence-based intervention.

Component C - Interperability. Building on the discovered clinical courses and digital biomarkers, we will 1) make use of this collected data to develop enhanced clinical persona simulators that can be used for ongoing testing and refinement of models over extended conversational turns (i.e., >5,000 turns), and 2) use advanced interpretability techniques to understand how safeguards in modern large-parameter LLMs (including Qwen, Llama, and gpt-oss) may fail in these settings and how they can be adapted to avoid these failure modes.

This project will be executed as part of the UCSF AI in Mental Health Research Group within the Department of Psychiatry and Behavioral Sciences; the group has both the clinical and technical resources and personnel to execute. The team is led by PI Karthik V. Sarma, MD, PhD (UCSF), founder of the UCSF AI in Mental Health Research Group, a clinician and technical AI researcher experienced in evaluating the mental health impact of chatbots and co-author of a published case report on AI-associated psychosis, and includes Kaitlin Hanss, MD, MPH, Sachin Pendse, PhD, Anne Glowinski, MD, MPE, and Andrew Krystal MD (UCSF), and collaborator Nina Vasan, MD, MBA, Founder and Executive Director of Brainstorm: The Stanford Lab for Mental Health Innovation.

AI systems that distort human judgment at scale are a core pathway to catastrophic risk. Though there is no scientific evidence at this time that the use of LLM-based systems can directly cause mental illness, it is less clear if these platforms can acclerate or exacerbate illness, and certainly clear that these platforms may be able to better help people with such illnesses.

Researchers have theorized mechanisms of unhealthy interaction modes, including sycophancy, delusion reinforcement, and engagement-optimized emotional dependency; however, these theories are not based on robust interpretability research or high-quality clinical evidence. Collecting and analyzing such evidence will allow the AI community to understand the nature of the potential problem, and the degree to which it poses a heightened existential risk to humans.

Epistemic vulnerability to persuasive AI is one of the least-measured components of AI risk. Our project to estavlish rigorous, clinically grounded measurement of these epistemic effects, starting with the populations in which where harms could be most acute, is a prerequisite for building systems that are helpful, honest, and harmless.

We have already raised $67,500 over the required $100,000 to cover all three components in the outpatient setting and are looking for a minimum of $32,500 additional. The min funding covers:

• Some researcher effort from Dr. Sarma, Dr. Hanss, and Dr. Pendse (between 5-15%; Drs. Krystal and Glowinski are contributing time to the project without cost)
• Some staff assistant effort (approx 0.3 FTE) to assist in electronic participant recruitment + phone call follow up + IRB communications
• Some research assistant effort (approx 0.3 FTE) to assist in child and adolescent recruitment and ethics protocols (child assent/parent assent protocol, etc.)
• System costs to facilitate outpatient research recruitment and data collection (MyChart/EHR analyst programming time, etc.) at UCSF
  
  Additional funding over the $32,500 up to the ideal would cover:
• Expansion to Stanford outpatient (requires additional recruitment system costs and staff assistant time)
• Expansion to UCSF and Stanford inpatient (requires substantial additional research assistant time)