When therapists graded LLMs, the AI did not get A’s.
Large language models (LLMs) can answer questions, provide supportive and affirmative responses, and even recognize emotional tone, and these are just the basic capabilities of the technology. For those struggling with mental health issues and the professionals helping them, this is a cause for both consternation and hope. As LLMs get better at writing essays, coding software, and passing licensure exams, a reasonable question has become harder to ignore: can AI think like a therapist?
To find out, a team of New Zealand researchers gave seven AI models three clinical cases and asked them to complete a series of psychotherapy tasks, including case formulation, treatment planning, session structure, coping recommendations, and goal setting. Experienced clinical psychologists then evaluated the responses. The results were mixed. The models generally handled structured tasks fairly well, including discussing treatment goals and suggesting session length, but struggled when asked to integrate information into coherent case formulations or nuanced treatment plans. The strongest responses emphasized collaboration, humility, and clear therapeutic boundaries. The weaker responses tended to oversimplify clinical complexity. The authors concluded that AI systems may be most useful as clinician support tools rather than stand-alone therapists.
For therapists, perhaps the most interesting finding is not that AI can help generate goals, but that it appears to do best when the work of therapy is already well-defined and structured. The parts of psychotherapy that rely on shared understanding, formulation, judgment, and the ability to hold complexity remain much harder for AI systems to replicate.
A huge asterisk on these conclusions is the rapid advancement of the technology. While this study was published just one year ago, it may already be outdated, having examined LLMs that were released in 2023 and 2024. The “reasonable question” will be asked repeatedly. And it will be answered, repeatedly, not only by researchers and professionals, but a public that demands more and better help for mental health disorders.
Citation:
Tan, K. S., Cervin, M., Leman, P., Nielsen, K., Kumar, P. V., & Medvedev, O. (2025). AI meets psychology: An exploratory study of large language models' competence in psychotherapy contexts. Journal of Psychology and AI, 1(1), Article 2545258. https://doi.org/10.1080/29974100.2025.2545258