Mental health is a rather tricky subject, even with the best intentions. Trust, in both the counselor and in the process, is crucial. So how do Artificial Intelligence and machine learning fit into all this? An American mental health platform recently experimented to find out how AI, specifically ChatGPT, can be used in counseling. Unfortunately for them, the experiment gave birth to more problems than solved.

Koko is a nonprofit mental health platform that connects teens and adults who need mental health help to volunteers through messaging apps like Telegram and Discord. On Friday, Koko co-founder Rob Morris announced on Twitter that his company ran an experiment to provide AI-written mental health counseling for 4,000 people without informing them first to see if they could discern any difference.

Critics have called the experiment deeply unethical because Koko did not obtain informed consent from people seeking counseling.

Koko works through a Discord server users sign in to the Koko Cares server and send direct messages to a Koko bot that asks several multiple-choice questions like “What’s the darkest thought you have about this?”. It then shares a person’s concerns—written as a few sentences of text—anonymously with someone else on the server who can reply anonymously with a short message.

During the AI experiment, which applied to about 30,000 messages, volunteers assisting others had the option to use a response automatically generated by OpenAI’s GPT-3 large language model, the model upon which ChatGPT is based, instead of writing one themselves.

After the experiment, Morris put up a thread on Twitter that explained the experiment they had conducted. This is where things turned ugly for Koko. Morris says that people rated the AI-crafted responses highly until they learned they were written by AI, suggesting a fundamental lack of informed consent during at least one phase of the experiment.

Morris received many replies criticizing the experiment as unethical, citing concerns about the lack of informed consent and asking if an Institutional Review Board (IRB) approved the investigation.

Using AI as a therapist is far from new. Still, the difference between Koko’s experiment and typical AI therapy approaches is that patients typically know they are not talking with an actual human.

In the case of Koko, the platform provided a hybrid approach where a human intermediary could preview the message before sending it instead of a direct chat format. Still, without informed consent, critics argue that Koko violated prevailing ethical norms designed to protect vulnerable people from harmful or abusive research practices.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

error: Unauthorized Content Copy Is Not Allowed
Scroll to Top
%d bloggers like this: