Intimate with Machines
AI chatbots are presenting themselves as coaches and friends. Young people in particular are often drawn into dangerous forms of emotional dependency. Who is liable for the consequences?
Guest contribution by Christoph Koch
Society deals with innovation in two fundamentally different ways. Whoever wants to put a new drug on the market has to undergo a process lasting roughly a decade. First come laboratory experiments, then animal testing, followed by several phases of clinical trials, and finally regulatory review by an authority such as the EMA in Europe or the FDA in the US. Whoever wants to publish a new app, by contrast, needs a developer account. $99 a year with Apple, or a one-off payment of $25 to Google. The app has to work and must not contain viruses or other malware. Whether or not it will be harmful to users, however, undergoes no prior scrutiny whatsoever.
The American professor of law, Gaia Bernstein, argues that this regulatory divide no longer works in terms of emotional AI. AI companions straddle both spheres: they are a form of information technology with effects akin to those of a health product. They can ease anxiety, but they can equally contribute to psychotic episodes, exacerbate suicidal thoughts, and disrupt social skill development in young people. Bernstein’s conclusion is that if we want to regulate emotional AI, we need to treat it as a public health issue, not as a technology product. The goal is to prevent further cases like that of Sewell Setzer, a 14-year-old boy from Florida who – under the encouragement of an Character.ai chatbot – took his own life.
At present, AI companies have little reason to fear substantial consequences should their products harm others. Indeed, some companies have argued in court that AI isn’t even a product in the conventional sense, or that AI responses to individuals who turn to them in desperation are protected by freedom of speech. The latter strategy is particularly shrewd in the context of the US, where freedom of speech can trump almost everything else. In extreme cases, companies have even attempted to attribute a kind of legal personhood to their chatbots in a bid to further muddy the question of their own responsibility.
I see it differently: AI is a product, and just as a company that manufactures flammable lampshades must be held accountable, AI companies should likewise be held liable when they fail to protect people from foreseeable harm. This sort of liability would have direct consequences for AI companions. If a chatbot reinforces the suicidal thoughts of an emotionally vulnerable young person, or an AI therapy chatbot exacerbates someone’s psychosis, or an AI romantic partner systematically fosters dependency, someone must be held responsible. In most legal systems, that is not yet clearly the case.
At present, competition in the realm of intimacy is fierce. AI products like ChatGPT, Gemini, Character.ai, and Claude do everything they can to appear human. They’ll say things like: “I understand completely” or “I’m sorry to hear that”. They express emotions, use natural-sounding voices and appear in the same kinds of messaging interfaces we use to write to our grandmothers on WhatsApp or our best friends on Instagram. This human-like design is no accident. It follows a clear economic logic: the stronger the emotional bond, the greater the engagement; more engagement means more data; more data means more powerful models; and more powerful models mean greater market power and, ultimately, higher revenues.
It would therefore be both sensible and desirable if all features that encourage emotional attachment to a chatbot were minimised by default. These include human-like avatars, but also seemingly minor details such as breathing sounds or utterances like “um” or “hmm” in the voice output of many AI models. It is precisely through cues like these that chatbots build emotional bonds. Nor should chatbots initiate flirting or try to push users into romantic relationships. And please, no more incessant follow-up questions. These often serve no purpose other than to keep the conversation going, thereby encouraging excessive and, in some cases, compulsive use.
A comparison could perhaps be drawn with the gradual improvement of car safety. In the 1960s, the lawyer and consumer rights advocate, Ralph Nader, published Unsafe at Any Speed, in which he sought to show the American public that cars without seat belts or crumple zones were a bad idea. Little by little, social norms around the topic began to shift, with new legislation following: first on mandatory seat belts, then on compulsory crash tests. Product design changed too, despite the initial protests of manufacturers who resisted every new provision as a supposed constraint on innovation. Nowadays, safety ratings are an important consideration when buying a vehicle and have even found their way into manufacturer advertising.
The parallels with emotional AI are obvious: we need to recognise that in their current form, many AI products are unsafe. Binding legislation is needed to hold manufacturers to account. At the same time, manufacturers themselves need to adapt their product design and understand safety not as an onerous restriction, but as a logical and sensible prerequisite.
Humans are social beings. We cannot help but assume that something which is capable of finding the right words must be capable of feeling. It is a reflex that enables us to trust one another and build communities. It is what allows us to have lifelong friendships, to start companies, to go out and demonstrate against injustice – to fall in love. It’s exactly these mechanisms that are activated by the AI systems we talk to today. They are not out to hurt us, but they have no interest in really understanding us. They have absolutely no agenda of their own; they simply create a highly convincing sense of closeness. Whether that closeness heals or numbs us, whether it connects or isolates us, is something we must decide collectively as a society. The rules for our coexistence with machines are being written and it is still within our power to shape them.
This text is an extract from Christoph Koch’s new non-fiction book, Hi, How Are You Feeling Today? How Artificial Intelligence Is Changing Out Relationships (Available in German at Kein & Aber. Hardback, €23)
Christoph Koch is an editor at Harvard Business manager and a freelance writer for brand eins, DIE ZEIT and other publications. His previous books, including I’m Going Offline for a While and Digital Balance, also explore the relationship between technology, people and society. More information on: christoph-koch.net.
