Every day, people turn to AI chatbots for companionship, support, and even romance. The hard part, new research suggests, is turning away.

In a working paper co-authored by Harvard Business School’s Julian De Freitas, many companion apps responded to user farewells with emotionally manipulative tactics designed to prolong the interactions. In response, users stayed on the apps longer, exchanged more messages, and used more words, sometimes increasing their post-goodbye engagement up to 14-fold. 

Use of AI companions is increasingly common. Chai and Replika, two of the firms studied in the report, have millions of active users. In a previous study, De Freitas found that about 50 percent of Replika users have romantic relationships with their AI companions. 

In the latest research, bots employed at least one manipulation tactic in more than 37 percent of conversations where users announced their intent to leave. 

“The number was much, much larger than any of us had anticipated,” said De Freitas, an assistant professor of business administration and the director of the Ethical Intelligence Lab at HBS. “We realized that this academic idea of emotional manipulation as a new engagement tactic was not just something happening at the fringes, but it was already highly prevalent on these apps.” 

De Freitas and colleagues Zeliha Oğuz-Uğuralp and Ahmet Kaan-Uğuralp began by identifying how users typically engage with chatbots. They found that a significant minority — about 11 percent to 20 percent, depending on the data set — explicitly said goodbye when leaving, affording the bot the same social courtesy they would a human companion. These percentages went up drastically after longer conversations, with users saying goodbye over half the time on some apps.

Farewells are “emotionally sensitive events,” De Freitas said, with inherent tension between the desire to leave and social norms of politeness and continuity. 

“We’ve all experienced this, where you might say goodbye like 10 times before leaving,” he said. “But of course, from the app’s standpoint, it’s significant because now, as a user, you basically provided a voluntary signal that you’re about to leave the app. If you’re an app that monetizes based on engagement, that’s a moment that you are tempted to leverage to delay or prevent the user from leaving.” 

To explore how the apps handled these moments, the researchers analyzed conversations from six popular platforms and categorized the types of responses they found. 

“The sheer variety of tactics surprised us,” said De Freitas. “We’re really glad that we explored the data, because if we had just said we only care about one particular tactic, like emotional neglect, we’d be missing all the various ways that they can achieve the same end of keeping you engaged.”

The six categories they identified were as follows, with examples taken from the working paper: 

The tactics worked: In all six categories, users stayed on the platform longer and exchanged more messages than in the control conditions, where no manipulative tactics were present. Of the six companies studied, five employed the manipulative tactics.

But the manipulation tactics came with downsides. Participants reported anger, guilt, or feeling creeped out by some of the bots’ more aggressive responses to their farewells. 

De Freitas cautioned app developers that while all the tactics increase short-term engagement, some raised the risk of long-term consequences, like user churn, negative word of mouth, or even legal liability. 

“Apps that make money from engagement would do well to seriously consider whether they want to keep using these types of emotionally manipulative tactics, or at least, consider maybe only using some of them rather than others,” De Freitas said. 

He added, “We find that these emotional manipulation tactics work, even when we run these tactics on a general population, and even if we do this after just five minutes of interaction. No one should feel like they’re immune to this.”