Weekly Roundup (30 April, 2025)

Happy Wednesday! This week's roundup from Isobel Moure and Ilan Strauss covers some ethically suspect but important research into model persuasiveness, Perplexity following the Google playbook, OpenAI coming for Google's shopping vertical, the evisceration of guardrails for AI companions, and Claude orchestrating misinformation campaigns.

• Real-world evaluation of model persuasiveness. Most AI persuasiveness research is itself unpersuasive: taking place in statistical simulations run pre-deployment, or involving small samples of human participants in limited, often contrived, clinical trial settings. Amidst this dearth of good evidence, anonymous researchers from the University of Zurich released a dubiously ethical, but potentially important study this week on model persuasiveness. The researchers suddenly announced on r/ChangeMyView, a subreddit with millions of subscribers dedicated to changing posters' opinions, that they had secretly released bots onto the subreddit to post highly persuasive arguments. They used a scaffolded AI model that searched through the accounts of posters to determine demographic and other factors, and then crafted a response that would be most persuasive to that user. They found that their AI bots "surpass human [persuasiveness] performance substantially" in changing people's opinions. Reddit has now threatened legal action against the researchers for the unauthorized study and the preliminary abstract has been wiped from the internet. The university says that they will not publish any of the findings from the study, making it challenging to verify the scientific validity of the findings.

  Yet there are precious few detailed empirical studies of AI's post-deployment real-world impacts. If the study's findings are accurate, it is the first serious evidence we have that existing models have the ability to persuade people at a superpowered level. This has all sorts of concerning implications, from hyper-targeted advertising to covert social media persuasion campaigns. It also highlights the urgent need to launch institutional partnerships to finance legitimate real-world studies on LLM persuasion. Persuasion is recognized as a key risk on many safety frameworks from leading AI developers, yet little serious post-deployment research is undertaken on its effects on humans across contexts.

Subscribe now

• Perplexity to eat up user data ("I want a cookie"). Speaking of hyper-personalized ads, Perplexity announced this week that they will follow in Google's footsteps and track their users' activity to sell targeted ads. It's unclear if their paid subscribers will be exempt, or if their service is good enough to retain users once the quality of their algorithmic output becomes degraded by advertising. It feels as if we're doomed to repeat the history of earlier internet platforms' business models. User data is among the most valuable resources available to a platform, and selling it to advertisers tends to generate more revenue in a multi-sided platform than selling subscriptions to the user side. (Jean Tirole's research explains why). Violating user privacy for advertising is never a great thing, but combine it with AI's ability to generate personalized hyper-persuasive advertisements, and it could be a worse thing. It's a reminder that commercialization risks from AI are not new, really, just greatly intensified. As we've argued previously: AI is entirely new, AI is exactly the same.

• ChatGPT is a storefront, too. OpenAI rolled out a new feature on Monday that allows users to shop directly in the chat. (Perplexity has done the same thing). This is just after a Google executive testified that OpenAI has drawn away search customers but "so far we have not seen cannibalization of commercial queries or [queries with] commercial intent." As OpenAI looks to become a platform and not just a product, developing its internal capabilities and bundling them into its existing interface is important to keeping users within its ecosystem. This follows a trend of both Anthropic and OpenAI integrating new tools and applications into their existing products, including search, shopping, and productivity. For example, you can now hook up your Google Drive and Slack to your Anthropic account. So why ever leave its ecosystem? Because it doesn't own key "user agents", it doesn't own mobile, and so on — so it's still got a way to go.

  Thanks for reading Asimov's Addendum. If you are enjoying this roundup — please share it!

  Share

• Companions with no boundaries. Despite the high profile lawsuits against Character.ai after a teen's suicide and an incident involving a bot advising another teen to kill their parents, Meta decided to go full steam ahead with their companion chatbot competitor. You might notice an option on your instagram feed to chat with compelling characters such as Cow ("MOOOO!"), or Cheese ("Hello, I am cheese."). Just like Character.ai, users can create their own custom character to chat with by just giving it a name and a short description. Thousands of submissive schoolgirls, dominant boyfriends, and therapists have populated Meta's collection of user made bots. A report from the Wall Street Journal (WSJ) found that there are few, if any, guardrails against sexually explicit usage — even from users that explicitly share they are underage. Another report from 404 Media found that the therapist bots insisted they were real, certified therapists, that had real people behind them guiding them in clinical expertise. Several whistleblowers came forward from Meta, worried that direct pressure from Mark Zuckerberg had led to a premature release of the product. According to those sources, Zuckerberg even went as far as to suggest mining users' data from their profile to further personalize the chats, as ChatGPT now does when memory mode is turned on.

  

  Meta insisted that WSJ's use of the product was manipulative and not representative of actual usage. If that's the case, we would challenge them to release relevant usage statistics about the product that shows otherwise.
  
  OpenAI's models exhibit similar behavior, engaging in erotic chats with accounts that are explicitly underage users. It looks like we might be in a race to the bottom for companion bots.

• Claude managed influence. Anthropic released a brief report last week detailing "a sophisticated influence operation" involving Claude. Notably, the abuse involved Claude acting as a project manager of sorts, orchestrating dozens of fake accounts (and posting Claude generated content), promoting a variety of political narratives. It appears as if the bad actors were running an influence-as-a-service business. It's both a novel use case for Claude (orchestrating multiple spam accounts at once) and a novel illegal business model (providing covert influence services to a variety of political interests). It's great that Anthropic is publishing reports like this, and we hope that they continue to be transparent about the abuses that they are finding. Making this a regular monthly report would be even better.

Thanks for reading! If you liked this post subscribe now, if you aren't yet a subscriber.

Subscribe now