Weekly Roundup (April 2, 2025)

Editor's note from Ilan: Isobel is away this week in Mexico rock climbing, so Sruly is taking over this week's roundup.

AI is coming for engagement maximization. Character.ai's highly addictive chatbots have probably been the starkest illustration of LLMs maximizing for user engagement so far. (Ilan tested out its service and now receives weekly notifications from their one chatbot, asking me: "Are you doing OK?"). Tinder, the original swipe left dating app, has now officially entered the AI engagement maximizing game. Tinder has introduced an in-app game that allows users to flirt with artificially intelligent chatbots, aiming to boost user engagement. For now, games are limited to five per day, with around 3 minutes a game, per user. The games are free and voice-based, and use OpenAI's GPT-4o and GPT-4o mini at the backend. How exactly Tinder (owned by Match Group) is fine-tuning these models for engagement is not public knowledge. And when exactly such fine-tuning becomes overly aggressive and unhealthy engagement is unclear, especially without company disclosures on the metrics and datasets they are using for model fine-tuning, along with their testing benchmarks. And while "engagement" could in theory be regulated via mobile app stores (e.g., such as age restrictions), on the world wide web this becomes far more difficult.

China's AI labeling laws meet the gatekeepers. In an important regulatory move, four key Chinese regulatory authorities have jointly issued comprehensive measures requiring all AI-generated synthetic content to be clearly identified, effective September 1, 2025. According to South China Morning Post: "The directive says there must be explicit and implicit labels for AI-generated text, images, audio, video and virtual content. Explicit markings must be clearly visible to users, while implicit identifiers, such as digital watermarks, should be embedded in the metadata." Reportedly, many large social media companies have already started implementing these rules. However, Chinese app stores have so far not required AI content labeling for apps wanting to be listed in their app stores. (Thibault Schrepel, associate professor at Vrije Universiteit, notes, these requirements exceed those in the EU's AI Act. However, the Chinese focus is squarely on output identification rather than input transparency – a critical distinction from the EU approach.)

China's regulation seems to recognize that the efficacy of content-labeling remains questionable if not hard-coded into technical standards, since otherwise it becomes reliant on the rules and monitoring systems of the digital infrastructure giants.

Such infrastructure is ubiquitous online, Robin Berjon notes in a wide-ranging essay from late February 2025. Central to digital infrastructure is a "structural power" that can decide on standards and data governance. Though it may be a mistake to ascribe too much power to them. App Stores, despite being one such infrastructure bottleneck, seem unable to properly monitor and regulate what happens within their shop, due ultimately to the complexity of what goes on there — by way of corporate ownership of apps and the phone's root network connection being difficult (AI agent apps take note). To the extent that technical standards at the appropriate layer can make broader standards (as governance mechanisms) easier to enforce, and less reliant on a gatekeeper's goodwill, the better. The problem is that such technical standards arise from market-based adoption in places like the U.S. And given concentrated digital markets there, such adoption tends to be determined by one or two internet giants, notes Berjon. This means that a different approach to standard-setting is required; what Berjon calls "post-voluntary" standards. But what exactly that should entail, we don't know. If you do, let us know!

AI safety is not yet a science — far from it. A new paper co-authored by researchers from Google DeepMind, UC Berkeley, Microsoft, Hugging Face, Stanford, MIT, and Anthropic argues that the current evaluation ecosystem for generative AI is inadequate for assessing real-world performance and safety. The authors highlight how, despite widespread deployment in critical sectors like medicine, law, and education, generative AI systems have not undergone the rigorous evaluation expected in other fields, leading to unforeseen issues like misinformation and inappropriate interactions. They, like us, argue that safety research should include both pre-deployment and post-deployment risks and that updates to both must be undertaken based on observed risks in the real world.

AI-powered shopping shows explosive growth. Adobe Analytics has released eye-opening data showing a remarkable 1,200% increase in traffic from generative AI sources to U.S. retail websites between July 2024 and February 2025, with the growth rate doubling approximately every two months since September 2024. Based on analysis of over 1 trillion visits to retail sites, the data shows that AI-directed shoppers are more engaged – browsing 12% more pages, spending 8% more time on sites, and exhibiting a 23% lower bounce rate compared to traditional traffic. While still modest compared to established channels like paid search, these trends suggest AI shopping assistants are reshaping consumer behavior. Important questions remain: How do the AI models themselves influence purchasing decisions? What percentage of potential purchases never make it past the AI conversation? Assessing these would require integrated model developers and AI services like Perplexity making available post-deployment conversations, the user-model interactions — through APIs to selective researchers, similar to Anthropic's Clio; or at the least disclosing operating metrics on AI conversation conversions, similar to how click-through rates underpinned operational assessments of web-based advertising.

Industry collaboration on AI safety takes shape. The Frontier Model Forum announced on March 28, 2025, that all member firms — including Anthropic, Google, Meta, Microsoft, OpenAI, and others — have formalized an agreement to share information about threats, vulnerabilities, and concerning capabilities in frontier AI systems. This marks a significant shift from the previous approach where each developer largely evaluated risks independently. The initial focus is fairly narrow – prioritizing chemical, biological, radiological, and nuclear (CBRN) threats and advanced cyber vulnerabilities. Information sharing will cover model vulnerabilities, threats targeting frontier AI systems, and capabilities that could potentially cause large-scale harm. While promising, this collaboration seems to primarily address hypothetical frontier risks, rather than the actual everyday harms currently affecting users (such as scams, disinformation, or various other risks that feature prominently in the AI Incident Tracker). Still, in our research, we found a distinct lack of standardized benchmarks and risk policy frameworks across competing models. Hopefully this will help address that at least for frontier risks.

"I cannot imagine us putting out the transformer papers for general use now". Feeling the pressure from its parent company, Google's DeepMind reportedly began to crack down on what papers are allowed to be published. Any "strategic papers" that harm Google's AI business, whether from criticizing its Gemini model or revealing new innovations that competitors could use, apparently now receive a six month embargo before having a chance of being released, according to reporting by the Financial Times. This has bad implications for AI safety & reliability if model weaknesses and vulnerabilities land up being kept confidential for longer than necessary. More than that though, it risks further concentrating the AI market if new technologies (as research innovations) are prevented from being disseminated and diffused throughout the economy. One counteracting force is labor mobility: many top computer science researchers can get a job anywhere and want to publish their research openly, such that measures that prevent their research from being published encourages them to leave for competitors, along with their ideas.