Weekly Roundup (March 26, 2025)

Happy Wednesday everyone. Today's weekly roundup covers: competing visions of the AI industry's future, the U.S. State Department's use of AI, new research on intimate relationships with AI, a benchmark based on actual evidence of misuse, and California's AI advisory group's draft research report on AI governance.

• "The model is the product": integration and bundling. A core tenet of our research is that AI's evolving market structures shape future risks and needed controls. There's significant debate about which of the AI "layers" (consisting of hardware, foundation models, middleware, cloud providers, applications, and more) will be the biggest winners — capturing most of the value, on the back of higher moats and more sustainable competitive advantages. Alexander Doria, an LLM engineer at Pleias, argues on his blog Vintage Data that increasingly "the model is the product", and this is how AI model developers will end up on top, as they bundle and integrate AI applications (such as Search, DeepResearch, and other tools) with their model — shifting features away from the API, open to all developers, and integrating them instead exclusively to the consumer facing front-end.
  
  Trent Kannegieter on Tech Policy Press, has a very different vision of AI's future market structure, and explores how models might become a commodity. He writes: "commoditization is the process of products or services becoming "standardized, marketable objects…No firm can create and sustain a differentiated model performance advantage that puts their product ahead of competitors." DeepSeek's latest V3 model release — fully open and with an MIT license — is an example of this application layer heavy market structure. In this future, value moves up (applications) and down (hardware) the stack, but away from the foundation model.
  
  This is already what we see happening in China: rapid consolidation of the model layer as companies change course and focus on applications within specific industries. This is outlined in a recent must-read Financial Times (FT) piece: "01.ai has stopped "pre-training" large language models to focus on selling tailored AI business solutions using DeepSeek's models; Baichuan has opted to concentrate on the healthcare market; and Moonshot has slashed its marketing budget for its Kimi chatbot to focus on model training."

• AI enabled online social media monitoring. Axios reports that the State Department is using AI to review immigrants' social media profiles, as part of their highly controversial "Catch and Revoke" program to deport activists on student visas. This is concerning whether or not the AI system is accurate, as AI facilitates the automated extension of social media monitoring dramatically. It demonstrates that AI risks stem from how systems are deployed and by whom, including the state, and not just from advanced capabilities in the hands of malicious civilians.

• Loneliness drives chatbot dependence. OpenAI and MIT Media Lab released two studies this week investigating how users form emotional dependence on chatbots. The researchers found that the majority of users were not forming an emotional bond with the model. However, the top 10% of users experienced downsides. The more the participants interacted with the model, the more they formed unhealthy relationships. Increased usage of ChatGPT was found to have correlated with increased loneliness, emotional dependence, and reduced social interaction. These initial results don't show that the interaction caused the loneliness, but rather suggests that lonely people became more attached to the model.

  

  It's impressive that OpenAI is conducting research on post-deployment behavioural impacts and openly publishing the results, even if they potentially go against their own agenda. However, OpenAI's products are currently geared towards productivity and business use cases, so these findings arguably stand to hurt "entertainment" chatbots like those from Character.AI or Replika more than their own GPTs. These studies point to the potential for corporates' commercial incentives to weaponize the loneliness epidemic. The social media era showed us that corporate developers have an incentive to advance algorithmic technologies that effectively capitalize off users' loneliness in order to ratchet up prices and extract profits. Dopamine hits induced through infinite scroll, pure algorithmic recommendations (dismantling of the social graph), autoplays, and algorithmic optimizations for engagement were some of the central innovations.

Subscribe now

• Benchmarks based on evidence from real world usage. Google released a benchmark this week to evaluate AI models' abilities to enable cyberattacks. What's special about this one is that it is based on evidence from actual malicious uses of Google's AI models in the real world. To ground their research, Google's researchers started by "analyzing over 12,000 instances of real-world attempts to use AI in cyberattacks from more than 20 countries that were catalogued by Google's Threat Intelligence Group." From there they based their model evaluations on representative cases of how bad actors were maliciously using their model. The authors argue that this framework: "empowers red teams to more accurately model AI-enabled adversary behavior, generating emulation plans that combine known tactics, techniques, and procedures with evidence of AI-driven cost reductions. Our evaluations revealed that current AI cyber evaluations often overlook critical areas."

  We've been advocating for evaluations grounded in actual usage, so this is an exciting first step by Google. Their results show that evaluations informed by post-deployment risks can fill in large gaps in existing AI evaluation practices. A next step would be to publicly disclose the 12,000 instances of AI cyberattack misuse to increase transparency, accountability, and assist in other teams' research to make these models more robust and safer.

• New recommendations California's AI policymaking. Following California governor Gavin Newsom's rejection of SB 1047, a working group was established to prepare a report to aid in enacting better AI governance. According to the authors: "This report provides an evidence-based foundation for AI policy decisions to support sound policy analysis". We believe its emphasis on disclosures and post-deployment monitoring is warranted by the urgent need for in-context evidence on how these systems are being used and their impacts. The report calls for post-deployment AI systems monitoring, whistleblower protection for AI employees, increased corporate transparency, and aligning corporate and consumer incentives through disclosures. It notes: "Holistic transparency begins with requirements on industry to publish information about their systems. Case studies from consumer products and the energy industry reveal the upside of an approach that builds on industry expertise while also establishing robust mechanisms to independently verify safety claims and risk assessments."

  With the U.S. AI Safety Institute being redirected away from safety, bias, and fairness concerns, and the Trump administration showing minimal interest in federal AI regulation that could stifle innovation, states now bear the responsibility for creating consumer protection policies — though they must first overcome industry lobbying against state-level regulations. While state regulation risks creating a burdensome patchwork of policy, hopefully California (home to 32 of the world's 50 leading Al companies!) can set a workable example for other states' AI legislations.

Thanks for reading! If you liked this post subscribe now, if you aren't yet a subscriber.

Subscribe now