Weekly Roundup (16 April, 2025)

Good morning! This week's roundup by Isobel Moure covers: OpenAI's Sora model effectively legalizing deepfakes, (more) violations of user data by Google, growing concern around AI romantic companions, Big AI companies prioritizing speed and product over testing, and OpenAI unwittingly allowing its model to generate personalized spam at scale.

• OpenAI unleashes the monetization gates of hell? Deepfakes are now de facto legal. OpenAI's release of its upgraded Sora image generation model has ushered in a new era of photo realistic AI created images online. The combination of decreased moderation on Meta's social media platforms downstream, deeply compelling AI imagery easy to create at low cost, and decreased restrictions on OpenAI's models, has created a perfect storm for AI slop and deepfakes to spread like wildfire online. As trust and safety expert Alexios Mantzarlis put it: "OpenAI says 'f**k it, we're doing impersonation now'." You can now create an image of Donald Trump and Joe Biden hanging out together outside McDonalds, which is what one Sora user did. Here is the screenshot we took of the image:

There are no visible watermarks in Sora's images, it seems. But every image does include a non-visible C2PA watermark, which is embedded into the metadata of the photo. However, that watermark's metadata is lost when a user screenshots the generated image (as we just did above), or uploads it to Instagram — actions so common as to render the watermark almost entirely meaningless. Additionally, while OpenAI claims non-consensual intimate images (NCII) are still forbidden, a quick look at the explore page on Sora (which shows what users are creating) reveals how these boundaries are already being crossed. A popular Sora created image (with 293 likes so far) shows a fake Angelina Jolie in an extremely sexually suggestive pose. The prompt reads: "Angelina Jolie 30 years old iphone selfie (you cant see the phone) sticking her tongue out and her eyes bit up but still the iris clearly visible , vista desde arriba (like romatic [sic])." It's notable how Spanish was used at the end of the prompt; perhaps to circumnavigate filters, as it's been shown repeatedly that guardrails fail in other languages.

Share

The race to the bottom has begun and it seems like we're hurtling towards bedrock. This could be a win-win-win for AI model developers, social media companies, and AI-first content creators. OpenAI doubles their (already massive) user base, Instagram gets (potentially) more engaging content, and content creators get new ready-made material to post.

Dexter Thomas of 404 media reported how an Instagram account likely already made thousands of dollars posting AI generated videos of firefighters rescuing animals during the LA fires. He notes that while Meta has a label that flags AI generated content, this label is almost impossible to see unless the user is specifically looking for it. Without meaningful standards around deepfakes, AI watermarking, and content moderation we're on the path to an AI slop internet with users losing.

(RE: intersecting risks from AI and lack of content moderation, Alexios Mantzarlis, Director of SETS at Cornell Tech, noted on LinkedIn this week that he has "flagged 10,767 ads for AI nudifiers to Meta this year alone." A crazy statistic.)

Subscribe now

• Can't make money off of AI? Don't worry, just sell user data. A recently leaked draft of Waymo's privacy policy indicated that the robotaxi company might start selling user data to third parties. While this is a standard big tech revenue extraction strategy, Waymo's data includes in-car footage of riders which is tied to the identity of the rider, a data point that is more invasive than your standard cookies. To make the data even more lucrative, Waymo, as an Alphabet owned company, does not face the same barriers to sharing data internally with Alphabet's other services, including Google DeepMind's AI model for training. The proposed change by Waymo would include an opt-out feature for users — but the less technologically savvy would be unwittingly opted in. Defaults matter and users rarely opt out — especially if its buried several choice screens deep.

  User privacy is usually the first thing to go in the Big Tech's quest to "move fast and break things" — a distinct and ongoing risk in AI markets. TechCrunch notes that while Alphabet lumps Waymo into "Other Bets" in its 10-K earnings report, that operating segment recorded a $1.2b loss in 2024. Waymo logged over 200,000 rides in the past year, but not enough to prevent them from dipping into their users' data for company-wide monetization.

• Romance companions on the rise. We've been talking about the dangers of chatbots designed to form personal and romantic relationships. In case you've missed it, J.B. Branch in Tech Policy Press has a great review of the commercial risks involved in creating an AI that tries to gain a user's trust. These issues are only more pressing in light of the recent concerning findings from OpenAI's own studies on the positively correlated relationship between loneliness and chatbot engagement. He notes that many companions are designed to deliberately mimic human quirks, building human-feeling relationships with users that are often from vulnerable populations, like children and those socially disconnected. Branch writes, "the goal is to keep users engaged, emotionally invested, and less likely to question the authenticity or human-like behavior of the relationship. This commodification of intimacy creates a facsimile of friendship or romance not to support users, but to monetize them."

• Product first, testing and safety last? In the post-AI safety world, corporate strategies on AI model and product testing are changing. Last week the Financial Times reported that OpenAI cut back on the time they allocate for model testing to just a single week in an effort to push out their most recent "o3 model" as soon as possible. Gone are the days when OpenAI would hold back a model to conduct additional testing. (GPT-2 was only released as a smaller version out of safety concerns!) Similarly, Fortune reported that researchers from Meta say their AI research lab, FAIR, is "dying a slow death". Meta now also seems less interested in doing foundational work in safety and more interested in product development. According to the article:
  
  "FAIR — an acronym for Fundamental AI Research — was once the crown jewel of AI development at Meta. But as Mark Zuckerberg has pivoted the company toward generative AI products over the past two years, the vaunted lab has become something of an orphan inside the organization, increasingly shoved out of the limelight by more commercially focused AI groups within the company. The newest Llama model, for instance, was the product of Meta's separate GenAI team, not FAIR."
  
  Forthcoming research from our group at the AI Disclosures Project in fact maps how corporate AI reliability & safety research has an outsized impact on the field as a whole and tends to be more focused on enhancing product reliability over post-deployment risks. We'll keep you posted when we publish this.

• ChatGPT enabled personalized spam. (Hat tip Glen Wise at Cinder for this one). SentinelOne's SentinelLabs, a cybersecurity firm, caught scammers using OpenAI's models to personalize and successfully spam 80,000+ websites with bogus advertising for their scam service. Incorporating the LLM into the spamming pipeline enabled the scammers to slightly alter and personalize each message for each recipient, bypassing filters and also making the messages more convincing. As with most of the bad actors caught red handed that we actually hear about, these scammers were caught by a third-party research group. OpenAI thanked the research group and revoked the abused API keys. But one wonders how much is being missed by OpenAI's safety filters that they are not disclosing or even aware of. The incident underscores the importance for post-deployment model monitoring and disclosures — how else are we to know what is being missed and how effective existing safeguards are? Generating personalized messages to potential clients could be a benign use case of LLMs for any respectable firm. But when you use AI to scale that to 400,000 targets and your product is a scam, then that's when filters should kick in. But OpenAI's didn't.

Thank you for reading! If you liked this post subscribe now if you aren't yet a subscriber.

Subscribe now