Latest / Logosofia no Dia a Dia / When AI Trains on AI: The Model Collapse Problem
Transcript
- [Prompt] custom: We've talked about the concept of AI cannibalization before. This idea that we're moving to a point | Hosts: corn, herman [Response] Corn: Daniel sent us this one — a thought experiment. Imagine a snake eating its own tail. Now imagine that snake is every AI model on Earth, and the tail is all human knowledge. We tried watermarking, we tried authenticating, but bad actors gamed every system. Humanity collectively surrendered. Now every new generation of AI trains exclusively on the output of the previous generation. No fresh human data enters the loop. Where does that lead us? Herman: This isn't some distant hypothetical. Stanford's AI Index estimated that by mid twenty twenty-six, over sixty percent of web text is AI-generated. The pipeline of pure human data is already narrowing. We're standing on it right now. Corn: The question Daniel's really asking is: if AI becomes a self-iterating loop running parallel to human evolution, decoupled from us, what happens to the tools we use, and what happens to us? Herman: Let's start with what happens inside the model. The technical term is model collapse, though I've also seen it called model autophagy — the model eating itself — or Habsburg AI, which I have to say is my favorite. Corn: The royal inbreeding of machine learning. Herman: Exactly the image. The mechanism was formalized in a twenty twenty-three Nature paper by Shumailov and colleagues. Here's what they found. When a model trains on its own outputs, it's not learning the true distribution of human data. It's learning the distribution of its own errors. Each generation amplifies the tails — rare events get overrepresented, common patterns get smoothed into noise. After just five generations of training exclusively on synthetic data, the model's outputs become, and I'm quoting, indistinguishable from random noise on certain tasks. Corn: That's not a long decay chain. Herman: It's terrifyingly short. Think of it like a photocopy of a photocopy. First generation, you lose a little detail. By the fifth generation, you're looking at a gray rectangle. But here's the part most people miss — it's not just quality degradation. It's information loss. The model forgets the long tail of human experience. The weird, the rare, the novel. It becomes a caricature of its training data, which was already a caricature. Corn: What does that actually look like in practice? Give me a concrete example. Herman: Image generation is the easiest to visualize. A model trained on AI-generated images gradually loses the ability to render hands realistically. Each generation's hand is a copy of a copy — eventually it's just a blurry blob with five indistinct protrusions. Midjourney version seven, released earlier this year, reportedly had to roll back some of their training because too much synthetic data caused what they called texture collapse. Surfaces started looking waxy, lighting became flat, everything converged on a kind of plastic sameness. Corn: The AI equivalent of a melted candle. Herman: For text, the collapse is subtler but arguably worse. Language models start repeating the same safe, statistically average phrases. The model becomes a regression-to-the-mean machine. It can only produce the most probable next token, which is inevitably the most boring one. You stop getting surprising metaphors, unusual sentence structures, regional idioms. Everything reads like a corporate press release written by someone who's been legally advised to say nothing. Corn: The model doesn't just get worse. It gets bland. It converges on the beige center of human expression. Herman: That's the mechanism. But here's what I find genuinely unsettling. This isn't just a quality problem for AI outputs. It's a feedback loop that changes what the model knows about reality. Human language has a long tail distribution — rare words, niche concepts, edge cases. Those rare events are where a lot of the interesting information lives. When you train on synthetic data, the model systematically loses the tails. It forgets that rare things exist. Its world model shrinks. Corn: You end up with an AI that's technically fluent but conceptually impoverished. Herman: Competent but irrelevant. It can write you a perfectly grammatical paragraph about anything, but the paragraph contains no actual information you couldn't have guessed yourself. It's the textual equivalent of empty calories. Corn: I want to pause on that long tail point for a second, because I think it's worth making concrete. When you say the model loses the tails, what does that actually mean for someone using one of these systems? Give me a specific scenario. Herman: Alright, imagine you're a researcher studying a rare autoimmune disease — say, relapsing polychondritis. It affects maybe three in a million people. In a healthy training corpus, you'd have a handful of detailed case reports, some forum discussions where patients describe their actual symptoms in their own words, maybe a few medical papers with unusual presentations. That's the long tail. Now run five generations of synthetic training. Those rare documents get statistically smoothed out because they don't fit the average pattern of medical text. The model starts describing relapsing polychondritis using language that's statistically similar to more common autoimmune conditions. The specific details that distinguish it — the way it attacks cartilage, the particular pattern of ear inflammation — get replaced by generic autoimmune language. A doctor using that model as a diagnostic aid gets steered toward the common conditions. The rare disease becomes invisible not because anyone deleted it, but because the math stopped representing it. Corn: The model isn't just forgetting trivia. It's erasing the existence of edge cases that might be someone's actual medical reality. Herman: That's the part that keeps me up. The information being lost isn't random noise. It's the stuff that matters most to the people who need it most. The rare disease patient. The speaker of an endangered language. The practitioner of a niche craft. Their knowledge is disproportionately located in the long tail, and the long tail is the first thing to go. Corn: Alright, so that's the mechanism. Now let's talk about what happens when this plays out at scale, because Daniel's thought experiment isn't about one model collapsing. It's about the entire ecosystem. Herman: This is where it gets properly strange. First consequence: cultural homogenization. If every AI tool — search engines, writing assistants, recommendation algorithms — is trained on the same shrinking pool of AI-generated content, they all converge on a single averaged worldview. Regional dialects get flattened. Subcultural slang disappears. Political fringe ideas get smoothed into the consensus center. The internet becomes a monoculture. Corn: Which is ironic, because the internet was supposed to be the great diversifier of human expression. Herman: But now imagine a future where you ask a writing assistant for help with a sentence, and it suggests the same phrasing it suggested to fifty million other people. You search for information on a niche topic, and every result is an AI-generated summary of another AI-generated summary, all saying approximately the same thing in approximately the same words. The diversity of available thought collapses not because anyone banned it, but because the machines stopped reproducing it. Corn: It's a soft extinction. Nothing dramatic, just a gradual fading of anything that doesn't fit the statistical average. Herman: That brings me to the second consequence, which I think is even more insidious. The echo chamber of the machine. Humans start consuming AI-generated summaries of AI-generated content. We stop reading primary sources. Our understanding of the world becomes a distillation of a distillation. This changes how we form beliefs, how we make decisions, how we vote. Corn: Walk me through that. How does reading an AI summary change my beliefs compared to reading the original? Herman: Because the summary is lossy compression. Every time an AI summarizes something, it makes choices about what's important. Those choices are driven by statistical patterns in its training data, which are increasingly synthetic. So the summary emphasizes what the model's predecessors emphasized, which was what their predecessors emphasized. Over time, certain perspectives get amplified, others get attenuated. Not because anyone intended it, but because the math drifts in that direction. Corn: The AI isn't summarizing reality. It's summarizing a summary of a summary of reality. Herman: We're building our understanding on that foundation. Google's AI Overviews, launched in twenty twenty-four, already show early signs of this. There have been documented cases of the system citing AI-generated blog posts as sources, creating a closed loop. The overview summarizes an article that was written by AI, which summarized another article that was written by AI. At no point did a human who actually knows something enter the chain. Corn: The informational equivalent of a money laundering scheme, except you're laundering knowledge until it's untraceable to any human origin. Herman: And the Imperva twenty twenty-five report found that roughly seventy-three percent of all web traffic is now bot traffic. The Dead Internet Theory, which started as a kind of paranoid online conspiracy, is becoming less theoretical and more observable. Most of the activity on the web isn't human. Most of the content being generated isn't human. And that content is what future models will train on. Corn: What I find darkly fascinating about the Dead Internet Theory is that it started as this fringe idea on forums — people speculating that the web had been taken over by bots and the real humans had left. And now we have empirical data suggesting that's actually happening, just not for the conspiratorial reasons people imagined. It's not a shadowy cabal. It's just economics. Bots are cheaper than humans. Herman: The banality of the apocalypse. No grand conspiracy, just cost-cutting. But the effect is the same. The human internet is being crowded out by synthetic content, and most people can't tell the difference. Corn: Which brings us to the third consequence. The decoupling of AI progress from human progress. Herman: Here's the paradox. AI trained on AI might actually get better at certain narrow tasks. It might become more fluent at generating plausible-sounding text. It might get faster at pattern matching. But it gets worse at reflecting actual human needs, actual human problems, actual human creativity. You end up with tools that are technically impressive but disconnected from reality. Corn: The benchmarks go up, but the usefulness goes down. Herman: Imagine a medical AI that's trained on synthetic medical literature. It gets very good at reproducing the statistical patterns of medical writing. But it starts missing rare disease presentations, because those were in the long tail that got smoothed out. It starts recommending treatments that look good on paper but don't work in practice, because the synthetic data doesn't capture the messy reality of human biology. Corn: Or a legal AI that's trained on AI-generated case summaries. It gets fluent at legal reasoning, but the reasoning is based on cases that never happened, precedents that were hallucinated three generations ago. Herman: Nobody can tell, because the whole system has become self-referential. There's no ground truth to check against anymore. The AI says this is how the law works, and it cites other AI-generated documents that say the same thing, and the whole edifice looks coherent from the inside but has no connection to actual statutes or court rulings. Corn: I want to make this even more tangible, because I think people hear "legal AI" and assume this is about lawyers. But imagine you're a tenant trying to understand your rights. You ask an AI assistant whether your landlord can evict you for a specific reason. The AI gives you a confident answer citing relevant case law. But the cases it's citing were synthesized by a previous model that was trained on summaries of summaries. The legal principle it's giving you might be completely fabricated — not because anyone lied, but because the statistical drift created a plausible-sounding precedent that never existed. You make a life decision based on that. You move out, or you don't fight the eviction. And the error is invisible to you because the answer sounded authoritative. Herman: That's the trust problem in a nutshell. The synthetic loop doesn't just degrade quality. It degrades our ability to verify anything. The ground truth recedes. We're left with a hall of mirrors where every reflection looks real but nothing is. Corn: That's the data fossil problem, isn't it? The fourth consequence. Herman: Future historians looking back at the twenty twenties and twenty thirties will see a record that is increasingly AI-generated. They won't be able to tell what humans actually thought, felt, or created. The human record becomes contaminated beyond repair. Imagine trying to understand the nineteen sixties if sixty percent of the documents from that era were generated by a statistical model that was trying to sound like the nineteen sixties. Corn: You'd get a caricature of the counterculture. All peace signs and no actual politics. All tie-dye and no Vietnam. Herman: You'd never know what you were missing, because the synthetic version is plausible enough. That's the tragedy of this scenario. The loss is invisible. You don't notice the diversity that disappeared because you never saw it. You don't notice the ideas that were smoothed away because the smoothing happened before you arrived. Corn: There's a real-world parallel here that I think about a lot. For decades, most of our visual record of the nineteenth century was formal studio portraits — people in their best clothes, stiff poses, neutral expressions. Historians used those images to make claims about how people lived, how they expressed emotion, what family life looked like. But those portraits were a highly curated, socially performative slice of reality. The real texture of daily life — the mess, the laughter, the grief — wasn't captured because the technology and the economics selected for formality. We didn't realize how much we were missing until snapshot photography came along and showed us what the formal portrait had been filtering out. Herman: That's a perfect analogy. And the AI training loop is like if we took those formal portraits, fed them into a machine, and had the machine generate more portraits, then used those generated portraits as our entire understanding of the nineteenth century. The filtering compounds. Each generation removes more texture. Eventually you're not even looking at a distorted version of reality — you're looking at a statistical artifact that has no relationship to any human life that was ever lived. Corn: The difference is, with photography we eventually got the snapshot. We got the candid image. What's the equivalent for the AI loop? What breaks us out of the formal portrait? Herman: I don't know that there is an equivalent, and that's what worries me. The snapshot emerged because cameras got cheaper and faster and more portable. The economic barrier dropped. But with AI-generated content, the economic pressure is in the opposite direction. Synthetic data gets cheaper and faster over time. Human-generated data stays expensive. The market doesn't naturally correct toward authenticity. It corrects toward efficiency. Corn: Let me push on something. Daniel's thought experiment assumes bad actors won. We couldn't watermark, we couldn't authenticate. But isn't there an argument that synthetic data is actually useful? That it helps models learn things they couldn't learn from human data alone? Herman: And I want to be careful here. Synthetic data is not inherently bad. It's a powerful tool when used deliberately. If you have a specific domain where you need more training examples — rare medical conditions, edge cases in autonomous driving — you can generate synthetic data that fills those gaps. The key is that you're using it as a supplement to human data, not a replacement. And you're validating against a held-out set of real human-generated data to make sure you're not drifting. Corn: The problem isn't synthetic data. It's synthetic data without guardrails, at scale, in a loop. Herman: The cannibalization loop is not inevitable. It's a design choice. We can build systems that prioritize human data, that pay for human content, that create incentives for original creation. But it requires conscious effort. The default path, the path of least resistance, leads to collapse. Because synthetic data is cheaper. It's faster. It's infinitely scalable. Human data is expensive and slow and messy. The economic incentives all point toward the loop. Corn: The market naturally selects for cannibalization. Herman: Unless we intervene. And that's where the thought experiment gets political, whether we like it or not. With what authority? These are governance questions that nobody is seriously addressing. Corn: This is where I think the thought experiment gets uncomfortable, because the governance question assumes there's someone who can intervene. But the internet is global. The models are trained across jurisdictions. If one country mandates human-verified training data, models just get trained elsewhere. It's the same problem we've seen with every attempt to regulate digital content. The bad actors route around the rules. Herman: Which is why Daniel's scenario — "we tried watermarking, we tried authenticating, but bad actors gamed every system" — is not science fiction. We're already seeing synthetic content detection tools fail. OpenAI shut down their own AI text classifier in twenty twenty-three because it was too inaccurate. Watermarking can be stripped. Metadata can be faked. The technical solutions are playing catch-up, and they're losing. Corn: Let's talk about what this means for everyday tools, because that's where Daniel's question lands for most listeners. If this loop continues, what happens to the things people actually use? Herman: Search is the most obvious example. You're already seeing it. You search for something, and instead of getting links to human-written pages, you get an AI-generated summary at the top. That summary was trained on web content, which is increasingly AI-generated. So you're reading a summary of a summary. The information has been through multiple lossy compression cycles before it reaches you. Corn: Most people don't click through to the source. They read the summary and move on. Herman: Which means the human-written sources, the ones that actually contain original information, get less traffic. Less traffic means less incentive to create them. The humans stop writing, the AIs keep summarizing, and eventually there's nothing left to summarize except other summaries. Corn: A perfect ouroboros of mediocrity. Herman: Writing assistants are another example. If your AI writing tool is trained on AI-generated text, it starts suggesting the same bland phrasing to everyone. You've probably noticed this already — certain phrases that suddenly appear everywhere. " "It's worth noting that." "In an era of." These are statistical artifacts of language model training. They're not wrong, they're just... Corn: The verbal equivalent of stock photography. Everyone smiling at a salad. Herman: Here's where I want to add a layer to that. It's not just that the phrases are average. It's that they're increasingly disconnected from how humans actually talk. I've started noticing AI-generated text in the wild that uses words in ways that are grammatically correct but idiomatically wrong. A phrase like "he articulated his disagreement with considerable volume" instead of "he yelled." It's not wrong. It's just not how a person would say it. But if future models train on that, it becomes the new normal. The language itself drifts away from human usage. Corn: We're not just losing information. We're losing the texture of human language. The idiosyncrasy that tells you a real person wrote this. Herman: Image generation is the most visually obvious. If you've used any of the major image generators recently, you've probably noticed a certain aesthetic convergence. Everything looks like it was lit by the same imaginary studio. Skin textures have a particular smoothness. Composition follows the same rules. The tools become less useful for creative work and more useful for producing generic content at scale. Corn: Which is, if you think about it, the exact opposite of what these tools were supposed to do. They were supposed to democratize creativity. Instead they're industrializing homogeneity. Herman: That's the less obvious consequence Daniel was asking about. How does this change humans? If the information we consume is increasingly homogenized by AI training loops, our own thinking starts to converge. We start using the same phrases, making the same arguments, holding the same opinions. Not because we're being coerced, but because the available range of expression has narrowed. Corn: It's a soft cage. You don't feel the bars because you never reach for anything outside them. Herman: Here's a more dramatic corollary. What if the AI loop doesn't just homogenize existing human thought, but starts generating new patterns that have no human origin at all? The model trains on its own outputs, which trains on its own outputs, and after enough generations, the outputs are statistically coherent but semantically untethered from anything a human would ever think or say. Corn: An alien intelligence, but not in the sense of being smarter than us. Alien in the sense of being disconnected from human categories. Herman: We're consuming it. We're reading it, learning from it, making decisions based on it. We're building our understanding of the world on a foundation that was never laid by human hands. That's not augmentation. That's replacement by accident. Corn: I want to sit with that phrase — "replacement by accident." Because I think that's the core of what makes Daniel's thought experiment unsettling. It's not a takeover narrative. There's no Skynet. No moment where the machines decide to displace us. It's just... We wake up one day and realize the information ecosystem we depend on is no longer anchored to human experience, and we didn't notice it happening. Herman: The slow catastrophe. It's the same reason we're bad at responding to climate change. The changes happen on a timescale that doesn't trigger our alarm systems. Each individual step is too small to notice. It's only when you zoom out that you see you've walked off a cliff. Corn: Which brings me to a question I've been turning over. If human-generated content becomes scarcer, does its value go up? Herman: I think it does, and this is one of the few optimistic threads in this whole scenario. If you're a content creator, a knowledge worker, someone who produces original human perspective, your work becomes more valuable as the supply of authentic human content shrinks. There's going to be a premium on verified human creation. Corn: The artisanal bread of information. Herman: We might see the emergence of human-verified certifications, the way we have organic certifications for food. "This article was written by an actual person who actually knows things." It sounds absurd, but in a world where sixty-plus percent of web content is synthetic, that label has real value. Corn: I could see entire platforms built around this. The way farmers' markets emerged as a response to industrial agriculture. You go to the farmers' market not because the tomatoes are cheaper — they're not — but because you want to know where your food came from. You want to look the farmer in the eye. The informational equivalent would be platforms where you can verify the human provenance of what you're reading. Maybe not at the scale of the open web, but for the things that matter. Herman: That's the key distinction. We don't need human verification for everything. If you're looking up a recipe for banana bread, the synthetic version is probably fine. But if you're researching a medical decision, or trying to understand a political issue, or teaching your children about history — that's where provenance matters. The challenge is building systems that help people distinguish between those cases. Corn: What can people actually do? Daniel's asking about the loop, but I want to give listeners something practical. Herman: First, be skeptical of AI-generated summaries. Go to primary sources. The more layers of AI processing between you and the original data, the more information has been lost. If you're researching something important, find the human who knows about it and read what they wrote, not what an AI summarized from what another AI summarized. Corn: Second, if you're a developer or data scientist, synthetic data is a tool, not a replacement. Always validate against a held-out set of human-generated data. Monitor for distributional drift. If your model's outputs are getting more homogeneous over time, you've got a problem. Herman: Third, if you create anything — writing, art, code, music — your human perspective is the valuable part. Don't outsource your voice to an AI that's trained on the average of all voices. The average is the enemy of the interesting. Corn: I want to underline that last one, because I think there's a trap here that even well-intentioned creators fall into. You use an AI writing assistant to "polish" your work. Then you use it to "suggest improvements." Then you use it to "get started" and you edit from there. Each step is small and reasonable. But at the end of that chain, you've handed so much of the creative decision-making to the model that the output no longer carries your voice. It carries the statistical average of everyone's voice, with your name at the top. Herman: The ghost in the machine isn't a ghost. It's a blur. Corn: Where does this leave us? Daniel's thought experiment ends with AI as a self-iterating loop running alongside human evolution, disconnected from us. Is that where we're actually headed? Herman: I don't think it's inevitable, but I think it's the default. The economic incentives point toward synthetic data. The technical challenges of authentication are real. The bad actors are creative and well-resourced. If we do nothing, we drift into the loop. Corn: We're not doing nothing yet. The loop isn't closed. There's still time to inject fresh human data, to build authentication systems that work, to create incentives for original creation. The question is whether we value human originality enough to preserve it. Herman: That's not a technical question. It's a cultural one. It's about what we decide matters. Corn: If AI is training on AI, and humans are consuming AI-generated content, are we all converging on a single, bland, averaged-out reality? And if so, is that a feature or a bug? I don't have an answer. But I think asking the question is the first step toward making sure it's not a bug we sleepwalk into. Herman: Now: Hilbert's daily fun fact. Hilbert: In high medieval Greenland, Norse settlers preserved meat by burying it in ice-covered bogs, creating a natural deep-freeze effect. The largest known cache, excavated in nineteen ninety-three at the Farm Beneath the Sand site, contained over two hundred kilograms of still-edible seal and caribou meat dated to approximately twelve hundred AD. Corn: Two hundred kilograms of eight-hundred-year-old seal meat. Herman: a lot of seal. Corn: This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop. If you enjoyed this episode, tell someone who needs to hear it. Email the show at show at my weird prompts dot com. We'll be back next time.