The Hidden Architecture of AI's Training Data: Why 40% of the Web Is a Permanent Collection

The Invisible Architecture of the Internet's Greatest Library

Somewhere in the cloud, a server farm is building the most comprehensive library in human history. Every month, a system called Common Crawl reaches into the living web and pulls out somewhere between 2 and 3 billion URLs—pages, documents, images, the raw substrate of the internet. This library is not a building with shelves. It's a stream of data, constantly refreshed, that underlies some of the most powerful artificial intelligence systems ever created. GPT. Llama. Falcon. PaLM. Every time you interact with a large language model, you're touching content that Common Crawl collected.

But here's what nobody had fully understood until now: what exactly is in that library? What fraction of the web's enduring knowledge does it capture? What fraction does it miss? And crucially—why?

A new paper from researchers Michael Paris, Hande Celikkanat, and Luca Foppiano at the Common Crawl Foundation tackles these questions with unusual mathematical rigor. Their answer upends a simplifying assumption that the field had been using for years. The web, it turns out, is not one population. It has structure. And that structure has consequences for everyone who builds AI.

---

The Science: Reading the Crawler's Mind Without Touching the Crawler

To understand what Paris and colleagues discovered, you first need to understand how web crawlers work—and why measuring them is surprisingly hard.

A web crawler is a program that starts with a list of URLs and fetches the content at those addresses. It then extracts new URLs from those pages, adds them to its list, and repeats. The process is recursive: each page points to other pages, which point to more pages, and so on. Common Crawl runs this process monthly, building snapshots of what the web looked like at a given moment.

The challenge is that the web is not static. URLs disappear. New pages appear. Existing pages change. A crawler always sees a partial, evolving sample of something that itself is constantly mutating. The researchers call this a "longitudinal web crawl": a sequence of partial samples from an evolving population.

The traditional way to analyze such a crawl has been pairwise containment—measuring what fraction of URLs in one crawl also appeared in another crawl some number of rounds later. If crawl A and crawl B overlap by 30%, and crawl A and crawl C overlap by 20%, you can work backward to estimate how fast URLs are being churned out and how thoroughly each crawl samples what's left.

This approach works, but it has a flaw: it treats the web as if every URL behaves identically. Every page has the same chance of persisting, the same chance of being fetched, the same life expectancy. This is the "homogeneous urn model," and it's the assumption Paris and colleagues set out to test.

Their insight was elegant. If you have two different ways of measuring the same underlying process, and they give you the same answer, your model is probably right. But if they disagree—watch closely. The disagreement itself becomes a measurement.

The first projection is the standard pairwise containment method. The second is something new: the discovery curve. Instead of looking at overlap between two specific crawls, the discovery curve tracks how the total count of unique URLs grows as you accumulate crawls over time. Start with crawl 1, add crawl 2, then crawl 3, and so on. How fast does the pool of seen URLs expand? Does it keep growing linearly, or does it plateau?

Under the homogeneous urn model, both of these projections—containment and the discovery curve—should be governed by the same two parameters: survival rate α (the fraction of URLs that persist round to round) and coverage c (the fraction of surviving URLs that get fetched in a given round). The researchers derived the closed-form mathematics for both. And then they applied them to real data and watched the model break.

The data came from two very different archives. Common Crawl, the monthly megacrawl that has produced over 90 archives since 2013, running in the 2020–2025 window. And the German Academic Web (GAW), a smaller focused crawl built on different architecture, seeded from roughly 150 German university homepages. Two archives, built by different teams, using different crawling strategies—one recursive and graph-based, one hop-by-hop and re-harvesting. If the homogeneous model was correct, both should yield the same (α, c) parameters from both measurement approaches.

They did not.

---

What They Found: The Web Has a Core and a Shell

The failure of the homogeneous model was immediate and unmistakable. When Paris and colleagues fit the containment curve and the discovery curve independently using the same two parameters, the fits diverged. Containment pulled toward high survival and moderate coverage—it was dominated by the URLs that kept showing up, crawl after crawl. The discovery curve pulled toward lower survival and coverage—it was dominated by the churn of fresh discoveries, URLs that appeared once and vanished.

Neither single parameter pair could explain both patterns simultaneously. The homogeneous urn was the wrong model.

The minimal extension that closed the gap was a two-component urn. Imagine the web's URLs divided into two populations. There's a core—call it K—of persistent, well-linked pages that survive indefinitely and are reliably fetched every time they appear. This is the content that shows up in every crawl, the stable architecture of the web. And there's a shell—call it ∂K—of ephemeral content that appears, gets fetched for a while, and then churns out, replaced by new URLs that will themselves churn out in turn.

When Paris and colleagues refit their model with this two-component structure, something striking happened: the two projections agreed. Both the containment curve and the discovery curve, fit independently, converged on the same core fraction κ. For Common Crawl at domain granularity—meaning we're talking about entire websites rather than individual pages—the core fraction came out to approximately 0.4. Forty percent of the domains in Common Crawl's archive are part of that persistent, reliable core.

This is the scalar entry point to a more general decomposition. The core, once identified, opens a window into the rest of the structure. The shell—the remaining 60%—has its own survival and coverage parameters (α_∂ and c_∂), which describe how URLs behave once they leave the immortal core. And those shell parameters showed a residual disagreement between the two measurement approaches—a hint that the shell itself may not be uniform, that there may be deeper structure waiting to be resolved.

But even this first-order split is revelatory. It means the standard practice of treating the web as a single homogeneous population was averaging over two fundamentally different kinds of content. The AI systems trained on Common Crawl data aren't sampling uniformly from the web. They're sampling from a library with a permanent collection and a circulating collection, and those two collections behave very differently.

---

Why This Changes Things: What the Models Got Wrong

To appreciate why this matters, you need to understand how large language models are built.

When an AI lab trains a large language model, they start with a corpus of text. For most of the leading open and commercial models, that corpus includes content from Common Crawl—often the single largest source. The pretraining process exposes the model to billions of web pages, and the quality of what the model learns is bounded by the quality of what it was shown.

The filtering pipelines that process Common Crawl data—language identification, quality classification, perplexity scoring, deduplication—are sophisticated. But they're downstream of a more fundamental choice: what the upstream fetch policy chose to capture in the first place. As Paris and colleagues put it, "If a URL is never fetched, no downstream filter can recover it."

The fetch policy is driven by a small set of knobs: host budgets (how many pages per website), harmonic-centrality thresholds (which URLs are considered important enough to recrawl), deletion cutoffs (when to stop trying to fetch a URL that has disappeared). These knobs are set at the domain and host granularity—exactly where the two-component model reveals its structure.

Under the homogeneous model, the implicit assumption was that each URL had the same expected lifetime in the crawl, the same probability of being fetched each time it appeared. This made the analysis tractable, but it also made it wrong in a specific, measurable way. The homogeneous fit underweighted the persistent core (it looked like just a higher survival rate) and overweighted the ephemeral shell (it looked like lower survival, but not as a distinct population).

The implications are concrete. Consider URL lifetime—a concept the researchers translate into operator-facing terms. Lifetime ℓ is the expected number of crawls a URL persists in the urn before churning out. Under the homogeneous model, you could estimate this from pairwise containment alone. Under the two-component model, the core has effectively infinite lifetime—it persists crawl after crawl, year after year—while the shell has a finite, policy-sensitive lifetime that the operators can actually affect with their knobs.

The hidden time η̄ is the expected number of crawls a URL is alive but unsampled—present on the web, but not fetched. This is where the policy levers act most directly. A URL that survives a round has some probability of being in the next crawl. The coverage parameter c determines that probability. Push c higher, and more surviving URLs get fetched. Push it lower, and more URLs spend rounds in the shadows, present but unseen.

Under the two-component model, these parameters apply to the shell. The core doesn't need tuning—it persists forever and is always fetched. But the shell's behavior is policy-responsive, and it's where the tradeoff between coverage and efficiency lives. A crawler that wants to capture more of the ephemeral web needs to push coverage higher, fetching more of what survives. A crawler that's resource-constrained might accept lower coverage, accepting that some surviving URLs will spend time hidden.

---

The Discovery Curve: A New Window into Crawl Dynamics

One of the paper's key contributions is methodological: the discovery curve as a new probe into crawl behavior.

The discovery curve U(s, T) is defined as the cumulative number of distinct URLs observed over T consecutive crawls starting at crawl s. Think of it as asking: if I look at the last six months of crawls, how many unique domains have I seen? How about the last year? The last two years?

Under the homogeneous urn model, this curve has a closed-form expression. It has a geometric transient that decays as you look at longer windows, plus a linear asymptote with slope ν_∞ (the asymptotic new-discovery rate) and an intercept I (the integrated transient). The mathematical derivation is in the paper, but the intuition is simple: early crawls add lots of new URLs, but each successive crawl adds fewer new ones, until you're growing linearly—replacing churned URLs with new ones at a steady rate.

The researchers applied this to Common Crawl's publicly available rolling-window statistics, which track unique URL counts over the last N crawls for various N. When they pivoted these across all valid starting points and window lengths, they got a heatmap of discovery over the 2020–2025 archive—and clear evidence of the decelerating growth the model predicts.

More importantly, dividing each discovery curve by its single-crawl cardinality collapsed the family across starting points. Every trajectory followed the same normalized shape. This is the discovery-formula shape, the regression target that the two-component model explains.

The beauty of this approach is that it uses data Common Crawl already publishes. The per-crawl URL counts and rolling-window unique-URL statistics are available on their crawl-statistics page. The researchers didn't need to instrument the crawler differently or ask for new telemetry. They just read the existing output through a new mathematical lens.

---

Two Archives, One Structure

One of the paper's underappreciated findings is that the two-component structure appears in both archives, despite their architectural differences.

Common Crawl uses Nutch-based recursive crawling, where each monthly fetch list is derived from the previous crawl database and ranked by harmonic centrality. This is a graph-based approach: URLs that link to important pages get priority.

The German Academic Web uses Heritrix-based focused crawling, seeded from academic homepages and re-harvesting via hops at each round. This is more like following links by hand: start with a university's homepage, follow the links you find, follow the links on those pages, and repeat.

These are fundamentally different crawling philosophies. One is global and recursive; one is focused and hop-by-hop. One derives its priorities from link structure; one follows explicit seeds. Yet both archives show the same signature: containment and discovery curve disagree under the homogeneous model, and both converge on a two-component decomposition with a persistent core fraction.

This suggests the structure is real—it's not an artifact of a particular crawling strategy. The web itself has a core and a shell, and different crawlers are all sampling from the same underlying population. The homogeneous model was wrong everywhere, not just for Common Crawl.

The fitted values do differ across archives. Common Crawl's κ ≈ 0.4 at domain granularity reflects its broad, recursive coverage. GAW's parameters, measured at URL granularity rather than domain granularity, will differ in their specifics. But the qualitative structure—the persistent core, the ephemeral shell, the convergence of two projections on a single triple—is shared.

---

The Residual and What It Opens

The paper doesn't claim to have fully solved the problem. There's a residual on the shell coverage parameter c_∂. The two projections agree on the core fraction κ, but they still disagree slightly on how thoroughly the shell gets sampled.

The researchers interpret this as evidence that the shell itself is not homogeneous—that there's structure within the ephemeral population that a single scalar cannot capture. Some URLs in the shell are more persistent than others. Some get fetched more reliably. The single "shell" is actually a distribution of behaviors, not a single urn component.

This is the entry point to a rank-resolved generalization. The core/shell split is the scalar summary of a deeper structure: the web ranked by persistence, from immortal links at one end to one-time appearances at the other. The next step would be to move from two components to a continuous spectrum, resolving the shell by rank. The paper leaves this as follow-up work, but the direction is clear.

The implications for AI training are significant. If the shell has rank-resolved structure—if some ephemeral URLs are more "important" than others in ways that correlate with persistence—then a fetch policy that doesn't capture that structure will systematically miss a particular kind of content. And if that content is the kind that matters for language model training, the bias is built into the corpus from the start.

---

What This Means for AI Development

Let's be concrete about what the two-component structure implies for AI training pipelines.

The persistent core (40% of domains, likely more of the actual content) represents the stable web—the articles, documentation, and pages that have been around for years and will likely be around for years more. This is the content that current filtering pipelines are good at capturing and evaluating. It's stable enough to be quality-classified reliably, diverse enough to represent most topics reasonably well.

The ephemeral shell represents the living web—news articles, social posts, discussion forums, temporary pages, the content that appears in response to events and disappears when the event fades. This is where the coverage question becomes urgent. A language model trained primarily on core content will have a particular temporal profile: it will represent the world as it was when the core was stable, missing the churn that characterizes real information ecosystems.

The paper doesn't claim that Common Crawl has too little shell content, or too much. It just shows that the shell exists and behaves differently from the core, and that the homogeneous model was blind to this difference. Any estimate of coverage—how much of the web a model has seen—was an average over two populations with different statistical properties.

This matters for the ongoing debate about AI training data. Researchers have documented various biases in large language models: demographic skew, source skew, temporal bias toward older content. The two-component model provides a structural explanation for some of these biases. A crawler with fixed host budgets and harmonic-centrality thresholds is structurally biased toward persistent, well-linked content. The shell—the more volatile, less linked content—will always be undersampled relative to its share of the live web.

Whether that's a problem depends on what you think AI training should be doing. If you want a model that represents the stable, authoritative web, the current policy is appropriate. If you want a model that captures the full dynamics of information—including the churn, the news cycles, the ephemeral discourse—then the policy needs to change, or the architecture needs to be redesigned to oversample the shell.

---

What's Next: Open Questions and Practical Steps

The paper opens several threads. The rank-resolved generalization—moving from two components to a continuous spectrum—is the most technically ambitious. It would require more parameters and more data, but it would also be more complete, explaining not just the core/shell split but the full distribution of persistence in the web.

The operator implications are more immediate. The paper translates the urn parameters into operator language—lifetime, sampled time, hidden time—in units of crawls per URL life. This gives the people setting host budgets and coverage thresholds a way to reason about their choices quantitatively. If you want to increase the average URL lifetime in your crawl, you can calculate what change to α or c is required. If you want to reduce hidden time, you can estimate the coverage increase needed.

There's also the question of what "granularity" means for the analysis. The paper applies the model at domain granularity for Common Crawl and URL granularity for GAW. These are different levels of aggregation, and the fitted parameters won't be directly comparable. A domain-level κ of 0.4 means something different from a URL-level κ. Understanding the mapping between granularity levels is a practical question for anyone applying this framework.

The residual on c_∂ is the most interesting empirical hint. It says the shell is not uniform, but it doesn't yet specify the structure. Is the shell power-law distributed, with a few moderately persistent URLs and many one-time appearances? Is it structured by topic, with some domains churning faster than others? Is it structured by crawl strategy, with some crawling philosophies capturing more shell than others? The paper gestures at these questions without answering them.

Finally, there's the question of temporal evolution. The analysis covers 2020–2025, a period that includes the COVID-19 pandemic, the explosion of AI-generated content, and major shifts in web platform policies. How the core/shell structure evolves over time—whether the core is growing or shrinking, whether shell lifetime is increasing or decreasing—is a question for future work with longer time series.

---

The Bottom Line

Common Crawl has been called the internet's greatest library. It's the raw material for most of the open large language models, and for the web portions of the commercial ones. Understanding what it contains—and what it misses—is therefore a question with outsized consequences.

Paris and colleagues have shown that this library has an architecture. The persistent core (roughly 40% of domains) is the permanent collection: stable, well-linked, reliably fetched. The ephemeral shell (the remaining 60%) is the circulating collection: URLs that appear, get fetched for a while, and churn out, replaced by new URLs that will themselves churn out.

This structure was invisible under the homogeneous model, which treated every URL as average. The discovery curve—a new probe that uses the full time series of crawl statistics rather than just pairwise overlaps—reveals it. So does the agreement between two independent projections: when containment and discovery curve converge on the same core fraction, the two-component model is confirmed.

The implications for AI development are concrete. The quality of training data is bounded by fetch coverage. Fetch coverage is shaped by policy levers that act on the shell. The shell is where the action is, where the tradeoffs between coverage and efficiency live, and where the biases that affect language models are built in.

Knowing that the structure exists doesn't automatically tell us what to do about it. But it's the first step toward a more honest accounting of what Common Crawl contains—and what it means for the AI systems built on top of it.