Web scraping won't train the next AI

For fifteen years, one implicit assumption has carried all of modern AI: everything a model needs is already on the internet — you just have to go and get it. That assumption is dying. And with it, an entirely new market is being born — between those who build tomorrow's models and those who own the data of the real world.

Here's why I believe 2026 is the year data stops being a free raw material and becomes a strategic asset that gets traded.

1. The web is hitting its ceiling — and starting to poison itself

The tipping point isn't an opinion. It's arithmetic.

According to Epoch AI, the total stock of high-quality human text available publicly sits around 300 trillion tokens — and large models will have exhausted that stock between 2026 and 2032, with a median projection around 2028 (Epoch AI, "Will we run out of data?", ICML 2024).

This isn't the worry of a few isolated researchers. In December 2024, at NeurIPS, Ilya Sutskever — co-founder of OpenAI — put it bluntly: "Pre-training as we know it will end," because "compute is growing, but the data is not growing, because we have but one internet." He called data "the fossil fuel of AI": it was created once, we've consumed it, and we've reached "peak data" (reported by The Verge, December 2024).

Worse: the well is being contaminated. A study published in Nature (Shumailov et al., July 2024) demonstrated "model collapse" — a model trained recursively on AI-generated content degrades and loses information about the real distribution of the world. And the web is filling up with precisely that content: according to Graphite, the share of published articles generated by AI overtook human-written ones as early as November 2024 (≈ 52% by May 2025). The reservoir we drew from is now filling with the reflection of the models themselves.

And the doors are closing. Since July 1, 2025, Cloudflare — which fronts roughly a fifth of the web — blocks AI crawlers by default. On the publisher side, nearly half of all news sites now block at least one AI crawler. On the legal front, The New York Times v. OpenAI (filed late 2023) survived the motion to dismiss in 2025 and is proceeding on the merits.

The takeaway from part one: the resource that made generative AI — free public text — is becoming finite, contaminated, locked, and contested, all at the same time.

2. The labs have already started paying

The best proof that scraping is no longer enough is that the companies that lived off it are reaching for the checkbook.

• OpenAI – News Corp: a licensing deal reportedly worth more than $250M over 5 years (reported by the WSJ, May 2024).
• Google – Reddit: roughly $60M a year for data access (reported by Reuters, February 2024).
• OpenAI – Axel Springer, Financial Times, Le Monde, Associated Press… a cascade of deals across 2024–2025, from "tens of millions" to undisclosed sums.

Meanwhile, the market's infrastructure is being built: Microsoft announced in early 2026 a "Publisher Content Marketplace" to broker content licensing between publishers and AI developers. And the staggering valuation of Scale AI — ~$29B after Meta's ~$14.3B investment for a ~49% stake (June 2025) — says something simple: training data is now a strategic-tier asset.

The message is clear. Data is no longer scraped. It is licensed, negotiated, bought.

3. The real shift: tomorrow's models don't need more web — they need something else

Here's the point most analyses miss.

The next frontier of AI is not one more LLM. It's world models and physical AI: systems that don't just manipulate language, but model, simulate, and act in the real world.

• NVIDIA launched Cosmos at CES (January 2025), a family of world foundation models for physical AI. Jensen Huang calls it "the ChatGPT moment for robotics." These models feed on petabytes of video and sensor data — NVIDIA claims it processed 20 million hours of video in 14 days.
• Google DeepMind unveiled Genie 2 (December 2024) and then Genie 3 (August 2025): models capable of generating playable, interactive worlds to train embodied agents.
• Fei-Fei Li — the godmother of computer vision — raised $230M as early as September 2024 for World Labs, built around "spatial intelligence," followed by ~$1B more in 2026.

But these models hit a wall the web cannot climb. There is no "internet of physical interaction" to scrape. Robotics training data remains tiny: the reference datasets in embodied AI are counted in hundreds of thousands of demonstrations (RT-1: ~130,000; VIMA: ~650,000), whereas a vision-language corpus like LAION-5B aligns 5.7 billion. Real-world data has to be captured one gesture, one sensor, one trip at a time.

That's exactly why Tesla (over 10 billion cumulative FSD miles, May 2026) and Waymo (over 100 million autonomous miles) treat their fleets as a defensible asset: it isn't on the web, it can't be bought from a text broker — it is produced in the real world.

The data the next generation of AI is missing was never online. It's in factories, fleets, hospitals, energy grids, supply chains. It belongs to operators, not to labs.

4. What we're seeing on the ground — and why it's a two-sided market

This is where our work at d-nvest meets the thesis — because we don't just comment on it, we measure it.

On our platform, we have mapped 311 real data holders to date — organizations that produce, often without realizing it, exactly the kind of data tomorrow's AI demands:

• 66% are time series (206 of 311) — sensors, telemetry, machine logs: the raw signal of the physical world.
• The dominant sectors are industrial (149), mobility (91), and healthcare (24) — physical AI, not web text.
• The use cases read like the roadmap of industrial AI: predictive maintenance (136), industrial monitoring (64), document intelligence (34), regulatory RAG (19), diagnostic AI (15).
• On the demand side, the buyers are already identified by profile: foundation-model labs, computer-vision teams, vertical LLM builders, industrial-AI vendors.
• All spread across the markets that matter — UK, France, United States, Germany, Canada — and backed by nearly 1,000 press signals attesting to these holders' real activity.

These reserves address markets that aren't promises: predictive maintenance alone is worth ~$14B in 2025 (CAGR ~28%), industrial IoT exceeds $480B, industrial AI ~$44B.

On one side, holders sitting on a scarce resource they under-exploit. On the other, buyers — the builders of the next generation of AI — ready to pay for that resource, as their licensing deals already prove. What's missing between them is the infrastructure for matchmaking, qualification, and trust. That is exactly what we're building.

The bottom line

Web scraping trained the current generation of models. It will not train the next one. Public text is finite, it's polluting itself, and it's closing off. World models and physical AI demand a different kind of data — real, operational, multimodal — that was never published and never will be.

That data already exists. It belongs to tens of thousands of operators who often have no idea they're sitting on the gold of the next decade of AI.

The question is no longer "where do we find data." It's "how do we connect those who hold it with those who need it." It's a two-sided market, and it's only just beginning.

If you're an operator producing industrial, mobility, or healthcare data — or an AI player looking for proprietary real-world datasets — now is the time to talk.

— Salim Labriki, d-nvest

Methodology note: licensing-deal amounts are those reported by the press (WSJ, Reuters, Bloomberg) and rarely officially confirmed by the parties. Inventory figures (311 holders, modalities, sectors) come from our own mapping as of July 1, 20