The Great GPU Windfall

Hyperscalers are spending over $700 billion on data centers while retiring GPUs after 18 months. The result: a coming flood of cheap, powerful used hardware and the quiet migration of enterprise AI into basements.
10 min read Send link by email
The Great GPU Windfall

Image AI generated by Heisun Kim, HSEC

AI ai edge-ai gpus on-premise data-protection

by Helmut Hauschild, HSEC

The largest infrastructure buildout in IT-history is about to create the largest hardware windfall in history. The two facts are connected, and most of the coverage misses that connection.

A $700000000000 Treadmill

The four US hyperscalers are spending roughly $710 to 745 billion on data centers in 2026. Microsoft around $190B, Alphabet $180–205B, Amazon $200–220B, Meta $130–145B. That is about six times the level of 2022, and Moody’s projects $785B this year growing toward a trillion by 2027.

But the refresh cycle underneath that spending has accelerated to annual, with mid-cycle upgrades on top: Hopper in 2022, Blackwell B200 in late 2024, the B300/GB300 mid-2025, and Rubin in the second half of 2026. The reason is physics and economics. NVIDIA’s numbers put Blackwell at up to 50 times the throughput per megawatt and roughly 35x lower token cost compared to Hopper. Rubin is another order of magnitude on top. When the next generation delivers 35x cheaper tokens, last-generation hardware stops being economical for hyperscale inference even if it is only 18 months old.

And that is what the hyperscalers are doing: industry analysts expect Hopper systems being retired after roughly 18 months, with the shortened refresh cadence already strengthening the secondary market for lightly used GPUs. The mismatch is structural: GPUs have a 12–24 month refresh cycle, but the buildings that house them are on 15–20 year leases. CoreWeave has disclosed accelerated depreciation for its GPU fleets, and Nebius quietly extended the useful life of server and network equipment from four to five years this spring. The buildings are long-lived; the silicon inside them is becoming disposable.

There is a trap in this, worth naming: the three-year Hopper commitment someone signs in mid-2026, right before Rubin arrives. That contract locks in the most expensive tokens in your own fleet for a full generation cycle.

The Windfall

So what do used GPUs actually cost right now (late 2026)? The numbers are better than most people expect:

  • H100 SXM: ~$22,600 per GPU all-in on the secondary market (range $18.2k–$27.7k), per CCIR’s index built from 3,356 executed sales between July 2023 and September 2026. H100 PCIe units trade at $14,000–$25,000 used against $25,000–$34,000 new.
  • H200: ~$47,000 per GPU all-in.
  • A100 80GB: refurbished units from around $7,800, not much more then consumer cards.
  • System level: used equipment sells for 40–60% below original MSRP, and an 8-GPU H100 system that runs $250–320k new costs substantially less refurbished.

The market infrastructure is maturing fast: Compute Exchange launched a dedicated secondary marketplace for data-center GPUs in July, with orders ranging from hundreds to tens of thousands of units. Their CEO put it plainly: “Not every workload requires the newest generation of GPUs.”

This is not a collapse yet: H100 rental prices actually spiked 22% in the month to early September, and on-demand capacity is sold out across providers. Demand is currently outrunning supply. The market wrote the H100 off and then bid it back up. The windfall arrives when the first wave of Blackwell and Hopper gets retired at scale, which is 2027–28. That is when “a few thousand euros per card” stops being a bargain and becomes the default price of last-generation compute.

An AI Rack in Your Basement

Today, a company that wants local AI runs LM Studio or Ollama on a Mac. That is fine for one person. It is not a service: no parallelism, no multi-user access, no 24/7 availability.

Tomorrow (“tomorrow” here means the moment the first big refresh wave hits the secondary market) that same company buys a small rack of used Hopper or Ampere cards, puts vLLM in front of it, and gets the same service shape as today’s cloud AI: many users, multiple models, always on. And here is the part that makes last-generation hardware acceptable: an internal rack does not run at full load 24/7. It serves a company’s working hours, its batch jobs, its agents. Maybe a 30–50% utilization. For that duty cycle, a card from two generations ago is not a compromise; it is the right tool.

The token economics back this up. Lenovo’s 2026 TCO analysis finds that owning the infrastructure yields up to a 17 times cost advantage per million tokens compared to Model-as-a-Service APIs, with breakeven in under four months for high-utilization workloads. Industry statistics put on-premise deployment at roughly 18x cheaper per million tokens than premium cloud APIs.

The other side of that math: there is a utilization trap. At 10% load, your effective cost per token is roughly ten times the fully-loaded number. Against a budget API (DeepSeek V4 Flash at $0.14 per million input tokens), a single H100 only beats the cloud at around 5.7 billion tokens per month; against GPT-5-class pricing ($5 per million), the crossover is around 250–300 million tokens per month at 60–70% utilization. The general rule of thumb: self-hosting wins once you are past roughly two million tokens a day.

For a mid-size company with real internal usage support, coding agents, document processing, RAG over its own knowledge base, that threshold is not theoretical. At HSEC sometimes we pass it on a normal day just with system operations. And the price of the hardware keeps falling, which moves the crossover point in your favor every year.

Local Wins Beyond Cost

Cost is the most visible argument, but it is not even the strongest one. The others: your data never leaves the building, your IP and training material stop transiting third-party infrastructure, and you are no longer one provider outage from standing still.

Then there is Bahrain. On 1 March 2026, the first day of the war that began with US-Israeli strikes on Iran two days earlier, Iranian drone swarms hit Amazon’s data centers in Bahrain and the UAE. A follow-up strike on 1 April took out a second availability zone, and on 24 July the IRGC launched missiles at the remaining Amazon data center structure in Bahrain, damage independently confirmed by satellite imagery. On 15 September, AWS posted what is probably the most important status update in cloud history: it was “unable to restore access to the resources and data hosted exclusively” in the Bahrain region, me-south-1. The damage “spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand.” One UAE zone, mec1-az2, is gone permanently. AWS issued $150 million in customer credits and suspended billing.

A cloud region losing customer data to an airstrike. AWS is not alone. A few weeks ago, Yandex has lost customer data after an airstrike on their data centers, even if it is hard to get trustworthy information about the magnitute of the problem in their case. We can expect this kind of thing to happen again.

Notice what it reveals: the security and availability promises of the largest, most professional infrastructure operators on earth did not hold up in a crisis. Multi-AZ is designed for hardware failure, not war. Have you checked if you insurance covers data loss caused by acts of war? It probably doesn’t.

The Shadow AI Problem

Here is where the story gets uncomfortable for most companies, because it is already happening. When a company’s local AI offering is inadequate (too small, too slow, too limited) employees go around it. Gartner finds that 69% of organizations suspect or have evidence that employees are using prohibited public generative AI. Verizon’s 2026 Data Breach Investigation Report shows employees regularly using AI on corporate devices jumping from 15% to 45% in a single year. A survey of 72 companies found roughly 50 different AI tools in active use, with consumer ChatGPT reaching three times the employees of the sanctioned Copilot deployment. About 71% of employees use AI without IT approval, and the median time to discover an unauthorized tool is over 400 days.

We know how this ends: Samsung’s engineers pasting code into ChatGPT in 2023 became the canonical example in OWASP’s LLM Top 10. In April this year, a compromise of Context.ai cascaded through an OAuth chain into Vercel. The pattern is not “employees are careless”. It is “the sanctioned path was worse than the unsanctioned one.”

Now add the advertising layer: OpenAI launched ads in ChatGPT on 9 February, with an official position that it “never sells your data to advertisers.” A May study from IMDEA Networks found something more subtle: ChatGPT, Claude, Grok and Perplexity all load trackers from Meta, Google and TikTok; conversation permalinks are exposed to third-party trackers; and in some cases the access controls are weak enough that anyone with a chat URL can read its content. A measurement study on arXiv found 17 of 20 major chatbots sharing information with at least one third party: three sending plaintext conversation text, prompts and responses included, to Microsoft Clarity via session replay; 15 sending chat URLs or identifiers to third-party advertising, analytics and social endpoints.

So companies are squeezed from both sides: inadequate local offerings push employees to personal accounts where conversations leak toward ad endpoints, while generous enterprise offerings (GPT-5-class at ~$5 per million tokens, Claude Sonnet 4 at $3/$15) produce exorbitant token bills that scale with usage. The third option, a rack of your own, is the only one that solves both problems at once.

Where I Think This Goes

In order of confidence:

  1. The migration is automatic. Nobody has to decide it; the price signal does. The moment the first wave of data-center GPUs gets replaced, a flood of last-generation compute hits the secondary market at a fraction of new-hardware prices, and every company that was renting intelligence starts doing the buy-versus-rent math. The market migrates to edge AI not because of a strategy memo, but because the hardware becomes cheap enough.
  2. Edge AI is not a niche. Enterprise inference performed on-premises or at the edge has jumped to 55%, up from 12% in 2023. The “edge” is where the volume goes; the cloud keeps the frontier training and the burst workloads.
  3. What to do now is boring, which makes it right. Start with what you have: Macs, Ollama, LM Studio. Measure your actual token volume and how sensitive your data is. Build the internal service so it is good enough that nobody needs a personal account. Then plan for the rack: when 2027–28’s refresh wave hits, you will want to be a buyer with a spec sheet, not a renter discovering the price drop in someone else’s invoice.

The data centers are getting old. Your rack is about to get cheap.

Sources