COMPUTE TERMINAL · SILICON DEEP DIVE · DATA AS OF 2026-10-19
Training built the models. Serving them is building an industry.
For a decade the chip race was about teaching machines; from 2026 on it is about answering with them. Inference — running trained models, billions of times a day — passed half of all AI compute in 2025 and is heading toward three-quarters by 2030. That inversion changes what a chip is for: from peak FLOPs to cost per token, from one giant cluster to a fragmented fleet of GPUs, custom ASICs, wafer-scale engines and in-memory parts. The closest historical analogue is the x86 server's long commoditisation — but this cycle has no true precedent: the incumbent is also the supply chain's biggest customer, and the bottleneck is not the transistor but the memory and packaging around it.
Evidence graded: company primary > industry cross-check > broker relay > derived [flagged] · 40 dated anchors · No ratings, no price targets
§0 · Key findings
Inference crossed half of AI compute in 2025; the silicon, the supply chain and the capital map are being redrawn around cost per token.
Five findings carry this report. First, the crossover is observed, not projected: compiled across Gartner, Deloitte and vendor disclosures, inference rose from roughly a third of AI compute in 2023 to about half in 2025, and sits at 55–67% in 2026 depending on what is counted. Second, the money followed: the AI inference market is sized near $106 billion in 2025 and forecast to roughly double by 2030, while NVIDIA's Data Center revenue reached $193.7 billion in fiscal 2026 — a business that was $15 billion three years earlier. Third, the unit economics collapsed: serving a GPT-3.5-level model cost about $20 per million tokens in November 2022 and $0.07 by October 2024 — a 280-fold fall that turned experiments into utilities.
Fourth, the fleet is fragmenting: JPMorgan projects custom ASICs will pass GPUs in accelerator unit shipments in 2027 (54% versus 47%), even as NVIDIA retains roughly 85% of the revenue pool — units decentralise, dollars concentrate. Fifth, the binding constraints are not transistors: high-bandwidth memory and CoWoS advanced packaging — ~1.15 million wafers in 2026, ~60% pre-allocated to NVIDIA — decide who can actually ship. China's fork is the extreme case: cut off from both, it is rebuilding the stack domestically around Huawei's Ascend.
How to read this report. Every figure hangs on a dated anchor in the source register (K1–K40); clicking any number in any chart opens its basis and provenance. Company-reported performance claims are labelled as such; broker relays and derived figures are flagged in the register. Nothing here is a rating or a price target.
§1 · The crossover
Training is capital expenditure with a finish line; inference is operating expenditure that scales with every user — procurement, geography and capacity planning all flip with it.
The distinction is older than the boom: Google deployed its first TPU for inference in 2015, years before "AI infrastructure" was an asset class. What changed is the denominator. A frontier training run is enormous but finite; a successful model then answers queries continuously, and reasoning models multiply that load — DeepSeek-R1-class systems consume on the order of 150× more compute per query than single-pass inference. Epoch AI estimates token demand growing near 10× a year against supply growth of 3–4×. The revenue line of the industry's supplier of record tells the same story: NVIDIA's Data Center segment grew 233-fold from FY2017 to FY2026, with the steepest leg coming after inference crossed into the majority of deployments.
NVIDIA's data-centre business crossed $1 billion in FY2018 and $6.7 billion by FY2021 — respectable, but the segment was still smaller than gaming. Inference existed mostly as a cloud-side afterthought: Google's TPU v1 (2015) and NVIDIA's T4 (2018) served ranking and vision workloads while the industry's capital chased training clusters.
A100s ramped through FY2022; then November 30, 2022 happened. Within a year, export controls (October 2022) had split the map and every hyperscaler was rationing GPU allocations. Revenue nearly doubled — and the bottleneck moved from fabs to a packaging line called CoWoS.
The H100 cycle tripled the segment in a year. This was the training buildout at full throttle — 10,000-GPU clusters, sold-out supply, and a gross margin structure that made every challenger an economics essay.
August 2024: NVIDIA says inference passed 40% of trailing data-centre revenue. Blackwell arrives explicitly inference-optimised — the GB200 NVL72 claims 30× the H100's LLM throughput and a 25× cut in cost and energy per query. The segment reached $193.7 billion in FY2026.
Q2 FY2027 printed $89.0 billion of data-centre revenue, up 117% year on year. Yet the same quarters show the counter-move: ASIC units forecast to pass GPUs in 2027, a licensed Groq architecture inside NVIDIA's own racks, and a China market that no longer buys American silicon at all. The tape is the boom; the composition is the story.
Inference is now the centre of gravity. The training runs still generate the headlines, but the majority of chips, power, and dollars now go to serving models rather than building them.
— Industry compilation, 2026-06
Context Vendor and analyst reads converged on the same inversion during 2026. Why it matters A market that runs forever is priced, financed and engineered differently from one that trains and rests.
THE COMPUTE-BOUND PHASE
Reading the prompt: parallel, dense matrix work that GPUs were built for.
Prefill processes the entire input context at once — throughput-bound, high utilisation, friendly to big batched GPUs. It is the phase where NVIDIA's roadmap is strongest, and why rack-scale systems like GB200 NVL72 exist.
Hover for the full story
THE MEMORY-BOUND PHASE
Writing the answer one token at a time: each step re-reads all model weights.
Autoregressive decode moves billions of parameters from memory to compute for every single token. Bandwidth, not FLOPs, sets the speed — the opening attacked by SRAM-rich LPUs, wafer-scale engines and in-memory compute. NVIDIA's ~$20B Groq licence (press-reported figure) was explicitly framed around pairing GPU prefill with LPU decode.
Hover for the full story
THE ONLY METRIC THAT COMPOUNDS
Cost per served token is the industry's true exchange rate.
Every architectural bet in this report — ASICs, wafer-scale, in-memory, low-precision FP4 — is ultimately a claim about dollars per million tokens. Stanford's AI Index documented a 280× fall for GPT-3.5-level serving between late 2022 and late 2024; each succeeding chip generation is marketed on extending that curve.
Hover for the full story
§2 · The physics of inference
Four silicon generations in four years quadrupled on-package memory while compute merely doubled — the industry is re-engineering around bandwidth, not arithmetic.
Watch what changed fastest. From H100 to the Rubin generation, per-package high-bandwidth memory grew 3.6× and memory bandwidth grew roughly 6.6×, while dense compute grew more modestly. AMD's MI455X pushes the same vector to 432 GB and 19.6 TB/s. The scarcest commodity in the stack is no longer logic but the HBM stacked beside it — three vendors (SK hynix, Samsung, Micron) supply the world's accelerator memory, and 2026 is the HBM4 transition year.
§3 · The merchants
The incumbent holds ~85% of accelerator revenue, a networking moat, and 60% of the packaging line — but the second source is now funded, specified and shipping.
NVIDIA's position is a system, not a chip: CUDA's software gravity, NVLink's rack-scale fabric (networking revenue grew 142% in FY2026), and pre-allocation of the industry's scarcest inputs. Its answer to the inference turn is to absorb the challengers' ideas — FP4 low precision, disaggregated serving, and the reported ~$20 billion Groq licence to cover the decode phase GPUs handle least well. AMD, meanwhile, converted from bystander to second source: $5.8 billion of data-centre revenue in Q1 2026, MI355X volume shipments, and two six-gigawatt commitments from OpenAI and Meta that function as escrowed demand for MI450/Helios racks. Intel, holder of the original inference franchise (the CPU), is the absence in this market — Gaudi is winding down and its next attempt, Crescent Island, is a 2026 trial.
§4 · The ASIC insurgency
Every dollar a cloud giant spends on NVIDIA silicon carries a ~70%-gross-margin tax; custom ASICs are the rebate programme — and 2027 is when the units tip.
The logic is arithmetic, not ideology: at hyperscaler volumes, a workload migrated from merchant GPU to in-house ASIC saves the vendor margin on every chip. Google's TPU programme (running since 2015, now at v7 Ironwood with 9,216-chip pods) covers most of Google's own inference and won Anthropic as an external anchor — up to a million chips and more than a gigawatt in 2026. AWS has deployed over half a million Trainium2 chips and locked Anthropic into a ten-year, $100-billion-plus commitment. Microsoft's Maia 200 and Meta's MTIA cover their own serving stacks; OpenAI's Broadcom-designed part tapes out around Q3 2026. JPMorgan's fall 2026 update makes the call: ASIC/XPUs pass GPUs in unit shipments in 2027 — 12.5 million to 10.9 million — though units and dollars are different denominators.
§5 · The startup long tail
85 companies, five architectural bets, and a market that has already produced its first IPO, its first $20B licence, and its first casualties.
The survivors cluster around one insight: decode is memory-bound, so put memory next to (or inside, or across an entire wafer of) compute. Cerebras turned that into a Nasdaq listing (CBRS, May 2026, ~$66 billion) on $510 million of 2025 revenue and $24.6 billion of contracted backlog. Groq turned it into the industry's largest licensing event. d-Matrix put Corsair into mass production in June 2026; Positron raised at $5 billion on a bet that commodity LPDDR can dodge the HBM queue entirely. The cautionary path exists too — Graphcore's quiet absorption by SoftBank is what missing the window looks like.
| Company | Architecture bet | Cumulative | Latest event |
|---|---|---|---|
| Cerebras | Wafer-scale engine | $2.8B | IPO May-26 · ~$66B valuation |
| SambaNova | Reconfigurable dataflow | $2.5B | $1B Series F @ $11B · Jul-26 |
| Groq | Deterministic LPU | $2.0B | NVIDIA licence Dec-25 · $650M Jun-26 |
| Etched | Transformer-only ASIC | $1.6B | $700M Series D @ $21B · Aug-26 |
| Positron | Memory-centric (LPDDR5X) | $1.2B | $875M Series C @ $5B · Sep-26 |
| Tenstorrent | RISC-V + mesh | $1.0B | $693M Series D · Dec-24 |
Dataset: ~85 tracked companies, ~$26B cumulative; the top five hold ~39%. Private-company figures are disclosed-round reconstructions, not audited totals.
§6 · The China fork
NVIDIA's China share fell from 95% to single digits in four years; the constraint on the domestic stack is memory, not architecture.
The sequence is stark: H20 banned April 2025, Beijing's directive against NVIDIA procurement September 2025, H200 production halted March 2026. Huawei's answer is a yearly cadence — 950PR (inference-prefill, Q1 2026), 950DT (Q4 2026), 960 and 970 through 2028 — plus system-level compensation: 384 Ascend chips in a CloudMatrix pod trading 3.9× the power for competitive cluster throughput, subsidised by $0.03–0.06/kWh electricity. Per chip, an Ascend 910C delivers roughly 60% of an H100 on inference; the Council on Foreign Relations estimates leading US silicon could sit ~17× ahead of Huawei's best by 2027. The binding constraint is not the NPU but HBM: zero domestic HBM3e, a finite ~11.4-million-stack stockpile, and CXMT years from the frontier.
§7 · The bottleneck
CoWoS capacity roughly triples in three years and is still 18–24% short; every accelerator roadmap is a claim on the same interposer supply.
TSMC's CoWoS advanced packaging — the step where logic dies and HBM stacks become one device — grew from ~35,000 wafers a month in 2024 to ~80,000 at end-2025, heading for ~120,000 by end-2026; UBS sees the industry total, including OSATs, reaching 250,000 by end-2027. NVIDIA pre-allocated ~60% of it. The interesting signal is the dilution: NVIDIA's share of demand slides from ~56% (2026) toward ~45% (2027) as Broadcom's Google business, AMD's MI455X and AWS's Trainium3 triple their allocations — the ASIC insurgency is visible in the packaging queue before it shows up in revenue.
§8 · The verdict
The evidence weighs toward a hybrid fleet: ASICs take the unit crown in 2027 while NVIDIA defends the revenue pool with systems, networking and software.
Both sides of the scale carry real mass. The fragmentation case rests on unit forecasts, hyperscaler capex guidance, and the observed exit of the China market. The concentration case rests on IDC's 85.2% revenue share, the networking moat, and the fact that NVIDIA controls the input queue. The falsification line is shared: if Rubin slips, the ASIC window widens; if ASIC units pass 54% while NVIDIA keeps four-fifths of the dollars, the "crossover" was a units story all along. The closest historical analogue is the x86 server's commoditisation under Linux and the cloud — but this cycle has no true precedent, because the incumbent owns the bottleneck it is being attacked through.
Appendix
Vendor-stated specifications for the flagships discussed; forecast parts in italics. Peak theoretical figures, vendor conventions.