HBM: High Bandwidth Memory Explained — The Technology Deciding How Fast AI Can Scale

Every AI GPU headline focuses on compute. The real story — and the real bottleneck — is memory.

Key Takeaways

• HBM (High Bandwidth Memory) is stacked DRAM mounted directly beside a GPU’s processor die, built to move enormous amounts of data at once rather than to be fast on any single request.
• AI GPUs need it because modern AI workloads are memory-bound: the compute cores can process data faster than conventional memory can deliver it.
• NVIDIA’s H100 and H200 use the exact same compute die — the only difference is memory. The H200’s HBM3E upgrade alone delivers roughly 1.9x faster real-world inference on large models.
• Only three companies on Earth make HBM — SK hynix, Samsung, and Micron — and their combined capacity has been sold out through 2026 and into 2027.
• Big Tech is on track to spend roughly $725 billion on AI infrastructure in 2026, and memory now consumes an estimated 30% of hyperscaler data center budgets — four times its 2023 share.
• Micron is the only U.S.-headquartered HBM maker, and it’s now spending roughly $200 billion to bring HBM manufacturing home to Idaho, New York, and Virginia.

What Is HBM (High Bandwidth Memory)?

HBM, or High Bandwidth Memory, is a type of 3D-stacked DRAM built to sit directly next to a processor and move very large volumes of data per second. Unlike conventional memory chips, which are soldered around the edge of a circuit board, HBM dies are stacked vertically and mounted on a shared silicon base alongside the GPU — making it the standard memory architecture for every modern AI accelerator, from NVIDIA’s H100 to AMD’s MI300X.

That’s the one-paragraph version. The rest of this article explains why that architecture exists, how it physically works, how it evolved from HBM1 to HBM4, and why a memory chip most people have never heard of has become one of the tightest chokepoints in the global AI economy.

Why AI GPUs Needed HBM in the First Place

Big Tech is set to spend roughly $725 billion on AI infrastructure in 2026 — Amazon, Microsoft, Alphabet, and Meta alone, up 77% from 2025’s already-record $410 billion. A striking share of that money is now going somewhere unglamorous: memory. Industry estimates put memory at roughly 30% of hyperscaler data center spending this year, a fourfold jump from 2023. The reason is HBM.

For most of computing history, memory and compute improved on separate tracks, and the gap was tolerable. AI broke that arrangement. A large language model isn’t really “computed” so much as moved — every forward pass shuttles billions of weight values, intermediate activations, and (during inference) a growing cache of prior context between memory and the processing cores. The arithmetic itself is cheap and highly parallelizable, which is exactly what GPUs are built for. The bottleneck isn’t doing the math — it’s delivering the numbers to do math on.

Engineers call this the memory wall: processor compute throughput has scaled faster than memory bandwidth for decades. A GPU that can theoretically execute trillions of operations per second is worthless if it sits idle waiting for data. Conventional GPU memory — GDDR, the same family used in gaming cards — was never built to solve this at data-center scale, because its edge-mounted PCB layout puts a hard ceiling on how wide the connection to the processor can be.

HBM is the industry’s answer: stack the memory dies vertically, wire them with thousands of internal connections, and mount the whole stack immediately beside the processor on a shared substrate. The result is a connection so wide and so short that it can move data at rates conventional memory can’t approach.

Bandwidth, Capacity, and Latency: Three Different Problems

These three terms get conflated constantly in AI hardware coverage, so it’s worth separating them clearly:

• Memory capacity — how much data a GPU’s memory can hold at once (in GB). Determines whether a model fits on one GPU or has to be split across several.
• Memory bandwidth — how much data can move per second (in GB/s or TB/s). Determines how fast the GPU can actually be fed.
• Memory latency — how long a single request takes to be fulfilled. HBM barely touches this; DRAM cell physics hasn’t fundamentally changed. HBM’s advantage is almost entirely bandwidth and capacity, achieved through massive parallelism rather than raw speed.

Think of it as a warehouse: capacity is the size of the building, bandwidth is the width of the loading dock, and latency is how long one forklift trip takes. HBM’s genius is building an enormous loading dock.

HBM vs. GDDR vs. DDR: Three Memory Families, Three Jobs

DDR, GDDR, and HBM all descend from the same DRAM cell technology, but each was engineered for a different physical layout and a different job.

The core design choice is width over speed. GDDR pushes very high signaling rates through a narrow interface. HBM uses a comparatively modest per-pin speed multiplied by an enormous interface width — 1,024 or 2,048 parallel lines per stack versus 32 on a single GDDR chip. That’s only physically possible because HBM sits immediately next to the processor rather than at PCB distances, which is also why it’s dramatically more power-efficient per bit moved.

Anatomy of an HBM Stack: What’s Actually Inside

Physically, an HBM module is a small tower built through a manufacturing process fundamentally different from conventional memory chips:

• DRAM dies — the storage layers, stacked vertically. A stack may contain 8, 12, or 16 dies depending on generation.
• Through-Silicon Vias (TSVs) — thousands of microscopic vertical channels drilled through each die, letting signals and power pass directly between layers. This is HBM’s single most distinctive manufacturing technology.
• Microbumps — tiny solder connections joining each die to the one below it.
• The base (logic) die — the bottom layer, which manages the interface between the DRAM above and the processor below, and (in newer generations) increasingly handles active control logic.
• The interposer — a thin silicon layer beneath both the HBM stacks and the GPU die, carrying the dense wiring between them. This is 2.5D packaging: GPU and memory aren’t on one die, but they’re co-packaged closely enough to communicate as if they were.

This is also why HBM can’t simply be bolted onto any chip — it requires advanced packaging capacity (interposer fabrication, TSV etching, precision die stacking) that’s a separate manufacturing discipline from DRAM or GPU logic fabrication. That distinction matters enormously later, in the supply-chain section.

How HBM Connects to an AI GPU

A modern AI accelerator surrounds its GPU die with several HBM stacks, not one. Each stack connects through its own independent memory channels — HBM3 and HBM3E define 16 channels per stack, each acting as a semi-autonomous pathway servicing requests in parallel rather than queuing behind one shared bus.

The bandwidth math is simple multiplication: interface width × data rate per pin × number of stacks ÷ 8 (bits to bytes). One HBM3E stack at roughly 9.2–9.8 Gb/s per pin across a 1,024-bit interface delivers a bit over 1 TB/s. Multiply by five or six stacks around a single GPU die, and you land on the 4–8 TB/s aggregate figures now standard on frontier AI accelerators — numbers that would be physically implausible with edge-mounted GDDR at any reasonable power budget.

The Generational Story: HBM1 to HBM4

HBM’s evolution isn’t the same chip getting incrementally faster — each generation solved a specific bottleneck the last one exposed.

Two inflection points matter most. HBM3E kept HBM3’s interface and simply pushed signaling faster, letting it ship quickly. HBM4 is different in kind: the first generation since HBM1 to widen the interface itself, and the first to move toward a genuinely customizable base logic die — meaning GPU vendors can increasingly co-design memory’s control layer alongside the compute die, rather than treating HBM as an off-the-shelf part.

HBM in Real AI Accelerators

The cleanest way to see why memory, not compute, decides real-world performance is to compare GPUs that share compute silicon but differ only in memory.

The H100-to-H200 comparison is the cleanest natural experiment in the entire AI hardware market: identical GH100 compute die, identical on-paper TFLOPS — and yet NVIDIA’s own benchmarks show roughly 1.9x faster real-world inference on large models after the memory upgrade alone. Nothing about the compute changed. The GPU simply stopped starving.

Looking ahead, NVIDIA’s Vera Rubin platform and AMD’s MI400/MI430X generation both commit to HBM4, targeting per-GPU bandwidth in the 13–20 TB/s range and capacities approaching 300–400+ GB. (Note: figures vary by SKU and cooling configuration — cite the exact configuration, e.g. SXM vs. PCIe, from NVIDIA’s/AMD’s own datasheets at publish time.)

Why AI Workloads Are Unusually Memory-Hungry

Not every workload cares this much about bandwidth. AI is unusual because of how transformer models actually execute.

Training holds a model’s weights plus activations from the forward pass (needed for backpropagation), optimizer states, and gradients — often several multiples of the raw parameter count — and touches most of it, repeatedly, on every step.

Inference has its own pressure point: the KV cache. As a transformer generates tokens one at a time, it retains a running cache of key/value vectors for every prior token, so each new token’s attention calculation can reference everything before it. As context windows have grown from a few thousand tokens to hundreds of thousands, this cache has exploded — and unlike model weights, it grows dynamically during generation and must be read on every single step.

This is what “memory-bound” means: the processor’s arithmetic units sit idle waiting for data far more often than they’re actually computing. In a compute-bound workload, more FLOPS makes things faster. In a memory-bound workload — which describes the majority of real-world LLM inference — more FLOPS does almost nothing, because the bottleneck was never the math.

From Compute-Centric GPUs to Memory-Centric AI Accelerators

Put the pieces together and a framework emerges: the industry has moved from compute-centric design, where memory was sized to whatever the compute die needed, to memory-centric design, where bandwidth and capacity are now primary constraints that compute is designed around. HBM4’s shift toward integrated logic in the base die is the clearest evidence — memory is no longer just fed by the processor, it’s becoming an active participant in the pipeline itself.

It isn’t that AI GPUs “use” HBM the way a laptop uses RAM. HBM’s bandwidth ceiling increasingly determines what an accelerator’s usable performance actually is, regardless of how much compute sits on the die beside it.

The HBM Supply Chain — and Why It’s Becoming an American Manufacturing Story

Only three companies on Earth manufacture HBM at meaningful scale: SK hynix, Samsung, and Micron. That concentration alone would make HBM a supply-chain concern; combined with surging AI demand, it’s made HBM one of the tightest bottlenecks in the AI hardware stack.

Two constraints compound each other. First, HBM production — DRAM fabrication plus TSV processing plus die stacking — requires specialized capacity that can’t be quickly repurposed from standard DRAM lines, and expanding it typically takes over a year. Second, finished stacks still need advanced packaging — most notably TSMC’s CoWoS process — which has, at various points, been the tighter of the two bottlenecks. A GPU vendor can have all the compute die and HBM stacks it needs and still be shipment-limited by how many finished packages CoWoS can assemble in a quarter.

The results show up directly in pricing: HBM capacity across all three suppliers has reportedly been sold out well into 2026 and, in places, 2027, with contract pricing rising sharply and hyperscalers reportedly prepaying billions to secure future allocation. This is the technical foundation underneath the broader memory-pricing story hitting everything from AI accelerators to consumer laptops, since HBM and conventional DRAM draw on overlapping fabrication capacity.

Here’s the part that matters most for a U.S. reader: almost all of today’s leading-edge HBM is manufactured in Taiwan, Japan, and Korea. That’s now changing. Micron — the only U.S.-headquartered company among the three HBM makers — is in the middle of an approximately $200 billion domestic expansion, backed by CHIPS Act funding, that includes two new leading-edge memory fabs in Boise, Idaho; up to four fabs planned in Clay, New York; and an expanded Virginia facility that will bring HBM packaging and assembly to U.S. soil for the first time. The first Idaho fab is targeting first output in mid-2027. Roughly 90,000 direct and indirect jobs are tied to the buildout. In short: the chip quietly deciding how fast American AI companies can scale is also becoming a genuine domestic manufacturing bet.

Why This Matters for AI Infrastructure Economics

Pull the thread all the way through: HBM bandwidth → GPU utilization → training/inference throughput → advanced packaging demand → AI accelerator architecture → HBM supply constraints → AI infrastructure economics. A memory technology most people outside semiconductors have never heard of now sits directly upstream of how fast new AI models can be trained, how much it costs to serve them, and how fast the industry can scale compute capacity at all. When HBM is scarce or expensive, it doesn’t just raise GPU prices — it reshapes which companies can afford frontier-scale AI infrastructure in the first place.

Frequently Asked Questions

What is HBM used for? HBM is used as the primary memory in AI accelerators, high-performance computing systems, and some high-end graphics cards — anywhere a processor needs to move very large amounts of data at very high speed.

Is HBM faster than DDR or GDDR? HBM delivers far higher aggregate bandwidth than DDR or GDDR, mainly because of its much wider interface (1,024 or 2,048 bits vs. 32–64 bits), not because any single connection is faster in isolation.

Why can’t GPUs just use more GDDR instead of HBM? GDDR’s edge-mounted, PCB-based layout physically limits how wide its interface can be, and pushing per-pin speeds higher to compensate costs disproportionate power. HBM’s stacked, co-packaged design sidesteps that limit entirely.

What’s the difference between HBM3 and HBM3E? HBM3E keeps HBM3’s 1,024-bit interface but pushes per-pin data rates higher — roughly 600–819 GB/s to over 1 TB/s per stack — with taller (12-Hi) stack options for more capacity.

Why is HBM4 a bigger deal than HBM3E was? HBM4 doubles the interface width to 2,048 bits for the first time since HBM1, and moves toward a customizable logic base die — a structural redesign, not just a speed bump.

Which companies make HBM? Only three: SK hynix, Samsung, and Micron — and Micron is the only one headquartered in the United States.