Serve GenAI 56× faster without buying GPUs: the IBM, NVIDIA and Supermicro paper
As context grows, it stops fitting in GPU memory. The system throws away previous work and has to redo it as soon as the session returns — and that's where your requests per second go. IBM, NVIDIA and Supermicro published in June 2026 the Context Without Limits Redbook with the numbers of a different approach: keep those computations on fast shared storage. Time to first token drops from 32 s to half a second, and the system serves 22× more requests.
Two years of debate on GenAI infrastructure have been about how many GPUs to buy. IBM, NVIDIA and Supermicro change the question in the Context Without Limits Redbook (June 2026): if the GPU is slow because it keeps redoing computations it did seconds ago, the problem isn't how much GPU you have — it's where you keep those computations. Moving them to fast shared storage gives, in their tests, a response 56× faster to the user, 22× more requests per second and 95 % less total time. Below we break down what they measured, how it works, when it makes sense and what your options are if you'd rather build it on Ceph than on IBM.
A large model (Llama 70B, gpt-oss 120B) responding with a 100,000-token context no longer fits entirely in GPU memory. The system has to evict old work and, when the session returns, redo it from scratch — like re-reading the whole book every time someone asks a question about it. Keeping that work on fast shared storage removes the repetition. That's what IBM Redbook MD260021 measures (June 29, 2026), on 8 Supermicro nodes with IBM Storage Scale ECE, NVIDIA Dynamo and vLLM 0.14.1.
What is the KV cache and why is it the new bottleneck?
The KV cache (Key-Value cache) is the structure where a transformer model stores the attention keys and values of every token in the context. Without it, each new token would force a recomputation of attention over everything before it; with 100,000-token contexts that's no longer viable. Put differently: the KV cache is what makes serving long contexts at acceptable speeds possible.
The problem shows up when GPU HBM fills. HBM is blistering-fast (microseconds) but expensive and limited: an NVIDIA H100 has 80 GB, an RTX PRO 6000 Blackwell reaches 96 GB. On a 120-billion-parameter model with a 130,000-token context, a single request can consume 60–80 GB of KV cache. With several concurrent users, the GPU evicts older caches and, if that session returns, recomputes them from scratch. Each recomputation is a full forward pass through the model — GPU cycles going by without producing a single new token.
Buying more GPUs adds HBM to the cluster, but each HBM stays isolated per GPU. The cache generated for user A isn't available when their next request lands on GPU B. You pay for hardware without improving the hit rate — until the KV cache is moved to a shared tier.
How much KV cache does your deployment consume?
Pick a model, a context length and a concurrency. The calculator applies the standard KV cache formula and tells you if it fits in a typical GPU's HBM or if you need a shared G4-style tier. Per-token values come from the official configurations of each model (Llama 3.1, gpt-oss 120B) — they are not estimates.
NVIDIA Dynamo's memory hierarchy: G1, G2, G3, G3.5 and G4
NVIDIA Dynamo defines five storage tiers for the KV cache, each with a different balance of latency, capacity and cost. Tiers G1–G3 are bounded by a single node's hardware. G4 is the only cluster-wide shared tier — the one that lets a cache generated by GPU 12 be reused when the same session lands on GPU 47.
What does IBM Storage Scale ECE do here?
IBM Storage Scale is the current commercial name of what was historically called GPFS (General Parallel File System). It's a distributed parallel file system with 25+ years in HPC — the same one behind systems like Summit or Sierra. ECE (Erasure Code Edition) is the software-defined variant that runs on commodity servers, uses erasure coding instead of classic RAID and scales to exabytes.
In the Redbook architecture, Storage Scale ECE plays the G4 tier role for six concrete reasons:
- POSIX, S3, NFS, SMB and CSI from the same backend — fits both vLLM/Dynamo and existing data pipelines.
- NVIDIA GPUDirect Storage (GDS): direct storage → GPU transfer bypassing CPU, via RDMA over lossless Ethernet or InfiniBand.
- Erasure coding 8+2P with 80 % usable capacity (vs 50–67 % on classic RAID6), without sacrificing durability.
- Automatic tiering NVMe / SSD / HDD / tape — hot KV cache lives on NVMe, cold moves to HDD.
- Instant snapshots and clones — useful for versioning datasets and training checkpoints.
- Scale without redesign: from 3 nodes to 256 without changing the architecture.
Three benchmarks from the Redbook (measured, not estimated)
IBM, NVIDIA and Supermicro built 8 Supermicro Petascale ASG-1115S-NE316R nodes (AMD EPYC 9535, 16 Micron E3 NVMe of 7.68 TB each, ConnectX-7 at 400 Gb/s), connected by three NVIDIA Spectrum-X SN5600 switches in spine-leaf with 800 Gb/s uplinks. Inference client: a single Supermicro SYS-212GB-FNR with 4 RTX PRO 6000 Blackwell GPUs. Software: IBM Storage Scale ECE v6.0.0.1 on RHEL 9.6, clients on Ubuntu 24.04, NVIDIA Dynamo v0.9.0+, vLLM v0.14.1 and openai/gpt-oss-120b quantised to MXFP4.
Benchmark 1 — Time-to-first-token (TTFT) by context length
G1 (HBM) and G2 (DRAM) tiers were set to zero capacity to force the scenario: all KV cache — 1.4 million tokens — went through G4.
Without persistent KV cache, time-to-first-token grows quadratically with prompt length. With Storage Scale it stays under a second across the whole measured range. At 130 K tokens it's 32 seconds vs half a second: at 32 seconds the user has already closed the tab; at half a second they stay.
Benchmark 2 — Throughput under concurrent load
200 requests on the 120-billion-parameter model with 28 concurrent connections (100 unique prompts, 24 million tokens, 825 GB of KV cache):
22× more throughput, 95 % less total time. And the cache started cold: the speedup built up over the 200 requests. With a pre-warmed cache, the real figure would be even better.
Benchmark 3 — Under "noisy neighbor" stress
Four concurrent clients generating 200 GB/s of junk traffic in parallel, to simulate a realistic multi-tenant cluster.
18× improvement under stress, only 18 % degradation vs the clean scenario. Proof that the architecture doesn't collapse when the cluster is full.
How many nodes, GPUs and bandwidth do I need?
The Redbook itself publishes three sizing profiles. Here they are side by side — click the one closest to your case.
Multi-tenant production, LLMs of 70 B to 120 B parameters, RAG and long-context multi-turn. It's the exact configuration that appears in the Redbook with the 315 GB/s measured.
In the Redbook benchmark, the network hit the bottleneck before storage did: the 8 nodes deliver up to 315 GB/s of read, but the 2×400 GbE client-side network tops out at 80 GB/s. If you're going to build something similar, size the client network to match the storage or your GPUs will spend the day waiting for data.
Open source alternatives: Ceph, Lustre, DAOS
The core idea in the paper — pushing the KV cache to a shared G4 tier with GPUDirect and RDMA — doesn't depend on IBM. You can build it with other pieces.
If you're already an IBM customer with an SLA in place, Storage Scale ECE is the fastest path to production. If you're building from scratch and prioritise cost and control, start with Ceph. If you come from pure HPC and need 300 GB/s per node, Lustre or DAOS are legitimate options. If the diagnosis isn't clear, better validate it before ordering hardware.
The long comparison with real benchmarks of the three open source options is here: IBM Storage Ceph vs Storage Scale (GPFS), GFS2, NFS and SMB.
Six questions that save you hardware
A well-built G4 tier gives the paper's numbers. A poorly sized one is hundreds of thousands of euros of expensive hardware spending the day waiting for data. These six questions separate the case where it pays off from the one where it doesn't.
If your GenAI is slow or too expensive, chances are the GPU isn't the culprit
We work with IBM Storage Scale, Ceph, Lustre and GPUDirect in production systems. IBM Business Partner with 15+ years on 24/7 critical infrastructure. If you suspect the bottleneck isn't your GPU, we help validate the diagnosis before it becomes a seven-figure project.
Frequently asked
What is the KV cache in an LLM?
The KV cache (Key-Value cache) is the structure where a transformer model stores the attention keys and values computed for every token in the context. Without it, the model would have to recompute attention over all previous tokens for every new token it generates. It's what makes serving long contexts feasible.
Why is the KV cache a bottleneck for serving LLMs?
Because GPU HBM is finite (80–96 GB on current models) and very expensive. With several concurrent sessions and long contexts, the KV cache doesn't fit: the GPU evicts older caches and recomputes them when the session returns, burning cycles that could be generating tokens. The fix is to push that cache to a shared high-speed storage tier.
What is IBM Storage Scale and how does it relate to GPFS?
IBM Storage Scale is the current commercial name of what was called GPFS (General Parallel File System). It's a distributed parallel file system with 25+ years in HPC. The ECE (Erasure Code Edition) variant is software-defined, runs on commodity servers and uses erasure coding instead of RAID.
Can I do the same with Ceph instead of Storage Scale?
Yes. Moving the KV cache to a shared G4 tier with GPUDirect and RDMA is independent of the file system. Ceph, Lustre and DAOS can play that role. Storage Scale is the lowest-friction path if you're already an IBM customer; Ceph is the first choice when cost, control and flexibility matter.
What is GPUDirect Storage and why does it matter?
GPUDirect Storage (GDS) is an NVIDIA technology that moves data directly between the file system and GPU memory, bypassing the CPU and host RAM. It prevents the CPU from becoming the next bottleneck when moving terabytes of KV cache between storage and GPU.
What's the realistic gain if I apply this to my deployment?
It depends on the workload. The 56× and 22× in the paper come from extreme context (130K tokens) and large KV cache (825 GB). For short contexts and low concurrency the gain is smaller. The longer the context, the more multi-turn the usage and the more concurrent the traffic, the bigger the G4 tier benefit.
Do I have to use NVIDIA Dynamo?
For orchestrating multi-tier KV cache (G1–G4) with smart eviction between tiers, Dynamo is the most complete tool today. You can build something similar with vLLM standalone and a custom scheduler, but you lose KV-aware routing and transparent offloading via NIXL. In production, vLLM + Dynamo + a shared G4 is the most consolidated combination as of August 2026.
Blog!
Our mission
SIXE ensures that your infrastructures based on IBM®, Red Hat®, Canonical®, and SUSE® technologies remain stable, secure, and optimized for as long as you need them.
At SIXE, we are committed to sustainability. That’s why we help you design, implement, and maintain durable and efficient technological solutions.
