AI and HPC storage with Ceph.
We design, build and run the cluster your GPUs read from, in your own data centre. Datasets, models, checkpoints and documents live in a single system that grows by adding servers.
Why storage holds back your GPUs
When storage can’t keep up, the GPU sits waiting and the most expensive machine in the data centre stops working. There are usually two causes: the metadata can’t cope with millions of small files, or the network between nodes runs out, and neither is solved by adding capacity.
Ceph brings together in one cluster what AI asks for separately: S3 objects for datasets and models, a shared file system for GPU nodes, and block storage for virtual machines and databases. How well it performs depends on the design, so we start by measuring your workload and size disks, network and metadata from that measurement.
What each workload needs
An AI cluster rarely does just one thing, and each workload puts pressure somewhere different. That’s why we split the data into separate pools and access paths within the same Ceph cluster.
| Workload | What it needs from storage | How we handle it with Ceph |
|---|---|---|
| Training | Reading the dataset from many nodes at once, often across millions of small files. | CephFS with metadata on NVMe and several active MDS, or RGW if your framework reads over S3. |
| Checkpoints | Large write bursts that don’t slow training down. | An NVMe pool in CephFS, then a copy to a capacity pool. |
| Inference | Loading the same weights on many nodes and starting quickly when scaling. | CephFS shared by every node, or RGW with a local NVMe cache. |
| RAG | Millions of documents and chunks, plus a latency-sensitive vector database. | RGW for the originals, CephFS for processed data and RBD on NVMe for Milvus, Qdrant or pgvector. |
| HPC | Shared user and project directories, and results that have to be kept. | CephFS for users and projects, and RGW with erasure coding for results. |
| Archive | Keeping datasets and model versions for years at the lowest cost per TB. | RGW with erasure coding on high-capacity disks and lifecycle rules. |
Block, file and object in the same cluster
CephFS
The shared directory every GPU node can see, which is what PyTorch or vLLM expect when you give them a path.
How we set it up
In Kubernetes it arrives as a ReadWriteMany volume through Rook. We size the MDS, the metadata servers, separately, and the metadata pool always goes on NVMe.
RGW
An S3 API inside your network for datasets, model weights and results. Anything that already speaks S3 only needs a new address.
How we set it up
With versioning, lifecycle rules and per-team quotas. Cold data moves to capacity disks with erasure coding, and several gateways share the requests.
RBD
Disks for virtual machines, databases and vector databases, with snapshots and clones that don’t copy the whole volume.
How we set it up
In Kubernetes it’s a ReadWriteOnce volume, and in OpenStack or Proxmox VE it’s the machines’ disk. Anything latency-sensitive gets its own NVMe pool.
What decides performance
The network Every replicated write crosses the network several times, and recovery traffic competes for the same links. We separate the cluster network from the client network and size it for what your GPUs need: with NVMe drives we start at 100 GbE, because changing it later means working on a cluster in production.
Metadata With millions of small files, the first bottleneck is usually here and not in the disks. We put the metadata pool on NVMe and share the work across several active MDS, pinning directories to each one when the workload calls for it.
Replication or erasure coding Three replicas write better but use three times the capacity. Erasure coding makes far better use of the disk in exchange for more CPU and network. We use replication for hot data and erasure coding for cold data, in separate pools of the same cluster.
Failures CRUSH rules spread copies across hosts, racks or rooms so that a fault never takes out more than one. We also tune recovery so that rebuilding a disk doesn’t slow your GPUs down.
When we suggest something else
If your main workload is large-scale training with GPUDirect Storage, or simulations where thousands of processes write to the same file at once, a parallel file system such as IBM Storage Scale or Lustre usually performs better.
Often the right answer is to combine them: the parallel file system for the most intensive work and Ceph for capacity and S3. We are an IBM Business Partner and we work with both, so we run these numbers with your workload in front of us. The full comparison is in Storage Scale or Ceph for inference and in which storage to choose for AI and HPC.
Hardware for the cluster
These are the Supermicro Gold Series configurations we use as a starting point. If you already have servers, we check which ones can be reused.
How we work
We can run the cluster for you or leave it ready for your own team to run.
We look at what your workload reads and writes, at what sizes and from how many nodes, and how much data you’ll have two years from now.
We define disks, network, pools and failure domains, and tell you what to buy and what to reuse.
We deploy with cephadm or Rook, connect Kubernetes and your inference platform, and move the data from your NAS or storage array in phases.
We repeat the measurement with fio and your own jobs before going into production, and keep those figures as the baseline.
Upgrades and incidents are handled by the same team that designed the cluster, through our Ceph technical support. We also train your team on our Ceph courses.
Frequently asked questions
Is Ceph any good for training models?
Yes, especially when training sits alongside inference, RAG and shared datasets, and the cluster has NVMe and a network of 100 GbE or more. If all that matters is getting maximum bandwidth to many GPUs, we suggest a parallel file system, alone or alongside Ceph.
Does the whole cluster have to be NVMe?
No. The usual approach is to keep NVMe for metadata, checkpoints and hot data, and leave what is rarely read on capacity disks. That way the flash budget stays where it shows.
How many nodes do I need?
A replicated cluster can start with three. Erasure coding needs more: as many as data and parity chunks plus one, so that data can be rebuilt when a node goes down. The real number comes from the capacity and performance you need, and we work it out from your measured workload.
Community Ceph or IBM Storage Ceph?
We implement both. The community version gives you independence and no licence cost. IBM Storage Ceph adds vendor support, which some companies need for contractual or audit reasons.
Does it work with Kubernetes?
Yes. With Rook, Ceph runs inside the Kubernetes cluster itself, and with Ceph CSI it connects to an external cluster. Either way pods get RBD and CephFS volumes, and applications can use S3 through RGW.
Can you bring the data over from a NAS or storage array?
Yes. We move it in phases: cold data first, then what is in use, which is synchronised and switched over in a window agreed with you.
Can we reuse the hardware we already have?
Often, yes. We review the disks, network and memory in your servers and tell you what works for Ceph, what only works for capacity and what is worth changing before you start.
What do your GPUs need to read?
Tell us what you train or serve, how many GPUs you have and how much data you handle. We’ll tell you whether Ceph fits and how we’d build it.
