SupermicroAS-2015CS-TNR-01-G2

Supermicro positions it as storage hardware for private clouds, and the parts list explains why. The 8 × 24 TB give 192 TB raw in a chassis with twelve 3.5-inch bays, and the two 3.8 TB U.2 NVMe drives (the 2.5-inch server form factor) do the thing that most improves a spinning-disk Ceph cluster: they hold each drive's internal database (what Ceph calls DB/WAL) on flash, four drives per NVMe, so small writes do not wait for the disk head. With 48 cores and 768 GB there is plenty of machine for the eight Ceph daemons, and whatever is left over can run virtual machines on the same node with Proxmox.

With 3-way replication each node contributes 64 TB usable and a three-node cluster ends up at 192 TB; with 4+2 erasure coding and six nodes you reach 768 TB usable out of 1,152 raw. Networking is 2 × 25 GbE, which keeps up with eight spinning disks but falls short if the cluster moves to all-NVMe, and for that day the chassis has four PCIe 5.0 x16 slots.

What we do

We set it up as a Proxmox VE node with built-in Ceph, the most direct way to get virtualisation and shared storage without a separate array, or as a node in a dedicated Ceph cluster if the virtual machines run on other servers. The OS is mirrored across the two 240 GB SSDs, and we hand it over with the BMC (the remote management controller), firmware and the split between VM traffic and replication traffic already in place. It fits as a node in a VMware to Proxmox migration.