SupermicroASG-2015S-E1CR24L-01-G2

This is the capacity machine of the family. The 24 drives sit in hot-swap 3.5-inch bays that open from the top of the chassis, and they hang off a Broadcom 3808 HBA (the controller that attaches drives without building a RAID) which passes them straight through to the operating system. That is exactly what Ceph and ZFS want, since each manages every drive on its own. With 24 × 22 TB you get 528 TB raw per node. The dual 100 GbE ports also let you split Ceph's public network from the replication network, which carries all the traffic when a drive fails and has to be rebuilt.

Those 528 TB come down to 176 TB usable with 3-way replication and 352 TB with 4+2 erasure coding (each object is split into four data chunks and two parity chunks), although replication needs at least three nodes and 4+2 needs six. There is one configuration detail to bear in mind: it ships with a single 960 GB SATA SSD for the OS and no NVMe, so the internal database Ceph keeps for each drive (BlueStore DB/WAL) lives on the spinning disk itself. For archive and backup that works well. If you will be moving lots of small files, the board has two M.2 NVMe slots, but adding drives takes the machine out of the Gold configuration and it becomes a build-to-order system.

What we do

We add it to a Ceph cluster alongside your other nodes, or set it up as a backup repository with Proxmox Backup Server or your own backup software on ZFS. Before building anything we agree with you on the failure domain (which set of machines can go down at once without losing data), how to use the two 100 GbE ports and which data goes on replication and which on erasure coding. Under contract, firmware, the BMC (the remote management controller) and Ceph updates are on us, with 24×7 on-call cover if you need it. If someone else already looks after your Ceph cluster, our third-party support may interest you.