AI server: how to choose oneFirst the model, then how many people use it at once, and last what your data centre can take. These are the sums we do before recommending a machine.
gpt-oss-120b, OpenAI’s open model, has 117 billion parameters and fits on a single 80 GB GPU. It gets there through the format: it stores almost all of its numbers in 4 bits instead of 16, and that changes the machine you need more than any new product in the catalogue.
If you are still at the “why would I want AI in-house?” stage, we cover that in why companies are building their own AI servers.
How much memory the model takes
A language model is a huge list of numbers (the parameters) that have to stay loaded in GPU memory all the time. The calculation is a simple multiplication: the parameters times the size of each one, which is two bytes in the original 16-bit format, one in 8 bits and half a byte in 4 bits. So a 70-billion-parameter model (a 70B, as model names put it) needs 140 GB straight out of the box, about 70 in 8 bits and about 35 in 4 bits.
Llama 3.3 70B: memory for the model alone
In 8 bits the quality barely suffers, and that is where we suggest starting. In 4 bits the model still gets it right almost every time, but check that “almost” against your own documents before you trust it.
In mixture-of-experts models, very common among open ones, only part of the model works on each token (each small piece the text is cut into, roughly a syllable). They answer quickly, but they still need all of their memory. Mistral Small 4 is one example.
How many people use it at once
Each conversation in progress takes its own slice of GPU memory while the model answers it, because the model notes down there what it has already read so it does not have to reread everything for each token it writes. And that slice grows as the conversation gets longer. On Llama 3.3 70B it is about 0.3 MB per token, so an 8,000-token conversation (a fifteen-page report and a few questions about it) takes about 2.5 GB. On an 8-billion-parameter model such as Llama 3.1 8B, the same conversation stays at 1 GB.
Now have twenty people asking questions at once about long documents. The 70B needs 50 GB just for their conversations, twice what an RTX PRO 6000 with 96 GB has left after loading the model in 8 bits. Those 25 GB are enough for about ten long conversations at a time, or twice that if that memory is also stored in 8 bits, and with short questions many more fit. For a hundred people at peak time you need another machine.
A 96 GB RTX PRO 6000 running Llama 3.3 70B in 8-bit
How fast the answer appears depends mostly on how fast that memory is, because for each token the model reads a large share of its parameters. On an H200, HBM (memory mounted right next to the chip) moves 4.8 TB/s, about three times as much as on an RTX PRO 6000, and you notice it when lots of people are waiting for an answer.
How many cards you need depends on how many people use it at once and with which documents. It is worth measuring that with your workload before you buy. We can do it for you, or your team can learn how on the local LLM and GPU infrastructure course.
The machine those numbers point to
If the model is small and few people use it, or the work runs in overnight batches, a CPU with plenty of memory is enough. But as soon as someone is waiting for an answer, the GPU stops being optional.
The usual choice for a company is a rack server with between one and eight PCIe cards (the kind that plug into a slot in the server). The L4, with 24 GB, suits small models; the L40S goes up to 48 GB, and the RTX PRO 6000 Blackwell to 96 GB, enough for a 70B in 8 bits. With these cards you can build anything from a 2U server (one U is 4.45 cm of rack height) with up to four to a 4U server with eight, where every model in the table up to DeepSeek-V4-Flash fits. DeepSeek-V3 would fit in memory, but split across eight cards that only talk over PCIe it would run much more slowly.
Above that are the HGX boards: eight data-centre GPUs (H200, B200 or B300, from 141 to 288 GB each) joined by NVLink, a direct link between them several times faster than PCIe. With it, a model with hundreds of billions of parameters is split across the eight as if they were a single GPU. It is the machine for models like DeepSeek-V3, for hundreds of users or for training your own models, and it comes with B300 and air cooling. Buying one so that a department can summarise contracts is using a sledgehammer to crack a nut.
From the simplest machine to the largest
CPU with plenty of memory
no GPUSmall models, few people or overnight batch jobs.
One to four PCIe cards
2UWith one 96 GB RTX PRO 6000, a 70B in 8-bit.
Up to eight PCIe cards
4UEvery model in the table up to DeepSeek-V4-Flash.
HGX board: eight GPUs with NVLink
141 to 288 GB per GPUDeepSeek-V3 and R1, hundreds of users or training your own models.
NVIDIA is not the only option: AMD sells its Instinct cards and vLLM runs on them too, though most AI software is still tested on NVIDIA first, so check that what you plan to use works well on AMD.
What your data centre has to handle
The figure that alarms people most is the one for the room. An eight-GPU B200 server draws up to 14.3 kW, about the same as five 3 kW kettles left on in one box. Before ordering anything you need to know how much power reaches the rack, whether the cooling can remove that heat and whether the floor can take the weight.
PCIe cards are easier to live with, though not as much as it seems: each RTX PRO 6000 can reach 600 W, and eight add up to almost 5 kW in GPUs alone. Some 2U servers cap them at 450 W so that four will fit. And if the room cannot take any more air conditioning, there are 5U servers with eight B200 and liquid cooling, provided the room has, or can have, a water circuit.
Power draw, in kW
If the machine is going to an office or a plant without a data centre, what matters is cabinet depth and temperature. There are short servers, about 44 cm deep, which in some configurations handle up to 55 °C with small GPUs such as the L4 and about 40 °C with larger ones.
Who looks after the software
Once the machine is chosen, it still needs software: the operating system, the GPU drivers and the program that serves the model. All three have to match, because an NVIDIA driver that does not fit the CUDA version turns the GPU into a very expensive paperweight. We work with two software stacks. One is Ubuntu, which since 26.04 LTS ships CUDA (NVIDIA's libraries for using the GPU) in its own repositories, with Canonical support. The other is Rocky Linux in its RLC Pro AI edition, with CUDA, the network drivers and PyTorch already validated and CIQ support, which is more convenient if your servers are already in the Red Hat family.
vLLM runs on top, the open-source program that shares memory between conversations. It speaks the same language as the OpenAI API, so the applications you already have barely change. If several teams are going to share the GPUs, Kubernetes is added to split them between teams, which you practise on the vLLM on Kubernetes course. As we are Canonical and CIQ partners, the same team looks after the server, the operating system and the support. And if you would rather not wrestle with any of this, we can set it up for you as an inference service on your own servers.
Frequently asked questions
How much GPU memory does a 70-billion-parameter model need?
For the model alone, 140 GB in 16 bits, about 70 in 8 bits and about 35 in 4 bits. On top of that comes about 2.5 GB for each long conversation. An RTX PRO 6000 with 96 GB serves it in 8 bits to a few people at once.
Can you run AI on-premise without a GPU?
Yes. With small models, few users or batch jobs, a CPU with enough memory will do. When people are waiting for answers or the model grows, you need a GPU.
How many people can a server with a single GPU serve?
It depends on the model and on how long the conversations are, because each one takes memory. A 70B in 8 bits on an RTX PRO 6000 handles about ten long conversations at a time, and an 8-billion-parameter model on the same card, dozens. The reliable way is to measure it with your workload.
How much does an AI server cost?
The biggest cost driver is the number and class of GPUs; then come memory, networking and years of support. The price follows from the model you will serve and how many people will use it. With those details and where the machine will go, in the configurator or by talking to us, we will prepare a proposal.
Which server do you need?
Tell us which model you want to serve, for how many people and where the machine will go. If needed, we measure your workload before you buy, and the same team looks after the server, the system and the support. If you would rather do the sums yourself, the configurator lets you build it step by step.
AI servers, with Ubuntu or Rocky Linux and supportBlog!
Our mission
SIXE ensures that your infrastructures based on IBM®, Red Hat®, Canonical®, and SUSE® technologies remain stable, secure, and optimized for as long as you need them.
At SIXE, we are committed to sustainability. That’s why we help you design, implement, and maintain durable and efficient technological solutions.
