One model, one server
The first step: the model that ran on a laptop moves to vLLM on a GPU server, and a team or department starts using it through an OpenAI-style API. From day one you can see how much it is used and how long it takes to answer.
vLLM · llm-d · Kubernetes
We deploy and support AI inference on your servers.
Your open models, served to the whole company with vLLM and Kubernetes behind the same API as OpenAI, and your data never leaves your data centre. If you have already tried them with Ollama, this is the next step.
Several teams
When a second model or a second team arrives, it makes sense for them to share the same GPUs instead of buying another machine. For that we set up Kubernetes with the NVIDIA GPU Operator, KServe to deploy each model and Kueue to share out quotas, and we store the models in Ceph.
Many users at once
Once hundreds of people use a model, a single vLLM replica falls short. llm-d sits in front and sends each request to the replica that already holds that conversation in memory, which saves the GPU work, and it can also read the prompt on one card and write the answer on another.
In production
By now AI is a service people depend on, so it is run like one: metrics for each model, traces for each request, every new vLLM release tested before it reaches production, and an annual support contract.
The servers
The machine for
each step.
The GPU depends on the size of the model and on how many people will use it at once, which is why we measure it before you buy anything. If you already have servers, we work on yours. The calculations are in our guide AI server: how to choose one.
To start
SupermicroAS-5126GS-TNRT-01-G22x RTX PRO 6000 · 1.5 TB
LenovoThinkSystem SR650a V4Up to 4 RTX PRO 6000 · Intel Xeon 6
Configure a custom server →All Supermicro AI servers →All Lenovo AI servers →
How it works
The journey of a request.
- Your applicationA chat, an agent or an integration calling the OpenAI API, only inside your own network.
- llm-dPicks the replica for each request, based on which one already has the conversation in memory.
- vLLMHandles many conversations at once on each GPU and streams the answer back word by word.
- The GPUShared in Kubernetes between models and teams, with quotas so nobody hogs the cards.
Support
Who answers
for each piece.
The operating system is supported by Canonical or CIQ, and Kubernetes, KServe and Ceph have Canonical support. No vendor covers serving the models, so we support that part ourselves, under an annual contract.
SIXE
- vLLM serving the models
- llm-d spreading the load
- Observability metrics and traces
Canonical or CIQ
- Kubernetes, KServe and Ceph Canonical
- Operating system Ubuntu with Canonical, Rocky Linux with CIQ
- NVIDIA drivers CIQ, in RLC Pro AI
How we work
Before you buy,
we measure.
Before you buy
We serve your model with your real workload and measure how long the first word takes and how many conversations it holds at once. Those figures decide how many GPUs you need.
Configure a server →Getting it running
We install the server, the operating system, Kubernetes and vLLM, plus llm-d or KServe when they are needed, until your model is serving with your data.
Kubernetes and containers →Afterwards
vLLM and llm-d ship new releases often, so each one is tested before it reaches production. And if your team wants to run it themselves, there is a course for that.
vLLM and llm-d course →Frequently asked questions
What people ask us
first.
vLLM or Ollama?
Ollama is the easiest way to try a model on a laptop or a server, and it is great for that. vLLM is built to serve it to many people at once, because it shares the GPU memory between conversations and gets more out of the card. The usual path is to try with Ollama and move to vLLM as usage grows. We compare them in detail in our guide vLLM vs Ollama.
What is llm-d and when do I need it?
It is an open-source project that spreads vLLM work across several replicas in Kubernetes: it sends each request to the replica that already holds the start of that conversation in memory, and it can split reading the prompt and writing the answer across different GPUs. You need it when one model serves many people at once; for a small team vLLM is enough.
Which models can be served?
The most widely used open models, such as Llama, Qwen, Mistral, Gemma, DeepSeek or gpt-oss, as they are or reduced to 8 or 4 bits so they take less memory. Which model and in which format depends on the task and the GPUs, and it is the first thing we measure.
How many GPUs do I need?
It depends on the size of the model and how many people use it at once, because each conversation takes its own GPU memory. A 70-billion-parameter model in 8 bits fits on a 96 GB card, but only with a few long conversations at a time; for more users you need more GPUs or llm-d. That is why we measure it with your workload before you buy. AI server configurator →
Does it work without a GPU?
For tests, or for small models with few users, vLLM also runs on CPU, including IBM Power. To serve several people at once with reasonable response times you need a GPU. vLLM on IBM Power, without a GPU →
Next step
Let’s talk.
Tell us which model you are interested in, for how many people, and which servers you have, if any, and we will suggest how to measure it and what you need to serve it.
