Description
A course for the platform team that has to take a model from an Ollama test to a service the whole company uses: vLLM serving the model on Kubernetes, llm-d spreading the load across replicas, and GPUs shared between teams. Everything is built on a lab cluster with GPUs and measured under load, so you know what each piece adds before taking it to production.
Course details
| SIXE code: SXIA08 | Category: Artificial Intelligence / AI platform |
| Delivery: Online and on-site | Duration: 3 days |
Who it is for
Platform, SRE and systems teams who already run Kubernetes and will operate language models for several teams in the company.
Prerequisites
Confident with Kubernetes (deployments, services and Helm) and Linux. Having served a model with Ollama or vLLM helps; if not, the local LLM and GPU infrastructure course covers that ground.
What you will learn
- From Ollama to vLLM: what changes when serving many people: the memory each conversation takes, continuous batching (requests join and leave the GPU without waiting for the rest) and the OpenAI-compatible API.
- vLLM on Kubernetes: the NVIDIA GPU Operator, deploying with KServe or Helm, and models on shared storage so they are not downloaded at every start.
- Spreading the load with llm-d: routing based on what each replica already holds in memory, splitting prompt reading from answer writing, and when it pays off over a standard load balancer.
- Sharing the GPUs: quotas and priorities with Kueue so several teams can use the same cards without stepping on each other.
- Running it: vLLM metrics in Prometheus (time to first token and tokens per second), alerts and how to test a new vLLM release before upgrading.
You work with an open model on a GPU cluster and finish with two replicas behind llm-d, a metrics dashboard and a load test comparing before and after.
What you take away
- Reusable deployment manifests for vLLM and llm-d.
- A dashboard with the model’s service metrics.
- A load-test script and the procedure for upgrading vLLM.
If you would rather we built and supported it for you, that is our AI inference service on your own servers. And if the team has not chosen a model or GPU yet, start with the local LLM and GPU infrastructure course.
Frequently asked questions
How long is the course?
3 days.
Is it online or on-site?
Both: at your offices or remotely, with a live instructor.
What do I need to know beforehand?
Confident with Kubernetes (deployments, services and Helm) and Linux. Having served a model with Ollama or vLLM helps; if not, the local LLM and GPU infrastructure course covers that ground.
Do we need our own GPUs?
Not for the course: the lab has GPUs. If you already have GPU servers, the labs can run on them.
How is it different from the local LLM course?
The local LLM course chooses the model, serves it on one server and sizes the GPU. This one starts from there and takes it to Kubernetes for many users and several teams.
Can it be tailored to my team?
Yes. We adapt the examples and labs to your team’s tools and processes, and it can be combined with other AI courses.
