We set up Kubernetes to share your GPUs across teams.

Production, data science and training work on the same cluster, on your own servers, and workloads that must not interfere with each other are kept apart in hardware.

Tell us which GPUs you have How we set it up ↓

With one GPU per team, some run flat out and others sit idle.

The production card can't keep up, the data scientists' card spends the afternoon with a notebook open and nothing running, and the training job that arrives on Thursday has nowhere to go.

Buying another card clears the queue for a while, but the ones you already have are still half idle and the next purchase comes sooner than planned.

Ways to share a GPU

A single cluster usually combines them. Which one fits each case depends on whether workloads can get in each other's way and how much memory each one needs.

GPU · 96 GB

Time-slicing

Several processes use the whole card and take turns to compute. It works on almost any NVIDIA GPU, but memory is shared and there is no isolation: if one process hogs it or the card fails, they all go down. It suits notebooks and tests, which spend a lot of time idle.

GPU · 96 GB

24 GB24 GB24 GB24 GB

Partitioned with MIG

The card is split in hardware into smaller GPUs, each with its own memory and cores, so a fault in one doesn't affect the others. It is the right choice for serving several small models in production.

Server · 4 GPUs

training · waiting

Whole GPUs, queued

Each job gets complete GPUs and waits its turn in its team's queue. That is where training and fine-tuning fit, and they can run overnight without taking room from production.

Production
Data
Testing and training

Example: a server with four GPUs

Three teams share the four cards over a working day.

08:00
The production model, served with vLLM, has GPU 1 to itself from start-up.
whole
09:30
Three small models (embeddings, a classifier and speech-to-text) share GPU 2, partitioned with MIG.
MIG
10:00
Five data science notebooks take turns on GPU 3.
time-sliced
13:00
A fine-tuning job asks for two GPUs. Only one is free, so it waits in its team's queue.
queued
19:00
The notebooks close and the fine-tuning job borrows GPU 3 alongside GPU 4.
borrowed
05:10
The job finishes and GPU 3 goes back to the data team before anyone arrives.
returned

How we set it up

Drivers with the GPU Operator It installs and maintains the NVIDIA drivers on every node, configures MIG partitions and time-slicing, and makes the cards visible to Kubernetes. Underneath runs Ubuntu or Rocky Linux, supported by Canonical or CIQ.

Requesting GPUs with DRA With DRA, built into Kubernetes since version 1.34, and NVIDIA's DRA driver, a workload asks for “a GPU with at least 40 GB” and the cluster finds one that fits.

A queue per team with Kueue Each team has its quota. What one team doesn't use another can borrow, and production has priority to reclaim its cards as soon as it needs them.

Measuring usage With NVIDIA's DCGM exporter, Prometheus and Grafana you can see how much each team uses, which is the figure you need to adjust quotas or decide whether another card is needed.

Servers and training

If you don't have the cards yet, we configure Lenovo AI servers and Supermicro AI servers with the GPUs that fit the plan. And so your team can run the cluster, there are courses on Kubernetes with MicroK8s, Kubernetes GPU sharing (2 days) and vLLM and llm-d on Kubernetes (3 days).

Frequently asked questions

Which GPUs support MIG?

The A100, A30, H100, H200, B200 and B300, plus the RTX PRO 6000 Blackwell. The L4 and L40S do not; on those cards you share by time-slicing or by whole GPUs.

Do you need Kubernetes for this?

For one team and one model, no: vLLM on a single server is enough. Kubernetes pays off when several teams or models share the same machines and someone has to decide who uses what and when. AI inference on your own servers

We already have the GPUs. Where do we start?

With the usage data from the cards you already have, we propose how to share them, and we only talk about adding more if that is not enough for everyone.

Contact

How many GPUs do you have and who uses them?

Tell us which cards you have or plan to buy and which teams will use them. We will suggest how to share them and what needs setting up.