On-premise AI · Guide

vLLM vs OllamaWhich one to choose depending on how many people use it at once and what changes when you move from one to the other.

To try a model on your laptop or serve one person, use Ollama. If several people will use it at once on a server GPU, use vLLM. Many teams use both, one for testing and the other for production. The difference lies in how each one shares the card when several requests arrive together.

What each one does

With Ollama, a model is running in five minutes. You install it, type ollama run with the name of the model and it starts answering, with the weights, the files that make up the model, already reduced to fit on a normal machine and nothing to choose. It works on laptops, on workstations and without a GPU, although more slowly.

vLLM is an inference server, the program that runs the model and answers requests, built to get as much as possible out of each GPU. It groups the requests that arrive at the same time and, rather than waiting for a batch to finish, slots in new requests as soon as another one ends. That is what lets it serve many people with the same card, but it asks more of you: you have to choose the weights, tune the parameters and work out how much memory each model needs.

Both offer an OpenAI-compatible API, so your application code barely changes: you can develop against Ollama and deploy against vLLM by changing little more than an address. If the model invokes functions in your application, known as tool calling, you will need to enable it when you start vLLM.

What happens when several people use it at once

One of the most cited benchmarks is Red Hat’s, with an NVIDIA A100, Llama 3.1 8B (8 billion parameters) and up to 256 simultaneous users. vLLM peaked at 793 tokens per second against 41 for Ollama, which could not catch up even when tuned to handle more requests in parallel. On Ollama, the time to the first word also rose sharply because requests were queuing.

That shows up on the bill: if the same card serves more people at once, you need fewer cards for the same service, and GPUs are the most expensive part of the machine.

Those figures are from 2025 versions and from a load pushed to the limit, and other comparisons find considerably smaller gaps because they use other cards, other models and other quantisation, which is how far the weights are reduced. The direction is the same across published comparisons: the more simultaneous users, the further vLLM pulls ahead.

If you are staying with Ollama for a small team, check one setting. According to its documentation, OLLAMA_NUM_PARALLEL handles a single request per model by default, and if you raise it, memory grows in proportion, because each parallel request reserves its own context. That value has changed between versions, so check yours. If Ollama feels slow with several users, start here: they may be waiting their turn.

When to use which

Simultaneous users matter more than the number of people with access: a hundred employees who ask something now and then may be three requests at once. Where the limit lies depends on the model, the card and how long the questions are, so the reliable way is to measure your peak hour, and that is the first thing we do before recommending a machine.

SituationOllamavLLM

Trying a modelBest option: running in minutesToo much work for this

One person, no server GPUYesOnly if you already have it set up

Small team, few requests at onceFine, if you tune parallelismFine, and it will cope as load grows

Several people at once in productionQueues build upYes, that is what it is built for

Getting more from an expensive GPUNot its goalYes

Once you have vLLM and the load no longer fits on one machine, a common next step is Kubernetes with llm-d, which shares requests across replicas.

If your machines are IBM Power, you do not need to buy others to get started: vLLM runs on Power without a GPU, which lets you try models on the hardware you already have.

What changes when you move from Ollama to vLLM

In the application, almost nothing. You change the port, which is usually 11434 on Ollama and 8000 on vLLM, and the name of the model, because Ollama uses its own tags and vLLM the Hugging Face name, where this model requires you to accept its licence and use an access token:

from openai import OpenAI

# Ollama
client = OpenAI(
  base_url="http://localhost:11434/v1",
  api_key="none",
)
model = "llama3.1:8b"

# vLLM
client = OpenAI(
  base_url="http://localhost:8000/v1",
  api_key="none",
)
model = (
  "meta-llama/Llama-3.1-8B-Instruct"
)

The real change is underneath. Ollama’s models come in GGUF format, which vLLM supports only on an experimental basis, so in production you have to download the weights in the format it expects. vLLM also works with quantised weights, but depending on which you pick the same model may take up more memory, which is why it pays to work out your memory and user numbers before buying anything.

The way the card is used changes too. By default vLLM reserves 92% of GPU memory at startup, whether or not it has work, and that catches people out when they share the machine with other processes. If you are going to split it between several teams, you need GPU sharing on Kubernetes.

And before you open the service to the network, put a lock on it. Ollama listens only on 127.0.0.1 by default and has no authentication, so to share it you have to put a proxy in front that requires it. vLLM accepts keys with --api-key, although they only protect the API paths, so a real service should also have an API gateway with per-user access.

If you want us to set it up, we serve models with vLLM on your own infrastructure, and if your team would rather run it, we have the vLLM and llm-d on Kubernetes course.

Frequently asked questions

Is vLLM or Ollama better?

It depends on the load. Ollama is better for trying models and for one person because it installs in minutes. vLLM is better when several people use the model at once on a server GPU.

Does vLLM need a GPU?

It is designed for GPUs, and that is where it gets its advantage, but it can also run without one, more slowly, on x86, Arm or IBM Power.

Can I use Ollama models in vLLM?

For production, no. Ollama uses GGUF and vLLM works with weights in the Hugging Face format. It has experimental GGUF support, but the normal route is to download the model again.

How many users can Ollama handle?

By default it handles one request per model and queues the rest. You can raise that with OLLAMA_NUM_PARALLEL at the cost of more memory, which works for a small team, but it does not scale like vLLM.

Does my application change when moving from Ollama to vLLM?

Very little. Both offer an API compatible with OpenAI’s, so changing the address and the model name is enough, plus enabling tool calling in vLLM if you use it.

AI inference

How many people will use your model at once?

Tell us which model you want to serve and how many people will use it at once, and we will tell you which engine and which machine you need. If you wish, we can also handle deployment and support.

Tell us what you need WhatsApp

Inference with vLLM on your infrastructure