Serving LLMs on Kubernetes with vLLM¶
To deploy vLLM on Kubernetes, run the vllm/vllm-openai image in a Deployment that requests a GPU, cache the model weights on a volume, give the pod a startup probe long enough for the model to load, and autoscale on the request queue rather than on CPU. This page builds that setup step by step and explains each choice.
What You'll Learn¶
- What a model server does that a normal web server doesn't: continuous batching, the KV cache, and streaming
- A production-shaped vLLM Deployment: a model cache,
/dev/shm, and probes for slow starts - Why CPU-based autoscaling fails for LLMs, and how to scale on queue depth instead
- When to use tensor parallelism, KServe, or llm-d instead of a plain Deployment
Why This Matters¶
vLLM is a widely used open-source inference engine with an OpenAI-compatible API. llm-d and KServe's LLM serving build on it. Deploying it like a stateless web app goes wrong in predictable ways. Liveness probes kill pods while the model is still loading. Every restart downloads 16 GB of weights again. The HPA watches CPU, which never moves. Tensor-parallel pods crash because a container's default /dev/shm is only 64 MB.
Mental Model¶
One vLLM replica is one copy of the model's weights on one or more GPUs, plus a KV cache filling the rest of their memory. The replica serves many requests at once by batching them one token step at a time (continuous batching). Batching gives you throughput, and the KV cache caps how much you can batch.
flowchart LR
Q["Waiting queue<br>vllm:num_requests_waiting"] --> PF["Prefill<br>read the prompt, fill the KV cache"]
PF --> DC["Decode<br>one token per step, streamed"]
DC --> D["Done<br>KV blocks freed"]
DC -. "preempted when the KV cache is full" .-> Q
Time to first token (TTFT) is time in the queue plus prefill. The pace of the stream after that (inter-token latency) depends on how many requests share each decode step.
How It Works¶
Deploy a model¶
This serves Qwen3-8B, an Apache-2.0 model that needs no Hugging Face token, on one 24 GB GPU:
apiVersion: v1
kind: Namespace
metadata:
name: llm
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hf-cache
namespace: llm
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: qwen3-8b
namespace: llm
labels:
app: qwen3-8b
spec:
replicas: 1
strategy:
type: Recreate # a ReadWriteOnce cache can't attach to two pods
selector:
matchLabels:
app: qwen3-8b
template:
metadata:
labels:
app: qwen3-8b
spec:
terminationGracePeriodSeconds: 120 # let in-flight streams finish
containers:
- name: vllm
image: vllm/vllm-openai:v0.34.0
command: ["vllm", "serve"]
args:
- Qwen/Qwen3-8B
- --max-model-len=8192
- --gpu-memory-utilization=0.90
- --port=8000
env:
- name: HF_TOKEN # only needed for gated models
valueFrom:
secretKeyRef:
name: hf-token
key: token
optional: true
ports:
- name: http
containerPort: 8000
resources:
requests:
cpu: "4"
memory: 16Gi
nvidia.com/gpu: "1"
limits:
memory: 24Gi
nvidia.com/gpu: "1"
volumeMounts:
- name: hf-cache
mountPath: /root/.cache/huggingface
- name: shm
mountPath: /dev/shm
startupProbe:
httpGet:
path: /health
port: http
periodSeconds: 10
failureThreshold: 90 # up to 15 minutes to download and load weights
readinessProbe:
httpGet:
path: /health
port: http
periodSeconds: 5
livenessProbe:
httpGet:
path: /health
port: http
periodSeconds: 10
failureThreshold: 6
volumes:
- name: hf-cache
persistentVolumeClaim:
claimName: hf-cache
- name: shm
emptyDir:
medium: Memory # vLLM uses shared memory for tensor parallelism
sizeLimit: 2Gi
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
---
apiVersion: v1
kind: Service
metadata:
name: qwen3-8b
namespace: llm
spec:
selector:
app: qwen3-8b
ports:
- name: http
port: 8000
targetPort: http
Why each piece is there:
--max-model-len=8192. Without it, vLLM uses the model's full context, 40,960 tokens for Qwen3-8B. If the KV cache can't hold one sequence that long, vLLM refuses to start. On a 24 GB GPU the cache holds roughly 25,000 tokens (see the sizing example), so set the limit to what your use case needs.- The cache volume keeps downloaded weights across restarts. Without it, every restart pulls 16 GB from Hugging Face.
strategy: Recreate. A rolling update would start a second pod that needs a second GPU and the sameReadWriteOncevolume. With several replicas, give each its own cache, or use aReadWriteManyvolume, and use a rolling update withmaxSurge: 0.- The probes. The server doesn't answer
/healthuntil the weights are loaded, which takes minutes. The startup probe gives it that time, and liveness checks only begin after startup succeeds.
Talk to it¶
kubectl -n llm logs deploy/qwen3-8b -f # wait for the server to finish starting
kubectl -n llm port-forward svc/qwen3-8b 8000:8000
curl -s localhost:8000/v1/models | jq -r '.data[].id'
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "Qwen/Qwen3-8B",
"messages": [{"role": "user", "content": "Explain a Kubernetes readiness probe in one sentence."}],
"max_tokens": 200,
"chat_template_kwargs": {"enable_thinking": false}
}' | jq -r '.choices[0].message.content'
Qwen3 writes out its reasoning before answering by default. enable_thinking: false skips that for short answers. Look in the startup log for the line that reports the KV cache size in tokens, which tells you how much concurrency you really have.
Get weights onto nodes faster¶
The first start of a new replica is dominated by moving weights. Options, roughly from simplest to fastest:
| Approach | Trade-off |
|---|---|
Download at startup into a ReadWriteOnce volume (above) |
Simple; each new volume pays the full download once |
Shared ReadWriteMany volume (NFS, Filestore, EFS) |
Download once for all replicas; network file systems can load slower than local disk |
| Pre-pull onto local NVMe on node startup | Fast loads; you build and maintain the node bootstrap |
| Bake weights into the container image | One artifact to version; images of tens of GB slow every pull |
Once weights are cached, setting HF_HUB_OFFLINE=1 stops vLLM from contacting the Hub at startup, so a Hub outage can't block a restart. Only set it after the first download has finished.
Autoscale on the queue, not on CPU¶
An LLM server's CPU stays low no matter how busy it is, and GPU utilization reads high even at modest load (see AI Observability). The honest signal is the queue: vllm:num_requests_waiting counts requests that arrived but can't join the batch yet.
First, let Prometheus scrape vLLM's /metrics endpoint. It's served on the same port as the API:
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: vllm
namespace: llm
spec:
selector:
matchLabels:
app: qwen3-8b
podMetricsEndpoints:
- port: http
path: /metrics
interval: 15s
Then scale on queue depth with KEDA:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: qwen3-8b
namespace: llm
spec:
scaleTargetRef:
name: qwen3-8b
minReplicaCount: 1
maxReplicaCount: 4
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 900 # a new replica takes minutes; don't flap
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus-operated.monitoring.svc:9090
query: sum(vllm:num_requests_waiting{namespace="llm", model_name="Qwen/Qwen3-8B"})
threshold: "4"
KEDA divides the query result by the threshold to get the target replica count, so 12 waiting requests means 3 replicas. KEDA drives an HPA underneath, so the scaling behavior in Autoscaling applies. Two cautions:
- A new replica takes minutes to become ready, so autoscaling smooths daily curves but can't absorb a sudden spike. Keep headroom in
minReplicaCountfor the traffic you can't afford to queue. - Scale-to-zero (
minReplicaCount: 0) works, but the first request then waits for a GPU node, an image pull, and a weight load. Use it for batch and development, not for interactive traffic.
Your PodMonitor is only picked up if your Prometheus instance's podMonitorSelector matches it. With kube-prometheus-stack, that usually means adding the chart's release label.
More than one GPU per replica¶
When the weights or the needed KV cache don't fit on one GPU, split the model across several with tensor parallelism: request the GPUs and pass the same number to vLLM. Qwen3-32B has about 66 GB of weights in BF16, so it needs two 80 GB GPUs, not two 24 GB ones:
Keep tensor parallelism inside one node, where GPUs talk over NVLink or PCIe. For models that need several nodes, use LeaderWorkerSet or llm-d's well-lit paths, covered on the next page.
Beyond a plain Deployment¶
| Option | What it is | Choose it when |
|---|---|---|
| vLLM Deployment (this page) | One engine per pod, plain Kubernetes objects | You want to understand and control every piece |
| SGLang | Another open-source engine with an OpenAI-compatible server | Your models benchmark better on it |
KServe LLMInferenceService |
CNCF incubating project; its LLM resource is built on llm-d | You want a platform API that many teams use to deploy models |
| llm-d | Kubernetes-native distributed inference stack running vLLM, SGLang, and others; CNCF Sandbox, v1.0 in October 2026 | Many replicas, prefix-cache-aware routing, prefill/decode disaggregation |
| vLLM production stack | Helm-based reference deployment with a router and dashboards | A quick start with the pieces already wired together |
What Most Engineers Miss¶
- The KV cache is the capacity number. A replica with three 8,000-token conversations can be full while a dashboard shows "3 requests". Watch tokens and KV cache usage, not request counts.
- Preemption is a warning sign. When the KV cache fills, vLLM pauses requests and recomputes them later. A rising
vllm:num_preemptions_totalmeans you're out of memory headroom, and it shows up as tail latency before anything errors. - Memory-backed
emptyDircounts against the pod's memory limit. A 2 GiB/dev/shmplus the process's own memory must fit underlimits.memory, or the kernel kills the container. - Pod termination cuts streams. When a pod is deleted, it's removed from endpoints at the same time as it receives SIGTERM. Give in-flight requests time with
terminationGracePeriodSeconds, and use apreStopsleep so the gateway stops sending traffic first. - The OpenAI-compatible API has no authentication by default. Don't expose vLLM directly. Put it behind a gateway, or at least set
--api-key.
Common Mistakes¶
- A liveness probe with no startup probe, so the pod is killed in the middle of loading weights, forever.
- Scaling on CPU or GPU utilization instead of queue depth.
- No persistent cache, so every restart re-downloads the model.
- Using the
latestimage tag, so a node replacement silently upgrades the engine. - Leaving
--max-model-lenat the model's maximum and wondering why vLLM won't start on a smaller GPU.
Check Yourself¶
Restart loop
New vLLM pods restart every few minutes and never become Ready. Events show Liveness probe failed: connection refused. What fixes it?
- Raise the memory limit
- Add a startup probe with a long
failureThresholdso liveness checks wait for the model to load - Switch the readiness probe to a TCP check
vLLM doesn't listen until the weights are loaded. Without a startup probe, the liveness probe counts that loading time as failure and restarts the pod, which then starts loading again.
The HPA that never scales
An HPA targeting 70% CPU never adds replicas, though users wait 20 seconds for the first token at peak. What should it scale on?
- GPU utilization from the DCGM exporter
- Queue depth:
vllm:num_requests_waiting - Container memory usage
LLM serving is GPU-bound and CPU stays low. GPU utilization reads high even at modest load. Waiting requests rise exactly when users start to wait.
Interview Questions¶
- Walk through what happens to a chat request inside vLLM, from arrival to the last token.
- Why does CPU-based autoscaling fail for LLM serving, and what would you scale on instead?
- How would you keep startup time under control for a 70B model?
- When would you choose KServe or llm-d over a plain Deployment?
Further Reading¶
Next¶
Continue to Inference Routing with the Gateway API Inference Extension to route traffic across replicas based on what each one is doing.