Skip to content

Serving LLMs on Kubernetes with vLLM

To deploy vLLM on Kubernetes, run the vllm/vllm-openai image in a Deployment that requests a GPU, cache the model weights on a volume, give the pod a startup probe long enough for the model to load, and autoscale on the request queue rather than on CPU. This page builds that setup step by step and explains each choice.

What You'll Learn

  • What a model server does that a normal web server doesn't: continuous batching, the KV cache, and streaming
  • A production-shaped vLLM Deployment: a model cache, /dev/shm, and probes for slow starts
  • Why CPU-based autoscaling fails for LLMs, and how to scale on queue depth instead
  • When to use tensor parallelism, KServe, or llm-d instead of a plain Deployment

Why This Matters

vLLM is a widely used open-source inference engine with an OpenAI-compatible API. llm-d and KServe's LLM serving build on it. Deploying it like a stateless web app goes wrong in predictable ways. Liveness probes kill pods while the model is still loading. Every restart downloads 16 GB of weights again. The HPA watches CPU, which never moves. Tensor-parallel pods crash because a container's default /dev/shm is only 64 MB.

Mental Model

One vLLM replica is one copy of the model's weights on one or more GPUs, plus a KV cache filling the rest of their memory. The replica serves many requests at once by batching them one token step at a time (continuous batching). Batching gives you throughput, and the KV cache caps how much you can batch.

flowchart LR
    Q["Waiting queue<br>vllm:num_requests_waiting"] --> PF["Prefill<br>read the prompt, fill the KV cache"]
    PF --> DC["Decode<br>one token per step, streamed"]
    DC --> D["Done<br>KV blocks freed"]
    DC -. "preempted when the KV cache is full" .-> Q

Time to first token (TTFT) is time in the queue plus prefill. The pace of the stream after that (inter-token latency) depends on how many requests share each decode step.

How It Works

Deploy a model

This serves Qwen3-8B, an Apache-2.0 model that needs no Hugging Face token, on one 24 GB GPU:

apiVersion: v1
kind: Namespace
metadata:
  name: llm
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hf-cache
  namespace: llm
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen3-8b
  namespace: llm
  labels:
    app: qwen3-8b
spec:
  replicas: 1
  strategy:
    type: Recreate                       # a ReadWriteOnce cache can't attach to two pods
  selector:
    matchLabels:
      app: qwen3-8b
  template:
    metadata:
      labels:
        app: qwen3-8b
    spec:
      terminationGracePeriodSeconds: 120 # let in-flight streams finish
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.34.0
          command: ["vllm", "serve"]
          args:
            - Qwen/Qwen3-8B
            - --max-model-len=8192
            - --gpu-memory-utilization=0.90
            - --port=8000
          env:
            - name: HF_TOKEN             # only needed for gated models
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
                  optional: true
          ports:
            - name: http
              containerPort: 8000
          resources:
            requests:
              cpu: "4"
              memory: 16Gi
              nvidia.com/gpu: "1"
            limits:
              memory: 24Gi
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: hf-cache
              mountPath: /root/.cache/huggingface
            - name: shm
              mountPath: /dev/shm
          startupProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            failureThreshold: 90         # up to 15 minutes to download and load weights
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            failureThreshold: 6
      volumes:
        - name: hf-cache
          persistentVolumeClaim:
            claimName: hf-cache
        - name: shm
          emptyDir:
            medium: Memory               # vLLM uses shared memory for tensor parallelism
            sizeLimit: 2Gi
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
---
apiVersion: v1
kind: Service
metadata:
  name: qwen3-8b
  namespace: llm
spec:
  selector:
    app: qwen3-8b
  ports:
    - name: http
      port: 8000
      targetPort: http

Why each piece is there:

  • --max-model-len=8192. Without it, vLLM uses the model's full context, 40,960 tokens for Qwen3-8B. If the KV cache can't hold one sequence that long, vLLM refuses to start. On a 24 GB GPU the cache holds roughly 25,000 tokens (see the sizing example), so set the limit to what your use case needs.
  • The cache volume keeps downloaded weights across restarts. Without it, every restart pulls 16 GB from Hugging Face.
  • strategy: Recreate. A rolling update would start a second pod that needs a second GPU and the same ReadWriteOnce volume. With several replicas, give each its own cache, or use a ReadWriteMany volume, and use a rolling update with maxSurge: 0.
  • The probes. The server doesn't answer /health until the weights are loaded, which takes minutes. The startup probe gives it that time, and liveness checks only begin after startup succeeds.

Talk to it

kubectl -n llm logs deploy/qwen3-8b -f        # wait for the server to finish starting
kubectl -n llm port-forward svc/qwen3-8b 8000:8000

curl -s localhost:8000/v1/models | jq -r '.data[].id'
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen/Qwen3-8B",
  "messages": [{"role": "user", "content": "Explain a Kubernetes readiness probe in one sentence."}],
  "max_tokens": 200,
  "chat_template_kwargs": {"enable_thinking": false}
}' | jq -r '.choices[0].message.content'

Qwen3 writes out its reasoning before answering by default. enable_thinking: false skips that for short answers. Look in the startup log for the line that reports the KV cache size in tokens, which tells you how much concurrency you really have.

Get weights onto nodes faster

The first start of a new replica is dominated by moving weights. Options, roughly from simplest to fastest:

Approach Trade-off
Download at startup into a ReadWriteOnce volume (above) Simple; each new volume pays the full download once
Shared ReadWriteMany volume (NFS, Filestore, EFS) Download once for all replicas; network file systems can load slower than local disk
Pre-pull onto local NVMe on node startup Fast loads; you build and maintain the node bootstrap
Bake weights into the container image One artifact to version; images of tens of GB slow every pull

Once weights are cached, setting HF_HUB_OFFLINE=1 stops vLLM from contacting the Hub at startup, so a Hub outage can't block a restart. Only set it after the first download has finished.

Autoscale on the queue, not on CPU

An LLM server's CPU stays low no matter how busy it is, and GPU utilization reads high even at modest load (see AI Observability). The honest signal is the queue: vllm:num_requests_waiting counts requests that arrived but can't join the batch yet.

First, let Prometheus scrape vLLM's /metrics endpoint. It's served on the same port as the API:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: vllm
  namespace: llm
spec:
  selector:
    matchLabels:
      app: qwen3-8b
  podMetricsEndpoints:
    - port: http
      path: /metrics
      interval: 15s

Then scale on queue depth with KEDA:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: qwen3-8b
  namespace: llm
spec:
  scaleTargetRef:
    name: qwen3-8b
  minReplicaCount: 1
  maxReplicaCount: 4
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 900   # a new replica takes minutes; don't flap
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-operated.monitoring.svc:9090
        query: sum(vllm:num_requests_waiting{namespace="llm", model_name="Qwen/Qwen3-8B"})
        threshold: "4"

KEDA divides the query result by the threshold to get the target replica count, so 12 waiting requests means 3 replicas. KEDA drives an HPA underneath, so the scaling behavior in Autoscaling applies. Two cautions:

  • A new replica takes minutes to become ready, so autoscaling smooths daily curves but can't absorb a sudden spike. Keep headroom in minReplicaCount for the traffic you can't afford to queue.
  • Scale-to-zero (minReplicaCount: 0) works, but the first request then waits for a GPU node, an image pull, and a weight load. Use it for batch and development, not for interactive traffic.

Your PodMonitor is only picked up if your Prometheus instance's podMonitorSelector matches it. With kube-prometheus-stack, that usually means adding the chart's release label.

More than one GPU per replica

When the weights or the needed KV cache don't fit on one GPU, split the model across several with tensor parallelism: request the GPUs and pass the same number to vLLM. Qwen3-32B has about 66 GB of weights in BF16, so it needs two 80 GB GPUs, not two 24 GB ones:

          args:
            - Qwen/Qwen3-32B
            - --tensor-parallel-size=2
          resources:
            limits:
              nvidia.com/gpu: "2"

Keep tensor parallelism inside one node, where GPUs talk over NVLink or PCIe. For models that need several nodes, use LeaderWorkerSet or llm-d's well-lit paths, covered on the next page.

Beyond a plain Deployment

Option What it is Choose it when
vLLM Deployment (this page) One engine per pod, plain Kubernetes objects You want to understand and control every piece
SGLang Another open-source engine with an OpenAI-compatible server Your models benchmark better on it
KServe LLMInferenceService CNCF incubating project; its LLM resource is built on llm-d You want a platform API that many teams use to deploy models
llm-d Kubernetes-native distributed inference stack running vLLM, SGLang, and others; CNCF Sandbox, v1.0 in October 2026 Many replicas, prefix-cache-aware routing, prefill/decode disaggregation
vLLM production stack Helm-based reference deployment with a router and dashboards A quick start with the pieces already wired together

What Most Engineers Miss

  • The KV cache is the capacity number. A replica with three 8,000-token conversations can be full while a dashboard shows "3 requests". Watch tokens and KV cache usage, not request counts.
  • Preemption is a warning sign. When the KV cache fills, vLLM pauses requests and recomputes them later. A rising vllm:num_preemptions_total means you're out of memory headroom, and it shows up as tail latency before anything errors.
  • Memory-backed emptyDir counts against the pod's memory limit. A 2 GiB /dev/shm plus the process's own memory must fit under limits.memory, or the kernel kills the container.
  • Pod termination cuts streams. When a pod is deleted, it's removed from endpoints at the same time as it receives SIGTERM. Give in-flight requests time with terminationGracePeriodSeconds, and use a preStop sleep so the gateway stops sending traffic first.
  • The OpenAI-compatible API has no authentication by default. Don't expose vLLM directly. Put it behind a gateway, or at least set --api-key.

Common Mistakes

  • A liveness probe with no startup probe, so the pod is killed in the middle of loading weights, forever.
  • Scaling on CPU or GPU utilization instead of queue depth.
  • No persistent cache, so every restart re-downloads the model.
  • Using the latest image tag, so a node replacement silently upgrades the engine.
  • Leaving --max-model-len at the model's maximum and wondering why vLLM won't start on a smaller GPU.

Check Yourself

Restart loop

New vLLM pods restart every few minutes and never become Ready. Events show Liveness probe failed: connection refused. What fixes it?

  • Raise the memory limit
  • Add a startup probe with a long failureThreshold so liveness checks wait for the model to load
  • Switch the readiness probe to a TCP check

vLLM doesn't listen until the weights are loaded. Without a startup probe, the liveness probe counts that loading time as failure and restarts the pod, which then starts loading again.

The HPA that never scales

An HPA targeting 70% CPU never adds replicas, though users wait 20 seconds for the first token at peak. What should it scale on?

  • GPU utilization from the DCGM exporter
  • Queue depth: vllm:num_requests_waiting
  • Container memory usage

LLM serving is GPU-bound and CPU stays low. GPU utilization reads high even at modest load. Waiting requests rise exactly when users start to wait.

Interview Questions

  • Walk through what happens to a chat request inside vLLM, from arrival to the last token.
  • Why does CPU-based autoscaling fail for LLM serving, and what would you scale on instead?
  • How would you keep startup time under control for a 70B model?
  • When would you choose KServe or llm-d over a plain Deployment?

Further Reading

Next

Continue to Inference Routing with the Gateway API Inference Extension to route traffic across replicas based on what each one is doing.