Lab 12: LiteLLM¶

By Sameer Alam · Updated 31 October 2026

What You'll Learn¶

• How LiteLLM puts many model providers behind one OpenAI-style API
• How the agent loop changes in the OpenAI message format
• How to count tokens and track cost per run across providers
• What a common interface translates, and what it loses

Why LiteLLM¶

Platform teams get asked to support several model providers: Claude for one team, a local model for sensitive data, another vendor for a cost experiment. LiteLLM gives them all one interface. It comes in two forms:

This lab uses the SDK. The proxy is the same translation layer, run as a service.

Step 1: The Loop in OpenAI Format¶

Save this as litellm_agent.py, next to tools.py and agent.py from Lab 2:

"""Lab 12: one agent loop, any provider, through LiteLLM's OpenAI-style interface.

    AGENT_MODEL=anthropic/claude-opus-5-5 python litellm_agent.py
    AGENT_MODEL=ollama_chat/llama3.1 python litellm_agent.py      # a local model
"""

import json
import os
import sys

import litellm

from agent import SYSTEM
from tools import TOOLS, ask_human, run_tool

MODEL = os.environ.get("AGENT_MODEL", "anthropic/claude-opus-5-5")
MAX_STEPS = 15

# LiteLLM takes tools in OpenAI's format. The JSON Schemas from Lab 2 carry over unchanged.
OPENAI_TOOLS = [
    {"type": "function",
     "function": {"name": t["name"], "description": t["description"], "parameters": t["input_schema"]}}
    for t in TOOLS
]


def run(task):
    messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": task}]
    cost = 0.0
    for step in range(1, MAX_STEPS + 1):
        response = litellm.completion(
            model=MODEL, messages=messages, tools=OPENAI_TOOLS, max_tokens=16000,
            reasoning_effort="medium",  # becomes output_config.effort on Claude
        )
        cost += litellm.completion_cost(completion_response=response) or 0.0
        message = response.choices[0].message
        messages.append(message.model_dump(exclude_none=True))

        if not message.tool_calls:
            return message.content, step, cost

        for call in message.tool_calls:
            args = json.loads(call.function.arguments)
            print(f"[step {step}] {call.function.name} {args}")
            content, is_error = run_tool(call.function.name, args, ask_human)
            messages.append({"role": "tool", "tool_call_id": call.id,
                             "content": f"ERROR: {content}" if is_error else content})
    return "(step limit reached)", MAX_STEPS, cost


if __name__ == "__main__":
    task = " ".join(sys.argv[1:]) or "Investigate the checkout alert. What is the root cause?"
    prompt_tokens = litellm.token_counter(model=MODEL, messages=[{"role": "user", "content": task}])
    print(f"{MODEL}: task is about {prompt_tokens} tokens")
    answer, steps, cost = run(task)
    print(f"\n--- {steps} steps, ${cost:.4f} ---\n{answer}")

The loop is Lab 2's loop with OpenAI's message shapes:

LiteLLM converts these to Claude's format on the way out and back on the way in. The JSON Schemas from Lab 2 work unchanged.

Step 2: Run It on Claude¶

python litellm_agent.py

A typical run:

anthropic/claude-opus-5-5: task is about 19 tokens
[step 1] list_files {'path': '.'}
[step 2] search_files {'pattern': 'error|timed out', 'path': 'logs'}
[step 3] read_file {'path': 'deploys.json', 'start_line': 1, 'max_lines': 200}

--- 4 steps, $0.1286 ---
The checkout alert fired because release 2.14.0 cut DB_POOL_SIZE from 20 to 5...

litellm.completion_cost() prices each response from LiteLLM's model price list, so you get a dollar figure per run without your own price table. Check that list when a new model comes out. A model it doesn't know yet has no price.

Step 3: Run It on Another Provider¶

Change one environment variable:

# A local model with Ollama (no API key, nothing leaves your machine)
ollama pull llama3.1
AGENT_MODEL=ollama_chat/llama3.1 python litellm_agent.py

# Another hosted provider, if you have a key
export OPENAI_API_KEY="sk-..."
AGENT_MODEL=openai/<model-name> python litellm_agent.py

Then run each model several times and compare. Smaller local models often call tools with wrong arguments, stop early, or skip the evidence. That's exactly what the eval in Lab 7 measures. To evaluate models through LiteLLM, add Lab 6's submit_report tool and stop_on handling to this loop, return the same result dict as run_agent, and point triage.py at it.

Step 4: Know What Gets Lost¶

A common interface can only offer what every provider has in common. Some Claude features translate and some don't:

Choosing an Approach¶

What Most Engineers Miss¶

• The same prompt isn't the same agent on another model. Tool descriptions, system prompts, and step limits tuned for one model need re-tuning, and re-evaluating, on another.
• A gateway is now a critical dependency. Every model call goes through it. Pin its version, and plan for its outages like any shared service.
• The gateway sees everything. Prompts, tool results, and logs pass through it. Apply the same data rules to it as to the model providers.
• Cost tables age. Prices change and new models appear. Check what completion_cost() returns for a new model before you trust a dashboard built on it.

Common Mistakes¶

• Assuming a prompt and tool set tuned on one model will work as well on another.
• Using drop_params to make errors go away instead of fixing the config.
• Expecting provider-specific features through a lowest-common-denominator API.
• Building cost dashboards on prices that haven't been updated.
• Putting a gateway in front of every model call with no plan for when it's down.

Interview Questions¶

• Why would a platform team run an LLM gateway such as the LiteLLM proxy?
• What changes in an agent loop when you move from the Claude Messages API to the OpenAI format?
• What can go wrong when you switch an agent to a different model provider?
• How would you compare two models for the same agent task?

Next¶

You've built the whole stack: the loop, guardrails, memory, MCP, evals, injection defenses, multiple agents, A2A, and two frameworks. Return to the AI Engineering Guide, or read AI Terminology to check the vocabulary.