CH·07.2Integrations

LiteLLM

Call deeprelay from the LiteLLM Python SDK or serve it behind the LiteLLM proxy with aliases, fallbacks, and a master key.

LiteLLM gives Python code one completion() call for every model provider, and ships a proxy server that exposes the same thing over an OpenAI-compatible endpoint with keys, budgets and logging. deeprelay works with both through LiteLLM's openai/ provider prefix: the prefix tells LiteLLM to speak the OpenAI protocol, and api_base points it at deeprelay.

This guide covers the Python SDK (sync, streaming, tool calling, JSON output, embeddings), the proxy server with a config.yaml, the router with fallbacks, and how to lock the proxy down.

Install

pip install litellm            # SDK only
pip install "litellm[proxy]"   # SDK plus the proxy server

Put your key in the environment. Every example reads it from there.

export DEEPRELAY_API_KEY=deeprelay_live_...

Model names

A LiteLLM model name is /. For deeprelay the provider is openai and the model is the id from GET /v1/models, so the full name has two slashes:

LiteLLM `model`	deeprelay model
`openai/deeprelay/deepseek-v4.1-flash`	`deeprelay/deepseek-v4.1-flash`
`openai/deeprelay/kimi-k3`	`deeprelay/kimi-k3`
`openai/deeprelay/qwen3-embedding-8b`	`deeprelay/qwen3-embedding-8b` (embeddings)

LiteLLM strips the first segment and sends the rest as the model field, so deeprelay receives exactly the id it expects. api_base must be https://api.deeprelay.ai/v1, including the /v1; LiteLLM appends /chat/completions and /embeddings itself.

Python SDK

import os
import litellm

response = litellm.completion(
    model="openai/deeprelay/deepseek-v4.1-flash",
    api_base="https://api.deeprelay.ai/v1",
    api_key=os.environ["DEEPRELAY_API_KEY"],
    messages=[{"role": "user", "content": "Explain a serverless inference relay in one sentence."}],
)
print(response.choices[0].message.content)
print(response.usage)

Streaming is the same call with stream=True:

stream = litellm.completion(
    model="openai/deeprelay/kimi-k3",
    api_base="https://api.deeprelay.ai/v1",
    api_key=os.environ["DEEPRELAY_API_KEY"],
    messages=[{"role": "user", "content": "Write a haiku about cold starts."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Embeddings use litellm.embedding with an embedding model id:

emb = litellm.embedding(
    model="openai/deeprelay/qwen3-embedding-8b",
    api_base="https://api.deeprelay.ai/v1",
    api_key=os.environ["DEEPRELAY_API_KEY"],
    input=["sunny day at the beach", "rainy afternoon in the city"],
)
print(len(emb.data[0]["embedding"]))

To stop repeating api_base and api_key, set them once:

litellm.api_base = "https://api.deeprelay.ai/v1"
litellm.api_key = os.environ["DEEPRELAY_API_KEY"]

Tool calling

tools and tool_choice pass through unchanged. The model answers with tool_calls; run the function, append a tool message with the result, and call again to close the loop.

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
messages = [{"role": "user", "content": "What is the weather in Paris? Use the tool."}]

first = litellm.completion(
    model="openai/deeprelay/deepseek-v4.1-flash",
    api_base="https://api.deeprelay.ai/v1",
    api_key=os.environ["DEEPRELAY_API_KEY"],
    messages=messages,
    tools=tools,
    tool_choice="auto",
)
call = first.choices[0].message.tool_calls[0]
print(call.function.name, call.function.arguments)  # get_weather {"city": "Paris"}

messages += [
    first.choices[0].message,
    {"role": "tool", "tool_call_id": call.id, "content": '{"temp_c": 18, "sky": "overcast"}'},
]
second = litellm.completion(
    model="openai/deeprelay/deepseek-v4.1-flash",
    api_base="https://api.deeprelay.ai/v1",
    api_key=os.environ["DEEPRELAY_API_KEY"],
    messages=messages,
    tools=tools,
)
print(second.choices[0].message.content)

Every chat model with tools in its supported_parameters accepts this. The Serverless Inference guide covers streaming tool calls and the tool_choice modes each model allows.

JSON output

response_format passes through as well. A JSON schema is enforced on the model side:

out = litellm.completion(
    model="openai/deeprelay/deepseek-v4.1-flash",
    api_base="https://api.deeprelay.ai/v1",
    api_key=os.environ["DEEPRELAY_API_KEY"],
    messages=[{"role": "user", "content": "A recipe name and three ingredients for pancakes."}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "recipe",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "ingredients": {"type": "array", "items": {"type": "string"}},
                },
                "required": ["name", "ingredients"],
                "additionalProperties": False,
            },
        },
    },
)
print(out.choices[0].message.content)  # {"name": "Classic Pancakes", "ingredients": [...]}

{"type": "json_object"} works too; as with OpenAI, the prompt has to mention JSON for that mode.

Proxy server

The proxy turns deeprelay models into short aliases behind one local endpoint, so any OpenAI client in your stack can use them without knowing about deeprelay at all. Define the aliases in config.yaml:

# config.yaml
model_list:
  - model_name: deepseek-flash
    litellm_params:
      model: openai/deeprelay/deepseek-v4.1-flash
      api_base: https://api.deeprelay.ai/v1
      api_key: os.environ/DEEPRELAY_API_KEY

  - model_name: kimi
    litellm_params:
      model: openai/deeprelay/kimi-k3
      api_base: https://api.deeprelay.ai/v1
      api_key: os.environ/DEEPRELAY_API_KEY

  - model_name: embed
    litellm_params:
      model: openai/deeprelay/qwen3-embedding-8b
      api_base: https://api.deeprelay.ai/v1
      api_key: os.environ/DEEPRELAY_API_KEY

os.environ/DEEPRELAY_API_KEY reads the key from the proxy's environment, so the file is safe to commit. Start the proxy:

litellm --config config.yaml --port 4000 --host 127.0.0.1

Binding to 127.0.0.1 keeps it reachable only from your machine. In a second terminal, check the aliases are served and make a call:

curl http://127.0.0.1:4000/v1/models

curl http://127.0.0.1:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-flash","messages":[{"role":"user","content":"hi"}]}'

Any OpenAI SDK now works against the proxy with the alias as the model:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:4000/v1", api_key="anything")
out = client.chat.completions.create(
    model="kimi",
    messages=[{"role": "user", "content": "hi"}],
    stream=True,
)
for chunk in out:
    print(chunk.choices[0].delta.content or "", end="")

Fallbacks with the router

The LiteLLM Router retries across deployments. Two deeprelay models with one as the fallback gives you a second model when the first is rate limited or drops out of the live catalog:

import os
from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "chat",
            "litellm_params": {
                "model": "openai/deeprelay/deepseek-v4.1-flash",
                "api_base": "https://api.deeprelay.ai/v1",
                "api_key": os.environ["DEEPRELAY_API_KEY"],
            },
        },
        {
            "model_name": "chat-fallback",
            "litellm_params": {
                "model": "openai/deeprelay/glm-5.3-flash",
                "api_base": "https://api.deeprelay.ai/v1",
                "api_key": os.environ["DEEPRELAY_API_KEY"],
            },
        },
    ],
    fallbacks=[{"chat": ["chat-fallback"]}],
    num_retries=2,
)

response = router.completion(
    model="chat",
    messages=[{"role": "user", "content": "hi"}],
)

The same fallbacks block goes under router_settings: in the proxy's config.yaml.

Secure the proxy

Before the proxy leaves your machine, set a master key. Without one, anyone who can reach the port spends your deeprelay credit.

# config.yaml
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

export LITELLM_MASTER_KEY=sk-$(openssl rand -hex 16)
litellm --config config.yaml --port 4000 --host 127.0.0.1

Clients then send the master key as their bearer token. Per-user keys with their own budgets are LiteLLM virtual keys, which need the proxy's Postgres database; see the LiteLLM docs for that setup.

Errors

deeprelay errors arrive as the matching LiteLLM exception types: AuthenticationError for a bad key, NotFoundError when a model has left the live catalog, RateLimitError on a 429, and BadRequestError for a request shape deeprelay rejects. The response body keeps deeprelay's code, for example insufficient_balance when credit runs out. The full list is in Conventions.

Next steps

Vercel AI SDK for the TypeScript side.
Portkey if you want a hosted gateway instead of running the proxy yourself.
Serverless Inference guide for what is on the wire.
Claude Code and Coding agents if the goal is an agent on deeprelay models: both talk to deeprelay directly, no proxy needed.

← All integrations