Portkey

Portkey is an AI gateway: one endpoint in front of every model provider, with retries, fallbacks, load balancing, caching, virtual keys and request logs. deeprelay sits behind it as an OpenAI-compatible custom host. Point Portkey's openai provider at https://api.deeprelay.ai/v1, forward your deeprelay key, and every Portkey feature applies to deeprelay traffic.

This guide covers the Portkey SDKs, the OpenAI SDK through the Portkey gateway, a saved provider in Portkey's Model Catalog, and a gateway config with a fallback.

Install

pip install portkey-ai      # Python
npm i portkey-ai            # Node

You need two keys: a Portkey API key from the Portkey dashboard, and a deeprelay key.

export PORTKEY_API_KEY=...
export DEEPRELAY_API_KEY=deeprelay_live_...

How the routing works

Three settings tell Portkey where to send a request:

Setting	Value	Why
provider	openai	deeprelay speaks the OpenAI protocol, so Portkey formats the request that way.
custom_host	https://api.deeprelay.ai/v1	Where to send it. Include the /v1; Portkey appends /chat/completions or /embeddings.
Authorization + forward_headers	Bearer $DEEPRELAY_API_KEY	Your deeprelay key, passed through untouched.

The model field is the plain deeprelay id, for example deeprelay/deepseek-v4.1-flash.

Portkey SDK

Python:

import os
from portkey_ai import Portkey

portkey = Portkey(
    api_key=os.environ["PORTKEY_API_KEY"],
    provider="openai",
    custom_host="https://api.deeprelay.ai/v1",
    Authorization=f"Bearer {os.environ['DEEPRELAY_API_KEY']}",
    forward_headers=["Authorization"],
)

response = portkey.chat.completions.create(
    model="deeprelay/deepseek-v4.1-flash",
    messages=[{"role": "user", "content": "Explain a serverless inference relay in one sentence."}],
)
print(response.choices[0].message.content)

Node:

import Portkey from "portkey-ai";

const portkey = new Portkey({
  apiKey: process.env.PORTKEY_API_KEY,
  provider: "openai",
  customHost: "https://api.deeprelay.ai/v1",
  Authorization: `Bearer ${process.env.DEEPRELAY_API_KEY}`,
  forwardHeaders: ["Authorization"],
});

const response = await portkey.chat.completions.create({
  model: "deeprelay/kimi-k3",
  messages: [{ role: "user", content: "Write a haiku about cold starts." }],
  stream: true,
});

for await (const chunk of response) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

Embeddings are the same client with portkey.embeddings.create and an embedding model such as deeprelay/qwen3-embedding-8b. tools, tool_choice and response_format pass through the gateway untouched, so tool calling and JSON-schema output work exactly as they do against deeprelay directly; see the Serverless Inference guide for the shapes.

OpenAI SDK through the Portkey gateway

If your code already uses the OpenAI SDK, keep it and swap the base URL for Portkey's gateway. createHeaders builds the x-portkey-* headers; the OpenAI client's own api_key carries the deeprelay key, which Portkey forwards.

import os
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

client = OpenAI(
    api_key=os.environ["DEEPRELAY_API_KEY"],
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key=os.environ["PORTKEY_API_KEY"],
        provider="openai",
        custom_host="https://api.deeprelay.ai/v1",
        forward_headers=["Authorization"],
    ),
)

stream = client.chat.completions.create(
    model="deeprelay/deepseek-v4.1-flash",
    messages=[{"role": "user", "content": "hi"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Or with plain curl against the gateway:

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-provider: openai" \
  -H "x-portkey-custom-host: https://api.deeprelay.ai/v1" \
  -H "x-portkey-forward-headers: Authorization" \
  -H "Authorization: Bearer $DEEPRELAY_API_KEY" \
  -d '{"model":"deeprelay/deepseek-v4.1-flash","messages":[{"role":"user","content":"hi"}]}'

Save deeprelay as a provider in the Model Catalog

To keep the deeprelay key out of application code, store it in Portkey once:

1. In the Portkey dashboard open Model Catalog and choose Add Provider.
2. Turn on Local/Privately hosted provider and pick the OpenAI API specification.
3. Set Custom Host to https://api.deeprelay.ai/v1.
4. Add an Authorization header with the value Bearer deeprelay_live_....
5. Name it, for example deeprelay, and create it.

Requests then reference the saved provider and drop the per-request headers:

portkey = Portkey(api_key=os.environ["PORTKEY_API_KEY"], provider="@deeprelay")

response = portkey.chat.completions.create(
    model="deeprelay/glm-5.3-flash",
    messages=[{"role": "user", "content": "hi"}],
)

Fallbacks and retries with a gateway config

A gateway config is JSON that Portkey applies per request. This one retries a deeprelay model on 429 and 5xx, then falls back to a second deeprelay model:

{
  "retry": { "attempts": 2, "on_status_codes": [429, 500, 502, 503, 504] },
  "strategy": { "mode": "fallback" },
  "targets": [
    {
      "provider": "@deeprelay",
      "override_params": { "model": "deeprelay/deepseek-v4.1-flash" }
    },
    {
      "provider": "@deeprelay",
      "override_params": { "model": "deeprelay/glm-5.3-flash" }
    }
  ]
}

Save it in the dashboard under Configs and pass its id, or pass the object inline:

portkey = Portkey(api_key=os.environ["PORTKEY_API_KEY"], config="pc-deeprelay-fallback")

Caching, load balancing across models, and per-team virtual keys with budgets all work the same way; they are Portkey features and need nothing from deeprelay beyond the custom host.

Errors

Portkey passes deeprelay's OpenAI-shaped error envelope through, so the code field is deeprelay's: invalid_api_key, insufficient_balance, model_not_found, rate_limit_exceeded. The full list is in Conventions.

Next steps

• Vercel AI SDK for TypeScript apps calling deeprelay directly.
• LiteLLM for a self-hosted proxy instead of a hosted gateway.
• Serverless Inference guide for the request and response shapes behind the gateway.