For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.
SDKs (Python, Node.js, Go)
Verified : Automated tests passed for: End-to-end run: Every step ran in a fresh test environment and the results were checked.Use the OpenAI SDK in Python, Node.js, or Go to call configured LLM providers through agentgateway in Kubernetes, and migrate from the LiteLLM SDK.
Call configured LLM providers from your application code through agentgateway in Kubernetes with the OpenAI SDK for Python, Node.js, or Go.
Agentgateway exposes an OpenAI-compatible API. Point the SDK’s base URL at the gateway and send OpenAI Chat Completions requests. Agentgateway forwards each request to the configured provider and translates it when the provider uses a different API, such as Anthropic, Amazon Bedrock, or Vertex AI. For per-provider API support, see About LLMs.
Before you begin
Get the gateway URL
Tip
Kind cluster? Kind does not support LoadBalancer services by default. To use this option with a Kind cluster, install and run cloud-provider-kind.
The following command reads the LoadBalancer IP address or hostname, whichever your cloud provider assigns.
export INGRESS_GW_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy \
-o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo "Gateway address: $INGRESS_GW_ADDRESS"If you use port-forwarding, keep it running and set the address in the terminal where you run the SDK examples:
export INGRESS_GW_ADDRESS=localhost:8080Set the base URL
End the SDK base URL with /v1, such as http://$INGRESS_GW_ADDRESS/v1. The SDK appends the API path, such as /chat/completions, to the base URL.
If your HTTPRoute matches a path prefix, such as /openai, put the prefix before /v1: http://$INGRESS_GW_ADDRESS/openai/v1.
Python
Install the OpenAI SDK in your Python project.
pip install openaiCreate and run the following script to send a request through agentgateway.
import os from openai import OpenAI gateway_address = os.environ["INGRESS_GW_ADDRESS"] client = OpenAI( base_url=f"http://{gateway_address}/v1", api_key="anything", # placeholder if gateway has no auth ) response = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": "Hello from Kubernetes!"}], ) print(response.choices[0].message.content)
Node.js
Install the OpenAI SDK in your Node.js project.
npm install openaiCreate and run the following script to send a request through agentgateway.
import OpenAI from "openai"; const gatewayAddress = process.env.INGRESS_GW_ADDRESS; const client = new OpenAI({ baseURL: `http://${gatewayAddress}/v1`, apiKey: "anything", }); const response = await client.chat.completions.create({ model: "gpt-4o-mini", messages: [{ role: "user", content: "Hello from Kubernetes!" }], }); console.log(response.choices[0].message.content);
Go
Add the OpenAI Go SDK to your module.
go get github.com/openai/openai-go/v3Create and run the following program to send a request through agentgateway.
package main import ( "context" "fmt" "os" "github.com/openai/openai-go/v3" "github.com/openai/openai-go/v3/option" ) func main() { client := openai.NewClient( option.WithBaseURL("http://"+os.Getenv("INGRESS_GW_ADDRESS")+"/v1"), option.WithAPIKey("anything"), ) resp, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{ Model: "gpt-4o-mini", Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Hello from Kubernetes!")}, }) if err != nil { panic(err) } fmt.Println(resp.Choices[0].Message.Content) }
Call more than one provider
Create an AgentgatewayModel for each provider with a provider prefix. The SDK code stays the same; only the model value changes.
Allow the AgentgatewayModel kind on the Gateway listener. Keep
HTTPRoutein the list to keep HTTPRoutes on other paths, such as/openai.Warning
After you allow AgentgatewayModel, the listener serves LLM paths such as
/v1/chat/completionsfrom its models instead of from HTTPRoutes. A request for a model name with no matching model, such asgpt-4o-miniwithout a prefix, returns a model-not-found error.kubectl apply -f- <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: agentgateway-proxy namespace: agentgateway-system spec: gatewayClassName: agentgateway listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All kinds: - group: gateway.networking.k8s.io kind: HTTPRoute - group: agentgateway.dev kind: AgentgatewayModel EOFSet
ANTHROPIC_API_KEYto your Anthropic API key and create a secret for it. The OpenAI secret,openai-secret, is from the prerequisites.export ANTHROPIC_API_KEY='<your-anthropic-api-key>' kubectl create secret generic anthropic-secret -n agentgateway-system \ --from-literal="Authorization=$ANTHROPIC_API_KEY"Create a model for each provider prefix.
kubectl apply -f- <<EOF apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayModel metadata: name: openai namespace: agentgateway-system spec: parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: agentgateway-proxy sectionName: http match: model: "openai/*" provider: OpenAI policies: auth: secretRef: name: openai-secret transformations: - field: model expression: 'llmRequest.model.stripPrefix("openai/")' --- apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayModel metadata: name: anthropic namespace: agentgateway-system spec: parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: agentgateway-proxy sectionName: http match: model: "anthropic/*" provider: Anthropic policies: auth: secretRef: name: anthropic-secret transformations: - field: model expression: 'llmRequest.model.stripPrefix("anthropic/")' EOFSend requests with the prefixed model names.
messages = [{"role": "user", "content": "Hello!"}] client.chat.completions.create(model="openai/gpt-4o-mini", messages=messages) client.chat.completions.create(model="anthropic/claude-sonnet-4-5", messages=messages)
Agentgateway strips the prefix and sends the request to the matching provider. For more about models, see Serve a model.
Migrate from the LiteLLM SDK
The example configuration in the previous section accepts openai/gpt-4o-mini and anthropic/claude-sonnet-4-5, so you can keep those LiteLLM model strings. Other model strings need matching gateway configuration. This example migrates Chat Completions calls. Review provider-specific parameters and other LiteLLM features separately. Replace the litellm call with the OpenAI SDK and move provider API keys from your application to Kubernetes secrets, as in Call more than one provider.
Before, with LiteLLM:
from litellm import completion
response = completion(
model="anthropic/claude-sonnet-4-5",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)After, with agentgateway:
import os
from openai import OpenAI
client = OpenAI(base_url=f"http://{os.environ['INGRESS_GW_ADDRESS']}/v1", api_key="anything")
response = client.chat.completions.create(
model="anthropic/claude-sonnet-4-5",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)For asynchronous calls, replace acompletion with AsyncOpenAI:
import asyncio
import os
from openai import AsyncOpenAI
async_client = AsyncOpenAI(base_url=f"http://{os.environ['INGRESS_GW_ADDRESS']}/v1", api_key="anything")
async def main():
response = await async_client.chat.completions.create(
model="anthropic/claude-sonnet-4-5",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
asyncio.run(main())| LiteLLM SDK | OpenAI SDK with agentgateway |
|---|---|
completion(...) | client.chat.completions.create(...) |
await acompletion(...) | await async_client.chat.completions.create(...) |
stream=True | stream=True |
embedding(...) | client.embeddings.create(...) |
num_retries=3 | OpenAI(..., max_retries=3) on the client, or a retry policy on the gateway |
Router(..., fallbacks=[...]) | A virtual model with failover routing. The client sends the virtual model name. |
completion_cost(...) | Agentgateway prices each request from its model cost catalog and records the USD cost in access logs, traces, and metrics. |
Provider keys, such as ANTHROPIC_API_KEY, in the application environment | A Kubernetes secret referenced by each AgentgatewayModel |
With LiteLLM fallbacks, the failed request moves to the next model. With agentgateway failover, the request that triggers eviction returns an error, and later requests go to the next priority group. The OpenAI Python SDK retries that error twice by default (max_retries=2), and the retry reaches a model in that group. If you set max_retries=0, configure a retry policy on the gateway instead.
Authenticate with a JWT
Configure an AgentgatewayPolicy with jwtAuthentication on the gateway. Use mode: Strict to require a valid token on every request.
Obtain a JWT access token for your gateway API from your identity provider. The token must satisfy the policy’s validation rules, including any configured issuer and audience requirements. Set AGENTGATEWAY_JWT in your application’s environment to the raw token, without the Bearer prefix.
Pass that token as the SDK API key. The SDK sends Authorization: Bearer <token>.
import os
from openai import OpenAI
jwt_token = os.environ["AGENTGATEWAY_JWT"]
client = OpenAI(
base_url=f"http://{os.environ['INGRESS_GW_ADDRESS']}/v1",
api_key=jwt_token,
)This token authenticates the caller to agentgateway. Keep the upstream provider API keys in the Kubernetes secrets referenced by your backend or model resources. Obtain a fresh token when it expires and update the SDK client before making more requests.