For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.
Nebius Token Factory
Verified : Automated tests passed for: Kubernetes resources: The Kubernetes resources on this page were checked against their API schemas.Route agentgateway LLM traffic to models served by Nebius Token Factory.
Configure Nebius Token Factory as an LLM provider in agentgateway. Nebius Token Factory serves open models through an OpenAI-compatible API.
You can connect to Nebius with an AgentgatewayModel or an AgentgatewayBackend. The Nebius model preset sends Chat Completions and Embeddings requests to Nebius. The AgentgatewayBackend approach sends Chat Completions requests only. Nebius model names include the vendor prefix, such as openai/gpt-oss-120b.
Before you begin
- Install and set up an agentgateway proxy.
- To use the AgentgatewayModel approach, allow the
AgentgatewayModelroute kind on thehttplistener of theagentgateway-proxyGateway. For the steps, see Enable LLM serving on a listener.
Set up provider credentials
Store your Nebius API key in a Secret. The model and backend examples both reference this Secret.
Get an API key from Nebius Token Factory.
Save the API key in an environment variable.
export NEBIUS_API_KEY='<your-api-key>'Create a Kubernetes Secret to store your API key.
kubectl apply -f- <<EOF apiVersion: v1 kind: Secret metadata: name: nebius-secret namespace: agentgateway-system type: Opaque stringData: Authorization: $NEBIUS_API_KEY EOF
Chat Completions and Embeddings requests with an AgentgatewayModel
The models attach directly to the Gateway listener, so you do not create an AgentgatewayBackend or an HTTPRoute.
Create the following AgentgatewayModel resources. The
nebius-chatmodel servesopenai/gpt-oss-120b, and thenebius-embeddingsmodel servesQwen/Qwen3-Embedding-8B.kubectl apply -f- <<EOF apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayModel metadata: name: nebius-chat namespace: agentgateway-system spec: parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: agentgateway-proxy sectionName: http match: model: openai/gpt-oss-120b provider: Nebius policies: auth: secretRef: name: nebius-secret --- apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayModel metadata: name: nebius-embeddings namespace: agentgateway-system spec: parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: agentgateway-proxy sectionName: http match: model: Qwen/Qwen3-Embedding-8B provider: Nebius policies: auth: secretRef: name: nebius-secret EOFReview the following table to understand this configuration. For more information, see the API docs.
Setting Description spec.match.modelThe model name that clients send in requests. The model forwards the same name to Nebius. spec.providerThe Nebiuspreset. The preset supplies the default URL,https://api.tokenfactory.nebius.com/v1, and the Chat Completions and Embeddings request formats.spec.policies.auth.secretRefThe Secret that holds your Nebius API key. Send a Chat Completions request to the
openai/gpt-oss-120bmodel.curl "http://$INGRESS_GW_ADDRESS/v1/chat/completions" \ -H 'Content-Type: application/json' \ -d '{ "model": "openai/gpt-oss-120b", "messages": [{"role": "user", "content": "Explain retrieval-augmented generation in one sentence."}] }' | jqSend an Embeddings request to the
Qwen/Qwen3-Embedding-8Bmodel.curl "http://$INGRESS_GW_ADDRESS/v1/embeddings" \ -H 'Content-Type: application/json' \ -d '{"model": "Qwen/Qwen3-Embedding-8B", "input": "Retrieval-augmented generation"}' | jq
Chat Completions requests with an AgentgatewayBackend
Use this approach to select the backend with an HTTPRoute.
Create an AgentgatewayBackend that points the
openaiprovider at Nebius.kubectl apply -f- <<EOF apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayBackend metadata: name: nebius namespace: agentgateway-system spec: ai: provider: openai: model: openai/gpt-oss-120b host: api.tokenfactory.nebius.com port: 443 path: /v1/chat/completions policies: auth: secretRef: name: nebius-secret tls: sni: api.tokenfactory.nebius.com EOFCreate an HTTPRoute that routes requests on the
/nebiuspath to the AgentgatewayBackend.kubectl apply -f- <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: nebius namespace: agentgateway-system spec: parentRefs: - name: agentgateway-proxy namespace: agentgateway-system rules: - matches: - path: type: PathPrefix value: /nebius backendRefs: - name: nebius namespace: agentgateway-system group: agentgateway.dev kind: AgentgatewayBackend EOFSend a Chat Completions request to the route.
curl "http://$INGRESS_GW_ADDRESS/nebius" \ -H 'Content-Type: application/json' \ -d '{ "model": "openai/gpt-oss-120b", "messages": [{"role": "user", "content": "Explain retrieval-augmented generation in one sentence."}] }' | jq
Next steps
To connect other providers that expose an OpenAI-compatible API, see OpenAI-compatible providers.
Cleanup
You can remove the resources that you created in this guide.kubectl delete AgentgatewayModel nebius-chat nebius-embeddings -n agentgateway-system
kubectl delete HTTPRoute nebius -n agentgateway-system
kubectl delete AgentgatewayBackend nebius -n agentgateway-system
kubectl delete secret nebius-secret -n agentgateway-system