Skip to content
Help guide the future of agentgateway — Take the survey

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Nebius Token Factory

Verified : Automated tests passed for: Kubernetes resources: The Kubernetes resources on this page were checked against their API schemas.
Page as Markdown

Route agentgateway LLM traffic to models served by Nebius Token Factory.

Configure Nebius Token Factory as an LLM provider in agentgateway. Nebius Token Factory serves open models through an OpenAI-compatible API.

You can connect to Nebius with an AgentgatewayModel or an AgentgatewayBackend. The Nebius model preset sends Chat Completions and Embeddings requests to Nebius. The AgentgatewayBackend approach sends Chat Completions requests only. Nebius model names include the vendor prefix, such as openai/gpt-oss-120b.

Before you begin

  1. Install and set up an agentgateway proxy.
  2. To use the AgentgatewayModel approach, allow the AgentgatewayModel route kind on the http listener of the agentgateway-proxy Gateway. For the steps, see Enable LLM serving on a listener.

Set up provider credentials

Store your Nebius API key in a Secret. The model and backend examples both reference this Secret.

  1. Get an API key from Nebius Token Factory.

  2. Save the API key in an environment variable.

    export NEBIUS_API_KEY='<your-api-key>'
  3. Create a Kubernetes Secret to store your API key.

    kubectl apply -f- <<EOF
    apiVersion: v1
    kind: Secret
    metadata:
      name: nebius-secret
      namespace: agentgateway-system
    type: Opaque
    stringData:
      Authorization: $NEBIUS_API_KEY
    EOF

Chat Completions and Embeddings requests with an AgentgatewayModel

The models attach directly to the Gateway listener, so you do not create an AgentgatewayBackend or an HTTPRoute.

  1. Create the following AgentgatewayModel resources. The nebius-chat model serves openai/gpt-oss-120b, and the nebius-embeddings model serves Qwen/Qwen3-Embedding-8B.

    kubectl apply -f- <<EOF
    apiVersion: agentgateway.dev/v1alpha1
    kind: AgentgatewayModel
    metadata:
      name: nebius-chat
      namespace: agentgateway-system
    spec:
      parentRefs:
      - group: gateway.networking.k8s.io
        kind: Gateway
        name: agentgateway-proxy
        sectionName: http
      match:
        model: openai/gpt-oss-120b
      provider: Nebius
      policies:
        auth:
          secretRef:
            name: nebius-secret
    ---
    apiVersion: agentgateway.dev/v1alpha1
    kind: AgentgatewayModel
    metadata:
      name: nebius-embeddings
      namespace: agentgateway-system
    spec:
      parentRefs:
      - group: gateway.networking.k8s.io
        kind: Gateway
        name: agentgateway-proxy
        sectionName: http
      match:
        model: Qwen/Qwen3-Embedding-8B
      provider: Nebius
      policies:
        auth:
          secretRef:
            name: nebius-secret
    EOF

    Review the following table to understand this configuration. For more information, see the API docs.

    SettingDescription
    spec.match.modelThe model name that clients send in requests. The model forwards the same name to Nebius.
    spec.providerThe Nebius preset. The preset supplies the default URL, https://api.tokenfactory.nebius.com/v1, and the Chat Completions and Embeddings request formats.
    spec.policies.auth.secretRefThe Secret that holds your Nebius API key.
  2. Send a Chat Completions request to the openai/gpt-oss-120b model.

    curl "http://$INGRESS_GW_ADDRESS/v1/chat/completions" \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "openai/gpt-oss-120b",
        "messages": [{"role": "user", "content": "Explain retrieval-augmented generation in one sentence."}]
      }' | jq
  3. Send an Embeddings request to the Qwen/Qwen3-Embedding-8B model.

    curl "http://$INGRESS_GW_ADDRESS/v1/embeddings" \
      -H 'Content-Type: application/json' \
      -d '{"model": "Qwen/Qwen3-Embedding-8B", "input": "Retrieval-augmented generation"}' | jq

Chat Completions requests with an AgentgatewayBackend

Use this approach to select the backend with an HTTPRoute.

  1. Create an AgentgatewayBackend that points the openai provider at Nebius.

    kubectl apply -f- <<EOF
    apiVersion: agentgateway.dev/v1alpha1
    kind: AgentgatewayBackend
    metadata:
      name: nebius
      namespace: agentgateway-system
    spec:
      ai:
        provider:
          openai:
            model: openai/gpt-oss-120b
          host: api.tokenfactory.nebius.com
          port: 443
          path: /v1/chat/completions
      policies:
        auth:
          secretRef:
            name: nebius-secret
        tls:
          sni: api.tokenfactory.nebius.com
    EOF
  2. Create an HTTPRoute that routes requests on the /nebius path to the AgentgatewayBackend.

    kubectl apply -f- <<EOF
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
      name: nebius
      namespace: agentgateway-system
    spec:
      parentRefs:
        - name: agentgateway-proxy
          namespace: agentgateway-system
      rules:
      - matches:
        - path:
            type: PathPrefix
            value: /nebius
        backendRefs:
        - name: nebius
          namespace: agentgateway-system
          group: agentgateway.dev
          kind: AgentgatewayBackend
    EOF
  3. Send a Chat Completions request to the route.

    curl "http://$INGRESS_GW_ADDRESS/nebius" \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "openai/gpt-oss-120b",
        "messages": [{"role": "user", "content": "Explain retrieval-augmented generation in one sentence."}]
      }' | jq

Next steps

To connect other providers that expose an OpenAI-compatible API, see OpenAI-compatible providers.

Cleanup

You can remove the resources that you created in this guide.
kubectl delete AgentgatewayModel nebius-chat nebius-embeddings -n agentgateway-system
kubectl delete HTTPRoute nebius -n agentgateway-system
kubectl delete AgentgatewayBackend nebius -n agentgateway-system
kubectl delete secret nebius-secret -n agentgateway-system
Was this page helpful?
Agentgateway assistant

Ask me anything about agentgateway configuration, features, or usage.

Note: AI-generated content might contain errors; please verify and test all returned information.

Tip: one topic per conversation gives the best results. Use the + button in the chat header to start a new conversation.

Switching topics? Starting a new conversation improves accuracy.
↑↓ navigate ↵ select esc dismiss

What could be improved?

Your feedback helps us improve assistant answers and identify docs gaps we should fix.

Need more help? Join us on Discord: https://discord.gg/y9efgEmppm

Want to use your own agent? Add the Solo MCP server to query our docs directly. Get started here: https://search.solo.io/.