For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

OpenAI-compatible providers

Configure agentgateway for providers without built-in support that implement the OpenAI API format.

Use this page for providers that implement the OpenAI API format but do not have a first-class provider: shortcut yet. For built-in providers such as Baseten, Cerebras, Cohere, DeepInfra, DeepSeek, Fireworks AI, Groq, Hugging Face, Mistral, OpenRouter, Together AI, xAI, and Ollama, use the dedicated provider pages instead.

If you need a different upstream endpoint for one of those built-in standalone providers, keep the first-class provider: value and set params.baseUrl on that provider instead of switching to provider: openAI.

In standalone mode, configure upstream authentication per model with llm.models[].auth and upstream TLS with llm.models[].tls. For an overview of the available auth and TLS options, see Providers.

Before you begin

You also need the following prerequisites.

• An API key for your chosen provider, unless you are pointing to a local endpoint such as vLLM or LM Studio.

Managed provider fallback

Perplexity

Perplexity exposes an OpenAI-compatible API for search-augmented models and does not currently have a first-class standalone provider shortcut.

cat > /tmp/test-perplexity.yaml << 'EOF'
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
  port: 3000
  models:
  - name: "*"
    provider: openAI
    auth:
      key:
        value: "$PERPLEXITY_API_KEY"
    params:
      model: llama-3.1-sonar-large-128k-online
      baseUrl: "https://api.perplexity.ai"
    tls: {}
EOF

Self-hosted OpenAI-compatible endpoints

vLLM

vLLM is a high-performance model server for self-hosted OpenAI-compatible inference.

cat > /tmp/test-vllm.yaml << 'EOF'
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
  port: 3000
  models:
  - name: "*"
    provider: openAI
    params:
      baseUrl: "http://localhost:8000/v1"
EOF

If your vLLM server uses HTTPS, set params.baseUrl to the HTTPS endpoint and add tls: {} to the model configuration. (In agentgateway versions prior to 1.3, this model-level setting was called backendTLS.)

LM Studio

LM Studio runs models locally and exposes an OpenAI-compatible API for desktop testing.

cat > /tmp/test-lmstudio.yaml << 'EOF'
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
  port: 3000
  models:
  - name: llama-3.2-90b
    provider: openAI
    params:
      baseUrl: "http://localhost:1234/v1"
EOF

Enable the local server in LM Studio: Settings > Local Server > Start Server.

Generic configuration

Use the following template for any OpenAI-compatible provider without built-in support:

llm:
  port: 3000
  models:
  - name: "*"
    provider: openAI
    auth:
      key:
        value: "$PROVIDER_API_KEY"
    params:
      model: "<upstream-model-name>"
      baseUrl: "https://provider.example.com/v1"
    tls: {}  # only for HTTPS providers

Set params.baseUrl to the provider’s API root. This can include provider-specific prefixes such as /v1, /openai/v1, or another base path. If the provider already has a first-class page, use that provider shortcut and its documented default base URL instead.