For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

LLM (OpenAI)

Route requests to OpenAI’s chat completions API with the agentgateway binary.

Configure the agentgateway binary to route requests to the OpenAI chat completions API.

Before you begin

1. Install the agentgateway binary.

   LatestSpecific versionNightly build

   To install the latest release:

   curl -sL https://agentgateway.dev/install | bash

   To install a specific version, pass the --version flag. Use any release tag from the agentgateway releases page, such as 1.3.0.

   curl -sL https://agentgateway.dev/install | bash -s -- --version 1.3.0

   To install the nightly build for development and testing:

   1. Go to the nightly release in GitHub Actions and click the release that you want to use.

   2. From the URL, get the release number, such as 24873456345 in https://github.com/agentgateway/agentgateway/actions/runs/24873456345.

   3. Using the gh CLI, download the release for your OS. The following example uses macOS.

      gh run download 24873456345 -R agentgateway/agentgateway -n release-binary-mac

   4. Make the binary file executable and move it to your binary location, such as in the following example.

      chmod +x agentgateway
      sudo mv agentgateway /usr/local/bin/agentgateway

   5. Verify that you have the nightly release.

      agentgateway --version

      Example output:

      {
        "version": "0.0.0-alpha.813d7d0",
        "git_revision": "813d7d0ab4757db7c8ed5a639bc63c0bb20ac116",
        "rust_version": "1.95.0",
        "build_profile": "release",
        "build_target": "aarch64-apple-darwin"
      }

2. Get an OpenAI API key.

Steps

Route to an OpenAI backend through agentgateway.

Step 1: Set your API key

Store your OpenAI API key in an environment variable so agentgateway can authenticate to the API.

export OPENAI_API_KEY='<your-api-key>'

Step 2: Start agentgateway

You add the model from the UI in the next steps, so you can start agentgateway with an empty config file.

echo '{}' > config.yaml
agentgateway -f config.yaml

Example output:

info  app  serving UI at http://localhost:15000/ui

Step 3: Enable LLM

1. Open the agentgateway UI.
2. On the Gateway Overview, find the LLM row and click Enable LLM.

Step 4: Add a model

1. In the LLM section of the navigation menu, click Models, and then click Add model.
2. For the Incoming model match, enter the model name that clients send, such as gpt-3.5-turbo.
3. From the Provider list, select OpenAI.
4. For the Provider API key, click Env var and enter OPENAI_API_KEY (the variable you set in Step 1).
5. Click Save model.

Step 5: Send a chat completion request

Send a request from the command line, or try it in the built-in playground.

From another terminal, send a request to the chat completions endpoint:

curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }' | jq .

Or open the LLM playground, enter a prompt in the User message box, and click Send.

Next steps

Check out more guides related to LLM consumption with agentgateway.