Bring your own LLM

Organizations often need to integrate multiple Large Language Models (LLMs) from different providers to optimize cost, performance, or compliance. Vectara's Bring Your Own LLM (BYO-LLM) capability enables seamless integration of third-party LLMs into Vectara's AI stack, supporting OpenAI-compatible models, resposes API for reasoning models, and Google Cloud Vertex AI.

By configuring LLMs with the Create LLM API, you can enhance flexibility in how Vectara generates summaries, answers, and content, leveraging your preferred LLM infrastructure while retaining full compatibility with Vectara's powerful RAG workflows.

For example, models like GPT-5, Claude Sonnet, and Opus excel at generating code and technical content as part of your text responses. In your applications, you could use advanced models to generate code within Vectara responses, while leveraging multimodal models' image generation capabilities through separate API calls.

Discover LLM configurations

Instead of hand-configuring every field, you can let Vectara figure out the configuration for you. Send an endpoint URL and any credential the provider accepts to the Discover LLMs endpoint, and Vectara infers the provider type and API dialect, enumerates the models the endpoint exposes, and live-tests up to three models per detected endpoint. It returns candidate configurations: add a name and the credential you supplied, then submit them to Create LLM.

Discovery does not persist anything, but it is not free of side effects: it makes live generation calls to the provider to verify a candidate, which the provider may bill.

The provider type is inferred from the credential shape and the URL host, so you often only need the credential:

• AWS credentials imply anthropic (Claude on Amazon Bedrock).
• A vertex_service_account or vertex_access_token credential implies anthropic (Claude on Vertex AI); a service_account or api_key credential implies vertex-ai (Gemini).
• An api.anthropic.com host implies anthropic (direct API). A generativelanguage.googleapis.com host implies vertex-ai via Google AI Studio.
• Anything else is treated as an OpenAI-compatible host, and both the openai-compatible (chat-completions) and openai-responses (reasoning) dialects are probed.

Set the optional type field to restrict discovery to a single provider when you already know it. The uri is optional when the credential itself identifies the provider: Anthropic (sk-ant-…) and OpenAI (sk-…) API keys, Bedrock credentials (region-derived endpoint), vertex_service_account and vertex_access_token credentials (project/region-derived endpoint), and the Gemini api_key credential, whose endpoint defaults to Google AI Studio. Any other credential — including the Gemini service_account credential, which declares no project or region — needs a uri.

Request Body

DISCOVER REQUEST EXAMPLE

Code example with json syntax.json

1

cURL Example

DISCOVER CURL EXAMPLE

Code example with bash syntax.bash

1

Successful Response

Each candidate carries the type and model to submit to Create LLM, inferred capabilities, and whether a live generation call against that model succeeded (verified). A normalized uri is included except for providers whose endpoint derives from the credential (Bedrock, and Vertex without a base URI). Candidates that verified are listed first.

DISCOVER RESPONSE EXAMPLE

Code example with json syntax.json

1

Model capabilities

The capabilities object records what a model can do. If you omit it when you create an LLM, Vectara infers the capabilities from the model name and provider type; any field you set explicitly overrides the inferred default.

Field	Description
image_support	Whether the model accepts image inputs.
context_limit	Maximum context window size in tokens (minimum 4096).
tool_calling	Whether the model supports tool and function calling.
structured_outputs	Whether the model supports structured output generation.
requires_role_alternation	Whether the model requires strict role alternation in a conversation. When true, Vectara groups consecutive messages of the same role together before sending them.

Limit requests and tokens

Each LLM configuration accepts seven optional quota fields. Vectara enforces each one independently, so you can combine, for example, a requests-per-second cap with a monthly token budget. All seven fields are available on all four LLM types.

Field	Counts	Window
requests_per_second	Requests sent to the model (1-10000).	Per second
input_tokens_per_minute	Input (prompt) tokens, as reported by the provider, including cached prompt tokens.	Per minute
output_tokens_per_minute	Output tokens (completion plus reasoning).	Per minute
input_tokens_per_day	Input tokens.	Per UTC calendar day
output_tokens_per_day	Output tokens.	Per UTC calendar day
input_tokens_per_month	Input tokens.	Per UTC calendar month
output_tokens_per_month	Output tokens.	Per UTC calendar month

The semantics differ between creating and updating a configuration:

• On create, omit a field or set it to null to apply no limit.
• On update, set a value to apply that limit, set the field to null to remove the limit, or omit the field to keep the current limit.

A read of the configuration omits a field entirely when the model has no limit for it.

Day and month windows are UTC calendar boundaries. The platform checks a quota before each call and charges the call's reported usage after it, so a window may exceed its limit by at most one in-flight call per concurrent request. Once any limit is reached, whether per-second, per-minute, per-day, or per-month, the platform refuses the LLM call. How the refusal reaches the caller depends on the operation.

createAgentInput and its agent-alias equivalents createAliasRoutedInput and createEndUserAliasRoutedInput, together with createChatCompletion (POST /v2/llms/chat/completions), return HTTP 429 with a Retry-After header. The header gives the seconds until the window resets.

query, queryCorpus, createChat, and createChatTurn return HTTP 429 with no Retry-After header. The message in the response body names the exhausted quota and the seconds until its window resets.

The A2A message:send operation returns HTTP 200 with a task in state FAILED whose status message names the exhausted quota and the seconds until its window resets; the agent accepts the message before it calls the LLM, so the refusal ends the task rather than the request. message:stream also returns HTTP 200 and ends its stream with a final status update putting the task in state FAILED, but that update carries a generic failure message: read the task back with GetTask to see the exhausted quota named.

A streamed query, queryCorpus, createChat, or createChatTurn that has already begun emitting receives the refusal as an error event in the stream on an HTTP 200. A streamed agent input — createAgentInput and its alias equivalents createAliasRoutedInput and createEndUserAliasRoutedInput — receives the refusal as an error event on an HTTP 200 as well, even when nothing has been emitted yet, because its stream opens when the request is accepted. The error event's message names the exhausted quota and the seconds until its window resets. A request that is not streamed gets the 429 instead.

Define a custom LLM configuration

The integration relies on defining a custom LLM configuration with the Create LLM endpoint. Vectara supports four LLM types:

Supported LLM Types

Type	Description	Use For
openai-compatible	OpenAI-style APIs	OpenAI, Azure OpenAI, self-hosted vLLM, OpenRouter
openai-responses	OpenAI Responses API	Reasoning models (o1, o3)
vertex-ai	Google Cloud Vertex AI	Gemini models
anthropic	Anthropic Claude (direct API, Amazon Bedrock, or Vertex)	Claude models

After you enter the type, continue with the remaining configuration fields:

Configuration Fields

Field	Description
type	One of the following: openai-compatible, openai-responses, vertex-ai, or anthropic
name	User-defined label for the LLM (referenced in queries)
description	(Optional) Metadata or notes about the model
model	Specific model version (gpt-4, claude-3.5-sonnet, gemini-2.5-flash)
uri	The API endpoint URL. Optional for anthropic: it defaults to the direct Anthropic API, and Bedrock and Vertex endpoints derive from the credential's region or project
auth	Authentication configuration (varies by type)
headers	(Optional) Additional HTTP headers for the API
test_model_parameters	(Optional) Test parameters to validate the configuration

Add custom LLM examples

Here are some examples for Anthropic, OpenAI, and Google LLMs.

Add Anthropic Claude 3.7 Sonnet

Request Body

REQUEST EXAMPLE

Code example with json syntax.json

1

cURL Example

CLAUDE EXAMPLE

Code example with json syntax.json

1

Successful Response

RESPONSE EXAMPLE

Code example with json syntax.json

1

Add OpenAI GPT-4o

GPT-4O MINI EXAMPLE

Code example with json syntax.json

1

Add Google Gemini (Vertex AI)

Using API Key Authentication

GOOGLE GEMINI EXAMPLE

Code example with json syntax.json

1

Using Service Account Authentication

SERVICE ACCOUNT EXAMPLE

Code example with json syntax.json

1

Add OpenAI Reasoning Models (o1, o3)

OPENAI O1 EXAMPLE

Code example with json syntax.json

1

Verify your configuration

To confirm your model was added successfully:

VERIFICATION EXAMPLE

Code example with bash syntax.bash

1

Look for your model in the response JSON and verify it has "enabled": true.

The response also echoes back the auth variant the model was created with, so you can confirm you configured the credential you intended. Every secret value is replaced by the literal string ****. The non-secret identifiers around it are preserved — header names, regions, AWS access key IDs, and the GCP project and region — so you can recognize the configured credential without exposing it.

To rotate a credential, submit a new auth object with a PATCH. Omit auth to keep the existing credentials.

Querying with a custom LLM

After you register a third-party LLM using the /v2/llms endpoint, you do not reference it directly by ID in your query. Instead, you associate the custom LLM with a generation preset, and then use that preset in your query with generation_preset_name.

However, when defining custom model_parameters, you can override the default preset and explicitly specify the registered model by name.

Example query

QUERY CUSTOM LLM EXAMPLE

Code example with json syntax.json

1

Add Openrouter models

Claude Sonnet 4.5

CLAUDE SONNET 4.5 EXAMPLE

Code example with bash syntax.bash

1

GLM 4.6

GLM 4.6 EXAMPLE

Code example with bash syntax.bash

1

Minimax 2

MINIMAX 2 EXAMPLE

Code example with bash syntax.bash

1

xAI Grok Code Fast 1

GROK CODE FAST 1 EXAMPLE

Code example with bash syntax.bash

1