Control

LLM Gateway

Your applications connect to BaseLLM's OpenAI-compatible endpoint instead of calling the model provider directly. Provider keys, fallback models and usage records are managed in one place.

Connection model

Your application sends requests to the OpenAI-compatible chat endpoint (POST /api/v1/chat/completions) with a project API key. The key identifies the project, not the person. Provider keys are stored in BaseLLM, encrypted with AES-256-GCM, not in your application.

from openai import OpenAI
client = OpenAI(
    base_url="<basellm-endpoint>/api/v1",
    api_key="sk-ffm-...",  # project API key
)
resp = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Hello"}],
    extra_headers={"X-FFM-User": "u-12345"},
)

Tool calling (tools, tool_choice, parallel_tool_calls) is passed through to the provider. You turn it on per project with tools_enabled; it is off by default.

Providers and usage records

ProviderNote
OpenAICached token counting
Anthropic (Claude)BaseLLM sets the cache marker
Google (Gemini)Cached token counting
OpenRouterProvider route, e.g. Claude Haiku 4.5
On-premise modelPricing and cost records; vLLM cache counting

At the end of every request to a model, the gateway writes a usage record: timestamp, project, model, input and output tokens, duration (latency_ms), cost and end-user ID. Conversation content is not stored in this record.

Fallback model

If the provider returns an error or doesn't respond, the request can move to the project's next model. This is off by default; an admin turns it on per project. One setting covers both chat and decision models.

ConditionFalls back?
No response within the set timeYes
Provider unreachableYes
"Too many requests" (429)Yes
Provider failure (5xx, overloaded)Yes
API key rejected, model not foundYes
The request itself is invalidNo
  • For streaming responses, fallback happens only before the first token is sent.
  • Every fallback is written to the decision list and the audit log.
  • Requests can also move to the next model when a model limit is reached (Policy Engine).
  • Requests do not fall back when the project's monthly budget is used up.

Added latency

MetricValue
Median added time (p50)0.29–0.30 ms
Slowest 1% (p99)1.73–1.76 ms
Slowest 0.1% (p99.9)~3 ms

API keys are checked with SHA-256 and settings are held in memory, so no database query is needed per request. Setting changes apply at once; with more than one server they reach the others through Redis (within 5 s at most without Redis). Usage and budget counters update with a delay of up to 10 s.

Routing

The model changes automatically in three cases: when a model limit is reached; when fallback is on and the provider fails or the circuit breaker holds a failing model back; and when smart routing is in on mode and sends a simple question to a cheaper model. With limit routing, the request completes with 200 and the model field in the response shows the model that was used.