Control
LLM Gateway
Your applications connect to BaseLLM's OpenAI-compatible endpoint instead of calling the model provider directly. Provider keys, fallback models and usage records are managed in one place.
Connection model
Your application sends requests to the OpenAI-compatible chat endpoint (POST /api/v1/chat/completions) with a project API key. The key identifies the project, not the person. Provider keys are stored in BaseLLM, encrypted with AES-256-GCM, not in your application.
from openai import OpenAI
client = OpenAI(
base_url="<basellm-endpoint>/api/v1",
api_key="sk-ffm-...", # project API key
)
resp = client.chat.completions.create(
model="<model-name>",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"X-FFM-User": "u-12345"},
)
Tool calling (tools, tool_choice, parallel_tool_calls) is passed through to the provider. You turn it on per project with tools_enabled; it is off by default.
Providers and usage records
| Provider | Note |
|---|---|
| OpenAI | Cached token counting |
| Anthropic (Claude) | BaseLLM sets the cache marker |
| Google (Gemini) | Cached token counting |
| OpenRouter | Provider route, e.g. Claude Haiku 4.5 |
| On-premise model | Pricing and cost records; vLLM cache counting |
At the end of every request to a model, the gateway writes a usage record: timestamp, project, model, input and output tokens, duration (latency_ms), cost and end-user ID. Conversation content is not stored in this record.
Fallback model
If the provider returns an error or doesn't respond, the request can move to the project's next model. This is off by default; an admin turns it on per project. One setting covers both chat and decision models.
| Condition | Falls back? |
|---|---|
| No response within the set time | Yes |
| Provider unreachable | Yes |
| "Too many requests" (429) | Yes |
| Provider failure (5xx, overloaded) | Yes |
| API key rejected, model not found | Yes |
| The request itself is invalid | No |
- For streaming responses, fallback happens only before the first token is sent.
- Every fallback is written to the decision list and the audit log.
- Requests can also move to the next model when a model limit is reached (Policy Engine).
- Requests do not fall back when the project's monthly budget is used up.
Added latency
| Metric | Value |
|---|---|
| Median added time (p50) | 0.29–0.30 ms |
| Slowest 1% (p99) | 1.73–1.76 ms |
| Slowest 0.1% (p99.9) | ~3 ms |
API keys are checked with SHA-256 and settings are held in memory, so no database query is needed per request. Setting changes apply at once; with more than one server they reach the others through Redis (within 5 s at most without Redis). Usage and budget counters update with a delay of up to 10 s.
Routing
The model changes automatically in three cases: when a model limit is reached; when fallback is on and the provider fails or the circuit breaker holds a failing model back; and when smart routing is in on mode and sends a simple question to a cheaper model. With limit routing, the request completes with 200 and the model field in the response shows the model that was used.