Models & Routing

Every LLM call from the runtime is brokered through the control plane, which routes it to a vLLM backend. The runtime never holds LLM API keys and never talks to LLM providers directly.

The request path

runtime → POST /api/runtime/chat  (JWT auth)
              ↓
        control plane resolves the model profile (+ tags)
              ↓
        fetches metrics from each vLLM backend, scores & selects one
              ↓
        vLLM backend (direct HTTP) → SSE stream back to the runtime

Model profiles

A model profile describes what the control plane can route to. Each profile has an id (e.g. llm, llm-smart, llm-image), a base model name, optional tags (used for backend filtering), and capability flags such as supportsTools and supportsImages.

The runtime discovers the catalog via GET /api/runtime/llms. Example response:

{
  "projectId": "proj-123",
  "defaultLlmId": "llm",
  "llms": [
    {
      "id": "llm-smart",
      "name": "Smart Model",
      "enabled": true,
      "supportsTools": true,
      "supportsImages": false,
      "contextWindowTokens": 130000,
      "maxOutputTokens": 16384,
      "tags": ["smart"]
    }
  ],
  "refreshedAt": 1718567890000
}

A conversation selects a profile via modelProfileId (aliases llmId and provider are also accepted on create/fork/patch).

Backend selection strategy

The control plane maintains a list of vLLM backends (managed by admins under Admin → vLLM Backends, seeded from config/vllm-backends.json). For each request it:

1. Filters by tags — request tags must match backend tags (e.g. a smart request only goes to backends tagged smart).
2. Scores by metrics — fetched from each backend's Prometheus endpoint:
   • score = running + (waiting * 10) + localInFlight + metricPenalty
   • Lower score is better; unavailable backends get a 1,000,000 penalty.
3. Applies cooldown — backends that recently failed enter a 30s cooldown.
4. Round-robins ties.
5. Retries up to 2 times on failure.

Metrics (vllm:num_requests_running, vllm:num_requests_waiting, GPU/CPU cache usage) are cached ~2s (stale) to 10s (expire).

Failover

Two failover mechanisms protect reliability:

• Latency-based (global, non-sticky): if a model's recent latency exceeds the slow threshold, requests are auto-redirected along a preference chain. Admins can toggle this and see live status on the dashboard.
• Disconnect-based (per-conversation, sticky): if the smart backend drops a stream mid-flight (client disconnect), that conversation is pinned to a cheaper fallback model until an admin clears it. Priority conversations bypass the latency failover and are never auto-redirected.

Bring-your-own keys (overrides)

You can attach your own LLM provider API key to a specific (conversationId, modelId) pair via the Projects API. When a matching request arrives:

1. The override key is decrypted (AES-256-GCM).
2. checkRequestLimit() is bypassed entirely — your key, your usage.
3. The override key is forwarded to the backend instead of the platform default.
4. Usage is logged with an isOverrideKey flag and excluded from project quotas.

Model matching is flexible: an override for qwen-qwq-32b also matches qwen-qwq-32b-high, so you don't need a rule per variant. See Security for how override keys are encrypted and protected.

Custom LLM backends (project-level BYO)

Beyond conversation-level overrides, a project can register its own OpenAI-compatible endpoints. Each enabled backend appears in that project's model catalog prefixed with custom: (e.g. custom:acme-gpt) and routes chat requests directly to the project's endpoint using the project's own credentials.

• Single-target routing — custom backends are excluded from the global round-robin, metrics, and availability pool. They retry on network errors in isolation; there is no cross-backend failover.
• Bypass quotas — requests use your credentials and do not count toward the project request limit.
• Secrets-backed credentials — a backend's key comes from a referenced project secret (preferred) or a raw encrypted API key.

Configure custom backends from Project → Settings → Custom LLMs, or via the Projects API (POST /api/projects/{projectId}/custom-llms).

Provider templates

When adding a custom LLM, you can select from pre-configured provider templates:

• Bob (default) — IBM Bob LLM Platform with locked-in configuration. All technical fields are pre-configured and hidden: API Base URL (https://api.us-east.bob.ibm.com/inference/v1), model (premium), auth scheme (ApiKey), provider (bob), max input tokens (160,000), and max output tokens (40,000). The profile name defaults to bob but can be customized. Users only provide a Profile Name and API Key (or select a secret).

  If the profile name conflicts with an existing backend, helpful alternatives are suggested with one-click application.

• Custom — Bring your own endpoint with full manual configuration of all fields.

Both templates support the project secrets system, allowing you to reference vault-backed credentials instead of storing raw API keys.