Skip to content
← All setup guides

Guide 05 / Operations

Routing is easy. Model lifecycle is the work.

Persistence, swap groups, alias drift, child processes, and protocol fidelity across heterogeneous machines.

2 MIN READ / REVIEWED Oct 23, 2026

01Give every role a capacity and ownership policy

Our peer-aware llama-swap coordinator routes the coder to the dual GB10 backend, agents to the Studio and utility/media work to the workstation. A role name can stay the same when its host changes. Inspect the target and process owner instead of inferring placement from an alias.

02Follow the label to the artifact

The October 23 coordinator labels now identify SuperQwen on the Studio and DeepSeek on the GB10 pair. Coder, scout and vision alias one four-slot backend; extra alias names do not create extra capacity. Resolve selector, peer, local alias and launch arguments before attributing a benchmark. September Nemotron Lightning measurements remain tied to their original artifact.

Legacy names also outlive placement: coder-spark resolves to Qwen on the Studio, and coder-dgx resolves to the Qwen3-Coder-Next candidate on the workstation. Studio-local scout remains SuperQwen even though the coordinator scout selector is DeepSeek. The October 23 workstation peer retains per-model TTLs and legacy matrix sets. Some newer pipelines fall outside those named sets; no concurrent media qualification was run.

03Make the resource group explicit

The following is a lifecycle excerpt from the active Studio configuration. It omits models, launch commands and endpoints. Match the schema to your exact gateway build; older upstream and peer-aware builds can differ.

Resource ownership, without private endpoints

# Lifecycle excerpt · current Studio config
routing:
  router:
    use: group
    settings:
      groups:
        primary:
          members: [lightning, qwen38, gemma-writer, agent-gemma, z-image, krea-2]
          swap: true
          exclusive: true
        embeddings:
          members: [embed]
          swap: false
          exclusive: false
          persistent: true

04Test the entire response contract

The original Studio metrics proxy dropped non-streaming tool calls while reconstructing text. Preserve tool-call IDs, names, arguments, streamed fragments, reasoning controls and usage fields. Compare direct and routed responses with deterministic fixtures; separate stochastic score movement from protocol correctness.

05Keep observation passive

Metadata polling previously woke models that should have remained unloaded. Passive metrics and inventory requests must not become implicit load requests. TTL controls idle unloading; persistence and exclusion decide what survives a competing model request.

06Expose a public record, keep the machines private

Publish model versions, limits, decisions and curated results. Keep credentials, network identities, user directories, private prompts and live control surfaces in the local deployment. This site has no connection to the inference gateway.

Primary references

Upstream docs describe supported behavior. Lab-specific configurations and numeric results retain their inspection or experiment dates.

Keep going

Related guides