Skip to content
← All setup guides

Guide 06 / Measurement

Measure the experience, not just tok/s

A repeatable protocol for latency, decode, context, concurrency, correctness, and honest comparisons.

2 MIN READ / REVIEWED Oct 23, 2026

01Open the tests and model history

The benchmark tests page publishes 16 methods, their scoring rules, adapted public prompts and historical promotion criteria. The new local repository handoff panel separates coding from review controls; the serving method distinguishes whole-wave throughput from cache replay. The model catalogue and both JSON downloads retain rejected, superseded and incomplete evidence.

Public evidence pages

/benchmarks/tests
/benchmarks/models

02Freeze the configuration and workload

Record hardware, artifact revision, quantization, engine, chat template, thinking mode, sampling, prompt length, output cap and concurrency. Separate cold startup from a loaded model and a cached prefix. Run one contender at a time unless contention is the experiment.

03Report distinct measurements

Missing measurements stay missing. A zero idle generation gauge is not a model performance result. Catalog estimates and configured limits do not become measured results.

The October speed harness sums individual request decode rates for its aggregate field. That differs from total wave output divided by wave wall time. Engine-log generation counters also describe aggregate mixed traffic. Keep all three denominators explicit.

October 23 DeepSeek capacity rates instead use total completion positions divided by whole-wave wall time, including prefill. The matched Hermes panel separately fixes tasks, budgets, sampling, scope, identity and cleanup; it is not an official SWE-bench score. Keep routine 1/1 and hard 0/3 visible instead of reporting a blended coding/review score.

  • TTFT: request start to first output; include prefill and queueing context.
  • Decode: output positions per second after generation begins. State the timing source and denominator.
  • Wall time: time to finish the real task, including loading, tools and validation.
  • Concurrent throughput: aggregate output rate plus per-stream latency and failures.
  • Capacity: memory, cancellations, admission behavior and retrieval near the tested context boundary.

04Use repeatable correctness checks

Keep raw failures, task counts and scoring rules. Repeat noisy custom gates and report medians and spread. Compile generated code or run executable tests where appropriate; regex framework checks alone cannot establish correctness. HumanEval chat adaptation, local tool suites and public leaderboard protocols must be identified separately.

05Compare within a shared protocol

Our Studio chart compares three models on the same September machine and short-probe protocol. It is not a Spark-versus-Mac hardware ranking. Model, runtime, quantization, prompts and dates differ across those studies. The older Spark tables remain separate.

A minimal result record

{
  "date": "YYYY-MM-DD",
  "hardware": "device, memory, driver",
  "artifact": "model revision and quantization",
  "runtime": "version or image digest",
  "protocol": "sampling, thinking, input/output lengths",
  "concurrency": 1,
  "samples": [],
  "correctness": "suite and denominator",
  "limitations": "what this result does not establish"
}

06Make a promotion reversible

A challenger needs a useful quality, latency, reliability or capacity improvement for its assigned role. Preserve the incumbent, rerun after runtime changes and keep a rollback path. The July Laguna reversal is the reason this journal records rejected and superseded decisions alongside wins.

Keep going

Related guides