01Open the tests and model history
The benchmark tests page publishes 16 methods, their scoring rules, adapted public prompts and historical promotion criteria. The new local repository handoff panel separates coding from review controls; the serving method distinguishes whole-wave throughput from cache replay. The model catalogue and both JSON downloads retain rejected, superseded and incomplete evidence.
Public evidence pages
/benchmarks/tests
/benchmarks/models02Freeze the configuration and workload
Record hardware, artifact revision, quantization, engine, chat template, thinking mode, sampling, prompt length, output cap and concurrency. Separate cold startup from a loaded model and a cached prefix. Run one contender at a time unless contention is the experiment.
03Report distinct measurements
Missing measurements stay missing. A zero idle generation gauge is not a model performance result. Catalog estimates and configured limits do not become measured results.
The October speed harness sums individual request decode rates for its aggregate field. That differs from total wave output divided by wave wall time. Engine-log generation counters also describe aggregate mixed traffic. Keep all three denominators explicit.
October 23 DeepSeek capacity rates instead use total completion positions divided by whole-wave wall time, including prefill. The matched Hermes panel separately fixes tasks, budgets, sampling, scope, identity and cleanup; it is not an official SWE-bench score. Keep routine 1/1 and hard 0/3 visible instead of reporting a blended coding/review score.
- TTFT: request start to first output; include prefill and queueing context.
- Decode: output positions per second after generation begins. State the timing source and denominator.
- Wall time: time to finish the real task, including loading, tools and validation.
- Concurrent throughput: aggregate output rate plus per-stream latency and failures.
- Capacity: memory, cancellations, admission behavior and retrieval near the tested context boundary.
04Use repeatable correctness checks
Keep raw failures, task counts and scoring rules. Repeat noisy custom gates and report medians and spread. Compile generated code or run executable tests where appropriate; regex framework checks alone cannot establish correctness. HumanEval chat adaptation, local tool suites and public leaderboard protocols must be identified separately.
06Make a promotion reversible
A challenger needs a useful quality, latency, reliability or capacity improvement for its assigned role. Preserve the incumbent, rerun after runtime changes and keep a rollback path. The July Laguna reversal is the reason this journal records rejected and superseded decisions alongside wins.