Skip to content

Benchmarks / evidence with context

Useful numbers. Honest boundaries.

Measured studies, operations records, and catalog estimates are separate kinds of evidence. Explore the original dates, runtimes, units, and decisions behind the configuration.

39 MEASURED STUDIES3 OPS RECORDS3 CATALOG ESTIMATESDOWNLOAD ALL RECORDS ↓

GB10 pair / Reviewed Oct 23, 2026

DeepSeek is serving. The coding panel is a tie.

Normal DeepSeek V4.1 Flash 1017 runs through TensorFold TP2 on the DGX Spark and ASUS GX10. The matched GLM comparison, short serving probes and memory observation answer different questions.

Shared inference slots
4

Coder, scout and vision share this capacity. Studio agent and embeddings keep independent routes.

Configured context
524,288

A serving limit. Local full-window retrieval and concurrent long-context capacity remain unqualified.

Initial memory minima / GiB
4.93 / 6.55

Head / worker available memory across one hour, 1,800 samples each. A bounded observation; guards remain active.

October 22–23 · Same frozen primary Hermes protocol, temperature 0.2, thinking off, 4,096 output allowance. Coding handoffs and diagnostic decisions keep separate denominators.
Model / dateRoutine codingHard codingDefective reviewOther controls
GLM · abliterated1 / 10 / 3Incorrect approvalAdmission 1 / 1
Healthy review 1 / 1
DeepSeek · normal1 / 10 / 3Correct rejectionAdmission 1 / 1
Healthy review 1 / 1

One routine task, three hard tasks, one visit each, one harness. Behavior, scope, report, exact identity and cleanup are required. This local panel is not an official SWE-bench score. Later production high-reasoning tuning is separate; no difficult-coding advantage was demonstrated.

Historical / September Studio evaluation

Speed is only one axis.

Protocol & limitations ↗

M5 Max · 40 GPU cores · 48 GB · Sep 22–23, 2026 · thinking off. Lightning here is the evaluated Nemotron artifact. The current SuperQwen alias has no score in this chart.

HumanEval / % ↑

050100080160

Code decode / tok/s →

Coding fallback / MLX 4-bit

95.1%

HumanEval · 156 / 164 tasks passed

All 164 original tasks, one greedy chat sample, original executable tests, no repair or retries. Chat-adapted pass@1; training contamination is possible. Gemma was a writing-role evaluation and did not run this suite.

Speed: median of three short 512-output code probes on a loaded backend. Timing and quality use different prompts; this is not full-task wall time.

See the experiments in the visual lab.Speculation, concurrency, promotion reversals, image budgets, and quality gates.

Explore the study records

3 OF 45 RECORDS

The evidence atlas / 3 records

Every study keeps its date.

Select a date to inspect its records. The dots count records, not performance.

MEASUREDOPERATIONSCATALOG
Measured study

Jul 21 two deep coder slots

Qwen3-Coder-Next · 2 × 262144

Historical lab pool

single
52.6tok/s
aggregate
76tok/s
warm turn
4.3s
All measurements & context
single
52.6 tok/s
aggregate
76 tok/s
warm turn
4.3 s
retrieval tokens
241,000

Historical qualification. Both deep sessions retrieved correctly near 241K. About 55.2 GiB. Warm-session affinity, not a 2× cold accelerator. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: llama.cpp · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Jul 21 three independent coder lanes

three-lane coder dispatcher

Historical lab pool

aggregate
99.6tok/s
per stream low
42tok/s
per stream high
56.8tok/s
All measurements & context
aggregate
99.6 tok/s
per stream low
42 tok/s
per stream high
56.8 tok/s
http 409s
0

Historical. Two coding-peer slots plus one local lane. Four concurrent 256-token streams; excess queued, no 409s. Session affinity keeps cache local. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: capacity-aware proxy · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Jul 19 multimodal creative agent bakeoff

Gemma-4-26B-A4B QAT · always-on agent

Historical lab pool

single
60.4tok/s
four way
129.9tok/s
tools
91.2%
All measurements & context
single
60.4 tok/s
four way
129.9 tok/s
tools
91.2 %
blind score
8.35 /10
warm prefix
0.303 s
deep edge
78.3 s

Historical. Gemma kept as always-on agent. Goetia and Fable rejected. Warm 17K prefix 0.303 s (23.2× over cold). Deep edge 78.3 s · 119,593 prompt tokens. Catalog ~65 tok/s on the later agent-gemma on-demand id is a separate roster figure. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: llama.cpp · four 163840 slots · dated bench page
Source record updated 2026-09-12

Read the decision ↗