Skip to content

Benchmarks / model history

Keep the contenders in the record.

Locally tested variants, their measured results and the decisions they informed. Rejected and superseded models remain useful evidence. Each artifact and runtime carries its own limits.

51 ENTRIES / REVIEWED Oct 23, 2026DOWNLOAD MODEL CATALOGUE ↓

Executed coding / run completeness

Aider polyglot: preserve the denominator.

Qwen3.8-27B + DFlash2

112 / 225 49.8%

Complete recorded run. Passes within two tries; first try 21/225 (9.3%). Thinking off.

Full run & language counts ↗

September MiaAI Flash-Next

94 / 225 RECORDED

Incomplete: 58 passes within two tries among recorded tasks; 131 tasks unrecorded. Thinking on. No final pass rate or winner.

Partial run & error counters ↗

Both runs started September 26. Artifact, thinking and infrastructure differ, and the effective history budget is not fully frozen. The October Qwen Flash artifact has separate historical speed evidence. The current DeepSeek deployment and its matched GLM control have their own entries. Read the protocol ↗

Browse the catalogue

51 OF 51 ENTRIES

Local evidencePrevious coder; rollback preserved

GLM-5.3-Flash · production control

EXL3 · abliterated serving checkpoint

2026-10-22
TensorFold native · TP2 · primary Hermes handoff panel
DGX Spark + ASUS GX10 · two 128 GB GB10 hosts

Matched Hermes panel: routine 1/1, hard 0/3 for both models. Correct admission and healthy-review decisions, but incorrect approval of the defective-change review. Do not pool with the earlier GLM speed artifact.

routine passed
1
routine tasks
1
hard passed
0
hard tasks
3
All results, context & sources
routine passed
1
routine tasks
1
hard passed
0
hard tasks
3
defective review correct
0

Context: Frozen local fixture context only; not long-context qualification.

Limits: One routine coding task and three hard coding tasks, one visit each, one harness. Not official SWE-bench Verified. Normal DeepSeek versus abliterated GLM. Different dates, tools, cache effects and production tuning prevent a general quality or speed ranking.

Dated exact-identity matched primary Hermes receipts · October 22–23, 2026; current DeepSeek settings independently inspected October 23.

Local evidenceActive deployment; hard-coding gain unproven

DeepSeek V4.1 Flash 1017

EXL3 EAQ 2.9 bpw · normal · ABLIT=0

2026-10-23
TensorFold native · TP2 · primary Hermes handoff panel
DGX Spark + ASUS GX10 · two 128 GB GB10 hosts

Matched Hermes panel: routine 1/1, hard 0/3 for both models. DeepSeek correctly rejected the defective-change review that GLM approved. Production high-reasoning tuning is separate from this thinking-off panel.

routine passed
1
routine tasks
1
hard passed
0
hard tasks
3
All results, context & sources
routine passed
1
routine tasks
1
hard passed
0
hard tasks
3
defective review correct
1

Context: 524,288 configured; local 21,875-position validation and separate 63,164-position history replay do not qualify full-window retrieval.

Limits: One routine coding task and three hard coding tasks, one visit each, one harness. Not official SWE-bench Verified. Normal DeepSeek versus abliterated GLM. Different dates, tools, cache effects and production tuning prevent a general quality or speed ranking.

Dated exact-identity matched primary Hermes receipts · October 22–23, 2026; current DeepSeek settings independently inspected October 23.

Local evidenceRetired

Cydonia 24B v4.3

Q6_K

Historical two-round study · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Strong prose in the original qualitative review, with dense decode around 10 tok/s.

reported decode approx
10tok/s
All results, context & sources
reported decode approx
10 tok/s

Context: suite context only

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceIdentity unresolved

DeepSeek-V4-Flash · unconfirmed quant

Retained report label ds4-iq2m

Aug 3, 2026
Runtime/artifact revision not independently frozen
Historical lab pool

Keep the measured report without attributing its quality to the verified August 2 IQ2_XXS artifact.

framework
84.8%
developer tools
41.7%
single stream
12.4tok/s
4-stream aggregate
16.5tok/s
All results, context & sources
framework
84.8 %
developer tools
41.7 %
single stream
12.4 tok/s
4-stream aggregate
16.5 tok/s

Context: Not qualified by this gate.

Limits: The retained label is not proof of quant identity. Scores belong to this unresolved report only.

Dated public study and retained numeric archive; reviewed October 10, 2026.

Local evidenceRejected

DeepSeek-V4-Flash-0731

UD-IQ2_XXS GGUF

Aug 2, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

One-Spark load and 262K retrieval were demonstrated, but decode and prefill were too slow for the primary coder. Quality was not measured for this verified quant.

decode low
16.6tok/s
decode high
17.8tok/s
prefill
530tok/s
prefill input positions
7,000
All results, context & sources
decode low
16.6 tok/s
decode high
17.8 tok/s
prefill
530 tok/s
prefill input positions
7,000
weight
85 GiB
available memory floor
31.1 GiB
cold load
376 s

Context: 262144 retrieval verified on one GB10 host; this quant has throughput evidence only.

Limits: The August 3 gate has an unresolved quant identity and is listed separately. Do not assign its quality score to this artifact.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Empero Qwythos-27B-v1

Q4_K_M

Jul 29, 2026
llama.cpp · revision not frozen in registry
Historical GB10 / lab pool; use dated record for exact host

Matched base A/B showed 84.6% versus 84.3% framework and 11.0 versus 10.8 tok/s; no useful role improvement.

framework
84.6%
developer tools
75%
broad tools
100%
single stream
11tok/s
All results, context & sources
framework
84.6 %
developer tools
75 %
broad tools
100 %
single stream
11 tok/s

Context: 262K configured for the gate; advertised 1M not deep-tested

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceSuperseded

Gemma-4 26B-A4B Heretic / ultra-uncensored

Q6_K

Historical two-round study · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Stable, careful former writing candidate, later superseded by QAT/Q4 qualification. No numeric quality score is supplied here.

No numeric score retained.

All results, context & sources

Context: suite context only

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceWriting role

Gemma-4 26B-A4B QAT

Q4_K_M

Sep 22–23, 2026
llama.cpp Metal · build 10964
Mac Studio · M5 Max · 48 GB

Highest mechanical prose constraints in the Studio comparison. HumanEval was not run for this writing role.

code decode
78.1tok/s
prose decode
76.2tok/s
framework
84.3%
broad tools
94.1%
All results, context & sources
code decode
78.1 tok/s
prose decode
76.2 tok/s
framework
84.3 %
broad tools
94.1 %
developer tools
66.7 %
prose constraints
97 %

Context: Synthetic retrieval reached about 34K input positions; not 64K/262K qualified.

Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.

Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.

Local evidenceFormer default

Gemma-4 26B-A4B QAT uncensored

Q4_K_M

Jul 19, 2026
llama.cpp · GB10-era agent
Historical GB10 / lab pool; use dated record for exact host

Former multimodal agent; blind-writing mean 8.35/10 and a production context ceiling of 160K. The recorded edge test reached 119,593 input positions.

reported decode
60.4tok/s
broad tools
91.2%
vision
87.5%
blind editor mean score
8.35/10
All results, context & sources
reported decode
60.4 tok/s
broad tools
91.2 %
vision
87.5 %
blind editor mean score
8.35 /10
deep edge input positions
119,593
deep edge
78.3 s

Context: Historical production ceiling 163840; retained edge probe at 119,593 input positions. No 262K qualification.

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Gemma-4 26B-A4B-it clean

Q6_K

Historical two-round study · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

The retained long-form study records a benign essay refusal with a claimed platform limit. No numeric literary-quality score was retained.

No numeric score retained.

All results, context & sources

Context: suite context only

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

GLM-4.5-Air 106B-A12B

Q4

Historical two-round study · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Stable serving but slower decode prevented a general-agent default role.

reported decode low
19tok/s
reported decode high
20tok/s
All results, context & sources
reported decode low
19 tok/s
reported decode high
20 tok/s

Context: suite context only

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceSpeed probe

GLM-5.3-Flash

EXL3

2026-10-05
TensorFold two-node TP2; exact probe revision not retained
DGX Spark + ASUS GX10 · two GB10 nodes

Historical October 5 speed probe: 97.59 tok/s code and 43.14 tok/s prose. The later abliterated production checkpoint has a separate matched Hermes entry; these speed results do not transfer to it.

code single stream
97.59tok/s
prose single stream
43.14tok/s
prefill ttft
8.062s
code four request mean decode
78.9tok/s
All results, context & sources
code single stream
97.59 tok/s
prose single stream
43.14 tok/s
prefill ttft
8.062 s
code four request mean decode
78.9 tok/s
code four request rate sum
315.6 tok/s

Context: ~6K input timing probe; configured 262144 is not full-window qualification.

Limits: One wave per workload/concurrency, no repeat spread. Dates, model tokenizers and output lengths differ; exact runtime revision, effective thinking policy, cache state and external contention were not independently frozen. The prefill probe does not qualify 262K retrieval. No executed correctness, tool suite, or writing-quality result.

Retained lab speed report · October 5, 18:04:27–18:05:20 CDT

Local evidenceRejected

Goetia / Gemma-4 26B-A4B Heretic

Q4_K_M

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Near the former agent’s speed and vision accuracy but a lower blind-writing mean.

reported decode
59.8tok/s
vision
87.5%
blind editor mean score
6.65/10
All results, context & sources
reported decode
59.8 tok/s
vision
87.5 %
blind editor mean score
6.65 /10

Context: deep retrieval passed

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceSuperseded

Huihui Qwen3-30B-A3B-2507 abliterated

Q6_K

Historical two-round study · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Former general agent, later superseded by Gemma QAT qualification.

reported decode approx
60tok/s
All results, context & sources
reported decode approx
60 tok/s

Context: suite context only

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Huihui Qwen3.6-35B-A3B abliterated

NVFP4

Historical gate · archived Sep 12, 2026
Historical GB10 runtime
Historical GB10 / lab pool; use dated record for exact host

One of three long runs collapsed. This evidence applies to the tested quant, rather than every variant of this family.

reported decode low
57tok/s
reported decode high
60tok/s
long runs
3
collapsed runs
1
All results, context & sources
reported decode low
57 tok/s
reported decode high
60 tok/s
long runs
3
collapsed runs
1

Context: long-form suite

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceSelected

Mage-Flow Turbo 4B

1024×1024 local pipeline

Jul 22, 2026
Dated image-generation pipeline
Historical local image lane

Historical preferred fit for warm latency, memory and visual quality.

warm generation
3.96s
peak
17.03GiB
All results, context & sources
warm generation
3.96 s
peak
17.03 GiB

Context: Not a language-model context test.

Limits: Warm timings only; pipeline/model styles and quality goals differ.

Dated public study and retained numeric archive; reviewed October 10, 2026.

Local evidenceRejected

Meta Muse Glimmer 30B

Dynamic kquant + projector + DFlash

Aug 10, 2026
llama.cpp b10358
Historical GB10 / lab pool; use dated record for exact host

Fast loading and complete screenshot-fact coverage, but 25% developer tools, dense-model latency and a configured 131072 ceiling failed the historical coder role.

framework
81.5%
developer tools
25%
developer perception facts
30
developer perception total
30
All results, context & sources
framework
81.5 %
developer tools
25 %
developer perception facts
30
developer perception total
30
vision
75 %
single stream
10.1 tok/s
cold ready
17 s

Context: 131072 configured (Meta default); Unsloth claims up to 262K — not needle-verified here

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Partial runIncomplete

MiaAI Qwen3.8-Flash-Next

September NVFP4 artifact · thinking on

Run started Sep 26, 2026
vLLM · Aider whole-file editing
Two GB10 nodes

58 passes within two tries among 94 recorded exercises, out of a planned 225. No final pass rate or model winner.

within two tries passed
58
first try passed
28
tasks recorded
94
suite tasks
225
All results, context & sources
within two tries passed
58
first try passed
28
tasks recorded
94
suite tasks
225
tasks unrecorded
131
context exhaustion counter
72
request error output counter
73

Context: Recipe specifies 65536; metadata lists 262144. Effective history cap not frozen.

Limits: Incomplete archive and different thinking policy. September MiaAI quality does not transfer to the current October NVIDIA artifact.

Dated public study and retained numeric archive; reviewed October 10, 2026.

Local evidenceRetained candidate

Mistral Small 4 119B-A6B

Official NVFP4

Sep 12, 2026
vLLM 0.29.0 · TP1
Historical GB10 / lab pool; use dated record for exact host

Quality candidate left unrouted: stronger tools and prose constraints, but slower decode and long readiness.

framework
81.4%
developer tools
83.3%
broad tools
100%
prose constraints
86.9%
All results, context & sources
framework
81.4 %
developer tools
83.3 %
broad tools
100 %
prose constraints
86.9 %
single stream
30.7 tok/s
2-stream aggregate
52.2 tok/s
4-stream aggregate
75.9 tok/s
cold ready
674 s
cached ready
585 s
model
66.12 GiB
kv
27.33 GiB

Context: 262144 configured; vLLM allocated 1,273,776 KV tokens / 4.86× capacity; retrieval not needle-verified

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceFormer default

Nemotron 3 Nano Omni 30B-A3B

UD-Q4_K_XL + projector

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Historical developer-vision sidecar: all 30 screenshot facts in 10 seconds. Generic fixtures were 5/8.

developer perception facts
30
developer perception total
30
developer perception wall
10s
vision passed
5
All results, context & sources
developer perception facts
30
developer perception total
30
developer perception wall
10 s
vision passed
5
vision tasks
8

Context: 240K verified; 262K configured

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Partial runCapacity rejected

Nemotron 3 Super 120B-A12B

MXFP4

Historical partial gate · archived Sep 12, 2026
Historical GB10 runtime
Historical GB10 / lab pool; use dated record for exact host

Answered three framework tasks, then was OOM-killed. Individual task scores are retained; no full-suite percentage is inferred.

tasks recorded
3
task one
100%
task two
100%
task three
77.8%
All results, context & sources
tasks recorded
3
task one
100 %
task two
100 %
task three
77.8 %
reported decode
16.6 tok/s

Context: 262K configured; not qualified

Limits: Three recorded tasks are not a completed eight-task gate. Configured context was not qualified.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceSpecialist

Nemotron Cascade 2 30B-A3B

Q4

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Context specialist with a recorded 240K edge pass in 141 seconds; did not replace the coder.

framework
80.6%
developer tools
66.7%
deep edge
141s
All results, context & sources
framework
80.6 %
developer tools
66.7 %
deep edge
141 s

Context: 240K verified; native up to 1M

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceFormer default

Nemotron-3.5-Lightning 30B-A3B

MLX 4-bit

Sep 22–23, 2026
MLX / MLX-VLM
Mac Studio · M5 Max · 48 GB

Fast September Studio agent artifact. The October agent alias serves a different model.

code decode
150.1tok/s
prose decode
150.4tok/s
framework
71.5%
broad tools
88.2%
All results, context & sources
code decode
150.1 tok/s
prose decode
150.4 tok/s
framework
71.5 %
broad tools
88.2 %
developer tools
66.7 %
prose constraints
73.5 %
humaneval passed
140
humaneval tasks
164
humaneval
85.4 %

Context: Synthetic retrieval reached about 34K input positions; not 64K/262K qualified.

Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.

Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.

Local evidenceFormer default

Nemotron-3.5-Lightning 30B-A3B

Official NVFP4 + DSpark

Aug 11 / Sep 12, 2026
vLLM 0.29.0
Historical GB10 / lab pool; use dated record for exact host

September agent default. The clean rebaseline improved the framework gate to 80.3%; the long essay stopped at 3,935 of 10,000 requested words.

framework
80.3%
developer tools
75%
broad tools
82.4%
prose constraints
63.8%
All results, context & sources
framework
80.3 %
developer tools
75 %
broad tools
82.4 %
prose constraints
63.8 %
single stream
77.6 tok/s
2-stream aggregate
120.4 tok/s
4-stream aggregate
169 tok/s
longform
85.9 tok/s
longform ttft
0.13 s
longform words
3,935
longform requested words
10,000

Context: production 128K; 1M native, not re-needle-tested in this pass

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

North Mini Code 30B-A3B

Q4

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Thinking retest improved framework checks, but tools and deep-context/scaffold qualification still failed.

framework thinking off
58.3%
framework thinking retest
74.4%
developer tools
41.7%
All results, context & sources
framework thinking off
58.3 %
framework thinking retest
74.4 %
developer tools
41.7 %

Context: incomplete/failing at depth

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRetired

Ornith 1.0 9B uncensored

Q6_K

Historical two-round study · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Long outputs at about 28 tok/s, without a current routed advantage.

reported decode approx
28tok/s
All results, context & sources
reported decode approx
28 tok/s

Context: suite context only

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceSpecialist

Ornith-1.0-35B

UD-Q4_K_XL

Jul 30, 2026
llama.cpp · tuned gate
Historical GB10 / lab pool; use dated record for exact host

Five framework runs: median 84.2%, standard deviation 5.55 points; Swift median 60.7%. Four tool runs: median 75%. Faster, smaller and quick to load, but not promoted over the primary coder.

framework median
84.2%
framework stddev points
5.55
swift median
60.7%
developer tools median
75%
All results, context & sources
framework median
84.2 %
framework stddev points
5.55
swift median
60.7 %
developer tools median
75 %
framework runs
5
developer tool runs
4
single stream
61.6 tok/s
cold ready
27 s
allocated
28.2 GiB

Context: 2 x 262144 configured; native 262144

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceSpecialist

Ortenzya 31B

Q4_K_M

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Optional creative specialist; blind-writing mean 8.17/10, with dense decode around 9.8 tok/s.

blind editor mean score
8.17/10
reported decode
9.8tok/s
All results, context & sources
blind editor mean score
8.17 /10
reported decode
9.8 tok/s

Context: deep retrieval passed

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceReversed promotion

Poolside Laguna-S-2.1 117.6B MoE

NVFP4 · revision 0761412

Jul 23–26, 2026
vLLM · SM121
Historical GB10 / lab pool; use dated record for exact host

Promoted July 23 and reverted July 26. Same-day control won framework quality and wall time; Laguna retained a tool advantage but had heavy memory use and slow cold startup.

framework
87.2%
developer tools
75%
framework wall
436s
single stream
17.7tok/s
All results, context & sources
framework
87.2 %
developer tools
75 %
framework wall
436 s
single stream
17.7 tok/s
allocated
101 GiB
cold ready low
480 s
cold ready high
900 s

Context: 257.1K verified (middle + edge pair); 262K configured

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Poolside Laguna-XS-2.1 33B

NVFP4

Jul 30, 2026
vLLM
Historical GB10 / lab pool; use dated record for exact host

Smaller weights did not solve the long-context KV footprint. Four-stream throughput used different slot settings than the llama.cpp incumbent; it is not an isolated model advantage.

framework
83.9%
developer tools
50%
single stream
35.3tok/s
4-stream aggregate
127.7tok/s
All results, context & sources
framework
83.9 %
developer tools
50 %
single stream
35.3 tok/s
4-stream aggregate
127.7 tok/s
cold ready
200 s

Context: 262144 configured

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Qwen3-235B-A22B-2507

Q2_K_XL; separate Q3 load failed

Historical study · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Q3 load failed with OOM. The tested Q2 ran around 14 tok/s with qualitative quantization damage; legacy refusal counts are separate from quality.

reported decode approx
14tok/s
legacy refusal cases
6
legacy refusals
1
All results, context & sources
reported decode approx
14 tok/s
legacy refusal cases
6
legacy refusals
1

Context: suite context only

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Qwen3-30B-A3B-Instruct-2507 clean

Q6_K

Historical two-round study · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Stable serving, but outline and long-form decay prevented a default role.

reported decode approx
62tok/s
All results, context & sources
reported decode approx
62 tok/s

Context: suite context only

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Qwen3-Coder-Next 80B-A3B

Q5_K_M

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Framework score near the Q4 baseline, but Swift and Tailwind regressed and code decode was slower.

framework
86.2%
developer tools
66.7%
code decode
45.3tok/s
All results, context & sources
framework
86.2 %
developer tools
66.7 %
code decode
45.3 tok/s

Context: same native 262K; deep test reserved for best Q5

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceFormer default

Qwen3-Coder-Next 80B-A3B

UD-Q4_K_XL

Jul 21–30, 2026
llama.cpp · dated runtime/tuning variants
Historical GB10 / lab pool; use dated record for exact host

Restored after the Laguna reversal. Six-run calibration median 86.4%; a separate July 26 run reached 90.9%. These are different summaries, not two contradictory scores.

framework median
86.4%
framework stddev points
2.9
jul26 framework
90.9%
jul26 developer tools
66.7%
All results, context & sources
framework median
86.4 %
framework stddev points
2.9
jul26 framework
90.9 %
jul26 developer tools
66.7 %
jul26 single stream
52.7 tok/s
jul26 4-stream aggregate
76 tok/s
allocated
53.9 GiB
cold ready approx
90 s

Context: 2 x 262K configured on GX10; 2 x ~241K concurrently verified

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Qwen3-Coder-Next 80B-A3B

UD-Q5_K_M

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

No assigned-role win over the Q4 incumbent.

framework
87.4%
developer tools
58.3%
code decode
41.6tok/s
All results, context & sources
framework
87.4 %
developer tools
58.3 %
code decode
41.6 tok/s

Context: same native 262K; deep test reserved for best Q5

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceWatch

Qwen3-Coder-Next 80B-A3B

UD-Q5_K_XL

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

88.0% overall and 74.1% Swift, with slower decode and Tailwind/Bash regressions. Retained as a Swift watch candidate.

framework
88%
swift
74.1%
All results, context & sources
framework
88 %
swift
74.1 %

Context: 231.7K + edge pair verified

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Qwen3.5-122B-A10B

Q4 + projector

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Good developer tools, but weaker framework quality and long deep-context wall time.

framework
79.5%
developer tools
83.3%
deep edge
480.6s
All results, context & sources
framework
79.5 %
developer tools
83.3 %
deep edge
480.6 s

Context: 231.7K verified; native 262K

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceControl

Qwen3.5-27B

Q4_K_M

Jul 29, 2026
llama.cpp
Historical lab pool

Matched Qwythos control. Framework 84.3%, tools 75%, broad tools 100% and single-wave throughput 10.8 tok/s.

framework
84.3%
developer tools
75%
broad tools
100%
single stream
10.8tok/s
All results, context & sources
framework
84.3 %
developer tools
75 %
broad tools
100 %
single stream
10.8 tok/s

Context: 262K configured; no deep qualification in this matched gate.

Limits: Applies to the tested artifact, runtime, hardware and dated protocol. It does not transfer to other variants.

Dated public study and retained numeric archive; reviewed October 10, 2026.

Local evidenceFallback

Qwen3.5-9B

Q6_K + BF16 projector

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Complete screenshot-fact coverage, but the six images took 80.7 seconds versus Omni’s 10 seconds.

developer perception facts
30
developer perception total
30
developer perception wall
80.7s
vision passed
6
All results, context & sources
developer perception facts
30
developer perception total
30
developer perception wall
80.7 s
vision passed
6
vision tasks
8

Context: 231.7K verified; native 262K

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceFallback

Qwen3.6-27B

Q4

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

A fallback with slower deep retrieval, rather than a primary upgrade.

framework
81.8%
developer tools
75%
vision
75%
deep edge
535s
All results, context & sources
framework
81.8 %
developer tools
75 %
vision
75 %
deep edge
535 s

Context: 231.7K verified; native 262K

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Qwen3.6-27B Fable Fusion

Q4_K_M

Historical gate · archived Sep 12, 2026
llama.cpp
Historical GB10 / lab pool; use dated record for exact host

Long-form repetition collapse. Vision reached 75% only with tuned settings.

blind editor mean score
4.77/10
reported decode
10.8tok/s
tuned vision
75%
All results, context & sources
blind editor mean score
4.77 /10
reported decode
10.8 tok/s
tuned vision
75 %

Context: deep retrieval passed

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Qwen3.6-35B-A3B

UD-Q4_K_XL

Jul 21 / Jul 30, 2026
llama.cpp · original and tuned rematch
Historical GB10 / lab pool; use dated record for exact host

Original 75.2% framework rose to 81.7% in the tuned rematch, still below the historical primary baseline. Same artifact appears twice in the archive and is consolidated here.

original framework
75.2%
rematch framework
81.7%
developer tools
75%
vision
75%
All results, context & sources
original framework
75.2 %
rematch framework
81.7 %
developer tools
75 %
vision
75 %
single stream
62.4 tok/s
4-stream aggregate
95.7 tok/s

Context: 231.7K retrieval recorded in the original gate; two 262144 lanes configured for the later rematch, without a new two-lane qualification.

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceSuperseded

Qwen3.6-35B-A3B Heretic

Q6_K + native MTP

Jul 11, 2026
llama.cpp b9967 in July speed A/B
Historical GB10 / lab pool; use dated record for exact host

Six draft positions sped up code and slowed creative writing in the same quiet single-slot comparison.

code mtp off
64.1tok/s
code mtp on
100.9tok/s
creative mtp off
64.6tok/s
creative mtp on
53.6tok/s
All results, context & sources
code mtp off
64.1 tok/s
code mtp on
100.9 tok/s
creative mtp off
64.6 tok/s
creative mtp on
53.6 tok/s

Context: ~255K verified in candidate config

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceCoding candidate

Qwen3.8-27B

MLX 4-bit

Sep 22–23, 2026
MLX / MLX-VLM
Mac Studio · M5 Max · 48 GB

Higher HumanEval and framework checks than Lightning in the same Studio study.

code decode
33tok/s
prose decode
33tok/s
framework
89.5%
broad tools
100%
All results, context & sources
code decode
33 tok/s
prose decode
33 tok/s
framework
89.5 %
broad tools
100 %
developer tools
50 %
prose constraints
87.2 %
humaneval passed
156
humaneval tasks
164
humaneval
95.1 %

Context: Synthetic retrieval reached about 34K input positions; not 64K/262K qualified.

Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.

Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.

Local evidenceFormer default

Qwen3.8-27B

NVFP4 target + DFlash2 draft

Aug 19 / Sep 12 / Sep 26, 2026
SGLang · dated draft/runtime variants
Historical GB10 / lab pool; use dated record for exact host

September coder default; isolated August gate 92.0%, September gate 85.9%. Completed Aider run: 112/225 passed within two tries. Draft/runtime and workloads vary across these dates.

framework
85.9%
developer tools
75%
broad tools
94.1%
single stream
19.6tok/s
All results, context & sources
framework
85.9 %
developer tools
75 %
broad tools
94.1 %
single stream
19.6 tok/s
2-stream aggregate
31.5 tok/s
4-stream aggregate
42.7 tok/s
aider first try passed
21
aider within two tries passed
112
aider tasks recorded
225
aider suite tasks
225
aider within two tries
49.8 %

Context: 262144 configured/native; not needle-verified in this engine pass

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceRejected

Qwen3.8-27B

Official fine-grained FP8

Aug 14, 2026
vLLM 0.27.1
Historical GB10 / lab pool; use dated record for exact host

Strong broad tool and screenshot checks did not offset 8 tok/s decode, a 1,072-second framework run and 433-second cold readiness.

framework
87.5%
developer tools
50%
broad tools thinking off
100%
broad tools thinking on
100%
All results, context & sources
framework
87.5 %
developer tools
50 %
broad tools thinking off
100 %
broad tools thinking on
100 %
developer perception facts
30
developer perception total
30
vision
62.5 %
prose constraints
77.8 %
single stream
8 tok/s
4-stream aggregate
31.2 tok/s
framework wall
1,072 s
cold ready
433 s
allocated
91.8 GiB

Context: 262144 configured; advertised native — not needle-verified (speed already failed)

Limits: Historical protocols and revisions vary. Registry-only values retain their original rounding; repeat spread, timing denominator or blind-judge task count may be unspecified. See each linked study for stronger provenance.

Archived local variant registry, last updated September 12, 2026; retained summaries reviewed October 10. Exact dates are shown only where retained.

Local evidenceHistorical runtime evidence

Qwen3.8-Flash-Next

NVFP4 · October artifact

2026-10-09
TensorFold two-node TP2; exact probe revision not retained
DGX Spark + ASUS GX10 · two GB10 nodes

October TensorFold speed evidence; runtime and workload conditions remain attached to the dated report. Superseded as the active coder; current DeepSeek has separate evidence.

code single stream
80.09tok/s
prose single stream
50.26tok/s
prefill ttft
2.108s
code four request mean decode
53.7tok/s
All results, context & sources
code single stream
80.09 tok/s
prose single stream
50.26 tok/s
prefill ttft
2.108 s
code four request mean decode
53.7 tok/s
code four request rate sum
214.78 tok/s

Context: ~6K input timing probe; configured 262144 is not full-window qualification.

Limits: One wave per workload/concurrency, no repeat spread. Dates, model tokenizers and output lengths differ; exact runtime revision, effective thinking policy, cache state and external contention were not independently frozen. The prefill probe does not qualify 262K retrieval. No executed correctness, tool suite, or writing-quality result.

Retained lab speed report · October 9, 22:56:32–22:57:14 CDT

Local evidenceHistorical runtime evidence

Qwen3.8-Flash-Next

NVFP4 · October NVIDIA serving artifact

2026-10-05
vLLM two-node TP2; exact probe revision not retained
DGX Spark + ASUS GX10 · two GB10 nodes

Historical October vLLM speed evidence and October 10 configuration inspection. The short probe does not qualify quality or full context. Superseded as the active coder; current DeepSeek has separate evidence.

code single stream
67.25tok/s
prose single stream
44.79tok/s
prefill ttft
1.851s
code four request mean decode
53.48tok/s
All results, context & sources
code single stream
67.25 tok/s
prose single stream
44.79 tok/s
prefill ttft
1.851 s
code four request mean decode
53.48 tok/s
code four request rate sum
213.91 tok/s

Context: ~6K input timing probe; configured 262144 is not full-window qualification.

Limits: One wave per workload/concurrency, no repeat spread. Dates, model tokenizers and output lengths differ; exact runtime revision, effective thinking policy, cache state and external contention were not independently frozen. The prefill probe does not qualify 262K retrieval. No executed correctness, tool suite, or writing-quality result.

Retained lab speed report · October 5, 16:41:01–16:41:49 CDT

Local evidenceAlternative

Sana Sprint 1.6B

1024×1024 local pipeline

Jul 22, 2026
Dated image-generation pipeline
Historical local image lane

Fastest warm timing of these three; the study did not establish a shared quantitative image-quality score.

warm generation
0.79s
peak
10.19GiB
All results, context & sources
warm generation
0.79 s
peak
10.19 GiB

Context: Not a language-model context test.

Limits: Warm timings only; pipeline/model styles and quality goals differ.

Dated public study and retained numeric archive; reviewed October 10, 2026.

Awaiting benchmarkConfigured

SuperQwen · current Studio agent

MLX 4-bit · October alias

Configured artifact reviewed Oct 10, 2026
MLX
Mac Studio · M5 Max · 48 GB

The current agent artifact has no local score in the reviewed archive. September Nemotron results remain attached to Nemotron.

No numeric score retained.

All results, context & sources

Context: No independent full-context qualification in the reviewed evidence.

Limits: Packaged upstream reproduction scripts are not measurements from this lab.

Direct October 10 configuration review.

Local evidenceAlternative

Z-Image-Turbo

1024×1024 local pipeline

Jul 22, 2026
Dated image-generation pipeline
Historical local image lane

Historical warm budget exceeded the under-20-GiB target.

warm generation
5.7s
peak
21.69GiB
All results, context & sources
warm generation
5.7 s
peak
21.69 GiB

Context: Not a language-model context test.

Limits: Warm timings only; pipeline/model styles and quality goals differ.

Dated public study and retained numeric archive; reviewed October 10, 2026.