Skip to content

Benchmarks / evidence with context

Useful numbers. Honest boundaries.

Measured studies, operations records, and catalog estimates are separate kinds of evidence. Explore the original dates, runtimes, units, and decisions behind the configuration.

39 MEASURED STUDIES3 OPS RECORDS3 CATALOG ESTIMATESDOWNLOAD ALL RECORDS ↓

GB10 pair / Reviewed Oct 23, 2026

DeepSeek is serving. The coding panel is a tie.

Normal DeepSeek V4.1 Flash 1017 runs through TensorFold TP2 on the DGX Spark and ASUS GX10. The matched GLM comparison, short serving probes and memory observation answer different questions.

Shared inference slots
4

Coder, scout and vision share this capacity. Studio agent and embeddings keep independent routes.

Configured context
524,288

A serving limit. Local full-window retrieval and concurrent long-context capacity remain unqualified.

Initial memory minima / GiB
4.93 / 6.55

Head / worker available memory across one hour, 1,800 samples each. A bounded observation; guards remain active.

October 22–23 · Same frozen primary Hermes protocol, temperature 0.2, thinking off, 4,096 output allowance. Coding handoffs and diagnostic decisions keep separate denominators.
Model / dateRoutine codingHard codingDefective reviewOther controls
GLM · abliterated1 / 10 / 3Incorrect approvalAdmission 1 / 1
Healthy review 1 / 1
DeepSeek · normal1 / 10 / 3Correct rejectionAdmission 1 / 1
Healthy review 1 / 1

One routine task, three hard tasks, one visit each, one harness. Behavior, scope, report, exact identity and cleanup are required. This local panel is not an official SWE-bench score. Later production high-reasoning tuning is separate; no difficult-coding advantage was demonstrated.

Historical / September Studio evaluation

Speed is only one axis.

Protocol & limitations ↗

M5 Max · 40 GPU cores · 48 GB · Sep 22–23, 2026 · thinking off. Lightning here is the evaluated Nemotron artifact. The current SuperQwen alias has no score in this chart.

HumanEval / % ↑

050100080160

Code decode / tok/s →

Coding fallback / MLX 4-bit

95.1%

HumanEval · 156 / 164 tasks passed

All 164 original tasks, one greedy chat sample, original executable tests, no repair or retries. Chat-adapted pass@1; training contamination is possible. Gemma was a writing-role evaluation and did not run this suite.

Speed: median of three short 512-output code probes on a loaded backend. Timing and quality use different prompts; this is not full-task wall time.

See the experiments in the visual lab.Speculation, concurrency, promotion reversals, image budgets, and quality gates.

Explore the study records

45 OF 45 RECORDS

The evidence atlas / 45 records

Every study keeps its date.

Select a date to inspect its records. The dots count records, not performance.

MEASUREDOPERATIONSCATALOG
Measured study

Short native capacity probe · 1, 2 and 4 concurrent requests

DeepSeek V4.1 Flash 1017 · EXL3 · normal

NVIDIA DGX Spark

single stream
41.16tok/s
single stream output tokens
160
single stream wave wall
3.887s
All measurements & context
single stream
41.16 tok/s
single stream output tokens
160
single stream wave wall
3.887 s
2-stream aggregate
75.777 tok/s
2-stream aggregate output tokens
307
2-stream aggregate wave wall
4.051 s
4-stream aggregate
91.568 tok/s
4-stream aggregate output tokens
598
4-stream aggregate wave wall
6.531 s
prompt tokens
2,787
later cached tokens per request
2,048
requests completed
7
requests attempted
7

Measured aggregate output divided by complete wave wall time: about 41.16, 75.78 and 91.57 tok/s at 1, 2 and 4 requests. Includes prefill; not pure decode speed or a quality score.

Protocol: One sequential wave at each concurrency. Temperature zero, thinking disabled, output cap 160. Each request had 2,787 prompt positions. The first wave was uncached; all requests in the later waves reported 2,048 cached positions. Sum each wave’s completion positions and divide by the measured wave wall seconds. Record stop versus length termination and verify the exact native model identity for all seven requests.

Limits: Tiny synthetic workload, seven requests and no repeat spread. Warm/cache state differs between waves; several responses hit the output cap. Scaling is not a cold-cache concurrency comparison. No correctness grading, long-context retrieval or concurrent full-window soak. The earlier October harness’s sums of individual decode rates use a different denominator.

Source: Dated native capacity receipt · October 23, 2026, 03:09 CDT

Runtime: TensorFold native · two-node TP2 · four shared slots
Source record updated 2026-10-23

Read the decision ↗
Measured study

Matched local repository handoff panel · coding and review controls

DeepSeek V4.1 Flash 1017 · EXL3 · normal

NVIDIA DGX Spark

routine passed
1
routine tasks
1
hard passed
0
All measurements & context
routine passed
1
routine tasks
1
hard passed
0
hard tasks
3
defective review correct
1
scope admission correct
1
healthy review correct
1
routine wall
74.698 s
menu behavior wall
350.188 s
strict types wall
412.824 s
socket repair wall
814.588 s

Measured local portfolio panel. Routine coding and hard coding have separate denominators; admission, defective-change review and healthy-change review are separate diagnostic controls. Both models tied on coding; no difficult-coding advantage was demonstrated.

Protocol: Same frozen tasks, sources, sampling, budgets and independent acceptance gates. Temperature 0.2, thinking disabled, 4,096 output allowance per request. Routine: 360 seconds / 32 requests; hard visits: 1,440 seconds / 96 requests including repairs and reporting. Acceptance requires behavior, allowed scope, structured report, exact native identity and verified cleanup. Each control is one decision, encoded 1 for correct and 0 for incorrect. One visit per case; the healthy-menu positive control is reported separately from the six-case panel.

Limits: One primary harness, one routine coding task, three hard coding tasks, and no statistical ranking. This is not an official SWE-bench Verified result. Wall time includes tools and cache effects. Normal DeepSeek is compared with the previously serving abliterated GLM checkpoint. Later high-reasoning production tuning and historical mixed-revision controls are not pooled into this comparison. Public commands passing alone do not override behavior or scope failures.

Source: Dated frozen primary Hermes comparison receipts · October 22–23, 2026; native identities and source fingerprints checked

Runtime: TensorFold TP2 · primary Hermes harness · frozen completion-v11 protocol
Source record updated 2026-10-23

Read the decision ↗
Operations record

One-hour initial memory observation · both GB10 ranks

DeepSeek V4.1 Flash 1017 · EXL3 · normal

NVIDIA DGX Spark

head min available
4.927GiB
worker min available
6.546GiB
head samples
1,800
All measurements & context
head min available
4.927 GiB
worker min available
6.546 GiB
head samples
1,800
worker samples
1,800
runtime guard floor
2.5 GiB
head recorded supervisor restarts
0

Operations observation covering initial validation, cache replay, benchmark visits and early live work. Minimum sampled MemAvailable: head 4.93 GiB, worker 6.55 GiB. No recorded OOM or supervisor restart during this bounded observation.

Protocol: 1,800 samples on each host over roughly one hour: head 08:02:22–09:02:22 UTC and worker 08:02:24–09:02:23 UTC. Report minimum host MemAvailable separately, retaining unchanged recipe admission, reserve and runtime guards. Samplers stopped after this window.

Limits: Not a controlled capacity experiment, a leak diagnosis, days of uptime, or a heavy concurrent long-context soak. Sampled minima can miss brief peaks; head and worker minima occurred at different instants. Swap occupancy is not an active paging rate.

Source: Dated bounded memory sampler and runtime receipts · October 23, 03:02–04:02 CDT

Runtime: TensorFold native · two-node TP2
Source record updated 2026-10-23

Read the decision ↗
Measured study

Held-history compatibility replay · cold and warm requests

DeepSeek V4.1 Flash 1017 · EXL3 · normal

NVIDIA DGX Spark

prompt tokens
63,164
history messages
220
complete tool pairs
110
All measurements & context
prompt tokens
63,164
history messages
220
complete tool pairs
110
cold wall
43.9 s
warm wall
8.8 s
warm cached tokens
61,440
structured responses passed
2
requests attempted
2

Measured compatibility/cache observation. Two high-reasoning requests used a copy of an existing 220-message history with 110 complete tool-call/result pairs. Both returned valid structured responses; tools were not executed.

Protocol: Replay the same copied history twice against the new native model without changing the original held session or executing proposed tools. Record total prompt length, response structure, request wall time and second-request cache usage. The warm request reported 61,440 cached positions out of 63,164 input positions.

Limits: One copied conversation and two requests, with no repeat spread, task-success grading or retrieval test. These wall times are not TTFT measurements. Cache reuse does not qualify the configured 524,288-position window, other conversations, or simultaneous full-context requests. Private history text is not published.

Source: Dated copied-history compatibility receipts · October 23, before the 03:57 CDT loop resume

Runtime: TensorFold native · TP2 · high reasoning
Source record updated 2026-10-23

Read the decision ↗
Measured study

Matched local repository handoff panel · coding and review controls

GLM-5.3-Flash · EXL3 · abliterated

NVIDIA DGX Spark

routine passed
1
routine tasks
1
hard passed
0
All measurements & context
routine passed
1
routine tasks
1
hard passed
0
hard tasks
3
defective review correct
0
scope admission correct
1
healthy review correct
1
routine wall
63.487 s
menu behavior wall
376.562 s
strict types wall
190.034 s
socket repair wall
580.339 s

Measured local portfolio panel. Routine coding and hard coding have separate denominators; admission, defective-change review and healthy-change review are separate diagnostic controls. Both models tied on coding; no difficult-coding advantage was demonstrated.

Protocol: Same frozen tasks, sources, sampling, budgets and independent acceptance gates. Temperature 0.2, thinking disabled, 4,096 output allowance per request. Routine: 360 seconds / 32 requests; hard visits: 1,440 seconds / 96 requests including repairs and reporting. Acceptance requires behavior, allowed scope, structured report, exact native identity and verified cleanup. Each control is one decision, encoded 1 for correct and 0 for incorrect. One visit per case; the healthy-menu positive control is reported separately from the six-case panel.

Limits: One primary harness, one routine coding task, three hard coding tasks, and no statistical ranking. This is not an official SWE-bench Verified result. Wall time includes tools and cache effects. Normal DeepSeek is compared with the previously serving abliterated GLM checkpoint. Later high-reasoning production tuning and historical mixed-revision controls are not pooled into this comparison. Public commands passing alone do not override behavior or scope failures.

Source: Dated frozen primary Hermes comparison receipts · October 22–23, 2026; native identities and source fingerprints checked

Runtime: TensorFold TP2 · primary Hermes harness · frozen completion-v11 protocol
Source record updated 2026-10-23

Read the decision ↗
Operations record

One hour of mixed coder traffic · October 10

Qwen3.8-Flash-Next · NVFP4

NVIDIA DGX Spark

nonzero sample median
47.9tok/s
peak sample
103.4tok/s
throughput samples
319
All measurements & context
nonzero sample median
47.9 tok/s
peak sample
103.4 tok/s
throughput samples
319
nonzero generation samples
310
min nonzero sample
0.1 tok/s
max sampled running
4
max sampled waiting
1
peak kv cache
14.6 %
first prefix hit gauge
74.5 %
last prefix hit gauge
76.8 %
head logged error lines
0
head logged oom lines
0
head logged preemption lines
0
head kv cache
20.95 GiB
startup kv tokens
1,014,349
startup full context equivalent
3.87
api log to ready
869.18 s

Dated operations observation, not a controlled decode benchmark or task-success study. Generation rates are engine-wide mixed-traffic samples. Startup cache capacity is the engine's calculation, not a full-context concurrency test.

Protocol: Fixed interval 2026-10-11 01:00–02:00 UTC = October 10 20:00–21:00 America/Chicago. Parsed 319 head log rows; median over 310 rows with generation > 0. Peak running/waiting/cache values are sampled gauges. Prefix-hit values are counter-derived gauges, not an isolated interval hit rate. Counted ERROR/Traceback, OutOfMemory/out of memory and preempted lines in the same head log. Startup timing is first API argument log to Application startup complete on the same date.

Limits: No prompts or responses published. Irregular log intervals and mixed workloads; sampled maxima may miss peaks. Zero matching head log lines does not certify request correctness, worker or kernel health, or long-context reliability. The startup memory calculation and timing describe one boot. No current throughput test, retrieval qualification, or full-window soak was run.

Source: Direct read-only head-container log summary · October 10, 20:00–21:00 CDT; separate October 10 startup log

Runtime: vLLM 0.30.0 · two-node TP2 + expert parallelism + MTP3
Source record updated 2026-10-10

Read the decision ↗
Measured study

October short-prompt runtime probe · Qwen / TensorFold

Qwen3.8-Flash-Next · NVFP4

NVIDIA DGX Spark

code single stream
80.09tok/s
prose single stream
50.26tok/s
prefill ttft
2.108s
All measurements & context
code single stream
80.09 tok/s
prose single stream
50.26 tok/s
prefill ttft
2.108 s
code 4-request mean
53.7 tok/s
code 8-request mean
36.51 tok/s
prose 4-request mean
39.36 tok/s
prose 8-request mean
31.8 tok/s
code 4-request rate sum
214.78 tok/s
code 8-request rate sum
292.04 tok/s
prose 4-request rate sum
157.46 tok/s
prose 8-request rate sum
254.43 tok/s
code single stream ttft
0.119 s
code single stream completion tokens
228
prefill input tokens
6,033
wave requests
26
wave successes
26
prefill requests
1
http errors
0

Measured historical speed probe on the DGX Spark + ASUS GX10 pair. Every wave request and the separate prefill probe returned HTTP 200. Code was not executed and prose was not scored. A speed probe is not a model promotion or full-context qualification.

Protocol: One prose prompt and one Python merge-function prompt; one wave each at concurrency 1, 4 and 8. Temperature 0, requested reasoning_effort medium, streaming with usage, wave output cap 320. TTFT includes first content or reasoning. Per-request decode = reported completion positions / elapsed time after first output. Concurrent means average these request rates; rate sums add them and are not wave-wall throughput. The separate ~6K input probe used output cap 16.

Limits: One wave per workload/concurrency, no repeat spread. Dates, model tokenizers and output lengths differ; exact runtime revision, effective thinking policy, cache state and external contention were not independently frozen. The prefill probe does not qualify 262K retrieval. No executed correctness, tool suite, or writing-quality result.

Source: Retained lab speed report · October 9, 22:56:32–22:57:14 CDT

Runtime: TensorFold two-node TP2; exact probe revision not retained
Source record updated 2026-10-10

Read the decision ↗
Measured study

October short-prompt runtime probe · GLM / TensorFold

GLM-5.3-Flash · EXL3

NVIDIA DGX Spark

code single stream
97.59tok/s
prose single stream
43.14tok/s
prefill ttft
8.062s
All measurements & context
code single stream
97.59 tok/s
prose single stream
43.14 tok/s
prefill ttft
8.062 s
code 4-request mean
78.9 tok/s
code 8-request mean
46.2 tok/s
prose 4-request mean
28.06 tok/s
prose 8-request mean
23.49 tok/s
code 4-request rate sum
315.6 tok/s
code 8-request rate sum
369.61 tok/s
prose 4-request rate sum
112.25 tok/s
prose 8-request rate sum
187.92 tok/s
code single stream ttft
0.181 s
code single stream completion tokens
207
prefill input tokens
6,032
wave requests
26
wave successes
26
prefill requests
1
http errors
0

Measured historical speed probe on the DGX Spark + ASUS GX10 pair. Every wave request and the separate prefill probe returned HTTP 200. Code was not executed and prose was not scored. A speed probe is not a model promotion or full-context qualification.

Protocol: One prose prompt and one Python merge-function prompt; one wave each at concurrency 1, 4 and 8. Temperature 0, requested reasoning_effort medium, streaming with usage, wave output cap 320. TTFT includes first content or reasoning. Per-request decode = reported completion positions / elapsed time after first output. Concurrent means average these request rates; rate sums add them and are not wave-wall throughput. The separate ~6K input probe used output cap 16.

Limits: One wave per workload/concurrency, no repeat spread. Dates, model tokenizers and output lengths differ; exact runtime revision, effective thinking policy, cache state and external contention were not independently frozen. The prefill probe does not qualify 262K retrieval. No executed correctness, tool suite, or writing-quality result.

Source: Retained lab speed report · October 5, 18:04:27–18:05:20 CDT

Runtime: TensorFold two-node TP2; exact probe revision not retained
Source record updated 2026-10-10

Read the decision ↗
Measured study

October short-prompt runtime probe · Qwen / vLLM

Qwen3.8-Flash-Next · NVFP4

NVIDIA DGX Spark

code single stream
67.25tok/s
prose single stream
44.79tok/s
prefill ttft
1.851s
All measurements & context
code single stream
67.25 tok/s
prose single stream
44.79 tok/s
prefill ttft
1.851 s
code 4-request mean
53.48 tok/s
code 8-request mean
45.47 tok/s
prose 4-request mean
31.33 tok/s
prose 8-request mean
26.62 tok/s
code 4-request rate sum
213.91 tok/s
code 8-request rate sum
363.78 tok/s
prose 4-request rate sum
125.32 tok/s
prose 8-request rate sum
213 tok/s
code single stream ttft
0.317 s
code single stream completion tokens
244
prefill input tokens
6,031
wave requests
26
wave successes
26
prefill requests
1
http errors
0

Measured historical speed probe on the DGX Spark + ASUS GX10 pair. Every wave request and the separate prefill probe returned HTTP 200. Code was not executed and prose was not scored. A speed probe is not a model promotion or full-context qualification.

Protocol: One prose prompt and one Python merge-function prompt; one wave each at concurrency 1, 4 and 8. Temperature 0, requested reasoning_effort medium, streaming with usage, wave output cap 320. TTFT includes first content or reasoning. Per-request decode = reported completion positions / elapsed time after first output. Concurrent means average these request rates; rate sums add them and are not wave-wall throughput. The separate ~6K input probe used output cap 16.

Limits: One wave per workload/concurrency, no repeat spread. Dates, model tokenizers and output lengths differ; exact runtime revision, effective thinking policy, cache state and external contention were not independently frozen. The prefill probe does not qualify 262K retrieval. No executed correctness, tool suite, or writing-quality result.

Source: Retained lab speed report · October 5, 16:41:01–16:41:49 CDT

Runtime: vLLM two-node TP2; exact probe revision not retained
Source record updated 2026-10-10

Read the decision ↗
Catalog estimate

roster catalog · on-demand agent-gemma

agent-gemma · Gemma-4-26B-A4B QAT

Historical lab pool

catalog
65tok/s
All measurements & context
catalog
65 tok/s

UI catalog copy only. On-demand llama-swap id, ttl 600, 160k. Do not read as live tok/s. Dated / historical / not-live.

Source: Coordinator roster catalog (text extract). Not the Oct 1 idle probe.

Runtime: UI catalog copy
Source record updated 2026-09-12

Catalog estimate

roster catalog · on-demand omni

omni · Nemotron 3 Nano Omni 30B-A3B

Historical lab pool

catalog
60tok/s
All measurements & context
catalog
60 tok/s

UI catalog copy only. On-demand multimodal OCR/GUI lane, ttl 600, 256k. Do not read as live tok/s. Dated / historical / not-live.

Source: Coordinator roster catalog (text extract). Not the Oct 1 idle probe.

Runtime: UI catalog copy
Source record updated 2026-09-12

Measured studyINCOMPLETE RUN

Aider polyglot · incomplete Flash-Next run

MiaAI Qwen3.8-Flash-Next · NVFP4 (September artifact)

Historical lab pool

within two tries passed
58
tasks recorded
94
suite tasks
225
All measurements & context
within two tries passed
58
tasks recorded
94
suite tasks
225
tasks unrecorded
131
first try passed
28
test timeouts
0
malformed response counter
1
context exhaustion counter
72
request error output counter
73
cpp tasks recorded
12
cpp first try passed
1
cpp within two tries passed
5
go tasks recorded
11
go first try passed
4
go within two tries passed
6
java tasks recorded
22
java first try passed
6
java within two tries passed
15
javascript tasks recorded
24
javascript first try passed
8
javascript within two tries passed
16
python tasks recorded
14
python first try passed
4
python within two tries passed
10
rust tasks recorded
11
rust first try passed
5
rust within two tries passed
6

Measured partial archive: 94/225 task result files, 28 first-try passes and 58 passes within two tries among the recorded tasks. No final suite percentage or winner is reported. The missing 131 tasks are unrecorded, not imputed failures. This is the September MiaAI artifact, not the October NVIDIA serving artifact.

Protocol: 225 Exercism tasks across six languages; whole-file editing; temperature 0; one worker; up to two tries with test feedback. Thinking on. Run recipe specifies 65536 context; metadata lists 262144 input and 32768 output. Per-language pass counts are preserved.

Limits: Effective chat-history cap is not independently frozen; earlier notes differ. Harness context-exhaustion, malformed-response and request-error-output counters are event counters, not counts of distinct failed tasks or HTTP requests. Incomplete run with different artifact, thinking policy and infrastructure; observed pass counts cannot support a final 225-task rate.

Source: Aggregated retained per-exercise Aider result files and local run recipe. Run start September 26; archive reviewed October 10, 2026.

Runtime: vLLM two-GB10 · Aider whole-file editing
Source record updated 2026-10-10

Read the decision ↗
Measured study

Aider polyglot · complete 225-task run

Qwen3.8-27B · NVFP4 + DFlash2

NVIDIA DGX Spark

within two tries
49.8%
first try
9.3%
tasks recorded
225
All measurements & context
within two tries
49.8 %
first try
9.3 %
tasks recorded
225
suite tasks
225
first try passed
21
within two tries passed
112
test timeouts
2
malformed response counter
0
context exhaustion counter
0
request error output counter
0
cpp tasks recorded
26
cpp first try passed
2
cpp within two tries passed
19
go tasks recorded
39
go first try passed
5
go within two tries passed
19
java tasks recorded
47
java first try passed
1
java within two tries passed
21
javascript tasks recorded
49
javascript first try passed
3
javascript within two tries passed
26
python tasks recorded
34
python first try passed
4
python within two tries passed
14
rust tasks recorded
30
rust first try passed
6
rust within two tries passed
13

Measured full recorded run: 21 first-try passes and 112 passes within two tries across all 225 exercises. EQ-Bench long-form generation for this arm was also recorded as completed with judging skipped; it provides no writing-quality score.

Protocol: 225 Exercism tasks across six languages; whole-file editing; temperature 0; one worker; up to two tries with test feedback. Thinking off. Run recipe specifies 65536 context; metadata lists 262144 input and 32768 output. Per-language pass counts are preserved.

Limits: Effective chat-history cap is not independently frozen; earlier notes differ. Harness context-exhaustion, malformed-response and request-error-output counters are event counters, not counts of distinct failed tasks or HTTP requests. One local run, without repeat spread or a matched completed Flash result.

Source: Aggregated retained per-exercise Aider result files and local run recipe. Run start September 26; archive reviewed October 10, 2026.

Runtime: SGLang + DFlash2 · Aider whole-file editing
Source record updated 2026-10-10

Read the decision ↗
Measured study

Mac Studio · writing-specialist evaluation

Gemma-4 26B-A4B QAT · Q4_K_M

Mac Studio

code decode
78.1tok/s
prose decode
76.2tok/s
framework
84.3%
All measurements & context
code decode
78.1 tok/s
prose decode
76.2 tok/s
framework
84.3 %
broad tools
94.1 %
developer tools
66.7 %
prose constraints
97 %

Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval was not run for this writing role; no coding benchmark score is inferred. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.

Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.

Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.

Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.

Runtime: llama.cpp Metal · build 10964
Source record updated 2026-09-23

Read the decision ↗
Measured study

Mac Studio · fast-agent evaluation

Nemotron-3.5-Lightning 30B-A3B · MLX 4-bit

Mac Studio

code decode
150.1tok/s
prose decode
150.4tok/s
framework
71.5%
All measurements & context
code decode
150.1 tok/s
prose decode
150.4 tok/s
framework
71.5 %
broad tools
88.2 %
developer tools
66.7 %
prose constraints
73.5 %
humaneval passed
140
humaneval tasks
164
humaneval
85.4 %

Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval: all 164 original tasks; one greedy chat sample; 2048-output cap; thinking off; original sandboxed tests; no repair or retries. Chat-adapted pass@1, not a leaderboard reproduction. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.

Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.

Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.

Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.

Runtime: MLX / MLX-VLM
Source record updated 2026-09-23

Read the decision ↗
Measured study

Mac Studio · coding evaluation

Qwen3.8-27B · MLX 4-bit

Mac Studio

code decode
33tok/s
prose decode
33tok/s
framework
89.5%
All measurements & context
code decode
33 tok/s
prose decode
33 tok/s
framework
89.5 %
broad tools
100 %
developer tools
50 %
prose constraints
87.2 %
humaneval passed
156
humaneval tasks
164
humaneval
95.1 %

Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval: all 164 original tasks; one greedy chat sample; 2048-output cap; thinking off; original sandboxed tests; no repair or retries. Chat-adapted pass@1, not a leaderboard reproduction. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.

Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.

Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.

Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.

Runtime: MLX / MLX-VLM
Source record updated 2026-09-23

Read the decision ↗
Catalog estimate

roster catalog · resident agent

agent / lightning · Nemotron-3.5-Lightning 30B-A3B

Historical lab pool

catalog low
80tok/s
catalog high
82tok/s
All measurements & context
catalog low
80 tok/s
catalog high
82 tok/s

UI catalog copy only. Coordinator roster still prints ~80–82 tok/s for resident agent / lightning. Oct 1 live snapshot saw this selector unloaded. Do not read these figures as live tok/s.

Source: Coordinator roster + benches catalog text, page last-modified 2026-09-12

Runtime: UI catalog copy
Source record updated 2026-09-12

Measured study

Sep 12 four concurrent streams (c4 aggregate)

Lightning 30B-A3B · Qwen3.8-27B + DFlash2 · Mistral Small 4

Historical lab pool

lightning 4-stream aggregate
169tok/s
qwen38 27b 4-stream aggregate
42.7tok/s
mistral 4-stream aggregate
75.9tok/s
All measurements & context
lightning 4-stream aggregate
169 tok/s
qwen38 27b 4-stream aggregate
42.7 tok/s
mistral 4-stream aggregate
75.9 tok/s

Historical Sep 12 table. Lightning 2.2× Mistral at c4. Four concurrent streams. Not re-measured on 2026-10-01.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: vLLM 0.29 / SGLang · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Archived coder gate · baseline-lightning-v029

Nemotron-3.5-Lightning 30B-A3B · NVFP4 + DSpark

Historical lab pool

framework
80.3%
developer tools
75%
single stream
77.6tok/s
All measurements & context
framework
80.3 %
developer tools
75 %
single stream
77.6 tok/s
2-stream aggregate
120.4 tok/s
4-stream aggregate
169 tok/s
four request mean decode
45.8 tok/s
framework wall
42 s
gate first response
0 s
bash framework
88.9 %
nuxt framework
85.7 %
repo framework
75 %
swift framework
74.1 %
tailwind framework
44.4 %
vue framework
100 %

Measured historical gate. Clean September rebaseline. Historical agent, superseded later.

Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.

Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.

Source: Retained numeric coder-gate summary, 2026-09-12; reviewed October 10, 2026.

Runtime: vLLM 0.29.0
Source record updated 2026-10-10

Read the decision ↗
Measured study

Archived coder gate · baseline-qwen38-sglang-v255

Qwen3.8-27B · NVFP4 + DFlash2

Historical lab pool

framework
85.9%
developer tools
75%
single stream
19.6tok/s
All measurements & context
framework
85.9 %
developer tools
75 %
single stream
19.6 tok/s
2-stream aggregate
31.5 tok/s
4-stream aggregate
42.7 tok/s
four request mean decode
22.6 tok/s
framework wall
154 s
gate first response
1 s
bash framework
77.8 %
nuxt framework
100 %
repo framework
100 %
swift framework
74.1 %
tailwind framework
77.8 %
vue framework
91.7 %

Measured historical gate. Clean September rebaseline. Historical coder, superseded later.

Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.

Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.

Source: Retained numeric coder-gate summary, 2026-09-12; reviewed October 10, 2026.

Runtime: SGLang Qwen3.8 image · retained gate revision
Source record updated 2026-10-10

Read the decision ↗
Measured study

Sep 12 generic generation (c1 decode)

Lightning 30B-A3B · Qwen3.8-27B + DFlash2 · Mistral Small 4

Historical lab pool

lightning single stream
77.6tok/s
qwen38 27b single stream
19.6tok/s
mistral single stream
30.7tok/s
All measurements & context
lightning single stream
77.6 tok/s
qwen38 27b single stream
19.6 tok/s
mistral single stream
30.7 tok/s

Historical Sep 12 2026 production rebaseline. Lightning 2.5× Mistral at c1. Not live telemetry. Live generation on 2026-10-01 was idle (0 tok/s).

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: vLLM 0.29 / SGLang · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Sep 12 Lightning long-form generation

Lightning 30B-A3B · long-form pass

Historical lab pool

decode
85.9tok/s
ttft
0.13s
essay words
3,935
All measurements & context
decode
85.9 tok/s
ttft
0.13 s
essay words
3,935

Historical long-form pass cited on the Sep 12 2026 rebaseline. A requested 10K-word essay stopped at 3,935 words. Not live telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Sep 12 Mistral quality watch (unrouted)

Mistral Small 4 119B-A6B NVFP4

Historical lab pool

broad tools
100%
prose constraints
86.9%
devcode
81.4%
All measurements & context
broad tools
100 %
prose constraints
86.9 %
devcode
81.4 %
devtool
83.3 %
cold ready
674 s
cached ready
585 s

Historical fast gate. Perfect broad tools and stronger prose than Lightning (63.8%). Fits TP1 at 256K (66.12 GiB weights + 27.33 GiB KV) but ten-minute reload keeps it an unrouted specialist. Retrieval was not needle-tested. Not live telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: vLLM 0.29 · one Spark · 256K
Source record updated 2026-09-12

Read the decision ↗
Measured study

Sep 12 role-split quality gate

Lightning 30B-A3B · Qwen3.8-27B + DFlash2 · Mistral Small 4

Historical lab pool

lightning devcode
80.3%
qwen devcode
85.9%
mistral devcode
81.4%
All measurements & context
lightning devcode
80.3 %
qwen devcode
85.9 %
mistral devcode
81.4 %
lightning broad tools
82.4 %
qwen broad tools
94.1 %
mistral broad tools
100 %

Historical. Keep the split: Lightning owns latency, Qwen owns code (perfect multi-turn and restraint), Mistral stays an unrouted quality watch. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: vLLM 0.29 / SGLang · non-reasoning · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Archived coder gate · qwen38-dflash-mf082

Qwen3.8-27B · NVFP4 + DFlash2

Historical lab pool

framework
92%
developer tools
50%
single stream
22.9tok/s
All measurements & context
framework
92 %
developer tools
50 %
single stream
22.9 tok/s
2-stream aggregate
42.8 tok/s
4-stream aggregate
43.3 tok/s
four request mean decode
21.2 tok/s
framework wall
106 s
gate first response
2 s
bash framework
100 %
nuxt framework
100 %
repo framework
100 %
swift framework
87.5 %
tailwind framework
77.8 %
vue framework
91.7 %

Measured historical gate. Single August gate; 92% is not a repeat median or a new universal champion score.

Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.

Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.

Source: Retained numeric coder-gate summary, 2026-08-19; reviewed October 10, 2026.

Runtime: SGLang · retained DFlash gate variant
Source record updated 2026-10-10

Read the decision ↗
Measured study

Archived dual-role gate · Qwen3.8 FP8

Qwen3.8-27B · official FP8

Historical lab pool

framework
87.5%
developer tools
50%
broad tools thinking off
100%
All measurements & context
framework
87.5 %
developer tools
50 %
broad tools thinking off
100 %
broad tools thinking on
100 %
developer perception facts
30
developer perception total
30
vision
62.5 %
vision tools
50 %
prose constraints
77.8 %
single stream
8 tok/s
2-stream aggregate
16 tok/s
4-stream aggregate
31.2 tok/s
four request mean decode
7.8 tok/s
framework wall
1,072 s
cold ready
433 s
allocated
91.8 GiB

Measured historical gate; rejected for latency and readiness despite strong broad tools and developer screenshot facts. Configured 262144 context was not needle-verified.

Protocol: Custom framework/tool/vision/writing suites plus warm prose concurrency waves. Broad tools were separately checked with thinking off and on. Context configuration and measured cold readiness are separate fields.

Limits: No executable framework validation. No full-window retrieval test. Different protocol axes cannot be combined into one universal model score.

Source: Retained August 14 dual-role numeric summary and archived model registry; reviewed October 10, 2026.

Runtime: vLLM 0.27.1
Source record updated 2026-10-10

Read the decision ↗
Measured study

Archived coder gate · lightning-gate

Nemotron-3.5-Lightning 30B-A3B · NVFP4 + DSpark

Historical lab pool

framework
70%
developer tools
66.7%
single stream
82.3tok/s
All measurements & context
framework
70 %
developer tools
66.7 %
single stream
82.3 tok/s
2-stream aggregate
108.5 tok/s
4-stream aggregate
161.5 tok/s
four request mean decode
45.7 tok/s
framework wall
44 s
gate first response
1 s
bash framework
66.7 %
nuxt framework
100 %
repo framework
50 %
swift framework
60.7 %
tailwind framework
55.6 %
vue framework
83.3 %

Measured historical gate. Earlier isolated gate. September rebaseline is a separate dated measurement, not a replacement for this result.

Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.

Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.

Source: Retained numeric coder-gate summary, 2026-08-11; reviewed October 10, 2026.

Runtime: vLLM · August gate revision not frozen
Source record updated 2026-10-10

Read the decision ↗
Measured study

Aug 11 Lightning coder gate

Nemotron-3.5-Lightning 30B-A3B NVFP4 + DSpark vs Coder-Next

Historical lab pool

lightning devcode
70%
champion devcode
86.4%
lightning single stream
82.3tok/s
All measurements & context
lightning devcode
70 %
champion devcode
86.4 %
lightning single stream
82.3 tok/s
champion single stream
50.9 tok/s
lightning 4-stream aggregate agg
161.5 tok/s
champion 4-stream aggregate agg
76 tok/s

Historical. Speed win (1.62× c1) and concurrency scales; quality −16.4 outside ±4 noise. Rejected as primary coder; kept as on-demand lightning. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: vLLM 0.27.1 · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Aug 10 Muse Glimmer dual-role reject

Meta Muse Glimmer 30B vs Coder-Next / Gemma agent

Historical lab pool

muse single stream
10.1tok/s
champion single stream
50.9tok/s
gemma single stream
60.4tok/s
All measurements & context
muse single stream
10.1 tok/s
champion single stream
50.9 tok/s
gemma single stream
60.4 tok/s
muse devcode
81.5 %
muse devtool
25 %
muse toolbench
91.2 %

Historical. Rejected for both fleet roles. Tools and sidecar quality were real; decode 0.20× the coder champion and 0.17× Gemma. Native context 131K fails the verified-256K coder gate. Not added to llama-swap. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: llama.cpp side image · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Archived coder gate · ds4-iq2m

DeepSeek-V4-Flash · quant identity unconfirmed

Historical lab pool

framework
84.8%
developer tools
41.7%
single stream
12.4tok/s
All measurements & context
framework
84.8 %
developer tools
41.7 %
single stream
12.4 tok/s
2-stream aggregate
12.7 tok/s
4-stream aggregate
16.5 tok/s
four request mean decode
17 tok/s
framework wall
371 s
gate first response
1 s
bash framework
88.9 %
nuxt framework
85.7 %
repo framework
100 %
swift framework
68.8 %
tailwind framework
66.7 %
vue framework
100 %

Measured historical gate. Measured report retained under its historical label. The label alone does not establish the quant. Do not transfer these quality scores to the verified August 2 UD-IQ2_XXS throughput-only artifact.

Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.

Limits: Quant identity is unresolved. Score attribution stays with the retained report label. No code compilation, tool execution or full-context qualification in this gate; one report does not establish repeat spread.

Source: Retained numeric coder-gate summary, 2026-08-03; reviewed October 10, 2026.

Runtime: Retained gate label ds4-iq2m; exact artifact not frozen
Source record updated 2026-10-10

Read the decision ↗
Measured study

Archived coder gate · coder-laguna-xs

Poolside Laguna-XS-2.1 33B · NVFP4

Historical lab pool

framework
83.9%
developer tools
50%
single stream
35.3tok/s
All measurements & context
framework
83.9 %
developer tools
50 %
single stream
35.3 tok/s
2-stream aggregate
70.1 tok/s
4-stream aggregate
127.7 tok/s
four request mean decode
32.3 tok/s
framework wall
141 s
gate first response
200 s
bash framework
88.9 %
nuxt framework
85.7 %
repo framework
100 %
swift framework
73.2 %
tailwind framework
66.7 %
vue framework
91.7 %

Measured historical gate. Rejected: slow cold readiness and single-stream latency. Four-slot vLLM wave is not a model-only comparison with two-slot llama.cpp.

Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.

Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.

Source: Retained numeric coder-gate summary, 2026-07-30; reviewed October 10, 2026.

Runtime: vLLM · exact revision not frozen
Source record updated 2026-10-10

Read the decision ↗
Measured study

Archived coder gate · coder-ornith

Ornith-1.0-35B · UD-Q4_K_XL

Historical lab pool

framework
84.2%
developer tools
75%
single stream
61.6tok/s
All measurements & context
framework
84.2 %
developer tools
75 %
single stream
61.6 tok/s
2-stream aggregate
93.9 tok/s
4-stream aggregate
94.2 tok/s
four request mean decode
48.1 tok/s
framework wall
115 s
gate first response
27 s
bash framework
88.9 %
nuxt framework
100 %
repo framework
100 %
swift framework
53.5 %
tailwind framework
77.8 %
vue framework
100 %

Measured historical gate. One retained run. Five-run framework median 84.2%, Swift median 60.7% and framework standard deviation 5.55 points are separate summaries.

Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.

Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.

Source: Retained numeric coder-gate summary, 2026-07-30; reviewed October 10, 2026.

Runtime: llama.cpp · tuned gate
Source record updated 2026-10-10

Read the decision ↗
Measured study

Archived coder gate · coder-qwen36

Qwen3.6-35B-A3B · UD-Q4_K_XL

Historical lab pool

framework
81.7%
developer tools
75%
single stream
62.4tok/s
All measurements & context
framework
81.7 %
developer tools
75 %
single stream
62.4 tok/s
2-stream aggregate
94.7 tok/s
4-stream aggregate
95.7 tok/s
four request mean decode
49 tok/s
framework wall
112 s
gate first response
17 s
bash framework
100 %
nuxt framework
85.7 %
repo framework
100 %
swift framework
61.6 %
tailwind framework
44.4 %
vue framework
100 %

Measured historical gate. Original framework 75.2%; tuned rematch 81.7%. Historical rejection remained.

Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.

Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.

Source: Retained numeric coder-gate summary, 2026-07-30; reviewed October 10, 2026.

Runtime: llama.cpp · tuned rematch
Source record updated 2026-10-10

Read the decision ↗
Measured study

Jul 30 SWE-bench leaderboard review (local gate)

Ornith-1.0-35B · Qwen3.6-35B · Laguna-XS-2.1 · Coder-Next

Historical lab pool

ornith devcode median
84.2%
qwen36 devcode
81.7%
laguna xs devcode
83.9%
All measurements & context
ornith devcode median
84.2 %
qwen36 devcode
81.7 %
laguna xs devcode
83.9 %
coder next devcode
86.4 %
ornith single stream
61.6 tok/s
coder next single stream
50.9 tok/s

Historical. All three published-rank challengers rejected as primary coder. Ornith took the tools-specialist slot; coder-laguna retired. One gate run is not a result — Ornith five-run spread 73.2–87.3. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: dated bench page · idle · thinking off
Source record updated 2026-09-12

Read the decision ↗
Measured study

Archived coder gate · qwen35-27b-base-q4

Qwen3.5-27B · Q4_K_M

Historical lab pool

framework
84.3%
developer tools
75%
single stream
10.8tok/s
All measurements & context
framework
84.3 %
developer tools
75 %
single stream
10.8 tok/s
2-stream aggregate
10.9 tok/s
4-stream aggregate
11 tok/s
four request mean decode
11.1 tok/s
framework wall
484 s
gate first response
1 s
bash framework
88.9 %
nuxt framework
100 %
repo framework
50 %
swift framework
73.2 %
tailwind framework
88.9 %
vue framework
100 %

Measured historical gate. Matched base for Qwythos: no meaningful quality or throughput gain from the challenger.

Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.

Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.

Source: Retained numeric coder-gate summary, 2026-07-29; reviewed October 10, 2026.

Runtime: llama.cpp · matched base A/B
Source record updated 2026-10-10

Read the decision ↗
Measured study

Archived coder gate · qwythos-27b-q4

Empero Qwythos-27B-v1 · Q4_K_M

Historical lab pool

framework
84.6%
developer tools
75%
single stream
11tok/s
All measurements & context
framework
84.6 %
developer tools
75 %
single stream
11 tok/s
2-stream aggregate
11.1 tok/s
4-stream aggregate
11.1 tok/s
four request mean decode
11.2 tok/s
framework wall
407 s
gate first response
1 s
bash framework
88.9 %
nuxt framework
100 %
repo framework
75 %
swift framework
73.2 %
tailwind framework
66.7 %
vue framework
100 %

Measured historical gate. Framework and tools within the matched base result; rejected for the historical coder/agent roles.

Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.

Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.

Source: Retained numeric coder-gate summary, 2026-07-29; reviewed October 10, 2026.

Runtime: llama.cpp · matched base A/B
Source record updated 2026-10-10

Read the decision ↗
Measured study

Archived coder gate · ub2048

Qwen3-Coder-Next 80B-A3B · UD-Q4_K_XL

Historical lab pool

framework
87.8%
developer tools
66.7%
single stream
51.1tok/s
All measurements & context
framework
87.8 %
developer tools
66.7 %
single stream
51.1 tok/s
2-stream aggregate
77 tok/s
4-stream aggregate
78.2 tok/s
four request mean decode
40 tok/s
framework wall
88 s
gate first response
1 s
bash framework
88.9 %
nuxt framework
100 %
repo framework
100 %
swift framework
67.8 %
tailwind framework
77.8 %
vue framework
100 %

Measured historical gate. Single tuning gate. Six-run calibration median 86.4% is a separate summary.

Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.

Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.

Source: Retained numeric coder-gate summary, 2026-07-27; reviewed October 10, 2026.

Runtime: llama.cpp · batch tuning gate at ubatch 2048
Source record updated 2026-10-10

Read the decision ↗
Measured study

Jul 26 Coder-Next restore / Laguna demote

Laguna S 2.1 NVFP4 vs Qwen3-Coder-Next UD-Q4_K_XL

Historical lab pool

laguna single stream
17.7tok/s
coder next single stream
52.7tok/s
laguna devcode
87.2%
All measurements & context
laguna single stream
17.7 tok/s
coder next single stream
52.7 tok/s
laguna devcode
87.2 %
coder next devcode
90.9 %
laguna wall
435.7 s
coder next wall
82 s

Historical same-day A/B. Coder-Next 2.98× decode and 5.3× wall; 47 GiB reclaimed (101 vs 53.9 GiB). Laguna kept on demand for tools 75.0% then retired July 30. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: same-node A/B · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Operations record

coder request log (ops summary)

coder ops note (post Coder-Next revert)

Historical lab pool

requests
507
errors
0
avg
59.3tok/s
All measurements & context
requests
507
errors
0
avg
59.3 tok/s
longest turn
123 s

Historical coder ops note from the July 26 revert write-up on the benches page: 507 requests, 0 errors, 59.3 tok/s average, longest turn 123 s. Not a live counter.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: dated bench page
Source record updated 2026-09-12

Measured study

Jul 22 image-gen gate

Mage-Flow Turbo 4B · Z-Image-Turbo · Sana Sprint 1.6B

Historical lab pool

mage warm
3.96s
mage peak
17.03GiB
zimage warm
5.7s
All measurements & context
mage warm
3.96 s
mage peak
17.03 GiB
zimage warm
5.7 s
zimage peak
21.69 GiB
sana warm
0.79 s
sana peak
10.19 GiB

Historical. Mage was the only tested model combining high visual quality, sub-10 s 1024², and under-20 GiB peak. Not uncensored (life-drawing refusal). Later moved on demand — warm ~4 s, cold ~120 s. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: local 1024² pipeline · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Jul 21 two deep coder slots

Qwen3-Coder-Next · 2 × 262144

Historical lab pool

single
52.6tok/s
aggregate
76tok/s
warm turn
4.3s
All measurements & context
single
52.6 tok/s
aggregate
76 tok/s
warm turn
4.3 s
retrieval tokens
241,000

Historical qualification. Both deep sessions retrieved correctly near 241K. About 55.2 GiB. Warm-session affinity, not a 2× cold accelerator. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: llama.cpp · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Jul 21 Laguna S first reject

Laguna S 2.1 NVFP4 vs Qwen3-Coder-Next

Historical lab pool

laguna code
79.7%
coder next code
87.8%
laguna single stream
18.6tok/s
All measurements & context
laguna code
79.7 %
coder next code
87.8 %
laguna single stream
18.6 tok/s
coder next 4-stream aggregate agg
109.9 tok/s
laguna 4-stream aggregate agg
57.8 tok/s

Historical. Not a quality or throughput upgrade. Native TP=2 deadlocked on 512-token generations. DFlash depth 15 cut four-way throughput 57.8 → 46.2 tok/s. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: dated bench page · incl. TP=2
Source record updated 2026-09-12

Read the decision ↗
Measured study

Jul 21 three independent coder lanes

three-lane coder dispatcher

Historical lab pool

aggregate
99.6tok/s
per stream low
42tok/s
per stream high
56.8tok/s
All measurements & context
aggregate
99.6 tok/s
per stream low
42 tok/s
per stream high
56.8 tok/s
http 409s
0

Historical. Two coding-peer slots plus one local lane. Four concurrent 256-token streams; excess queued, no 409s. Session affinity keeps cache local. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: capacity-aware proxy · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Jul 19 multimodal creative agent bakeoff

Gemma-4-26B-A4B QAT · always-on agent

Historical lab pool

single
60.4tok/s
four way
129.9tok/s
tools
91.2%
All measurements & context
single
60.4 tok/s
four way
129.9 tok/s
tools
91.2 %
blind score
8.35 /10
warm prefix
0.303 s
deep edge
78.3 s

Historical. Gemma kept as always-on agent. Goetia and Fable rejected. Warm 17K prefix 0.303 s (23.2× over cold). Deep edge 78.3 s · 119,593 prompt tokens. Catalog ~65 tok/s on the later agent-gemma on-demand id is a separate roster figure. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: llama.cpp · four 163840 slots · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Speculative decoding · code versus creative writing

Qwen3.6-35B-A3B · Q6_K · native MTP

NVIDIA DGX Spark

code off
64.1tok/s
code on
100.9tok/s
creative off
64.6tok/s
All measurements & context
code off
64.1 tok/s
code on
100.9 tok/s
creative off
64.6 tok/s
creative on
53.6 tok/s

Historical local A/B with six draft positions. Code used temperature 0.3; creative writing used 0.7. MTP helped code but hurt creative throughput. This does not qualify a different model, runtime, or sampling policy.

Protocol: Same model and quiet hardware; one slot. Workload and sampling settings are explicitly different between the code and creative probes.

Limits: Repeat count and spread were not recorded in the reviewed summary; do not treat this as a universal speedup.

Source: Local Spark tuning review, measured 2026-07-11; reviewed directly 2026-10-02.

Runtime: llama.cpp b9967 · single-slot quiet-box comparison
Source record updated 2026-07-11

Read the decision ↗