Measured studies, operations records, and catalog estimates are separate kinds of evidence. Explore the original dates, runtimes, units, and decisions behind the configuration.
Normal DeepSeek V4.1 Flash 1017 runs through TensorFold TP2 on the DGX Spark and ASUS GX10. The matched GLM comparison, short serving probes and memory observation answer different questions.
Shared inference slots
4
Coder, scout and vision share this capacity. Studio agent and embeddings keep independent routes.
Configured context
524,288
A serving limit. Local full-window retrieval and concurrent long-context capacity remain unqualified.
Initial memory minima / GiB
4.93 / 6.55
Head / worker available memory across one hour, 1,800 samples each. A bounded observation; guards remain active.
October 22–23 · Same frozen primary Hermes protocol, temperature 0.2, thinking off, 4,096 output allowance. Coding handoffs and diagnostic decisions keep separate denominators.
One routine task, three hard tasks, one visit each, one harness. Behavior, scope, report, exact identity and cleanup are required. This local panel is not an official SWE-bench score. Later production high-reasoning tuning is separate; no difficult-coding advantage was demonstrated.
M5 Max · 40 GPU cores · 48 GB · Sep 22–23, 2026 · thinking off. Lightning here is the evaluated Nemotron artifact. The current SuperQwen alias has no score in this chart.
HumanEval / % ↑
Code decode / tok/s →
Coding fallback / MLX 4-bit
95.1%
HumanEval · 156 / 164 tasks passed
All 164 original tasks, one greedy chat sample, original executable tests, no repair or retries. Chat-adapted pass@1; training contamination is possible. Gemma was a writing-role evaluation and did not run this suite.
Speed: median of three short 512-output code probes on a loaded backend. Timing and quality use different prompts; this is not full-task wall time.
Short native capacity probe · 1, 2 and 4 concurrent requests
DeepSeek V4.1 Flash 1017 · EXL3 · normal
NVIDIA DGX Spark
single stream
41.16tok/s
single stream output tokens
160
single stream wave wall
3.887s
All measurements & context
single stream
41.16 tok/s
single stream output tokens
160
single stream wave wall
3.887 s
2-stream aggregate
75.777 tok/s
2-stream aggregate output tokens
307
2-stream aggregate wave wall
4.051 s
4-stream aggregate
91.568 tok/s
4-stream aggregate output tokens
598
4-stream aggregate wave wall
6.531 s
prompt tokens
2,787
later cached tokens per request
2,048
requests completed
7
requests attempted
7
Measured aggregate output divided by complete wave wall time: about 41.16, 75.78 and 91.57 tok/s at 1, 2 and 4 requests. Includes prefill; not pure decode speed or a quality score.
Protocol: One sequential wave at each concurrency. Temperature zero, thinking disabled, output cap 160. Each request had 2,787 prompt positions. The first wave was uncached; all requests in the later waves reported 2,048 cached positions. Sum each wave’s completion positions and divide by the measured wave wall seconds. Record stop versus length termination and verify the exact native model identity for all seven requests.
Limits: Tiny synthetic workload, seven requests and no repeat spread. Warm/cache state differs between waves; several responses hit the output cap. Scaling is not a cold-cache concurrency comparison. No correctness grading, long-context retrieval or concurrent full-window soak. The earlier October harness’s sums of individual decode rates use a different denominator.
Matched local repository handoff panel · coding and review controls
DeepSeek V4.1 Flash 1017 · EXL3 · normal
NVIDIA DGX Spark
routine passed
1
routine tasks
1
hard passed
0
All measurements & context
routine passed
1
routine tasks
1
hard passed
0
hard tasks
3
defective review correct
1
scope admission correct
1
healthy review correct
1
routine wall
74.698 s
menu behavior wall
350.188 s
strict types wall
412.824 s
socket repair wall
814.588 s
Measured local portfolio panel. Routine coding and hard coding have separate denominators; admission, defective-change review and healthy-change review are separate diagnostic controls. Both models tied on coding; no difficult-coding advantage was demonstrated.
Protocol: Same frozen tasks, sources, sampling, budgets and independent acceptance gates. Temperature 0.2, thinking disabled, 4,096 output allowance per request. Routine: 360 seconds / 32 requests; hard visits: 1,440 seconds / 96 requests including repairs and reporting. Acceptance requires behavior, allowed scope, structured report, exact native identity and verified cleanup. Each control is one decision, encoded 1 for correct and 0 for incorrect. One visit per case; the healthy-menu positive control is reported separately from the six-case panel.
Limits: One primary harness, one routine coding task, three hard coding tasks, and no statistical ranking. This is not an official SWE-bench Verified result. Wall time includes tools and cache effects. Normal DeepSeek is compared with the previously serving abliterated GLM checkpoint. Later high-reasoning production tuning and historical mixed-revision controls are not pooled into this comparison. Public commands passing alone do not override behavior or scope failures.
Source: Dated frozen primary Hermes comparison receipts · October 22–23, 2026; native identities and source fingerprints checked
One-hour initial memory observation · both GB10 ranks
DeepSeek V4.1 Flash 1017 · EXL3 · normal
NVIDIA DGX Spark
head min available
4.927GiB
worker min available
6.546GiB
head samples
1,800
All measurements & context
head min available
4.927 GiB
worker min available
6.546 GiB
head samples
1,800
worker samples
1,800
runtime guard floor
2.5 GiB
head recorded supervisor restarts
0
Operations observation covering initial validation, cache replay, benchmark visits and early live work. Minimum sampled MemAvailable: head 4.93 GiB, worker 6.55 GiB. No recorded OOM or supervisor restart during this bounded observation.
Protocol: 1,800 samples on each host over roughly one hour: head 08:02:22–09:02:22 UTC and worker 08:02:24–09:02:23 UTC. Report minimum host MemAvailable separately, retaining unchanged recipe admission, reserve and runtime guards. Samplers stopped after this window.
Limits: Not a controlled capacity experiment, a leak diagnosis, days of uptime, or a heavy concurrent long-context soak. Sampled minima can miss brief peaks; head and worker minima occurred at different instants. Swap occupancy is not an active paging rate.
Source: Dated bounded memory sampler and runtime receipts · October 23, 03:02–04:02 CDT
Runtime: TensorFold native · two-node TP2 Source record updated 2026-10-23
Held-history compatibility replay · cold and warm requests
DeepSeek V4.1 Flash 1017 · EXL3 · normal
NVIDIA DGX Spark
prompt tokens
63,164
history messages
220
complete tool pairs
110
All measurements & context
prompt tokens
63,164
history messages
220
complete tool pairs
110
cold wall
43.9 s
warm wall
8.8 s
warm cached tokens
61,440
structured responses passed
2
requests attempted
2
Measured compatibility/cache observation. Two high-reasoning requests used a copy of an existing 220-message history with 110 complete tool-call/result pairs. Both returned valid structured responses; tools were not executed.
Protocol: Replay the same copied history twice against the new native model without changing the original held session or executing proposed tools. Record total prompt length, response structure, request wall time and second-request cache usage. The warm request reported 61,440 cached positions out of 63,164 input positions.
Limits: One copied conversation and two requests, with no repeat spread, task-success grading or retrieval test. These wall times are not TTFT measurements. Cache reuse does not qualify the configured 524,288-position window, other conversations, or simultaneous full-context requests. Private history text is not published.
Source: Dated copied-history compatibility receipts · October 23, before the 03:57 CDT loop resume
Runtime: TensorFold native · TP2 · high reasoning Source record updated 2026-10-23
Matched local repository handoff panel · coding and review controls
GLM-5.3-Flash · EXL3 · abliterated
NVIDIA DGX Spark
routine passed
1
routine tasks
1
hard passed
0
All measurements & context
routine passed
1
routine tasks
1
hard passed
0
hard tasks
3
defective review correct
0
scope admission correct
1
healthy review correct
1
routine wall
63.487 s
menu behavior wall
376.562 s
strict types wall
190.034 s
socket repair wall
580.339 s
Measured local portfolio panel. Routine coding and hard coding have separate denominators; admission, defective-change review and healthy-change review are separate diagnostic controls. Both models tied on coding; no difficult-coding advantage was demonstrated.
Protocol: Same frozen tasks, sources, sampling, budgets and independent acceptance gates. Temperature 0.2, thinking disabled, 4,096 output allowance per request. Routine: 360 seconds / 32 requests; hard visits: 1,440 seconds / 96 requests including repairs and reporting. Acceptance requires behavior, allowed scope, structured report, exact native identity and verified cleanup. Each control is one decision, encoded 1 for correct and 0 for incorrect. One visit per case; the healthy-menu positive control is reported separately from the six-case panel.
Limits: One primary harness, one routine coding task, three hard coding tasks, and no statistical ranking. This is not an official SWE-bench Verified result. Wall time includes tools and cache effects. Normal DeepSeek is compared with the previously serving abliterated GLM checkpoint. Later high-reasoning production tuning and historical mixed-revision controls are not pooled into this comparison. Public commands passing alone do not override behavior or scope failures.
Source: Dated frozen primary Hermes comparison receipts · October 22–23, 2026; native identities and source fingerprints checked
Dated operations observation, not a controlled decode benchmark or task-success study. Generation rates are engine-wide mixed-traffic samples. Startup cache capacity is the engine's calculation, not a full-context concurrency test.
Protocol: Fixed interval 2026-10-11 01:00–02:00 UTC = October 10 20:00–21:00 America/Chicago. Parsed 319 head log rows; median over 310 rows with generation > 0. Peak running/waiting/cache values are sampled gauges. Prefix-hit values are counter-derived gauges, not an isolated interval hit rate. Counted ERROR/Traceback, OutOfMemory/out of memory and preempted lines in the same head log. Startup timing is first API argument log to Application startup complete on the same date.
Limits: No prompts or responses published. Irregular log intervals and mixed workloads; sampled maxima may miss peaks. Zero matching head log lines does not certify request correctness, worker or kernel health, or long-context reliability. The startup memory calculation and timing describe one boot. No current throughput test, retrieval qualification, or full-window soak was run.
Source: Direct read-only head-container log summary · October 10, 20:00–21:00 CDT; separate October 10 startup log
October short-prompt runtime probe · Qwen / TensorFold
Qwen3.8-Flash-Next · NVFP4
NVIDIA DGX Spark
code single stream
80.09tok/s
prose single stream
50.26tok/s
prefill ttft
2.108s
All measurements & context
code single stream
80.09 tok/s
prose single stream
50.26 tok/s
prefill ttft
2.108 s
code 4-request mean
53.7 tok/s
code 8-request mean
36.51 tok/s
prose 4-request mean
39.36 tok/s
prose 8-request mean
31.8 tok/s
code 4-request rate sum
214.78 tok/s
code 8-request rate sum
292.04 tok/s
prose 4-request rate sum
157.46 tok/s
prose 8-request rate sum
254.43 tok/s
code single stream ttft
0.119 s
code single stream completion tokens
228
prefill input tokens
6,033
wave requests
26
wave successes
26
prefill requests
1
http errors
0
Measured historical speed probe on the DGX Spark + ASUS GX10 pair. Every wave request and the separate prefill probe returned HTTP 200. Code was not executed and prose was not scored. A speed probe is not a model promotion or full-context qualification.
Protocol: One prose prompt and one Python merge-function prompt; one wave each at concurrency 1, 4 and 8. Temperature 0, requested reasoning_effort medium, streaming with usage, wave output cap 320. TTFT includes first content or reasoning. Per-request decode = reported completion positions / elapsed time after first output. Concurrent means average these request rates; rate sums add them and are not wave-wall throughput. The separate ~6K input probe used output cap 16.
Limits: One wave per workload/concurrency, no repeat spread. Dates, model tokenizers and output lengths differ; exact runtime revision, effective thinking policy, cache state and external contention were not independently frozen. The prefill probe does not qualify 262K retrieval. No executed correctness, tool suite, or writing-quality result.
Source: Retained lab speed report · October 9, 22:56:32–22:57:14 CDT
Runtime: TensorFold two-node TP2; exact probe revision not retained Source record updated 2026-10-10
October short-prompt runtime probe · GLM / TensorFold
GLM-5.3-Flash · EXL3
NVIDIA DGX Spark
code single stream
97.59tok/s
prose single stream
43.14tok/s
prefill ttft
8.062s
All measurements & context
code single stream
97.59 tok/s
prose single stream
43.14 tok/s
prefill ttft
8.062 s
code 4-request mean
78.9 tok/s
code 8-request mean
46.2 tok/s
prose 4-request mean
28.06 tok/s
prose 8-request mean
23.49 tok/s
code 4-request rate sum
315.6 tok/s
code 8-request rate sum
369.61 tok/s
prose 4-request rate sum
112.25 tok/s
prose 8-request rate sum
187.92 tok/s
code single stream ttft
0.181 s
code single stream completion tokens
207
prefill input tokens
6,032
wave requests
26
wave successes
26
prefill requests
1
http errors
0
Measured historical speed probe on the DGX Spark + ASUS GX10 pair. Every wave request and the separate prefill probe returned HTTP 200. Code was not executed and prose was not scored. A speed probe is not a model promotion or full-context qualification.
Protocol: One prose prompt and one Python merge-function prompt; one wave each at concurrency 1, 4 and 8. Temperature 0, requested reasoning_effort medium, streaming with usage, wave output cap 320. TTFT includes first content or reasoning. Per-request decode = reported completion positions / elapsed time after first output. Concurrent means average these request rates; rate sums add them and are not wave-wall throughput. The separate ~6K input probe used output cap 16.
Limits: One wave per workload/concurrency, no repeat spread. Dates, model tokenizers and output lengths differ; exact runtime revision, effective thinking policy, cache state and external contention were not independently frozen. The prefill probe does not qualify 262K retrieval. No executed correctness, tool suite, or writing-quality result.
Source: Retained lab speed report · October 5, 18:04:27–18:05:20 CDT
Runtime: TensorFold two-node TP2; exact probe revision not retained Source record updated 2026-10-10
Measured historical speed probe on the DGX Spark + ASUS GX10 pair. Every wave request and the separate prefill probe returned HTTP 200. Code was not executed and prose was not scored. A speed probe is not a model promotion or full-context qualification.
Protocol: One prose prompt and one Python merge-function prompt; one wave each at concurrency 1, 4 and 8. Temperature 0, requested reasoning_effort medium, streaming with usage, wave output cap 320. TTFT includes first content or reasoning. Per-request decode = reported completion positions / elapsed time after first output. Concurrent means average these request rates; rate sums add them and are not wave-wall throughput. The separate ~6K input probe used output cap 16.
Limits: One wave per workload/concurrency, no repeat spread. Dates, model tokenizers and output lengths differ; exact runtime revision, effective thinking policy, cache state and external contention were not independently frozen. The prefill probe does not qualify 262K retrieval. No executed correctness, tool suite, or writing-quality result.
Source: Retained lab speed report · October 5, 16:41:01–16:41:49 CDT
Runtime: vLLM two-node TP2; exact probe revision not retained Source record updated 2026-10-10
Measured partial archive: 94/225 task result files, 28 first-try passes and 58 passes within two tries among the recorded tasks. No final suite percentage or winner is reported. The missing 131 tasks are unrecorded, not imputed failures. This is the September MiaAI artifact, not the October NVIDIA serving artifact.
Protocol: 225 Exercism tasks across six languages; whole-file editing; temperature 0; one worker; up to two tries with test feedback. Thinking on. Run recipe specifies 65536 context; metadata lists 262144 input and 32768 output. Per-language pass counts are preserved.
Limits: Effective chat-history cap is not independently frozen; earlier notes differ. Harness context-exhaustion, malformed-response and request-error-output counters are event counters, not counts of distinct failed tasks or HTTP requests. Incomplete run with different artifact, thinking policy and infrastructure; observed pass counts cannot support a final 225-task rate.
Source: Aggregated retained per-exercise Aider result files and local run recipe. Run start September 26; archive reviewed October 10, 2026.
Measured full recorded run: 21 first-try passes and 112 passes within two tries across all 225 exercises. EQ-Bench long-form generation for this arm was also recorded as completed with judging skipped; it provides no writing-quality score.
Protocol: 225 Exercism tasks across six languages; whole-file editing; temperature 0; one worker; up to two tries with test feedback. Thinking off. Run recipe specifies 65536 context; metadata lists 262144 input and 32768 output. Per-language pass counts are preserved.
Limits: Effective chat-history cap is not independently frozen; earlier notes differ. Harness context-exhaustion, malformed-response and request-error-output counters are event counters, not counts of distinct failed tasks or HTTP requests. One local run, without repeat spread or a matched completed Flash result.
Source: Aggregated retained per-exercise Aider result files and local run recipe. Run start September 26; archive reviewed October 10, 2026.
Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval was not run for this writing role; no coding benchmark score is inferred. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.
Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.
Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.
Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.
Runtime: llama.cpp Metal · build 10964 Source record updated 2026-09-23
Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval: all 164 original tasks; one greedy chat sample; 2048-output cap; thinking off; original sandboxed tests; no repair or retries. Chat-adapted pass@1, not a leaderboard reproduction. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.
Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.
Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.
Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.
Runtime: MLX / MLX-VLM Source record updated 2026-09-23
Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval: all 164 original tasks; one greedy chat sample; 2048-output cap; thinking off; original sandboxed tests; no repair or retries. Chat-adapted pass@1, not a leaderboard reproduction. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.
Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.
Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.
Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.
Runtime: MLX / MLX-VLM Source record updated 2026-09-23
UI catalog copy only. Coordinator roster still prints ~80–82 tok/s for resident agent / lightning. Oct 1 live snapshot saw this selector unloaded. Do not read these figures as live tok/s.
Measured historical gate. Clean September rebaseline. Historical agent, superseded later.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-09-12; reviewed October 10, 2026.
Runtime: vLLM 0.29.0 Source record updated 2026-10-10
Measured historical gate. Clean September rebaseline. Historical coder, superseded later.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-09-12; reviewed October 10, 2026.
Historical fast gate. Perfect broad tools and stronger prose than Lightning (63.8%). Fits TP1 at 256K (66.12 GiB weights + 27.33 GiB KV) but ten-minute reload keeps it an unrouted specialist. Retrieval was not needle-tested. Not live telemetry.
Lightning 30B-A3B · Qwen3.8-27B + DFlash2 · Mistral Small 4
Historical lab pool
lightning devcode
80.3%
qwen devcode
85.9%
mistral devcode
81.4%
All measurements & context
lightning devcode
80.3 %
qwen devcode
85.9 %
mistral devcode
81.4 %
lightning broad tools
82.4 %
qwen broad tools
94.1 %
mistral broad tools
100 %
Historical. Keep the split: Lightning owns latency, Qwen owns code (perfect multi-turn and restraint), Mistral stays an unrouted quality watch. Not live Oct 1 telemetry.
Measured historical gate. Single August gate; 92% is not a repeat median or a new universal champion score.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-08-19; reviewed October 10, 2026.
Measured historical gate; rejected for latency and readiness despite strong broad tools and developer screenshot facts. Configured 262144 context was not needle-verified.
Protocol: Custom framework/tool/vision/writing suites plus warm prose concurrency waves. Broad tools were separately checked with thinking off and on. Context configuration and measured cold readiness are separate fields.
Limits: No executable framework validation. No full-window retrieval test. Different protocol axes cannot be combined into one universal model score.
Source: Retained August 14 dual-role numeric summary and archived model registry; reviewed October 10, 2026.
Runtime: vLLM 0.27.1 Source record updated 2026-10-10
Measured historical gate. Earlier isolated gate. September rebaseline is a separate dated measurement, not a replacement for this result.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-08-11; reviewed October 10, 2026.
Runtime: vLLM · August gate revision not frozen Source record updated 2026-10-10
Nemotron-3.5-Lightning 30B-A3B NVFP4 + DSpark vs Coder-Next
Historical lab pool
lightning devcode
70%
champion devcode
86.4%
lightning single stream
82.3tok/s
All measurements & context
lightning devcode
70 %
champion devcode
86.4 %
lightning single stream
82.3 tok/s
champion single stream
50.9 tok/s
lightning 4-stream aggregate agg
161.5 tok/s
champion 4-stream aggregate agg
76 tok/s
Historical. Speed win (1.62× c1) and concurrency scales; quality −16.4 outside ±4 noise. Rejected as primary coder; kept as on-demand lightning. Not live Oct 1 telemetry.
Historical. Rejected for both fleet roles. Tools and sidecar quality were real; decode 0.20× the coder champion and 0.17× Gemma. Native context 131K fails the verified-256K coder gate. Not added to llama-swap. Not live Oct 1 telemetry.
Measured historical gate. Measured report retained under its historical label. The label alone does not establish the quant. Do not transfer these quality scores to the verified August 2 UD-IQ2_XXS throughput-only artifact.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: Quant identity is unresolved. Score attribution stays with the retained report label. No code compilation, tool execution or full-context qualification in this gate; one report does not establish repeat spread.
Source: Retained numeric coder-gate summary, 2026-08-03; reviewed October 10, 2026.
Runtime: Retained gate label ds4-iq2m; exact artifact not frozen Source record updated 2026-10-10
Measured historical gate. Rejected: slow cold readiness and single-stream latency. Four-slot vLLM wave is not a model-only comparison with two-slot llama.cpp.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-30; reviewed October 10, 2026.
Runtime: vLLM · exact revision not frozen Source record updated 2026-10-10
Measured historical gate. One retained run. Five-run framework median 84.2%, Swift median 60.7% and framework standard deviation 5.55 points are separate summaries.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-30; reviewed October 10, 2026.
Runtime: llama.cpp · tuned gate Source record updated 2026-10-10
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-30; reviewed October 10, 2026.
Runtime: llama.cpp · tuned rematch Source record updated 2026-10-10
Historical. All three published-rank challengers rejected as primary coder. Ornith took the tools-specialist slot; coder-laguna retired. One gate run is not a result — Ornith five-run spread 73.2–87.3. Not live Oct 1 telemetry.
Measured historical gate. Matched base for Qwythos: no meaningful quality or throughput gain from the challenger.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-29; reviewed October 10, 2026.
Runtime: llama.cpp · matched base A/B Source record updated 2026-10-10
Measured historical gate. Framework and tools within the matched base result; rejected for the historical coder/agent roles.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-29; reviewed October 10, 2026.
Runtime: llama.cpp · matched base A/B Source record updated 2026-10-10
Measured historical gate. Single tuning gate. Six-run calibration median 86.4% is a separate summary.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-27; reviewed October 10, 2026.
Runtime: llama.cpp · batch tuning gate at ubatch 2048 Source record updated 2026-10-10
Historical same-day A/B. Coder-Next 2.98× decode and 5.3× wall; 47 GiB reclaimed (101 vs 53.9 GiB). Laguna kept on demand for tools 75.0% then retired July 30. Not live Oct 1 telemetry.
Historical coder ops note from the July 26 revert write-up on the benches page: 507 requests, 0 errors, 59.3 tok/s average, longest turn 123 s. Not a live counter.
Runtime: dated bench page Source record updated 2026-09-12
Measured study
Jul 22 image-gen gate
Mage-Flow Turbo 4B · Z-Image-Turbo · Sana Sprint 1.6B
Historical lab pool
mage warm
3.96s
mage peak
17.03GiB
zimage warm
5.7s
All measurements & context
mage warm
3.96 s
mage peak
17.03 GiB
zimage warm
5.7 s
zimage peak
21.69 GiB
sana warm
0.79 s
sana peak
10.19 GiB
Historical. Mage was the only tested model combining high visual quality, sub-10 s 1024², and under-20 GiB peak. Not uncensored (life-drawing refusal). Later moved on demand — warm ~4 s, cold ~120 s. Not live Oct 1 telemetry.
Historical qualification. Both deep sessions retrieved correctly near 241K. About 55.2 GiB. Warm-session affinity, not a 2× cold accelerator. Not live Oct 1 telemetry.
Historical. Not a quality or throughput upgrade. Native TP=2 deadlocked on 512-token generations. DFlash depth 15 cut four-way throughput 57.8 → 46.2 tok/s. Not live Oct 1 telemetry.
Historical. Two coding-peer slots plus one local lane. Four concurrent 256-token streams; excess queued, no 409s. Session affinity keeps cache local. Not live Oct 1 telemetry.
Historical. Gemma kept as always-on agent. Goetia and Fable rejected. Warm 17K prefix 0.303 s (23.2× over cold). Deep edge 78.3 s · 119,593 prompt tokens. Catalog ~65 tok/s on the later agent-gemma on-demand id is a separate roster figure. Not live Oct 1 telemetry.
Speculative decoding · code versus creative writing
Qwen3.6-35B-A3B · Q6_K · native MTP
NVIDIA DGX Spark
code off
64.1tok/s
code on
100.9tok/s
creative off
64.6tok/s
All measurements & context
code off
64.1 tok/s
code on
100.9 tok/s
creative off
64.6 tok/s
creative on
53.6 tok/s
Historical local A/B with six draft positions. Code used temperature 0.3; creative writing used 0.7. MTP helped code but hurt creative throughput. This does not qualify a different model, runtime, or sampling policy.
Protocol: Same model and quiet hardware; one slot. Workload and sampling settings are explicitly different between the code and creative probes.
Limits: Repeat count and spread were not recorded in the reviewed summary; do not treat this as a universal speedup.
Source: Local Spark tuning review, measured 2026-07-11; reviewed directly 2026-10-02.