Measured studies, operations records, and catalog estimates are separate kinds of evidence. Explore the original dates, runtimes, units, and decisions behind the configuration.
Normal DeepSeek V4.1 Flash 1017 runs through TensorFold TP2 on the DGX Spark and ASUS GX10. The matched GLM comparison, short serving probes and memory observation answer different questions.
Shared inference slots
4
Coder, scout and vision share this capacity. Studio agent and embeddings keep independent routes.
Configured context
524,288
A serving limit. Local full-window retrieval and concurrent long-context capacity remain unqualified.
Initial memory minima / GiB
4.93 / 6.55
Head / worker available memory across one hour, 1,800 samples each. A bounded observation; guards remain active.
October 22–23 · Same frozen primary Hermes protocol, temperature 0.2, thinking off, 4,096 output allowance. Coding handoffs and diagnostic decisions keep separate denominators.
One routine task, three hard tasks, one visit each, one harness. Behavior, scope, report, exact identity and cleanup are required. This local panel is not an official SWE-bench score. Later production high-reasoning tuning is separate; no difficult-coding advantage was demonstrated.
M5 Max · 40 GPU cores · 48 GB · Sep 22–23, 2026 · thinking off. Lightning here is the evaluated Nemotron artifact. The current SuperQwen alias has no score in this chart.
HumanEval / % ↑
Code decode / tok/s →
Coding fallback / MLX 4-bit
95.1%
HumanEval · 156 / 164 tasks passed
All 164 original tasks, one greedy chat sample, original executable tests, no repair or retries. Chat-adapted pass@1; training contamination is possible. Gemma was a writing-role evaluation and did not run this suite.
Speed: median of three short 512-output code probes on a loaded backend. Timing and quality use different prompts; this is not full-task wall time.
Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval was not run for this writing role; no coding benchmark score is inferred. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.
Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.
Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.
Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.
Runtime: llama.cpp Metal · build 10964 Source record updated 2026-09-23
Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval: all 164 original tasks; one greedy chat sample; 2048-output cap; thinking off; original sandboxed tests; no repair or retries. Chat-adapted pass@1, not a leaderboard reproduction. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.
Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.
Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.
Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.
Runtime: MLX / MLX-VLM Source record updated 2026-09-23
Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval: all 164 original tasks; one greedy chat sample; 2048-output cap; thinking off; original sandboxed tests; no repair or retries. Chat-adapted pass@1, not a leaderboard reproduction. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.
Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.
Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.
Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.
Runtime: MLX / MLX-VLM Source record updated 2026-09-23
Measured historical gate. Clean September rebaseline. Historical agent, superseded later.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-09-12; reviewed October 10, 2026.
Runtime: vLLM 0.29.0 Source record updated 2026-10-10
Measured historical gate. Clean September rebaseline. Historical coder, superseded later.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-09-12; reviewed October 10, 2026.
Historical fast gate. Perfect broad tools and stronger prose than Lightning (63.8%). Fits TP1 at 256K (66.12 GiB weights + 27.33 GiB KV) but ten-minute reload keeps it an unrouted specialist. Retrieval was not needle-tested. Not live telemetry.
Lightning 30B-A3B · Qwen3.8-27B + DFlash2 · Mistral Small 4
Historical lab pool
lightning devcode
80.3%
qwen devcode
85.9%
mistral devcode
81.4%
All measurements & context
lightning devcode
80.3 %
qwen devcode
85.9 %
mistral devcode
81.4 %
lightning broad tools
82.4 %
qwen broad tools
94.1 %
mistral broad tools
100 %
Historical. Keep the split: Lightning owns latency, Qwen owns code (perfect multi-turn and restraint), Mistral stays an unrouted quality watch. Not live Oct 1 telemetry.
Measured historical gate. Single August gate; 92% is not a repeat median or a new universal champion score.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-08-19; reviewed October 10, 2026.
Measured historical gate; rejected for latency and readiness despite strong broad tools and developer screenshot facts. Configured 262144 context was not needle-verified.
Protocol: Custom framework/tool/vision/writing suites plus warm prose concurrency waves. Broad tools were separately checked with thinking off and on. Context configuration and measured cold readiness are separate fields.
Limits: No executable framework validation. No full-window retrieval test. Different protocol axes cannot be combined into one universal model score.
Source: Retained August 14 dual-role numeric summary and archived model registry; reviewed October 10, 2026.
Runtime: vLLM 0.27.1 Source record updated 2026-10-10
Measured historical gate. Earlier isolated gate. September rebaseline is a separate dated measurement, not a replacement for this result.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-08-11; reviewed October 10, 2026.
Runtime: vLLM · August gate revision not frozen Source record updated 2026-10-10
Nemotron-3.5-Lightning 30B-A3B NVFP4 + DSpark vs Coder-Next
Historical lab pool
lightning devcode
70%
champion devcode
86.4%
lightning single stream
82.3tok/s
All measurements & context
lightning devcode
70 %
champion devcode
86.4 %
lightning single stream
82.3 tok/s
champion single stream
50.9 tok/s
lightning 4-stream aggregate agg
161.5 tok/s
champion 4-stream aggregate agg
76 tok/s
Historical. Speed win (1.62× c1) and concurrency scales; quality −16.4 outside ±4 noise. Rejected as primary coder; kept as on-demand lightning. Not live Oct 1 telemetry.
Historical. Rejected for both fleet roles. Tools and sidecar quality were real; decode 0.20× the coder champion and 0.17× Gemma. Native context 131K fails the verified-256K coder gate. Not added to llama-swap. Not live Oct 1 telemetry.
Measured historical gate. Measured report retained under its historical label. The label alone does not establish the quant. Do not transfer these quality scores to the verified August 2 UD-IQ2_XXS throughput-only artifact.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: Quant identity is unresolved. Score attribution stays with the retained report label. No code compilation, tool execution or full-context qualification in this gate; one report does not establish repeat spread.
Source: Retained numeric coder-gate summary, 2026-08-03; reviewed October 10, 2026.
Runtime: Retained gate label ds4-iq2m; exact artifact not frozen Source record updated 2026-10-10
Measured historical gate. Rejected: slow cold readiness and single-stream latency. Four-slot vLLM wave is not a model-only comparison with two-slot llama.cpp.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-30; reviewed October 10, 2026.
Runtime: vLLM · exact revision not frozen Source record updated 2026-10-10
Measured historical gate. One retained run. Five-run framework median 84.2%, Swift median 60.7% and framework standard deviation 5.55 points are separate summaries.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-30; reviewed October 10, 2026.
Runtime: llama.cpp · tuned gate Source record updated 2026-10-10
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-30; reviewed October 10, 2026.
Runtime: llama.cpp · tuned rematch Source record updated 2026-10-10
Historical. All three published-rank challengers rejected as primary coder. Ornith took the tools-specialist slot; coder-laguna retired. One gate run is not a result — Ornith five-run spread 73.2–87.3. Not live Oct 1 telemetry.
Measured historical gate. Matched base for Qwythos: no meaningful quality or throughput gain from the challenger.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-29; reviewed October 10, 2026.
Runtime: llama.cpp · matched base A/B Source record updated 2026-10-10
Measured historical gate. Framework and tools within the matched base result; rejected for the historical coder/agent roles.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-29; reviewed October 10, 2026.
Runtime: llama.cpp · matched base A/B Source record updated 2026-10-10
Measured historical gate. Single tuning gate. Six-run calibration median 86.4% is a separate summary.
Protocol: Eight framework prompts scored by regex, twelve developer tool scenarios, and warm lighthouse-prose concurrency waves. Defaults: framework temperature 0.2 / output 4096; tools 0.1 / 1024; concurrency 0.7 / 512; thinking off. c1/c2/c4 divide total completion positions by whole-wave wall time. Mean per-request decode excludes each request’s TTFT. Gate first response is the short ping, not an independently measured model load time.
Limits: No code compilation or actual tool execution. One report does not establish repeat spread or context qualification. Different slot counts, runtime revisions and dates limit cross-engine comparisons.
Source: Retained numeric coder-gate summary, 2026-07-27; reviewed October 10, 2026.
Runtime: llama.cpp · batch tuning gate at ubatch 2048 Source record updated 2026-10-10
Historical same-day A/B. Coder-Next 2.98× decode and 5.3× wall; 47 GiB reclaimed (101 vs 53.9 GiB). Laguna kept on demand for tools 75.0% then retired July 30. Not live Oct 1 telemetry.