Skip to content

Benchmarks / evidence with context

Useful numbers. Honest boundaries.

Measured studies, operations records, and catalog estimates are separate kinds of evidence. Explore the original dates, runtimes, units, and decisions behind the configuration.

39 MEASURED STUDIES3 OPS RECORDS3 CATALOG ESTIMATESDOWNLOAD ALL RECORDS ↓

GB10 pair / Reviewed Oct 23, 2026

DeepSeek is serving. The coding panel is a tie.

Normal DeepSeek V4.1 Flash 1017 runs through TensorFold TP2 on the DGX Spark and ASUS GX10. The matched GLM comparison, short serving probes and memory observation answer different questions.

Shared inference slots
4

Coder, scout and vision share this capacity. Studio agent and embeddings keep independent routes.

Configured context
524,288

A serving limit. Local full-window retrieval and concurrent long-context capacity remain unqualified.

Initial memory minima / GiB
4.93 / 6.55

Head / worker available memory across one hour, 1,800 samples each. A bounded observation; guards remain active.

October 22–23 · Same frozen primary Hermes protocol, temperature 0.2, thinking off, 4,096 output allowance. Coding handoffs and diagnostic decisions keep separate denominators.
Model / dateRoutine codingHard codingDefective reviewOther controls
GLM · abliterated1 / 10 / 3Incorrect approvalAdmission 1 / 1
Healthy review 1 / 1
DeepSeek · normal1 / 10 / 3Correct rejectionAdmission 1 / 1
Healthy review 1 / 1

One routine task, three hard tasks, one visit each, one harness. Behavior, scope, report, exact identity and cleanup are required. This local panel is not an official SWE-bench score. Later production high-reasoning tuning is separate; no difficult-coding advantage was demonstrated.

Historical / September Studio evaluation

Speed is only one axis.

Protocol & limitations ↗

M5 Max · 40 GPU cores · 48 GB · Sep 22–23, 2026 · thinking off. Lightning here is the evaluated Nemotron artifact. The current SuperQwen alias has no score in this chart.

HumanEval / % ↑

050100080160

Code decode / tok/s →

Coding fallback / MLX 4-bit

95.1%

HumanEval · 156 / 164 tasks passed

All 164 original tasks, one greedy chat sample, original executable tests, no repair or retries. Chat-adapted pass@1; training contamination is possible. Gemma was a writing-role evaluation and did not run this suite.

Speed: median of three short 512-output code probes on a loaded backend. Timing and quality use different prompts; this is not full-task wall time.

See the experiments in the visual lab.Speculation, concurrency, promotion reversals, image budgets, and quality gates.

Explore the study records

7 OF 45 RECORDS

The evidence atlas / 7 records

Every study keeps its date.

Select a date to inspect its records. The dots count records, not performance.

MEASUREDOPERATIONSCATALOG
Measured study

Mac Studio · writing-specialist evaluation

Gemma-4 26B-A4B QAT · Q4_K_M

Mac Studio

code decode
78.1tok/s
prose decode
76.2tok/s
framework
84.3%
All measurements & context
code decode
78.1 tok/s
prose decode
76.2 tok/s
framework
84.3 %
broad tools
94.1 %
developer tools
66.7 %
prose constraints
97 %

Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval was not run for this writing role; no coding benchmark score is inferred. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.

Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.

Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.

Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.

Runtime: llama.cpp Metal · build 10964
Source record updated 2026-09-23

Read the decision ↗
Measured study

Mac Studio · fast-agent evaluation

Nemotron-3.5-Lightning 30B-A3B · MLX 4-bit

Mac Studio

code decode
150.1tok/s
prose decode
150.4tok/s
framework
71.5%
All measurements & context
code decode
150.1 tok/s
prose decode
150.4 tok/s
framework
71.5 %
broad tools
88.2 %
developer tools
66.7 %
prose constraints
73.5 %
humaneval passed
140
humaneval tasks
164
humaneval
85.4 %

Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval: all 164 original tasks; one greedy chat sample; 2048-output cap; thinking off; original sandboxed tests; no repair or retries. Chat-adapted pass@1, not a leaderboard reproduction. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.

Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.

Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.

Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.

Runtime: MLX / MLX-VLM
Source record updated 2026-09-23

Read the decision ↗
Measured study

Mac Studio · coding evaluation

Qwen3.8-27B · MLX 4-bit

Mac Studio

code decode
33tok/s
prose decode
33tok/s
framework
89.5%
All measurements & context
code decode
33 tok/s
prose decode
33 tok/s
framework
89.5 %
broad tools
100 %
developer tools
50 %
prose constraints
87.2 %
humaneval passed
156
humaneval tasks
164
humaneval
95.1 %

Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval: all 164 original tasks; one greedy chat sample; 2048-output cap; thinking off; original sandboxed tests; no repair or retries. Chat-adapted pass@1, not a leaderboard reproduction. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.

Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.

Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.

Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.

Runtime: MLX / MLX-VLM
Source record updated 2026-09-23

Read the decision ↗
Measured study

Sep 12 Mistral quality watch (unrouted)

Mistral Small 4 119B-A6B NVFP4

Historical lab pool

broad tools
100%
prose constraints
86.9%
devcode
81.4%
All measurements & context
broad tools
100 %
prose constraints
86.9 %
devcode
81.4 %
devtool
83.3 %
cold ready
674 s
cached ready
585 s

Historical fast gate. Perfect broad tools and stronger prose than Lightning (63.8%). Fits TP1 at 256K (66.12 GiB weights + 27.33 GiB KV) but ten-minute reload keeps it an unrouted specialist. Retrieval was not needle-tested. Not live telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: vLLM 0.29 · one Spark · 256K
Source record updated 2026-09-12

Read the decision ↗
Measured study

Sep 12 role-split quality gate

Lightning 30B-A3B · Qwen3.8-27B + DFlash2 · Mistral Small 4

Historical lab pool

lightning devcode
80.3%
qwen devcode
85.9%
mistral devcode
81.4%
All measurements & context
lightning devcode
80.3 %
qwen devcode
85.9 %
mistral devcode
81.4 %
lightning broad tools
82.4 %
qwen broad tools
94.1 %
mistral broad tools
100 %

Historical. Keep the split: Lightning owns latency, Qwen owns code (perfect multi-turn and restraint), Mistral stays an unrouted quality watch. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: vLLM 0.29 / SGLang · non-reasoning · dated bench page
Source record updated 2026-09-12

Read the decision ↗
Measured study

Archived dual-role gate · Qwen3.8 FP8

Qwen3.8-27B · official FP8

Historical lab pool

framework
87.5%
developer tools
50%
broad tools thinking off
100%
All measurements & context
framework
87.5 %
developer tools
50 %
broad tools thinking off
100 %
broad tools thinking on
100 %
developer perception facts
30
developer perception total
30
vision
62.5 %
vision tools
50 %
prose constraints
77.8 %
single stream
8 tok/s
2-stream aggregate
16 tok/s
4-stream aggregate
31.2 tok/s
four request mean decode
7.8 tok/s
framework wall
1,072 s
cold ready
433 s
allocated
91.8 GiB

Measured historical gate; rejected for latency and readiness despite strong broad tools and developer screenshot facts. Configured 262144 context was not needle-verified.

Protocol: Custom framework/tool/vision/writing suites plus warm prose concurrency waves. Broad tools were separately checked with thinking off and on. Context configuration and measured cold readiness are separate fields.

Limits: No executable framework validation. No full-window retrieval test. Different protocol axes cannot be combined into one universal model score.

Source: Retained August 14 dual-role numeric summary and archived model registry; reviewed October 10, 2026.

Runtime: vLLM 0.27.1
Source record updated 2026-10-10

Read the decision ↗
Measured study

Jul 19 multimodal creative agent bakeoff

Gemma-4-26B-A4B QAT · always-on agent

Historical lab pool

single
60.4tok/s
four way
129.9tok/s
tools
91.2%
All measurements & context
single
60.4 tok/s
four way
129.9 tok/s
tools
91.2 %
blind score
8.35 /10
warm prefix
0.303 s
deep edge
78.3 s

Historical. Gemma kept as always-on agent. Goetia and Fable rejected. Warm 17K prefix 0.303 s (23.2× over cold). Deep edge 78.3 s · 119,593 prompt tokens. Catalog ~65 tok/s on the later agent-gemma on-demand id is a separate roster figure. Not live Oct 1 telemetry.

Source: Internal benches page HTML, last-modified 2026-09-12

Runtime: llama.cpp · four 163840 slots · dated bench page
Source record updated 2026-09-12

Read the decision ↗