Skip to content

Benchmarks / evidence with context

Useful numbers. Honest boundaries.

Measured studies, operations records, and catalog estimates are separate kinds of evidence. Explore the original dates, runtimes, units, and decisions behind the configuration.

39 MEASURED STUDIES3 OPS RECORDS3 CATALOG ESTIMATESDOWNLOAD ALL RECORDS ↓

GB10 pair / Reviewed Oct 23, 2026

DeepSeek is serving. The coding panel is a tie.

Normal DeepSeek V4.1 Flash 1017 runs through TensorFold TP2 on the DGX Spark and ASUS GX10. The matched GLM comparison, short serving probes and memory observation answer different questions.

Shared inference slots
4

Coder, scout and vision share this capacity. Studio agent and embeddings keep independent routes.

Configured context
524,288

A serving limit. Local full-window retrieval and concurrent long-context capacity remain unqualified.

Initial memory minima / GiB
4.93 / 6.55

Head / worker available memory across one hour, 1,800 samples each. A bounded observation; guards remain active.

October 22–23 · Same frozen primary Hermes protocol, temperature 0.2, thinking off, 4,096 output allowance. Coding handoffs and diagnostic decisions keep separate denominators.
Model / dateRoutine codingHard codingDefective reviewOther controls
GLM · abliterated1 / 10 / 3Incorrect approvalAdmission 1 / 1
Healthy review 1 / 1
DeepSeek · normal1 / 10 / 3Correct rejectionAdmission 1 / 1
Healthy review 1 / 1

One routine task, three hard tasks, one visit each, one harness. Behavior, scope, report, exact identity and cleanup are required. This local panel is not an official SWE-bench score. Later production high-reasoning tuning is separate; no difficult-coding advantage was demonstrated.

Historical / September Studio evaluation

Speed is only one axis.

Protocol & limitations ↗

M5 Max · 40 GPU cores · 48 GB · Sep 22–23, 2026 · thinking off. Lightning here is the evaluated Nemotron artifact. The current SuperQwen alias has no score in this chart.

HumanEval / % ↑

050100080160

Code decode / tok/s →

Coding fallback / MLX 4-bit

95.1%

HumanEval · 156 / 164 tasks passed

All 164 original tasks, one greedy chat sample, original executable tests, no repair or retries. Chat-adapted pass@1; training contamination is possible. Gemma was a writing-role evaluation and did not run this suite.

Speed: median of three short 512-output code probes on a loaded backend. Timing and quality use different prompts; this is not full-task wall time.

See the experiments in the visual lab.Speculation, concurrency, promotion reversals, image budgets, and quality gates.

Explore the study records

2 OF 45 RECORDS

The evidence atlas / 2 records

Every study keeps its date.

Select a date to inspect its records. The dots count records, not performance.

MEASUREDOPERATIONSCATALOG
Measured studyINCOMPLETE RUN

Aider polyglot · incomplete Flash-Next run

MiaAI Qwen3.8-Flash-Next · NVFP4 (September artifact)

Historical lab pool

within two tries passed
58
tasks recorded
94
suite tasks
225
All measurements & context
within two tries passed
58
tasks recorded
94
suite tasks
225
tasks unrecorded
131
first try passed
28
test timeouts
0
malformed response counter
1
context exhaustion counter
72
request error output counter
73
cpp tasks recorded
12
cpp first try passed
1
cpp within two tries passed
5
go tasks recorded
11
go first try passed
4
go within two tries passed
6
java tasks recorded
22
java first try passed
6
java within two tries passed
15
javascript tasks recorded
24
javascript first try passed
8
javascript within two tries passed
16
python tasks recorded
14
python first try passed
4
python within two tries passed
10
rust tasks recorded
11
rust first try passed
5
rust within two tries passed
6

Measured partial archive: 94/225 task result files, 28 first-try passes and 58 passes within two tries among the recorded tasks. No final suite percentage or winner is reported. The missing 131 tasks are unrecorded, not imputed failures. This is the September MiaAI artifact, not the October NVIDIA serving artifact.

Protocol: 225 Exercism tasks across six languages; whole-file editing; temperature 0; one worker; up to two tries with test feedback. Thinking on. Run recipe specifies 65536 context; metadata lists 262144 input and 32768 output. Per-language pass counts are preserved.

Limits: Effective chat-history cap is not independently frozen; earlier notes differ. Harness context-exhaustion, malformed-response and request-error-output counters are event counters, not counts of distinct failed tasks or HTTP requests. Incomplete run with different artifact, thinking policy and infrastructure; observed pass counts cannot support a final 225-task rate.

Source: Aggregated retained per-exercise Aider result files and local run recipe. Run start September 26; archive reviewed October 10, 2026.

Runtime: vLLM two-GB10 · Aider whole-file editing
Source record updated 2026-10-10

Read the decision ↗
Measured study

Aider polyglot · complete 225-task run

Qwen3.8-27B · NVFP4 + DFlash2

NVIDIA DGX Spark

within two tries
49.8%
first try
9.3%
tasks recorded
225
All measurements & context
within two tries
49.8 %
first try
9.3 %
tasks recorded
225
suite tasks
225
first try passed
21
within two tries passed
112
test timeouts
2
malformed response counter
0
context exhaustion counter
0
request error output counter
0
cpp tasks recorded
26
cpp first try passed
2
cpp within two tries passed
19
go tasks recorded
39
go first try passed
5
go within two tries passed
19
java tasks recorded
47
java first try passed
1
java within two tries passed
21
javascript tasks recorded
49
javascript first try passed
3
javascript within two tries passed
26
python tasks recorded
34
python first try passed
4
python within two tries passed
14
rust tasks recorded
30
rust first try passed
6
rust within two tries passed
13

Measured full recorded run: 21 first-try passes and 112 passes within two tries across all 225 exercises. EQ-Bench long-form generation for this arm was also recorded as completed with judging skipped; it provides no writing-quality score.

Protocol: 225 Exercism tasks across six languages; whole-file editing; temperature 0; one worker; up to two tries with test feedback. Thinking off. Run recipe specifies 65536 context; metadata lists 262144 input and 32768 output. Per-language pass counts are preserved.

Limits: Effective chat-history cap is not independently frozen; earlier notes differ. Harness context-exhaustion, malformed-response and request-error-output counters are event counters, not counts of distinct failed tasks or HTTP requests. One local run, without repeat spread or a matched completed Flash result.

Source: Aggregated retained per-exercise Aider result files and local run recipe. Run start September 26; archive reviewed October 10, 2026.

Runtime: SGLang + DFlash2 · Aider whole-file editing
Source record updated 2026-10-10

Read the decision ↗