Mac Studio · coding evaluation
Qwen3.8-27B · MLX 4-bit
Mac Studio
- code decode
- 33tok/s
- prose decode
- 33tok/s
- framework
- 89.5%
All measurements & context
- code decode
- 33 tok/s
- prose decode
- 33 tok/s
- framework
- 89.5 %
- broad tools
- 100 %
- developer tools
- 50 %
- prose constraints
- 87.2 %
- humaneval passed
- 156
- humaneval tasks
- 164
- humaneval
- 95.1 %
Measured on M5 Max, 40 GPU cores, 48 GB unified memory. Single-stream decode is the median of three short 512-output probes on a loaded backend, with warm/fresh prefixes retained. Framework scores are regex criteria, not compilation. Broad/developer tools and prose constraints are custom suites. HumanEval: all 164 original tasks; one greedy chat sample; 2048-output cap; thinking off; original sandboxed tests; no repair or retries. Chat-adapted pass@1, not a leaderboard reproduction. Synthetic retrieval checks reached about 34K input positions, not 64K/262K qualification.
Protocol: Sequential large-model tests; thinking off; repaired proxy path. Quality suites and timing probes use different prompts.
Limits: Old HumanEval can be contaminated by training data. Custom suite percentages are not autonomous coding success rates; regex criteria can penalize valid alternatives.
Source: Mac Studio evaluation scorecard and retained numeric summary, 2026-09-22/23; reviewed directly 2026-10-02.
Runtime: MLX / MLX-VLM
Source record updated 2026-09-23