#llm #omlx #mlx #qwen #benchmark #apple-silicon #local-llm #recipe # Overview A shareable recipe for benchmarking **Qwen3.8-27B with MTP enabled in oMLX** on Apple Silicon, so results are comparable across machines. Takes about 15 minutes to run. Post your numbers in [Results](#results). # Setup **1. Install oMLX 0.6.4 from the [official DMG](https://github.com/jundot/omlx/releases).** Use the DMG. **2. Pull an MTP-preserved checkpoint model.** ``` fcmeyer/Qwen3.8-27B-MLX-oQ4e-mtp (~17 GB) ``` ![[SCR-20260910-kkct 1.png]] # Settings MTP is **off by default** and there is no GUI toggle. Edit `~/.omlx/model_settings.json`: ```json { "version": 1, "models": { "Qwen3.8-27B-MLX-oQ4e-mtp": { "max_context_window": 262144, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "force_sampling": false, "mtp_enabled": true, "mtp_num_draft_tokens": 2, "turboquant_kv_enabled": false, "qwen35_ane_prefill_enabled": false, "dflash_enabled": false, "vlm_mtp_enabled": false } } } ``` Two things that will silently cost you the run: - **The model key is bare** — `Qwen3.8-27B-MLX-oQ4e-mtp`, no org prefix, no double dash. Copy it exactly as the dashboard lists it. - **Settings are read only at server start.** ```bash omlx restart ``` # Run Dashboard → **Benchmark** --> Performance, then: | | | | -------------------- | -------------------------------------------- | | Model | `Qwen3.8-27B-MLX-oQ4e-mtp` | | Engine | Auto | | Context | Code (Mixed) | | Single Request Tests | pp1024 / pp4096 / pp8192 / pp16384 / pp32768 | That runs pp1024 / 4096 / 8192 / 16384 / 32768 at tg128, then continuous batching at 1× / 2× / 4×. **Capture the log too — this part is not optional:** ```bash grep -hE "accept=|tok/cycle|parked" ~/.omlx/logs/server.log | tail -40 ``` The benchmark output does not report MTP acceptance, so a row where speculative decoding **parked** looks identical to a row that was simply slow. Without the log you cannot tell whether you measured MTP or measured nothing. You can map log lines to benchmark rows by matching the gaps between timestamps against the reported E2E times. # What to report - Chip, RAM, and memory bandwidth - oMLX version, and DMG vs pip - Both benchmark tables, verbatim - The MTP log lines above # Results ## Reference — Mac Studio M2 Max, 64 GB, 400 GB/s, oMLX 0.6.4 | Test | TTFT | pp TPS | tg TPS | Peak Mem | |---|---|---|---|---| | pp1024/tg128 | 11.2 s | 91.5 | 18.4 | 18.35 GB | | pp4096/tg128 | 39.4 s | 103.9 | 18.8 | 22.60 GB | | pp8192/tg128 | 82.7 s | 99.1 | 17.0 | 23.62 GB | | pp16384/tg128 | 172.0 s | 95.2 | 16.5 | 24.59 GB | | pp32768/tg128 | 380.6 s | 86.1 | 15.5 | 25.65 GB | Continuous batching, pp1024/tg128 — **1× 18.4 · 2× 26.3 · 4× 57.2 tok/s** MTP ran at 2.17–2.84 tok/cycle at 83–100% acceptance, except pp8192, which parked. Two things worth knowing before you compare against this: - **Prefill is the constraint, not generation.** 86–104 tok/s against roughly 1,000 tok/s for comfortable agent work. A 32K prompt takes over six minutes to first token, cold. - ==**Speculative decoding produced no measurable gain here.** pp4096 hit 2.84 tok/cycle at 100% acceptance — the ceiling for depth 2 — and still returned 18.8 tok/s, which is exactly this machine's MTP-**off** baseline. Whether that holds on other chips is the most interesting thing this recipe could answer.== ## Add yours Same format, one section per machine. # References - [oMLX](https://github.com/jundot/omlx) · [releases](https://github.com/jundot/omlx/releases) - [fcmeyer/Qwen3.8-27B-MLX-oQ4e-mtp](https://huggingface.co/fcmeyer/Qwen3.8-27B-MLX-oQ4e-mtp) - [[oMLX Benchmark Results]] — full measured ladders and analysis - [[oMLX Experimentation]] — setup traps, MTP investigation