Qwen 3.8 27B mac m5: 200 tok/s, audited

Qwen 3.8 27B on Mac M5 reaches 200 tokens per second only as an unqualified peak in the lithos-metal blog post, not as a measured end-to-end rate. The 4-bit model weighs 16.1 GB, and Apple's 614 GB/s memory bandwidth sets a plain-decoding ceiling near 38 tokens per second, so any faster figure depends on speculative decoding and must be audited closely.

What does the memory-bandwidth ceiling say about Qwen 3.8 27B on Mac M5?

Qwen 3.8 27B on Mac M5 cannot exceed roughly 38 tokens per second with plain decoding, because each token requires one full read of 16.1 GB of 4-bit weights over a memory bus rated at 614 GB/s. Reading those weights once takes about 26 milliseconds, which caps ungapped generation near that 38 tokens per second figure.

Apple specifies the M5 family memory bandwidth on its Mac product page, and the 614 GB/s number is a peak, so sustained decoding sits lower. The arithmetic above comes from the video creator, not from lithos. That estimate also ignores the key-value cache, which grows with the prompt and adds traffic per token.

The ceiling explains why any local inference stack can improve toward 38 tokens per second but not beyond it by tidying up kernel launches. A tool that fuses work and removes launch overhead gets closer to the ceiling; it still performs one weight read per pass. To pass the ceiling, an engine must produce several tokens from a single read, which is what speculative decoding does.

The number that matters for buyers is therefore not a single tokens-per-second rate. It is the ceiling for plain decoding, the measured round time for the accelerated path, and the acceptance rate that determines how many tokens each read actually yields. lithos-metal publishes the first two and leaves the third implicit for this model.

What did Stanford's Hazy Research measure about megakernels?

Hazy Research, a Stanford research group, showed on an NVIDIA H100 that fusing one pass through a model into a single megakernel lifted memory-bandwidth utilization from about 50% to 78% for a Llama 1B model. Those runs compared vLLM and SGLang, two widely used serving engines, against a fused kernel, and the result was about 2.5 times vLLM throughput and more than 1.5 times SGLang.

That measurement comes from the Hazy Research blog post on megakernels, not from Apple hardware. Serving engines typically split one model pass into around 100 small GPU kernels, and each kernel pays setup and teardown costs while the GPU waits. A fused kernel keeps more work resident and fewer units idle.

Apple's own research team reached a related conclusion for the M-series: after the first token, memory bandwidth limits generation more than raw compute does. That reasoning appears in Apple's work on on-device inference and explains why utilization, not FLOPs, dominates the decode phase for a model of this size.

The H100 result transfers as an idea, not as a number. lithos-metal brings megakernel fusion to Metal, Apple's GPU programming interface, but the 78% utilization figure belongs to the H100 experiment with a 1B model. No published measurement in the sources shows which fraction of 614 GB/s lithos achieves in real use.

How does lithos-metal fuse layer mixers on Metal?

lithos-metal fuses only the layer mixer, either the gated delta net block or the attention block, and leaves the feed-forward network as a separate kernel, so the popular description of a whole layer running in one program overstates the design. The lithos-metal GitHub repository documents that split in its design notes.

The 64 layers of this model divide into 48 gated delta net layers and 16 full-attention layers. The pattern repeats 16 times as three gated delta net layers followed by one attention layer. In a gated delta net layer the mixer keeps a 32-bit recurrent state with a key side and a value side; a conventional kernel reads and writes the entire state for every token.

lithos-metal assigns a worker a slice of the state's columns and holds it in on-chip fast memory across a block of input. Full-attention layers get a similar treatment, with each worker keeping small running softmax totals in 32-bit numbers so the engine never writes the full attention-score table to memory. A progress protocol tells a worker when another worker's result is ready.

Fewer memory round trips for the same answer is the gain. It does not change the read itself, and the weights still arrive from memory once per pass, so fusion raises the share of 614 GB/s in use without moving the 38 tokens per second ceiling.

How does DSpark speculative decoding create more than one token per read?

DSpark, a speculative decoding method from DeepSeek and Peking University, lets a compact drafter guess a block of tokens so the large model verifies them in a single pass over its weights. The method is described in the DSpark paper dated July 2026, which also reports 60% to 85% faster per-user generation in DeepSeek production serving, against an MTP1 baseline at the same total server load.

Guessing several tokens in parallel has a known weakness: each later guess cannot see the earlier ones, so acceptance falls toward the end of the block. A low-rank mask head lets each guess attend to its predecessor. A confidence head predicts how far into a block verification is worthwhile. Both are data-center techniques and neither measurement belongs to Apple hardware.

Inside lithos-metal the drafter is small, with five layers occupying 1.136 GB in NVFP4, a 4-bit format. That is roughly 7% of the large model, and on a 40-core M5 Max at a 128-token context the draft stage took 4.392 milliseconds against the 26 milliseconds a full weight read needs.

By default the engine accepts up to seven guessed tokens plus one anchor, the last token already confirmed. Each token becomes one row, so the large model verifies eight rows in one pass and reads its weights once for all of them. Extra rows cost compute, not another weight read.

Acceptance follows exact speculative sampling. In greedy mode a guess survives only if it matches the large model's token. In sampled mode a guess is accepted with probability equal to the model's probability divided by the drafter's, capped at one, followed by a correction token or a bonus token when every guess was right. A pull request merged on October 7, 2026 added that sampled path.

Rewind follows every acceptance decision, and three state structures must be brought back in line with the accepted tokens: the key-value cache, the short history a small filter layer keeps, and the recurrent state. The readme speed figures leave that rewind work out.

Which Qwen 3.8 27B on Mac M5 numbers are measured and which are projected?

Only one published round on this chip is measured end to end, and the 200-plus tokens per second headline is not it. Auditing the numbers means separating a real round time from readme projections and from a blog peak with no stated context length.

Figure	Scope	Status
~38 tok/s	plain decoding, 16.1 GB weights, 614 GB/s	derived ceiling
43.188 ms	full round, 40-core M5 Max, 128-token context	measured
168 tok/s	128-token context, six accepted tokens per pass	projection
127 tok/s	32K context, six accepted tokens per pass	projection
200+ tok/s	blog intro and announcement summary	peak, no context given

The measured round took 43.188 milliseconds on a 40-core M5 Max at a 128-token context. Six accepted tokens in that round works out to about 139 tokens per second, and a full eight tokens would give about 185. Both rates are derived. The model card never states how many tokens that particular round accepted.

The readme projects 168 tokens per second at a 128-token context and 127 tokens per second at 32K on a 40-core M5 Max with 48 GB, assuming six accepted tokens per eight-row pass. It states plainly that these are not measured end-to-end generation rates and not a direct speedup comparison. The 200-plus figure appears in the blog introduction and in a summary of the announcement post that carries the word peak, with no context length, prompt shape, or acceptance rate.

The blog also places lithos-metal at roughly 1.8 to 2 times MLX and Ollama at short contexts and above 2 times vLLM-metal, labeling those as target-compute projections that exclude drafting and rewind. If that ratio applied to 168 tokens per second, MLX would run near 90 tokens per second, far above the plain-decoding ceiling, so either the baselines use different settings or the ratio refers to another figure. The sources do not say which.

One separate quality check exists: all 512 generated tokens across four short prompts matched a BF16 reference. That says the faster path produced identical tokens on those four prompts. It says nothing about speed and nothing about other prompts.

What does long-context prefill cost on Apple silicon?

Prefill dominates long-context use: a 30K-token prompt on a 40-core M5 Max with 48 GB takes 53.3 seconds before the first output token, and the 127 tokens per second projection only applies after that wait. A pull request merged on October 7, 2026 reported the improvement, cutting time to first token from 71.3 seconds to 53.3 seconds on that prompt size.

The same pull request lists a 4K-token prompt dropping from 13.0 seconds to 6.6 seconds and a 17K-token prompt dropping from 39.1 seconds to 26.9 seconds. Those are time-to-first-token measurements with a clear prompt size attached, unlike the headline decode figure.

A 32K prompt would take a little longer than the 30K case, since prefill scales with prompt length. Anyone publishing a long-context decode rate should pair it with the corresponding prefill wait, because a 53-second pause before the first token changes how the tool feels in an agent loop.

Prefill and decode answer different questions. Prefill measures how long the model reads your prompt; decode measures how fast it writes the answer. Quoting a decode rate for long context without the prefill number gives an incomplete picture of interactive latency.

Which Macs run lithos-metal fully accelerated?

Full acceleration is limited to the 40-core M5 Max, while other chips fall back to slower paths with less published measurement behind them. The lithos-metal readme sorts Apple silicon into tiers rather than claiming uniform support.

• 40-core M5 Max: measured recipes for large models.

• M5 Pro: measured kernels for smaller models only.

• 32-core M5 Max and M4 Pro: unmeasured native fallback, no automatic megakernel fusion, tensor accelerator off.

The project also carries constraints outside the chip tier. It requires macOS 26 or later, uses the Apache 2.0 license, supports one model family plus an experimental 35B mixture-of-experts model, and runs one generation at a time. If you need several concurrent generations, this engine is not the tool for that workflow.

Compatibility here means kernel-level support per chip, not a promise that any Apple silicon machine reaches a published rate. The 43.188-millisecond round and every projection above belong to the 40-core M5 Max with 48 GB.

Should you try lithos-metal, and what is still untested?

Try lithos-metal if you own a 40-core M5 Max with 48 GB and run the 4-bit Qwen 3.8 27B on short prompts, and measure your own throughput rather than trusting 200 tokens per second. The sources do not tell you what speed you will get, because acceptance rate for this model is unpublished and drives everything past the ceiling.

The largest gap is the absence of an ablation. No published result isolates the megakernel gain from the DSpark gain on Apple hardware, so the share of the speedup contributed by fusion and the share contributed by speculative decoding remain unknown. The megakernel evidence comes from an H100 experiment with a 1B model, and the DSpark evidence comes from data-center serving on DeepSeek infrastructure.

For context on how the rest of the local-AI stack is covered, Dev doido publishes Brazilian Portuguese developer material, and Crazystack TypeScript shows how tooling write-ups can stay readable; Skala blog follows the same principle of pairing claims with their scope.

The mechanism is sound. Fewer idle gaps around each weight read and several tokens per read both raise throughput, and the memory wall stays exactly where the hardware puts it. The open question is narrower than the headline: how much of the gain survives when drafting, verification, and rewind are timed as part of the round.

Frequently asked questions

• Is 200 tokens per second for Qwen 3.8 27B on Mac M5 measured? No. The 200-plus figure appears in the lithos-metal blog introduction and in a summary of the announcement post that labels it a peak, with no context length or acceptance rate published. The only measured round is 43.188 milliseconds on a 40-core M5 Max at a 128-token context.

• Why can't plain decoding pass 38 tokens per second? Each output token requires one full read of the 16.1 GB weight file, and the M5 Max memory bus is rated at 614 GB/s. That arithmetic puts the ceiling near 38 tokens per second, and the 614 GB/s figure is itself a peak.

• What is DSpark and who published it? DSpark is a speculative decoding method from DeepSeek and Peking University, described in a paper dated July 2026. A compact drafter guesses several tokens and the large model verifies them in one pass over its weights, with exact speculative sampling after a pull request merged on October 7, 2026.

• Do 168 and 127 tokens per second include drafting and rewind? No. The readme states both are projections that assume six accepted tokens per eight-row pass and exclude drafting, acceptance processing, and state rewind.

• Which chip gets full acceleration? The 40-core M5 Max. The M5 Pro has measured kernels for smaller models only, and the 32-core M5 Max and M4 Pro fall back to an unmeasured native path with megakernel fusion disabled.

• What does a 30K prompt cost before the first token? About 53.3 seconds of prefill on a 40-core M5 Max with 48 GB, down from 71.3 seconds before the October 7, 2026 pull request.

• Does lithos-metal run this model in mixed precision safely? The model card reports all 512 generated tokens across four short prompts matching a BF16 reference, and the readme notes that the default fusion step changes 16-bit rounding.

• Can you run several generations at once? No. lithos-metal runs one generation at a time, which limits concurrent agent workflows on the same machine.

• Is the 1.8 to 2 times speedup over MLX and Ollama a measured result? No. The blog labels those ratios target-compute projections that exclude drafting and rewind, and the readme notes each backend's timer covers different work.

What a bandwidth audit changes for your buying decision

A bandwidth audit turns one headline into three numbers with three scopes, and that is more useful than any single rate. The hard limit near 38 tokens per second comes from hardware. The measured 43.188-millisecond round is real work on a 40-core M5 Max. The 168 and 127 figures are projections with a stated assumption, and the 200-plus figure has no context attached.

If you have a video that explains a technical mechanism like this one, the explanation usually outlives the numbers around it. Skala Blog turns a YouTube URL into a transcribed, structured article so the reasoning survives the headline.

Source video