Local AI Inference Speed: 7 Engines vs llama.cpp

Local AI inference speed claims are usually weaker than their headlines. Of seven engines promoted as faster than llama.cpp across 2026, only one published a table containing both its own number and a llama.cpp baseline, and even that table withholds the baseline's offload flags. The rest either measured a different slice of the job or published no comparable number at all.

Local AI inference speed: what the seven claims actually survive

Local AI inference speed is usually limited by memory bandwidth, and the published claims mostly fail to show a fair comparison against llama.cpp. Of seven engines promoted as faster in 2026, one published a table with both its own result and a llama.cpp baseline. Four reported numbers measured under different scopes, and two could not be traced to a primary source.

The pattern is not fraud. llama.cpp is a general-purpose runtime, and an engine tuned to one model can legitimately skip overhead that llama.cpp carries for everything. The problem is that the ratios travel without the build number, the flag list, the quantization, or the model those numbers belong to. A speed ratio missing a flag list is only half a measurement.

The audits below come from reading each project's own repository, model card, or release notes. Where a project published projections rather than measured runs, that distinction is preserved. Where a ratio has no baseline anywhere on the project's pages, the ratio is marked unsupported rather than assumed false.

The bandwidth ceiling that decides most local AI inference speed

Local AI inference speed for standard token-by-token decoding cannot exceed memory bandwidth divided by the bytes read per token. A dense model reads roughly every active weight once to produce each token, so the arithmetic sets a hard bound before any engine optimisation is applied.

An NVIDIA RTX 3090 has 936 GB/s of memory bandwidth. A four-bit Qwen3 27B-class file runs about 17.1 to 19 GB. Dividing bandwidth by file size gives a plain decoding ceiling of roughly 49 to 55 tokens per second. An Apple M5 Max, at 614 GB/s with a 16 to 17 GB four-bit stand-in file, works out to about 36 to 38 tokens per second.

Treat those as bounds, not predictions. The sum ignores the key-value cache, assumes a dense model, and uses bandwidth figures taken from spec listings rather than from a controlled test. Mixture-of-experts models read only their active experts per token, which changes the per-token byte count and can lift the ceiling.

The formula's practical value is as a filter. Before installing anything, divide your card's bandwidth by your model file's size. A claimed ratio that implies crossing the resulting ceiling by a wide margin needs speculative decoding, a smaller file, or an explanation the page does not contain.

Strata Pascal: the one claim with a real llama.cpp table

The Strata Pascal fork is the only engine in this set whose README prints a side-by-side table against llama.cpp, and it still leaves the baseline's flags undisclosed. The test ran on a laptop with a 4 GB GTX 1050 Ti mobile GPU and 48 GB of system RAM.

The model was a 35B Qwen mixture-of-experts build with 3B active parameters, shipped as a 14.1 GB file. Against llama.cpp build b11524 at an 8K prompt, the fork reported 24.53 tokens per second of decode against 7.88, a factor of 3.1. Prefill, which is prompt reading, was reported 2.26 times faster. At 16K context the decode gap held at 21.40 against 7.16.

The caveats sit in the same README. The fork averaged four runs against two for the baseline, and the llama.cpp offload flags are not disclosed. The document calls itself an independent experimental fork. Without the baseline's flags, the real size of that 3.1 factor is unknown, even though it is the strongest evidence in the whole set.

A separate Strata main README describes a three-tier layout for a 125B model with 24,576 experts: busiest experts in 12 to 24 GB of VRAM, the rest in 32 GB or more of system RAM, and lookup tables on NVMe. That layout is a general design note, not part of the Pascal fork's laptop result, and the two should not be merged.

The Mac, 5090 and 4090 claims that measure different slices

Several headline ratios compare numbers that were never measured the same way, so the ratio describes two different jobs rather than one speedup. The gaps show up most clearly on Lithos-Metal, PolyStrata and Glyd.

Lithos-Metal on the M5 Max

The Lithos-Metal README gives about 168 tokens per second at short context and 127 at 32K on a 40-core M5 Max, and labels both as target compute projections. The ratio that travels with the project, roughly twice as fast as Apple's MLX framework, assumes six accepted tokens per eight-row pass and leaves out drafting, acceptance processing and rewind. The project's own backend verification document says the different timing scopes prevent a direct end-to-end speedup claim, and the document times Ollama on a full eight-token pass, vLLM-metal on prefill proxies, and the Lithos and MLX lines by summing best single-layer timings. A separate post adds a 200-plus tokens per second per user figure at peak. Only the M5 Max is listed as a measured setup, and macOS 26 or later is required.

PolyStrata and the card swap

A headline pairs 57 tokens per second with an RTX 5090 and a llama.cpp figure of 26. The table puts GLM-5.3-flash on that 5090 at 59 to 66. The release notes carry 57 to 59 instead, and attribute it to an RTX 4070 laptop with 8 GB of VRAM. The 57 is a real number on the wrong card. The llama.cpp 26 appears in none of the README, the backends document, or the release notes. The project also notes that only about a fifth of the model's experts fit in the 5090's 32 GB, that the 3-bit file is a 114 GB download, and that setup needs Linux and around 100 GB of RAM.

Glyd and the missing baseline

Glyd's model card reports 45.5 tokens per second for a Nemotron 3 Super checkpoint, a 120B model on an RTX 4090 at 4K context, keeping 190 of 512 experts on the GPU and computing the rest on the CPU. The card also lists 37.0 on a 3090. The 2.5-times-llama.cpp ratio that travels with the checkpoint appears in neither the card, the package page, nor the homepage. The checkpoint is 48.7 GB, down from 241.4 GB at 16-bit, with experts squeezed to 2.99 bits, and its card states outright that it is not lossless. It scores 97.0% on 300 grade-school math questions and matches the original model's top choice 96.9% of the time. Full speed needs 64 GB of system RAM, Linux, an Ampere-or-newer GPU, and Glyd's own engine. The engine ships under the Business Source License 1.1, so free applies to personal and non-commercial use only.

Fused megakernels and speculative decoding: the two real tricks

The two mechanisms behind the surviving numbers are fused megakernels, which cut per-launch overhead, and speculative decoding, which reads the weights once for several tokens. A third idea, tiered expert offload, targets models larger than the card.

Why fusing helps small models most

Standard decoding launches roughly 100 small GPU kernels per token across a 24-layer model. Each launch pays a round trip to the CPU, a fresh fetch from global memory and a synchronisation. A fused megakernel collapses those into fewer dispatches and keeps activations on-chip. One public write-up reports fusing all 24 layers of a 0.8B model into a single dispatch, reaching 411 tokens per second at 220 W on a 3090 against llama.cpp's 267 at 350 W. That is a 0.8B model, where launch overhead is the largest share of total work, so the result should not be read as a 27B speedup.

Why drafting gets past the ceiling

A cheap drafter proposes several tokens and the main model verifies them in one parallel pass, so the weights are read once for the batch. The original speculative decoding work, published by Leviathan and colleagues in 2022, reported two to three times faster T5-XXL generation with identical outputs, because the acceptance rule preserves the output distribution. Everything after the first rejected guess is discarded, so the gain depends on how predictable the text is. A community MTP repository documented larger gains on code prompts than on prose, and its acceptance rate was 0.78 on a 3090 against 0.56 on a 4090, yet the 4090 gained more without an explanation published on the page.

Offload for models larger than the card

An archived Strata GLM experiment on an RTX 5090 reported about 1,000 tokens per second of prefill against llama.cpp's 29, but decode of only 15 to 18 against 2.4 to 2.9. In that run each token routed to 336 experts totalling 4.76 GB, with 2.2 to 3.2 GB moved over PCIe per token and 10 to 20 experts pulled from disk. The best possible expert hit rate was 73%, and the engine reached 55 to 58 tokens per second. Prefill wins because one expert fetch is reused across many tokens; decode pays the transfer bill again for every token.

llama.cpp keeps moving the baseline with MTP and rejection sampling

llama.cpp has merged multi-token prediction and rejection sampling, so the floor these engines must beat rose during 2026. A fork's ratio measured against an older build is stale by default unless it says which build it used.

llama.cpp merged pull request 22673 on 16 May 2026, letting a model's built-in MTP head draft two or three tokens for the main model to verify. At three drafts the pull request reports about 75% steady-state acceptance. On a 3090 with Q6K quantization it reports 22.97 rising to 42.45 tokens per second, about 1.85 times, for roughly 2.5 GB of extra VRAM. Listed limits include one sequence at a time and prefill dropping from 1,315 to 665 tokens per second on the same file.

Pull request 27694, merged on 2 October 2026, added a spec-draft sampling flag that switches the drafter from greedy to probabilistic guessing, with the main model accepting or rejecting each guess by comparing probabilities. At temperature zero nothing changes. Between 0.5 and 1.0, the pull request's tables on an RTX 5090 and a DGX Spark show throughput gains of roughly 5 to 15%. The widely repeated "plus 14% on prose" figure does not appear in the pull request; the closest value is 14.09% at temperature 0.9, which is accepted length on a draft test rather than throughput.

The practical effect is that a fair comparison now has to state whether MTP was on, at what temperature, and on which build. Without those three facts, a ratio can be reproduced only by accident.

Reading the unsourced claims: the V620 fork, the 140 tok/s digest and Lamina

Three claims in the set cannot support a purchase decision, because their pages contain no comparable llama.cpp figure or no primary source at all.

The V620 vLLM fork reports about 70 tokens per second for one stream across eight Radeon Pro V620 cards and about 64 on four, with prefill around 1,200 tokens per second on four cards. No eight-times figure appears anywhere in the repository. Its kernel changes are concrete, including a mixture-of-experts kernel for four-bit weights with eight-bit activations worth 27%. Building it takes about three hours and needs ROCm 7.1.4, AMD's GPU software stack. The README describes the repository as AI slop and disclaims any warranty.

The 140 tokens per second claim traces only to a German digest of a Reddit post about a CUDA megakernel written with Claude Code, Anthropic agentic coding tool. The digest reports 140 against 73 on code and 95 against 56 at 4K context, with llama.cpp plus MTP as the baseline. It links neither the post nor the repository and gives no settings. A 73 baseline is already above the 41.3 a community MTP repository documents on a 3090, so the quantisation or prompts must differ, and neither is published.

A Lamina claim of 44 against 29 on an RTX 3050 leads nowhere. The only published Lamina material is an unrelated paper on data-centre attention offloading. With no repository, post, or model card, the number cannot inform any decision.

A three-step check before you install anything

Three checks take about as long as reading one README and settle most local AI inference speed claims before you spend a weekend on setup.

Divide your card's memory bandwidth by your model file's size in GB. The result is the plain decoding ceiling in tokens per second for a dense model. A claim far above it needs drafting, a smaller file, or an explanation the page does not give.
Look for the baseline's build number, its offload flags, and whether MTP was enabled. A ratio without a flag list cannot be reproduced, which is exactly the gap in the Strata Pascal table.
Confirm the model, hardware, context length and quantisation the number was reported for. A published figure says little about the model you switch to next month.

Engine	Published evidence	Comparable llama.cpp figure	Best fit
Strata Pascal fork	Decode table at 8K and 16K, 4 GB GTX 1050 Ti laptop	Yes, build b11524, flags undisclosed	Older Pascal laptop GPUs with 48 GB RAM
Lithos-Metal	Target compute projections on M5 Max	No end-to-end figure	Apple silicon, projections only
PolyStrata	57 to 59 tok/s on RTX 4070 laptop	None published	Linux, around 100 GB RAM
Glyd	45.5 tok/s on RTX 4090, 4K context	None published	64 GB RAM, Ampere or newer, personal licence
V620 vLLM fork	70 tok/s across eight V620 cards	None published	Owners of V620 cards
Crazystack typescript tooling and community write-ups	Mixed, community testing notes	Varies by author	Learning the measurement method

For a 24 GB card running a 27B Qwen build, the first move is llama.cpp's own MTP flag, documented at plus 33% on a 3090 and plus 60% on a 4090 in the community repository. Drafting two tokens at a time and serving one request at a time are the documented limits. If you sample at 0.5 or above, the newer spec-draft sampling flag is worth trying, with the caveat that its tables cover only 8B and 9B models, so the gain on a 27B build is untested.

Frequently asked questions

Does any engine actually run local AI twice as fast as llama.cpp?

One relative in this set published a side-by-side table: the Strata Pascal fork, at 3.1 times decode on a 4 GB GTX 1050 Ti laptop against a specific llama.cpp build. Its own README withholds the baseline's offload flags, so the true size of the gap is unconfirmed. The other ratios lack a comparable run on the same page.

Why does plain decoding stop near 55 tokens per second on an RTX 3090?

Because each token requires reading roughly every active weight once, so speed is bounded by 936 GB/s divided by the file size. A four-bit 27B file of 17 to 19 GB gives about 49 to 55 tokens per second. Only speculative drafting or a smaller file moves past that bound.

Is llama.cpp's MTP flag worth enabling?

For single-user decoding it can be. The merged pull request reports 22.97 rising to 42.45 tokens per second on a 3090 with Q6K, about 1.85 times, for roughly 2.5 GB of extra VRAM. It serves one sequence at a time and reduced prefill from 1,315 to 665 tokens per second in that report.

Does rejection sampling help at temperature zero?

No. The spec-draft sampling flag changes nothing at temperature zero, because greedy output is already deterministic. Gains of roughly 5 to 15% appear in the pull request's tables between temperature 0.5 and 1.0 on an RTX 5090 and a DGX Spark.

How do I check a speed claim in a few minutes?

Divide your card's bandwidth by the model file's size to get the ceiling, then find the baseline's build, flags and MTP state, then confirm the model and hardware the number was reported for. If any of those three is missing, the ratio cannot be reproduced on your machine.

Turning a careful benchmark read into something people can find

The lesson in this audit is that a number without its baseline, build and flag list can mislead even when it is honestly reported. Anyone who has walked through that reasoning on video, or has a tested workflow like the ones around Crazystack, already holds material that is hard to find in prose. Skala blog turns that spoken analysis into a written draft you can review before publishing.

If you have recorded an explanation of how to read a benchmark, audit a repository, or compare tooling, you can paste the YouTube URL into Skalablog, get a transcription, and turn it into a structured article. The reasoning that took nineteen minutes to say becomes a page someone can search, quote and check against the sources you cited. That is the value of writing down the checks: other people can repeat them. The workflow starts at Skala Blog.

Source video