Running a 125B MoE model on an 8GB GPU works because a Mixture-of-Experts model activates only a fraction of its weights per token. In llama.cpp you keep the always-on attention layers, KV cache and shared experts in VRAM and push routed expert weights to system RAM with --cpu-moe. Enthusiast reports describe 17 to 19 tokens per second on an 8GB RTX 4060 Ti, but those numbers are unverified.
What does "running a 125B MoE model on a 8GB GPU" actually mean?
Running a 125B MoE model on an 8GB GPU means keeping the weights that every token touches in VRAM and the infrequently used expert weights in system RAM, so the card never has to hold all 125 billion parameters. The approach depends on a Mixture-of-Experts architecture, where each token activates only a small subset of the network. llama.cpp implements this split with two flags, and the result is a working setup rather than a 125B model resident in 8GB.
A Mixture-of-Experts (MoE) model stores many separate weight blocks called experts and routes each token to a few of them. The model file stays large, but the compute path per token stays small. That gap between stored size and active size is what makes a small card viable.
The llama.cpp project is an open-source C/C++ inference engine for running quantized language models on consumer hardware. Its current documentation and release history live in the ggml-org repository. The expert-offload flags discussed here are specific to that implementation, and the throughput figures belong to the users who measured them, not to the model's publisher.
Why does Qwen3.8-Flash-Next need only 6B active parameters?
Qwen3.8-Flash-Next needs only about 6B active parameters per token because it is a Mixture-of-Experts model with 48 layers, each holding 512 routed experts, and each token is sent to 10 of those 512 experts plus one shared expert. The other experts stay idle for that token, so the card is never asked to compute the whole 125B.
The arithmetic is straightforward. With 48 layers and 512 routed experts per layer, the model stores 24,576 experts in total. Each token touches 10 routed experts in every layer, so 480 expert activations run for one token, and the remaining experts do nothing until a later token selects them.
Some parts run for every token regardless of routing: the attention layers, the dense feed-forward layers, and the shared expert. The attention layers also read the KV cache, which stores attention state for every earlier token in the conversation and grows as the conversation lengthens. Those always-on parts are the natural residents of VRAM.
The routing pattern gives one rule worth remembering. The GPU keeps what every token touches. The system RAM keeps what only some tokens touch. The exact byte size of the always-on portion is not published in the sources reviewed for this article, and the creator of the original video estimated it by hand rather than measuring it.
How do expert split and layer split differ in llama.cpp?
Expert split and layer split differ in what fills VRAM: layer split stores whole transformer layers on the GPU, while expert split stores only always-on components and sends routed experts to CPU memory. On an 8GB card, layer split wastes precious VRAM on experts that may never run for the current token.
The default llama.cpp split mode assigns whole layers to the GPU up to the -ngl limit, and the rest go to the CPU. A whole layer includes its rarely used routed experts, so the card fills with weights that are inactive most of the time. Expert split inverts that priority.
| Setting | What goes to VRAM | What goes to system RAM |
|---|---|---|
| Layer split (default) | Whole layers up to the -ngl limit | Remaining whole layers |
Expert split (--cpu-moe) | Attention, dense layers, shared expert, KV cache | All routed expert weights |
Partial expert split (--n-cpu-moe N) | Always-on parts plus experts of later layers | Expert weights of the first N layers |
To enable the expert split, set --n-gpu-layers 999 so llama.cpp does not stop assigning layers at a low number, then add --cpu-moe. The --override-tensor flag with exps=CPU achieves a similar result. The --n-cpu-moe N flag keeps the expert weights of the first N layers on the CPU, so a smaller N leaves more expert layers on the card.
According to the project's own release history, llama.cpp merged the --n-cpu-moe pull request in early August 2025, a few days after the --cpu-moe flag landed. Anything written before that date describes a different set of options, and build numbers matter because the flags evolved quickly.
What crosses PCIe during a forward pass?
What crosses PCIe during a forward pass is a small activation vector per layer, not multi-gigabyte weight matrices. The token's activation travels from the GPU to system RAM, the CPU runs the selected experts, and the result returns over the same link. With 48 layers, one token can involve up to 48 such round trips.
Copying weights instead would be far slower. A developer-community explainer cited in the source video gives an example where 300GB of weights on the CPU side would take at least about 10 seconds for one batch over a PCIe 4.0 x16 link. Sending an activation vector takes a fraction of that time.
PCIe is the connection between the graphics card and the rest of the computer. Its latency, not its bandwidth, becomes the likely bottleneck once activations are tiny. The developer-community author named PCIe latency as the constraint but published no benchmark, so the size of that penalty remains untested in the reviewed material.
This has a practical consequence. A setup that moves small vectors repeatedly is sensitive to per-call overhead, which is why the operating-system layer and the number of sequential launches described later can matter more than raw memory bandwidth.
What are the three tuning knobs for 8GB VRAM?
The three tuning knobs for 8GB VRAM are moving the display to the CPU's integrated graphics, running native Linux instead of WSL2, and choosing the right llama.cpp build for each model. Users credit these settings with gains, but the sources do not decompose how much of the jump from about 4 to 17 or 19 tokens per second comes from each one.
The first knob is display routing. If the CPU has integrated graphics, plug the monitor into the motherboard output instead of the discrete card, which frees VRAM that the model can use. The user credited a 20 to 30 percent gain on the 8GB card. A separate report on an RTX 4070 Super running a different model showed 50 to 55 tokens per second, a 10 percent gain, while commenters reported 20 percent. The 8GB claim sits at the high end of that range.
The second knob is the operating system layer. NVIDIA's WSL2 documentation notes that GPU calls pass through a virtual link and that WSL limits pinned memory, which is RAM the GPU can copy from quickly. The user credited 33 to 38 percent from running native Linux instead. No reviewed source explains that specific magnitude.
The third knob is build selection. The user says the fastest llama.cpp build differs between the 35B model and Qwen3.8-Flash-Next but did not say which build wins. Record the build number for every run, because the flags and kernels change between releases.
RAM speed is a smaller factor. One user overclocked DDR5 for 36 percent more bandwidth and measured a 5 to 14 percent throughput gain. That suggests memory bandwidth is a limit, but not the only one.
How much VRAM and RAM does the memory math require?
The memory math requires roughly 72GB of fast storage plus system RAM, because the smallest published quantized file is 72.5GB and 8GB of VRAM plus 64GB of RAM totals 72GB. Something else has to hold part of the model, and in this setup it is the SSD.
Quantization stores each weight in fewer bits than the 16-bit original. Unsloth's model files list a smallest compressed file at 72.5GB, then 78.9GB, then 82GB, then 111GB, against 354GB for the original BF16 weights. At an assumed 3 to 3.5 bits per weight, the 6B active parameters come to about 2.25 to 2.6GB per token, which fits in 8GB with room for the KV cache.
The extra memory pressure comes from a 51B-parameter n-gram lookup table that sits on top of the 125B main model. Unsloth says the table can live on the SSD and be memory-mapped, so the system reads it from disk on demand. At 4 bits, a hand estimate puts that table near 25GB, which leaves about 47GB for the remaining weights inside a 72GB budget.
No source reviewed for this article gives a target disk speed for the memory-mapped table. Treat the SSD as a requirement without a published performance target, and keep the operating system and KV cache in the RAM budget.
What throughput do 8GB users report, and how reliable is it?
User reports describe 17 to 19 tokens per second from an 8GB RTX 4060 Ti with 64GB of RAM, against about 4 tokens per second on default settings. Those figures come from individual Reddit posts, so they are user results rather than a controlled benchmark.
Unsloth's Hugging Face discussion thread collects other speeds for the same model. A 24GB RTX 3090 with 128GB of DDR4 reports 19 tokens per second, and a 16GB RTX 5080 with 64GB of DDR5 reports 27.5 to 29, a figure that also uses n-gram speculation and is therefore not a plain comparison. The thread contains no 8GB report, so nothing there corroborates the 17 to 19 figure.
| Hardware | VRAM | System RAM | Reported speed | Notes |
|---|---|---|---|---|
| RTX 4060 Ti | 8GB | 64GB | 4 tok/s default, 17-19 tok/s tuned | User post, unverified |
| RTX 3090 | 24GB | 128GB DDR4 | 19 tok/s | Thread report |
| RTX 5080 | 16GB | 64GB DDR5 | 27.5-29 tok/s | Includes n-gram speculation |
| RX 7600 with Strata | 8GB | not stated | 24 tok/s | Depends on an unmerged fix |
The 35B model on the same 8GB card went from 25 tokens per second to between 52 and 65 after tuning. Different machines, different RAM and different SSD configurations all sit behind those numbers.
Does Strata change the answer for AMD cards?
Strata changes the answer for AMD cards only partly, because its published requirements call for 12GB of VRAM or more and its documentation lists no 8GB result. The engine is a dedicated implementation for this model family with a different split strategy, and the reported 8GB AMD result sits outside what its maintainers state as supported.
Strata keeps the few thousand most frequently used experts in VRAM while system RAM holds all of them, and the CPU runs the cache misses. Its README warns that 35 to 55GB loads into RAM at startup and that an SSD holds a lookup table. A small helper model guesses the next few words and the large model checks them together, which its README describes as the same answer 1.6 to 1.8 times sooner. That figure is the project's own and has not been independently checked.
In Strata's own benchmarks, a 12GB RTX 570 reaches 94 tokens per second on the Q2 file and 53 on the IQ3 file, while a 16GB RX 970 XT reaches 60 on the Q2 file. Those are vendor-reported numbers on specific quantizations, and they do not transfer to an 8GB card.
One user reports an 8GB RX 7600 reaching 24 tokens per second, but the result depends on a fix that is not merged into Strata, and there was no public pull request or issue for it as of October 2026. Treat that figure as unverifiable and follow the supported 12GB path if you own a listed card.
What quality do you give up with low-bit quantization?
Low-bit quantization trades answer quality for speed, and none of the reviewed sources measure how much quality the smallest files lose on this model. The 72.5GB file is far smaller than the 354GB BF16 original, so the compression is aggressive by construction.
Strata's README states that its smaller sizes sacrifice quality for speed. Unsloth keeps the n-gram lookup table at 4-bit or higher, so some components stay more precise than the main weights. A fast token rate only helps if the output is accurate enough for the task at hand.
The open question is worth testing directly. On your own tasks, does a 125B model compressed to a few bits answer better than a smaller model kept at higher precision? No source reviewed for this article measured that comparison, and a short side-by-side test on your own prompts would settle it for your workload.
What should an 8GB owner try first?
An 8GB owner should try the expert split first, measure tokens per second, and then change one setting at a time so each gain is attributable. Start with --n-gpu-layers 999 plus --cpu-moe, confirm that VRAM stays stable, and only then move the display, switch the operating system or swap builds.
- Launch llama.cpp with
--n-gpu-layers 999 --cpu-moeand a compressed model file, then record tokens per second on a fixed prompt. - If the CPU has integrated graphics, move the monitor cable to the motherboard output and re-measure.
- If you run WSL2, boot native Linux and re-measure, expecting a gain but not a guaranteed 33 to 38 percent.
- Try a different llama.cpp build for the specific model and note the build number for every run.
- Keep at least 64GB of system RAM, an SSD, and a compressed file of 72.5GB or larger in the plan.
Plan the storage budget before the software budget. The smallest file alone exceeds the combined 8GB of VRAM and 64GB of RAM, so the SSD and the memory-mapped table are part of the working setup rather than an optional extra.
Where the setup borrows ideas from a broader Brazilian stack
Tooling culture around local inference overlaps with the wider developer ecosystem in Brazil, where communities such as Dev doido discuss practical setups for constrained hardware. The same instinct behind a three-tier memory plan shows up in how developers evaluate other stacks, including the TypeScript tooling collected around Crazystack, a Brazilian project hub that gathers starter kits and templates.
A reader who wants to turn a working local setup into written notes for a team will find that habit familiar. Skala blog covers this kind of workflow documentation, and the same principle applies to any stack: record the flags, the versions and the measured result, then share the reproducible part.
None of these projects is required for the llama.cpp configuration described above. They are simply part of the same community practice of testing on modest hardware and publishing exact settings so others can repeat the result.
FAQ
- Can an 8GB GPU run a 125B parameter model?
It can when the model is a Mixture-of-Experts design and the inference engine keeps only the always-on weights in VRAM. llama.cpp does this with --cpu-moe, which sends routed expert weights to system RAM. The card still holds attention, dense layers, the shared expert and the KV cache.
- What do the llama.cpp flags --n-gpu-layers 999 and --cpu-moe do?
--n-gpu-layers 999 tells llama.cpp not to stop assigning layers to the GPU at a low count. --cpu-moe then keeps the routed expert weights on the CPU. Together they produce the expert split instead of the default whole-layer split.
- How much system RAM do I need for this setup?
Plan for about 64GB of RAM, an SSD and a compressed model file, because the smallest published file is 72.5GB and 8GB of VRAM plus 64GB of RAM totals 72GB. The n-gram lookup table can be memory-mapped from the SSD to free RAM.
- Is the reported 17 to 19 tokens per second a benchmark?
No. Those figures come from individual Reddit users on their own machines, and Unsloth's thread has no 8GB report to confirm them. Treat them as user results and measure your own throughput after each change.
- Does Strata support 8GB AMD cards?
Strata's README states a requirement of 12GB of VRAM or more and lists no 8GB result. The reported 8GB RX 7600 run depends on an unmerged fix, so it is not a supported configuration as of October 2026.
Turning this working setup into a written guide
The lesson behind this setup is that the numbers only become useful when someone writes down the flags, the build number and the measured result in one place. If you have already recorded that journey on video, the knowledge exists but stays locked in a format that is hard to search and hard to reference in a team wiki.
That is the gap Skala Blog addresses. Paste a YouTube URL, let the video be transcribed, and generate a structured article you can edit, publish and cite. A video walkthrough of a llama.cpp expert split becomes a written reference with the same settings and the same caveats.
Visit Skala Blog at https://skalablog.com to turn a recorded explanation into an article that readers can follow step by step.
Fork this article
Start a new branch from the same video, shaped your way. You keep the credit; the original keeps the attribution.
A fork in another language is filed as a translation of this article, so the two pages point at each other. You can unlink it later from the editor.
0/240
You are creating
- Format
- For
- Language
- Source
- Your angle
You will be asked to sign in before it is generated.
Buy credits