If you buy frontier API access because local models aren't smart enough, Kimi K3 tested here may change the math. Moonshot AI's open-weight model reached 43.1 tokens/second, 88% reasoning, and 100% at 256K context, all at per-million prices well below the bigger labs.
Kimi K3 tested: what the benchmark suite measured
Kimi K3 tested on six tasks delivered 43.1 tokens/second decode, 88% on reasoning, 94% on HumanEval, and a perfect 256K context result. Moonshot AI’s open-weight model ran through performance, memory, reasoning, coding, sand physics, and Godot game generation using the Moonshot AI API at 1 million context.
The test suite comes from Luke’s Dev Lab, a YouTube channel that runs local and frontier models through the same hands-on tasks. The video is sponsored by Moonshot AI and published on 2026-09-18, but the creator states that sponsorship did not change the test execution.
The most important caveat is scope. The memory test ran at 256K, not the advertised 1M context, because of API timeouts during longer runs. The reasoning and HumanEval scores come from benchmark-style tasks that may be in training data, so treat them as directional evidence for this setup rather than a general ranking.
How fast is the Kimi K3 API at 1M context?
The Kimi K3 API generated about 43 tokens per second during decode at 1M context. Prefill speeds climbed from roughly 160 uncached and 240 cached tokens per second to just under 10,000 cached and 11,800 non-cached tokens per second near the end of the run.
That prefill pattern is odd. Uncached numbers usually sit below cached ones, and the two tracks moved together across the run, which led the creator to suspect the benchmark harness was reading cached numbers in both tracks. The decode figure of 43.1 tokens/second is the cleaner signal from this test.
For context, the model is a 3 trillion parameter model with a 1 million token context window, according to the video. Serving that footprint at 43 tokens per second on a hosted API is a practical result for interactive use, though the creator notes the suite was not tailored for frontier-scale tasks.
Did the 256K memory test pass?
The model found the inserted datum on all 15 runs at every tested context depth. The memory test inserts a piece of information at 0, 25, 50, 75, or 100% of context depth, then asks the model to retrieve it, repeating each depth three times at 256K context.
The full 1M run did not complete. The creator said the benchmark kept timing out from the API when pushed to 1 million context, so the practical memory result is 256K, not the advertised ceiling. All other tests also ran at 256K for comparison with earlier local-model results.
Output quality varied in presentation but not in correctness. Some answers came back clean, others carried padding or extra information, but none produced scroll bars or other signs of a badly malformed response.
How did Kimi K3 score on reasoning and HumanEval?
The reasoning suite produced 42 out of 48 passed tasks, an 88% pass rate, with an unexpected dip on medium-difficulty items. The model missed two medium tasks, one hard task, and three expert tasks. The creator expected medium to be easier, so the pair of medium misses is the notable pattern.
HumanEval, OpenAI’s 164 handwritten Python challenge set, gave the model an 86% pass rate across all questions and a 94% pass rate on questions it answered. That corresponds to 14 unanswered questions and a 91% answer rate within an 8,000 token budget.
The HumanEval result comes with the creator’s warning that the challenges are almost certainly in training data, so it is not a clean generalization measure. The 94% answered-question figure shows the model did not overthink within the token limit, but it does not prove coding ability on unseen problems.
Front-end and algorithmic coding results
The web development test produced a feature-complete Kanban board called Flowboard from a zero-shot prompt. The model delivered working features, an intuitive UI, a small logo, and a prompt on unsaved changes when cancel is pressed. It missed button colors, and the run consumed 7.1% of the 1 million context at a cost of $1.44.
Sand physics and dungeon crawler tests also passed from zero-shot prompts. The sand simulation was fully functional with splash physics, keyboard selection, brush sizing, placement, pause, and clear canvas, using 2.7% context at a cost of $0.47. The dungeon crawler delivered ray casting, dungeon generation, and god mode, using 3.2% context at $0.57.
One flaw appeared in the dungeon crawler: fog of war did not dim explored areas after the player left. The creator still marked it a pass because the required mechanics worked. A 3D dungeon crawler built in Godot as a bonus test delivered movement, fog of war, god mode toggle, map regeneration, and a finish goal, using 14.6% context at just under $6.
Kimi K3 benchmark results at a glance
| Task | Result | Context used | Reported cost |
|---|---|---|---|
| Decode speed | 43.1 tokens/sec | 1M context | Not stated |
| Memory retrieval | 15/15 runs passed | 256K context | Not stated |
| Reasoning | 42/48 (88%) | 1M context | Not stated |
| HumanEval | 94% of answered | 1M context | Not stated |
| Kanban (Flowboard) | Pass, missing button colors | 7.1% (70K tokens) | $1.44 |
| Sand physics | Pass | 2.7% (27K tokens) | $0.47 |
| Dungeon crawler (web) | Pass, fog of war flaw | 3.2% (30K tokens) | $0.57 |
| Blender lantern | Pass, glass panes off | 4.6% (46K tokens) | $0.92 |
| Godot moving platforms | Pass, very glowy | 17.2% (170K tokens) | Just under $7 |
| Godot dungeon crawler | Pass, dark scene fixed | 14.6% (146K tokens) | Just under $6 |
All costs are reported by the creator from the Moonshot AI API at max reasoning. The context percentages are calculated against 1 million tokens, which is why the raw token counts look low compared with tests scaled to a 128K window.
What do the Blender and Godot agentic tests show?
The Blender test asked for a Starfall lantern asset through MCP, a protocol that lets a model call external tools. The model delivered on a one-shot prompt after the creator asked for missing screenshots, then finished at 4.6% context and $0.92. The rendered asset looked strong overall, but the glass panes were too small, a common failure across models.
The Godot moving-platforms game was the hardest test and produced a playable 3D scene with moving platforms, collectible keys, respawning, hazards, and a goal. The creator had to interrupt the model’s own testing loop to test the game manually, which the model treated as an unexpected event in its reasoning.
The final Godot dungeon crawler reproduced the web dungeon crawler as a 3D game with a minimap. The only prompt during the run was a note that the scene was too dark. A missing facing-direction indicator on the minimap is the clearest usability gap, but map generation, god mode, and goal completion worked.
What does Kimi K3 cost per task?
Moonshot AI’s published API pricing for Kimi K3, as shown in the video, is $0.30 per million tokens for cache hits, $3 per million tokens for uncached input, and $15 per million tokens for output. That makes the hosted API far cheaper than comparable Claude or OpenAI tiers for the same token volume on paper.
The per-task totals in this suite ranged from $0.47 for sand physics to just under $7 for the Godot moving-platforms game, all at max reasoning. The creator notes that max reasoning was probably overkill for simpler tasks and that dropping to medium reasoning should cut costs further with similar end results.
Token efficiency is the second factor. The coding tasks used between 2.7% and 17.2% of the 1 million context, and the creator described those totals as low compared with what local models typically consume on the same tasks.
Frequently asked questions about Kimi K3 tested results
- Is Kimi K3 open-weight? Yes. The video describes Kimi K3, made by Moonshot AI, as an open-weight model that can run on your own hardware if that hardware is large enough. The tests here used Moonshot AI’s hosted API instead of a local deployment.
- What is Kimi K3’s context window? Moonshot AI advertises a 1 million token context window, and the performance and agentic tests ran at that size. The memory retrieval benchmark only reached 256K because of API timeouts at longer depths.
- How fast is Kimi K3 on the API? The measured decode speed was about 43 tokens per second at 1 million context. Prefill climbed toward 11,800 non-cached tokens per second near the end of the run, though the creator suspected cache tracking issues in that chart.
- How much did the full test suite cost? Individual tasks ranged from $0.47 to just under $7 at max reasoning. The total is not reported, but the per-task costs show that the Godot tests dominate the bill.
- Is Kimi K3 better than ChatGPT or Claude? This suite does not compare models head to head. The creator describes Kimi K3 as frontier level and a strong cheaper alternative, but the reasoning dip on medium tasks and the missing context test at 1M limit any broad claim.
Where the tests stop and what to verify yourself
Agentic benchmarks run by one creator on one API configuration show behavior on that suite, not a guarantee across production workloads. The reasoning misses on medium difficulty suggest the model’s behavior varies by prompt structure, and the HumanEval warning about training data applies to any coding score from that benchmark.
The performance and memory limits matter for anyone planning 1M-context work. The timeout at full context means the 256K pass is well evidenced while the 1M claim remains unverified by this suite. Anyone depending on the full window should run a retrieval test at their own target depth before committing a production workflow.
For developers who want a frontier-class hosted model at lower per-token prices, the Kimi K3 API is worth a scoped trial. Tools like Crazystack and communities such as Dev doido and Crazystack TypeScript are good places to compare notes on real workloads before switching a pipeline.
From a video review to a written guide
A benchmark review like this one carries useful detail that is easy to lose inside an 18-minute video: which test passed, which cost what, and where the harness hit a limit. If your own videos hold similar knowledge, Skala Blog turns a pasted YouTube URL into a transcribed, structured article draft. Visit Skala Blog to try the flow.
Fork this article
Start a new branch from the same video, shaped your way. You keep the credit; the original keeps the attribution.
A fork in another language is filed as a translation of this article, so the two pages point at each other. You can unlink it later from the editor.
0/240
You are creating
- Format
- For
- Language
- Source
- Your angle
You will be asked to sign in before it is generated.
Buy credits