Kimi K3 tops the intelligence index and cuts output tokens 21% — does the savings hold?

Short version: Kimi K3 now sits on top of the Artificial Analysis Intelligence Index, and Moonshot claims it burns ~21% fewer output tokens than K2.6 on the same tasks. Simon Willison ran it through his SVG-pelican test, and the results look real — the pelican is more coherent and the token count on that specific draw came out lower. But a 21% cut on a benchmark suite is not a 21% cut on your bill, and that distinction matters if you're the one paying per million tokens.

What is the pelican benchmark actually measuring?

Simon Willison's pelican-SVG test asks a model to draw a pelican riding a bicycle as raw SVG. It's a cheap proxy for spatial reasoning plus instruction-following plus the discipline to emit valid, compact markup. Kimi K3 producing a recognizable pelican with fewer tokens is a decent signal that the model isn't padding its output with dead XML and redundant path points — which is where a lot of models waste tokens.

Does the 21% token cut hold up in a real pipeline?

Here's the gotcha. The ~21% output-token reduction is an aggregate over Artificial Analysis's eval mix. Your workload isn't that mix. If you're doing structured extraction with tight JSON schemas, the savings shrink because there's not much fat to trim — the output length is bounded by the schema, not the model's verbosity. Where you'll actually see it is in open-ended generation: summaries, code, long-form reasoning traces where a chattier model runs longer.

The other thing people forget: output tokens are only half the bill. If K3 reasons more before answering, you can eat the savings back in hidden thinking tokens, depending on how Moonshot meters reasoning output. The pelican draw is a small output; it doesn't stress that path. I'd want to see it on a 4k-token generation task before trusting the headline number.

What broke when I reasoned it through?

Nothing broke in the pelican test itself — that's the point, it's a clean win there. What breaks is the extrapolation. "Tops the index AND cheaper per output token" reads like a free lunch, and free lunches in open-weights usually hide in one of three places: license terms, context-window pricing, or inference cost on the hardware you actually own. An open-weight model that beats closed frontier models on an index is genuinely useful, but the index score doesn't tell you how it behaves on your prompts at your temperature settings.

Should you use it?

If you're already self-hosting open weights, K3 is worth a real A/B against whatever you run now — the token efficiency is a legitimate cost lever and the benchmark position means you're not trading away quality to get it. If you're on a hosted closed model and happy, don't switch on the strength of one leaderboard and one pelican. Run K3 on a representative slice of your own traffic, measure output tokens AND latency AND any reasoning-token overhead, then decide. The pelican is a green flag, not a migration plan. I'd pull it into a bake-off; I wouldn't rip out production on it. That's the honest read from Simon Willison's writeup and my own back-of-envelope on where the savings evaporate.

FAQ

Is Kimi K3 open-weight?

Kimi K3 is from Moonshot AI, which has released its prior Kimi models as open weights. Confirm the specific K3 license and weight availability before you build on it — open-weight status and commercial-use terms are separate things worth checking.

Will I actually save 21% on tokens?

Probably not exactly. The ~21% figure is an aggregate over Artificial Analysis's eval mix. Schema-bound or short outputs will save less; long open-ended generation will save more. Measure on your own traffic before you count the money.

Is the pelican-SVG test a serious benchmark?

It's a lightweight community sanity check, not a rigorous eval. It probes spatial reasoning, instruction-following, and compact valid output in one cheap prompt. Good as a smell test, not a substitute for evaluating on your actual workload.