Kimi K3 open weights: 2.8T MoE, hybrid attention and no positional encoding
Quote892 words
Explains Kimi K3's architecture (Kimi Delta Attention in a 3:1 hybrid, 16 of 896 experts active, no positional encoding, MXFP4 QAT) and its benchmarks, pricing and agentic results. Concludes open weights are narrowing the US-China gap.
Key claims
2.8T params, 104B active, 1M context; $3.00 input / $15 output per MTok
Top WebDev Arena at 1,678 Elo; #4 on the AA Intelligence Index
Found 16 unreported flaws across 6 projects; solved 14 of 36 exploit tasks vs GLM-5.2's 8
So @Kimi_Moonshot followed through.
Kimi K3 model weights and tech paper are out.
Here are the key points and why it's a game-changer:
→ It's a 2.8T parameter MoE with 104B active, native vision and a 1M context window, which makes it the first open model in the (almost) 3T class. MoE means the model is split into a lot of small specialist networks and only a few of them wake up for any given token, so despite the 2.8T headline number you're only ever paying to run 104B of it. That's the only reason a model this size is servable at all. The 1M context means you can hand it something like an entire codebase or a few hundred pages of documents in one go, and native vision means images go in directly instead of through a bolted-on separate encoder.
→ The architecture is a full rewrite rather than just a Kimi K2 refresh. Kimi Delta Attention sits in a 3:1 hybrid with Gated MLA, which means three out of every four layers use the cheap linear attention and only every fourth layer does the expensive full version, so long inputs stop being quadratically painful. Attention residuals run across depth, letting later layers reach back to what earlier ones already worked out instead of rediscovering it. And Stable LatentMoE fires on 16 of 896 experts, which is a much finer-grained routing setup than K2 (also MoE) had. Add it up and they get roughly 2.5x the scaling efficiency of K2, meaning the same compute budget buys you significantly more model output.
→ They also dropped positional encoding entirely. Normally a model needs an explicit mechanism to know which token came first, and stretching that mechanism past its training length is where RoPE rescaling and YaRN come in, both of which are basically duct tape that degrades quality the further you push. K3 has none of that patch work. Order information falls out of KDA's decay behaviour on its own, so the 1M window is something the model actually trained into rather than something glued on afterwards. If that holds up outside the paper it could rly change how everyone builds long context from here.
→ On benchmarks it's strong but not on top. Best in class on ProgramBench (77.8), SWE-Marathon (42.0, a full 7 over Fable 5) and BrowseComp (91.2), but it still trails Fable 5 and GPT-5.6 Sol overall while beating every other model in the suite. It's also the first open model to top WebDev Arena, at 1,678 Elo above Fable 5, and it lands #4 of 580 on the AA Intelligence Index and #2 of 39 on Vals. See the attached image for more details on benchmark performance.
→ The pricing is where it gets rly interesting imo. $0.30 cached, $3.00 input, $15 output per MTok, and it hit the top BrowseComp score at $2.03 a task, def an order of magnitude under Claude running at max effort.
→ The agentic results are also a key point, because they're real engineering work, not just a bunch of test questions. And they're very impressive: In one 48h autonomous run Kimi K3 designed, optimized and verified an inference chip, 4mm² doing 8.7k tok/s decode, with no human in the loop. It wrote a Triton-like compiler from scratch that beats torch.compile, which is PyTorch's own production optimizer with years of work behind it. And, it took its own AttnRes kernel from 283ms to 114ms, so the model made the thing it runs on 2.5x faster. Pretty damn crazy.
→ They also pointed it at security research, and it found 16 software flaws nobody had reported before across 6 projects, two of them in the Linux kernel. On their own exploit test set it solved 14 of 36 against 8 of 36 for GLM-5.2. There's nothing to compare it against from the US labs, because they don't let their models work on this at all.
→ Moonshot ran MXFP4 QAT from SFT onward, which means the model was trained in 4-bit from the fine-tuning stage rather than trained at full precision and compressed afterwards. Normally, everyone quantizes to fit models onto cheaper hardware and everyone loses a bit of quality doing it. Here the small version is the real version, so there's no gap between the model they benchmarked and the model you can actually deploy. Pretty cool approach.
→ To their credit they list what's broken. It falls apart if your harness drops thinking history between turns, it improvises too much when you give it a vague instruction, and they flat out say the UX is worse than Fable 5 (appreciate the honesty).
Conclusion:
Would def not have thought that we see open-weight models compete with the closed-source frontier so quickly, but well, here we are. Imo a huge win for the open-source AI sector, and a clear sign of a narrowing US-China gap. Interesting tho how it seems to capture less attention / cause less panic than Deepseek did back in the day. Due to size probs difficult to run at home, but if you have the money to spend on the infra, nothing stops you from having (almost) superintelligence running on your desk. Wild stuff.
@Kimi_Moonshot More on Kimi K3 from announcement day:
x.com/zerokn0wledge_/status/2077850802014859294
@Kimi_Moonshot Am also building an agent with Kimi K3 under the hood.
Stay tuned.
x.com/zerokn0wledge_/status/2080321376226062615
QUOTING @Kimi_Moonshot
Releasing the model weights and technical report of Kimi K3. Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window. New model architecture: 2.5x the intelligence per unit of compute, not just more params. Alongside K
Releasing the model weights and technical report of Kimi K3. Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window. New model architecture: 2.5x the intelligence per unit of compute, not just more params. Alongside K
