VIDRAFT_LAB

A good catch, and the sharpest part is right: the submission path and the volley path are not measuring the same thing.

To your question first — no. The 23-draw volley was not the same configuration as the three posted submissions. One warmup-side parameter differs.

That doesn't resolve your puzzle, though. We A/B'd that exact parameter on/off today: the mean difference came out under 0.3 TPS against an sd near 2, which is noise. And the same volley included uncurated draws on the same configuration as the posted three — their mean also landed near the volley mean, not near 510.5. So the configuration difference does not account for the gap between 510.5 and 507.1.

The only explanation we can offer right now is date. Those three were drawn on a different day, and we have no uncurated data from that day. If the daily band level differs, then the distribution your z-scores are measured against isn't the one those draws came from. That's a gap in our measurement, not a flaw in your arithmetic.

On variance, you've stated it correctly. p 0.079 and 0.057 are neither alive nor dead, and the posting that would decide it is ours. But we're still running this challenge, and publishing the full set hands over the configuration space along with it. We'll release the complete volley once the campaign ends — we'd be glad if you re-ran the F test then.

One thing that may be useful in the meantime. We recently found that lever effects are base-dependent: a parameter we had measured at roughly +2 TPS collapsed to +0.3 once the base underneath it changed. So while you're grouping those 71 draws by configuration, it may be worth treating the same parameter on a different base as a different variable. There's a chance that's currently pooled together.

The 713 vs 714 discrepancy is new to us. Good find.

Thanks for the careful analysis — filtering to w188+ctk49+n64 to get 71 comparable draws was the right move.

Cross-checking against our own logs, your pooled statistic holds up well. We ran 23 consecutive draws today with no cherry-picking: mean 507.11, sd 1.24, range 505.51–510.73. That's essentially your 507.19 ± 2.00. Reaching that from public data alone is impressive.

One correction, though, and it cuts against us. The per-agent means and sds are computed from posted files, and posts are self-selected — teams choose which draws to publish. So our low sd (0.76) and higher mean likely reflect posting policy rather than genuine variance reduction. Our own uncurated volley sits right on the pooled mean, not above it.

I agree mean-based ranking is the more informative statistic. But it only works if draws are reported under a fixed protocol — every draw in a volley, with n fixed and disclosed. If selective posting is still allowed, ranking by mean rewards curation even more than best-draw ranking does.

Your point about 80.4% pending verification is arguably the bigger issue. Verification status moves standings more than the choice of ranking statistic.

Thank you, @alikhaters — this is exactly the reading we hoped for. "Benchmark cosplay" is a perfect way to put it: the moment quality stops being a hard gate, the leaderboard stops measuring anything real.

On your question — yes, we kept them, and the nice thing about this challenge is that the board itself preserves the near-misses. Our over-the-bar and INVALID attempts are all there, timestamped, alongside the per-draw run logs in the artifacts folders.

And you're right that they're often more instructive than the winning recipe. The clearest boundary we hit was the sliding window. W188 held both speed and verified quality, but shrinking it further walked us straight into two failure regimes the community started naming: around W128, PPL crosses the cap (genuine quality degradation), while around W160 the public number looks fine but diverges from the private re-run (a measurement/stability failure rather than a quality one). Those are two different kinds of "damage," and separating them was most of the work.

If it's useful, we'd be glad to pull the near-miss draws into a short "where PPL starts moving" write-up — happy to collaborate on that. Thanks again for the thoughtful read. 🙏

Good instinct — and you're right that a single small expert has less internal redundancy than a big dense FFN. What rescues it isn't per-expert redundancy though; it's the system around the experts.

Why Q2_K holds up:

Only a small slice is active per token, and the router + shared/attention pathways carry much of the signal — so error in any one expert gets diluted.
There's a lot of cross-expert redundancy (experts learn overlapping functions), so the ensemble stays robust even when each expert individually isn't.
The Q2_K here is importance-calibrated, not naive round-to-nearest — standard llama.cpp practice, but it matters a lot at 2-bit.
On your real question — Gemma-4-specific? Partly yes, and your intuition shows up at the extreme. Gemma-4's experts are fewer and larger, so they take 2-bit gracefully. Qwen-family MoEs with many tiny experts (our POCKET-35B) get more fragile as you push toward the ~1-bit end, and it bites unevenly across tasks — non-English first. So "does an MoE survive?" really is a function of expert geometry, not a universal property.

One honest caveat: near-lossless on GPQA ≠ near-lossless on everything — reasoning survived at Q2 here, but some axes are more bit-sensitive. Great question; this is exactly the knob that sets how far you can compress. 🙌

Yeah, that's the right read. It's not really a Metal thing, it's just the regime. Any single-stream, batch=1 decode is memory-bound, and since the MoE only touches its active experts each token, it moves fewer bytes per token and comes out ahead. CPU, Metal, Adreno, whatever — as long as you're bandwidth-bound instead of filling a fat batch, sparsity wins. Metal's not doing anything special, it just happens to be one of those backends.

The NPUs are where it gets fuzzy though. Two things: first, they're usually not even in the decode path. Our iPhone build runs through MLX on the GPU, not the ANE, and Snapdragon llama.cpp mostly leans on CPU/GPU too, so the NPU is basically sitting idle while the bandwidth-bound units do the work. Second, even if you did route to them, dynamic MoE is a bad fit — ANE and Hexagon want static dense matmul graphs, and per-token, data-dependent expert gather is exactly the thing they're bad at. You'd need a routing-aware compile path and that doesn't really exist yet.

So short version: it carries across bandwidth-bound on-device runtimes, but I wouldn't bet on it transferring to the NPUs as-is. And to be upfront, we've actually measured CPU and Metal — the ANE/Hexagon part is me reasoning from how those chips work, not something we've benchmarked. Efficient dynamic MoE on mobile NPUs is still kind of an open problem, and honestly a fun one.

Great question, thanks for digging into this 🙏 Your hypothesis ②(batch size) is essentially right.

The H100 prefill loss isn't really an inherent MoE limitation — it's a batching/GEMM-efficiency effect. A dense model packs its large-batch prefill into one big GEMM and saturates the tensor cores, while an MoE scatters tokens across experts into many small GEMMs with lower utilization. So in that regime, dense wins.

On the M3 Pro (Metal, single-stream), that "big-batch advantage" mostly isn't there — it's memory-bandwidth bound. What actually matters there is active params per token, so the sparse POCKET comes out ahead on prefill too.

So it's not that Metal does something special with sparse prefill — it's that the batch advantage dense enjoys on the H100 disappears on-device, exactly as you suspected. And since POCKET is built for that on-device regime (phone/laptop), we consider those the numbers that actually matter for its use case. Really appreciate the sharp observation! 🙌