HybridMamba-11

Parameter Golf is a simple, brutal constraint: build the most capable language model you can under a hard size budget. Every architectural choice becomes a trade you have to justify in bytes.

HybridMamba-11 alternates Mamba state-space blocks with Transformer attention blocks across eleven layers, wired together with U-Net-style skip connections between the early and late layers. The reasoning: state-space layers give you long context almost for free, attention gives you the precise token-to-token recall that pure SSMs are weak at, and the skips let the deep layers reuse early representations instead of re-learning them. It was the first State Space Model-based submission in the competition.

The hardest problem was not the architecture. Mamba's reference CUDA kernels were not traceable in torch.compile's fullgraph mode, which cost roughly 10× the throughput of the leading entries — enough to make the whole approach uncompetitive. The choice looked like giving up compilation or giving up Mamba.

I did neither. I reimplemented the selective scan as a Hillis–Steele parallel associative scan in pure PyTorch, replacing the sequential O(L) recurrence with O(log L) depth and making the whole graph compilable. More FLOPs on paper, far fewer in wall-clock: the optimised path measured a 48× speedup.

Getting from there to a submittable artifact meant clearing four integration problems — compile compatibility, a CUDA dtype mismatch, state-dict compatibility, and 12.5M dead parameters that were costing budget while contributing nothing. The result: 31.8M parameters, 13.6 MB compressed, comfortably under the 16 MB cap.

Submitted 30 April 2026 as PR #1365. It also ended my RunPod credits, which is its own kind of result.