My $30 Language Model Is #1 Under 50M on the Open SLM Leaderboard. It's Not My Highest Score.

As of 3 November 2026, Mantis-50M-Base is #1 in the under-50M class of the Open SLM Leaderboard, with an Intelligence Index of 18.36. I pretrained it from scratch: my own tokenizer, my own data mix, one H100, under 5 hours, under $30 of compute.

It isn't my highest score. A sibling model, Mantis-50M, scored 30.05 with the leaderboard's own eval tools. That's well past the previous under-50M leader at 17.93, and within 0.21 of the best under-100M model, which is twice its size. I never submitted it. Most of this post is about why, because that gap taught me more about small models than the training did.

Everything is public:

• Mantis-50M-Base: the submitted model, #1 under 50M, no synthetic data in its decay
• Mantis-50M: the 30.05 model, and its GGUF for llama.cpp
• Mantis-50M-arith: the synthetic arithmetic dataset
• kash8080/mantis-50m: all the code, Apache-2.0

Set your expectations on what this is. These are base models. They continue text. They aren't chatbots. The weights are CC-BY-NC-4.0 because one training source, ClimbMix, is non-commercial upstream.

Why 50M

I started out wanting 300M. On my $30 budget that would have been about 13 tokens per parameter, a badly undertrained model competing with SmolLM2-360M, which had 4 trillion tokens behind it. So I read the leaderboard first and let it pick the size.

The board accepts open models under 400M parameters and buckets them by size. It ranks on an Intelligence Index, which normalizes each benchmark against chance and combines them:

II = ( N(HellaSwag, 25) + N(mean(ARC-E, ARC-C), 25) + N(PIQA, 50)
       + 0.65 × N(ArithMark, 25) ) / 3.65

N(score, chance) rescales so chance = 0 and perfect = 100

On that formula one PIQA point is worth 0.55 II, a HellaSwag point 0.37, and an ArithMark point 0.24. ArithMark is the board's own test: 1,000 grade 1 to 5 word problems, written so the model has to continue a sentence with the right number out of four.

Here's what I saw on 24 Oct. Almost every model under 50M sat between 35 and 40% on ArithMark, barely above the 25% you'd get by guessing. Going from 37 to 70 was worth about 7 II points. #1 was 17.93. The class looked weak, and arithmetic looked like the way in.

That read was correct about the formula and naive about the game. It took me four days and nearly all my budget to find out how.

Gates before GPUs

With $30, you can't debug on paid hardware. So the project ran in gates, 0 through 6, and each one had to show evidence before the next could spend anything.

One rule I fixed before seeing any eval: the released model is the final checkpoint at the end of the decay. One builder on this board reports neighbouring checkpoints moving about ±0.8 II, so picking your best-scoring one is picking noise and calling it progress. That rule cost me later, and I kept it.

The architecture

A plain dense decoder: 25 layers, width 384, 6 query heads and 2 key-value heads, a 1,152-wide SwiGLU feed-forward, QK-norm, RoPE, tied embeddings. That comes to exactly 49,322,240 parameters. It exports as a standard Qwen3ForCausalLM, so transformers loads it with no custom code, and llama.cpp can convert it.

This was the most contrarian call in the project. The leaders use hybrids. 50M-pro is pure linear attention. GPT-X3 and KLM-1.5 mix Gated DeltaNet layers (a linear attention variant with a fixed-size memory, so cost per token stays constant) with normal attention, and KLM adds an n-gram lookup table at the input. All of those need a custom forward pass, which means trust_remote_code for anyone running them and a hard road to GGUF.

I wanted a model people could run on a laptop with one command. That's what "small" is for. So I gave up whatever those hybrids buy in exchange for portability. And at 50M the gap isn't obvious: on the four non-arithmetic benchmarks, my dense model contributes 15.91 points of index against 50M-pro's 15.00. That comparison is confounded by data and token count, since 50M-pro doesn't publish its token count, so read it as "dense isn't obviously losing here", nothing stronger.

The shape came from running the numbers against the 50M cap:

I never tested these shapes against each other. Deep and thin follows MobileLLM's published result at this size, and I didn't have the budget to re-derive it. Same for QK-norm, which normalizes queries and keys so attention logits can't blow up. I took it on evidence from OLMo 2 and Gemma, and it's the main reason I picked the Qwen3 layout over Llama. No run ever produced a NaN or a loss spike. That doesn't prove QK-norm did it. It proves I didn't need to find out.

I also wrote my own 150-line PyTorch model instead of using the HF class, because I wanted to understand every line. A test proves it matches HF's logits to a relative 1e-5. Bit-exact is impossible, since the two use different attention kernels, and that took a while to accept.

Context length is 1,024. Leaders use 2,000 to 4,000, but every benchmark item is under 300 tokens. I measured 2,048 at 12.3% slower per token, and my rule had been to switch only under 8%.

The tokenizer

At 50M, vocabulary is a budget line. Each token costs 384 parameters. With a standard 49k vocab, embeddings alone take 18.9M, which is 38% of the model. At 16,384 they take 6.29M, or 12.8%. That difference is the whole reason every other shape would have gone over the cap.

So I trained my own byte-level BPE: 256 bytes, 5 special tokens, 16,123 merges, on a 2 GB sample drawn in the same proportions as the training mix. Every digit is its own token, so the model can learn place value. "1234" costs 5 tokens, and that's the price.

When I looked through the vocab I saw a lot of long whole words and thought my corpus was too small. It wasn't. That's Zipf's law plus how BPE works: a bigger corpus makes frequent words more dominant. Prose comes out at 3.7 to 4.9 bytes per token, and word problems at 2.35, because of all the digits.

One surprise: llama.cpp recognizes pre-tokenizers by hashing token IDs, so any new vocab fails with "BPE pre-tokenizer was not recognized". I copied SmolLM2's exact pre-tokenizer and added a one-line hash mapping to a pinned llama.cpp checkout. That line is the only thing between a custom tokenizer and GGUF. I still owe them the upstream PR.

The data

Every batch draws from every source in these proportions. HobbyLM's public write-up showed why you do this: they fed sources in sequence, and train loss fell while evals got worse. No source repeats more than 4 times, which caps WikiHow (about 0.18B tokens, so it's small) and word problems. I threw out anything without a clear licence, including cc-math-finest, which 50M-pro used.

Decontamination: I dropped every training document sharing a 13-word sequence with any benchmark item. That removed 1,474 of 28.9M documents, about 1 in 20,000. HellaSwag was the biggest source of hits, 946 of them, which makes sense since it's partly built from WikiHow. Almost nothing, statistically, but it costs little and I can show the report. The limit is real too: items under 13 words can't match, and that's 22% of PIQA and 17% of ARC-Easy. The filter report says so.

Prep ran on Modal: one 2-core CPU container per parquet file, about 1M tokens a second each, with a manifest entry per shard recording repo, commit hash, file and token count. The whole data stage, all 27.67B tokens, cost under $2.

Making the H100 earn its money

Gate 3 changed how I think about GPU choice. The real benchmark, with real data and the real batch:

I saw 16 GB used on an 80 GB card and tried to fill it. Micro-batch 128 used 62 GB and gained 0.9%. The H200 has 43% more memory bandwidth and got 12.6% more speed. A narrow model doesn't care about either.

The profile explained it. In a 470 ms step, matmuls took 30% and ran at only about 56% of the H100's peak. Elementwise work, meaning RMSNorm, SwiGLU and the loss, took 32%. Attention took 14%. A width of 384 just doesn't give the GPU enough matrix work per byte of memory traffic. Big models live in a different world here, because wider matmuls use the hardware far better. The "H100 at 20% utilization" number looked like a bug to me for days. It's mostly physics, though a wider shape only moved it about 2% in my estimates.

The one real fix hid in the optimizer. torch.optim.Muon launches kernels per matrix, and it was launching 4,075 tiny kernels per step: 13 ms of work spread across an 85 ms span, with the GPU idling in between. I wrote a short subclass that groups the 175 matrices into 4 same-shape stacks. Now it's 150 kernels and 2.8 ms, bitwise identical to PyTorch's output. That's 9.1% more tokens for free.

That gave me the budget: 34,000 steps, 262,144 tokens per step, about $18 of expected step time. In tokens, 8.91B of compute. Token-Superposition makes that 22.28B of text read. It works out to 181 compute tokens per parameter, about 9 times the classic Chinchilla ratio. That's on purpose. Leaders in this game go far higher.

Testing the recipe on a smaller model

Every technique in the recipe had a rule written down before its test, then a run on a 13.6M proxy model. These were cheap, about 0.25B tokens and $0.40 each, so I could afford to be wrong.

Muon orthogonalizes each weight matrix's momentum before applying it, using a few Newton-Schulz iterations. At 13.6M it beat AdamW by 0.356 nats, on one seed and without tuning AdamW. That was the biggest gap in any proxy run. Muon is now built into PyTorch, so there's little reason not to try it.

Token-Superposition Training (Peng, Gigant and Quesnelle) is the strange one. For the first 30% of training you feed the model the average embedding of 6 neighbouring tokens and train it to predict all 6 at once. It reads text six times faster. It sees a blurry version. Then you switch to normal training.

TST failed my rule. At 0.25B tokens, no-TST won by 0.010 nats, and I needed TST to win by 0.02. The rule said to drop it. I ran a longer pair first, at 1B tokens, because the short run ended only 665 steps after the switch.

TST needed about 860 steps after the switch to catch up, then held a steady lead. The paper tested at 270M and above with AdamW, so seeing the same order of effect at 13.6M with Muon is new, at least to me. That $1.80 pair changed the decision for the whole main run. The lesson I took: a pre-written rule stops you arguing with results, but a short test can have the wrong sign, and the rule can't tell you that.

Synthetic arithmetic looked like the easy win: 77.8% against 29.8%, for almost no damage elsewhere. Hold on to that number. It's the mistake this post is about.

Main run: my laptop fell asleep

I launched on 27 Oct, and the first alarm fired early: val_rising, over and over, 77 times in the end. During TST, validation loss climbs, from 5.66 at step 500 to 6.65 at step 10,000. Validation feeds the model plain tokens, which it hasn't seen since step 0. The proxy had shown the same slope, so I left it alone. It recovered inside 50 steps of the switch: 2.94 at step 10,250.

That switch is also when the run died. My Mac dropped into idle sleep at 15:59. Modal cancelled train about 100 seconds later, one minute after the TST switch, in the middle of a recompile. I'd launched with --detach, but as best I can tell, detached runs only protect the last function triggered, and train had since spawned an eval. That cost 1,187 steps, about $0.60. I relaunched under caffeinate with the charger in, and it resumed cleanly.

A worse one got caught before launch. Modal retries a timed-out call with a fresh, full timeout. With my settings, a hung run would have gone 4.8 hours, then retried twice: 14.4 hours, about $60, on an account with $26 in it. --hours is now a wall-clock deadline across retries. And the trainer's cost cap only counted step time, so it wouldn't have helped.

Stuck at 37%

Everything else looked great. Val loss was on its fitted curve. My home-made arithmetic test climbed to 98.5%. And ArithMark sat at 37%, for 15,000 steps.

First guess: my test had easier wrong answers. I rebuilt it with ArithMark-style distractors and it still read 97%. So I ran a diagnosis on my Mac's CPU, for free, while the H100 kept training. I took ArithMark's own 128 addition number pairs and wrote them out in different sentences.

The model could do the arithmetic. It got 99% on ArithMark's own numbers. But the skill only switched on for the sentences it had been trained on, and my first generator was small: 27 templates, 50 names, 28 things. Only 7 of ArithMark's 50 names were on my list. It never used "lost", "sold", "used", "gave away" or "took out". Change Omar to Sam in a new sentence and accuracy falls from 92% to 23%.

Outside that gate, it fell back on a habit from web text: pick the round number. It chose the multiple of 10 about 73% of the time when one was offered, while ArithMark's correct answer is that number only about 23% of the time. That's why several topics scored below chance, and why "time" read 96%. Every time answer on ArithMark is a multiple of 5 and the distractors aren't, so the model didn't need to think.

This also reframed the whole class for me. A tiny 13.6M model with 65M tokens of training already scored 37.3% on ArithMark. So 35 to 40 is roughly what shortcuts get you, and the scores in this class say less about math than they appear to. The same "time" trick likely flatters everyone. That's my inference, and I haven't checked other people's models item by item.

Pausing instead of finishing

Warmup-stable-decay schedules have a property I'd been counting on. Learning rate stays flat until 80% of the way through, so the checkpoint just before the decay is a clean branching point. I had a script watch for it and stop the app. (It failed. modal app stop wanted a confirmation, so I stopped it by hand 30 seconds later and lost about 220 steps, $0.10.)

Then I built a new generator. 55 builders across all 17 ArithMark topics, about 500 names, 207 things, 90 gain-and-loss verbs, 110 lead phrases and 10 answer formats. I held back 87 names, 40 things, 6 verbs and 18 phrases to build a new transfer test from words the model would never train on. That test had a nice property: on the old checkpoint it read 33.8%, right where ArithMark was, reproducing the failure without my touching the benchmark.

I wasn't sure this version would do any better, and I'd already been fooled once by my own test. So I ran a 400-step test decay for $0.35 from that checkpoint, compressing the learning rate to zero.

ArithMark on that checkpoint: 37.3% to 85.2%. The index went from 16.54 to 28.37, already well past the leader, on about 13M tokens of new data. Addition went 21% to 94.5%. The "time" score dropped from 96% to 80%, because the shortcut had faded and the model had started computing.

I ran the full decay that night. Final numbers, at 01:40 on 28 Oct: II 30.05, ahead of 50M-pro on all five benchmarks, and 0.21 behind a model with twice the parameters. I was very pleased with myself.

Looking back, those 100 steps should have worried me more than they thrilled me. Skills that arrive that quickly don't sit very deep.

Did it learn anything?

I wanted outside numbers before posting, so I could say the model learned arithmetic and hadn't simply fit the board. So I built a held-out test from ASDiv and SVAMP, two public word-problem sets. Then I found out Orca-Math, one of my training sources, already covers 81.4% of SVAMP and 68.3% of ASDiv. After filtering those out, 232 problems remained.

85.7% on ArithMark. 31 to 41% on most formats of someone else's problems, and no gain at all on the hardest one. Some of the gain is real. Most of it isn't.

Bare addition really did improve. 93% on 2-digit addition is a true skill. Multiplication got worse, likely because the generators multiply small numbers. Letter-choice math stays at chance whatever you do, so that format tells you nothing at this size.

What the board said

I posted the question on discussion #275 before opening a PR: here's the data, here are my own bad held-out numbers, is this acceptable, and will you run the hidden sets? The community answered within hours. Models that did this didn't get accepted. The private arithmetic sets are very different from ArithMark and probe every weakness. Randomly generated data collapses on new phrasing even when the loss looks fine.

I should have read those threads first. Eight days earlier, on #242, another team had asked the same question about their 3M model. The maintainer ran it, it failed the hidden sets at around 35%, and their answer was that synthetic sets are allowed if they aren't based on ArithMark's format, but heavily discouraged, since none had generalized to the hidden sets so far. That team's held-out checks came from the same generators as their training data. So did my first transfer test. My 35.8% on ASDiv is a different test, but it's not a comforting match.

The rule is fair. Passing a 13-gram filter means I didn't copy the benchmark. It doesn't mean I didn't aim at it. I built v2 after studying exactly how ArithMark's items failed, and I covered its 17 topics. Even written from general English, with zero exact and zero 13-gram matches, that's benchmark-shaped data. A check on shorter sequences found 6-word overlaps with 123 of ArithMark's 1,000 items, mostly generic area and perimeter phrasing. The whole lead over the field came from arithmetic I'd pointed straight at the test.

Second decay, no synthetic data

The checkpoint at step 27,200 was still sitting there. Same trunk, same recipe, with the synthetic share set to zero and the other sources scaled up to fill it. A test enforces that this is the only difference from the first config. It cost about $4 and took 56 minutes.

I fixed the release rule first: submit the final checkpoint, whatever it scores. It scored 18.36, inside the 18.01 to 19.10 I'd forecast. That's 0.43 ahead of 50M-pro. PIQA noise alone is around ±0.6 II, so I'd call it a tie I happen to be on the right side of. Base beats 50M-pro on HellaSwag, ARC-Challenge and PIQA, and loses on ARC-Easy and ArithMark.

Base still has first-generation arithmetic in its history, 7% of the stable phase. I considered leaving that out of the PR and decided against it. It's in the weights and it's already public, so the PR discloses it. The maintainers might reject Base over that too. It's their board.

And "forgotten" turned out to be a sloppy word. On those templates accuracy dropped from 98.5% to 48.7% within 50 steps of decay, ending at 35.2%. But 2-digit addition also fell from 51.9% to 0.7%, which looks like total loss until you read the output. Base writes its working first: "59 + 63 = 122". Exact match scores that as wrong. On 100 random sums, the right answer showed up within 24 tokens 62 times. So it lost the templates and the bare-answer habit, and some addition stayed. The card says this, because I'd initially written it the simple way and was wrong.

What works at this size

Everything here comes with a label. "Measured" means I ran it. "Inferred" means it's from papers, other people's cards, or reasoning.

Noise is worth saying out loud. Seed-to-seed variance I never measured. Identical configs drifted about 0.005 nats and 1 point on a test from GPU nondeterminism alone. PIQA's error is about ±1.1 points. Half the "wins" on this board are smaller than that.

What solo pretraining looks like now

The headline fact is how cheap this got. Karpathy's GPT-2 reproduction in llm.c, 124M on 10B tokens, is about $20. nanochat's 561M run is about $100 on 8 H100s for 4 hours. HobbyLM's 1B mixture-of-experts model cost under $2,000 in total. And for about $20 of compute, my non-arithmetic scores land level with the leader of this size class.

Tokens still count, but they don't predict much. Here are numbers from other builders' cards in the under-100M class:

A few things have moved in the last year or so, and I'd bet on them continuing:

• Data beats tricks. Switching to ClimbMix cut nanochat's speedrun time by 27%, more than its other changes combined. ClimbMix now sits in nearly every top entry.
• The recipe has settled: Muon with GPT-X3's values, TST, warmup-stable-decay with a 1-sqrt cooldown, a decay that shifts toward Q&A and educational text.
• Hybrid attention is getting less painful. llama.cpp now converts Qwen3.5 Gated DeltaNet hybrids, which weakens my main reason for avoiding them.
• Benchmarks get gamed, and boards respond. Hidden sets are how this board answers that. It's the right call, even when it stings.

Making it usable

A release has its own bugs, and these two would've embarrassed me.

On transformers 4.x, my export silently used a RoPE theta of 10,000 instead of 100,000. The model loaded with no error and produced wrong text. I found that only by building a throwaway 680 MB environment to test the older version. It's fixed in metadata now, and logits match exactly on 4.57.6 and 5.20.2.

"Q4_K_M" isn't what it says on this model. k-quants need row counts that are multiples of 256, and mine are 384 and 1,152. So llama.cpp quietly falls back to a Q5_0/Q8_0 mix of about 38 MB. I shipped one honest file instead, F16 at 99.27 MB. On an M2 Pro it does 620 tokens a second on CPU, using about 214 MB of RAM. My goal was a real 4-bit model around 30 MB, and I missed it, because I picked the width without checking this. If file size matters to you, pick a width divisible by 256 from day one.

What it cost

Modal's Starter plan includes $30 of compute a month, so that's the budget I designed around, and I stayed inside it. The pretraining run was about $20. Everything for Mantis-50M, including data prep and all the tests, came to about $28. The second decay added about $4 on the next month's credits. I paid nothing out of pocket, and anyone with a Modal account can do the same.

The "credits left" column is what the Modal dashboard showed. The costs per stage are subtraction and my own estimates from the experiment log. The price list behind them: H100 at $3.95 an hour, A10 at $1.10, CPU at pennies. The trainer's own meter says $17.32 for the full main run, but it only counts step time.

What's next

PR #287 got merged. Base passed the hidden sets and is #1 under 50M. Next is a model near 100M, using the same pipeline. Reaching KLM-1.5 properly would mean around 118B tokens, and my rough model says $435 to $575 on Modal, so I'm looking at staged runs and cheaper GPU marketplaces. And no synthetic arithmetic unless it passes a test I didn't write.

If your team needs a small model trained for a constrained target (on-device, low RAM, one domain, a tight budget), this is the work I want to do more of. So is applied AI and forward-deployed engineering work more broadly. Email me, and read the code first if you want to check how I work.

Rahul Kashyap is CTO & Co-founder at Designare Studio and DeepStory, based in Bangalore.