He Paid $998 to Train a 3.8 Billion Parameter AI. Here's What Actually Happened

43 hours. 8 rented B200 GPUs. 65.3 billion tokens.

Hugo Vergnes published his technical report on September 10, 2026. The headline blew up on Hacker News: a 3.848 billion parameter model trained from scratch for $998.

I read the whole write-up. I checked the numbers. And honestly, there are things the viral headlines aren't telling you.

What's Actually True

The model is called little-lm. It hit a 0.384 score on the CORE benchmark after 43 hours of training on eight rented B200s.

For context: GPT-2, the model OpenAI released back in 2019, tops out at 0.2565 on the same benchmark.

So yeah, for about a grand, one person beat a model that represented the state of the art seven years ago.

What the Headlines Leave Out

The $998 figure only covers the final run. Vergnes failed first.

His first serious attempt: an 858 million parameter model trained for 5.8 days on an A100. Result: 60.45% on PIQA, below the score of a 124 million parameter GPT-2.

In other words, he spent time and money to get something worse than a model seven times smaller that came out in 2019.

His post-mortem identifies five problems: a learning rate that was too conservative, a cosine schedule that dropped the rate to zero too early, AdamW applied to every parameter, a dataset that converged too slowly, and sloppy dtype handling.

So the real cost of the experiment includes those failed runs. The report doesn't put a number on them. That's a big omission when you're talking about a $998 budget.

The Technical Choices That Made the Difference

Five changes turned the failure into a success:

1. Trapezoidal schedule instead of cosine. The learning rate stays high longer, then drops linearly over the second half. The loss curve kept falling all the way to the last checkpoint.

2. Muon for matrices, AdamW everywhere else. Muon costs about 25% more per step, but it converges so much faster that the overhead disappears.

3. Nemotron-ClimbMix instead of FineWeb-Edu. Vergnes calls it a huge jump in convergence speed. Andrej Karpathy saw the same thing.

4. FP8 and vocabulary padding. Going from 50,257 to 50,304 entries (a multiple of 64) lines up better with tensor cores. Combined gain: 33% throughput.

5. Context cut to 1,024 tokens. A run with half the context cost $820 and scored 0.338. Almost as good for 18% less.

The Architecture in Plain English

little-lm follows a modern Llama-style design: RMSNorm, RoPE, grouped-query attention (24 query heads, 8 KV), ReLU² for the MLPs, QK-norm.

Interesting detail: the ResFormer-style value embeddings account for 721.2 million parameters, or 19% of the model. They add almost no FLOPs (they're lookups) but they eat memory and optimizer state.

Vergnes ran an ablation: without those tables, the model loses 0.46% in loss and 3.2% in CORE. For 19% more parameters. The trade-off is debatable, and he debates it.

What This Means for You

If you're a solo dev or a small tech blogger, this story says one simple thing: the entry cost to experiment with serious models has dropped.

A thousand bucks for a 3.8B that beats GPT-2 is doable. Not comfortable, but doable.

The real lesson isn't the money. It's infrastructure discipline. Vergnes built a config-driven framework where every run is a YAML file. Swapping an optimizer or a dataset is a one-line change.

That's what let him iterate fast after his initial failure. That's what saved the project, not the budget.

FAQ

Is the $998 figure verifiable?

Vergnes reports that amount for the final run at 2,048 tokens of context. The report doesn't say whether it includes the failed runs, storage, or data prep. The full economic cost isn't documented.

Why use GPT-2 as a reference?

Because it's a known, reproducible comparison point. GPT-2 1.5B scores 0.2565 on CORE. little-lm hits 0.384 with 3.8B parameters. It's not the same scale, but it shows the progress.

Can you reproduce the experiment?

Yes. Vergnes published the little-lm framework on GitHub. All the YAML configs are there. You just need access to rented GPUs.

Does this threaten the big labs?

No. Frontier models cost tens of millions. little-lm is a teaching exercise and a proof of concept. But it shows the barrier is dropping for independent research.

My Take

This story deserves the attention it's getting, but not for the reasons everyone's repeating.

The $998 figure is a hook. The real value is in the post-mortem of the initial failure. Vergnes documents exactly what didn't work and why.

Most viral AI articles hide the mistakes. This one puts them on display.

That's rare. And that's exactly what makes this experiment useful for anyone who wants to understand pre-training without going broke.

Sources: Hugo Vergnes' technical report (little-lm), coverage from RuntimeWire, LavX News, and n1n.ai.