I trained a language model from scratch. On one RTX 3090.

It's called Loom-1, it's around 1B parameters, and if you ask it who it is, it will tell you, after a brief internal monologue that reads like a man waking up from anesthesia. This is a post about how it got there, what the architecture is, and why the honest verdict is "half win." No trophy, no breakthrough. Just a real account of what this much hardware and this much data actually buys you.

The short version

Loom-1 is roughly 1B parameters, trained from scratch on about 11B tokens of text, over 84,000 optimizer steps, on a single RTX 3090. Final pretraining perplexity landed around 11.2 and was still dropping at the last step. On top of that I ran a two-stage supervised fine-tune to make it conversational.

The result: format-correct, identity-aware, and semantically confused. Exactly what the setup predicts. If you only wanted the number, that's the number. Everything below is the interesting part.

The architecture is the actually novel bit

Loom-1 isn't a vanilla transformer. It's a hybrid of three things that don't usually live in the same model.

The first is a Recurrent Depth Transformer. Instead of a fixed stack of N layers, there's a core block that gets applied a variable number of times per forward pass. Depth is sampled from a Poisson distribution during training, mean 4, min 1, max 16. The idea is to let the model spend more "thinking" on harder positions by looping the same weights, rather than paying for depth in parameters. Prelude, then recurrent core, then coda.

The second is Mamba-2 / SSD state-space layers for part of the sequence mixing, with a chunked scan implementation, so I'm not paying full quadratic attention everywhere.

The third is MLA (Multi-head Latent Attention), the DeepSeek-flavored attention with a compressed latent KV cache. That last one matters a lot when your entire world is 24GB of VRAM.

The reason this combination exists is boring and honest: I wanted to see if I could make it train stably, and I wanted a compute and memory budget that fits on consumer hardware. Both turned out to be true. The thing trains without exploding for 84k steps, which, if you've ever tried to stabilize an exotic architecture from scratch, is not nothing.

The training run

The config that matters: d_model 2048, 16 heads, 2 prelude / 10 core / 2 coda blocks, sequence length 4096, bf16, gradient checkpointing (mandatory, the 3090 is not generous), 8-bit AdamW.

The run went from perplexity around 18 at step 22k down to about 11.2 at step 84k, and it was still improving at the final eval. The loss curve never flattened. The model was not done learning. I stopped it because I ran out of patience and the run was already eating days, not because it converged.

That gap between "still descending" and "I stopped" is the whole story of this project, and we'll come back to it.

The fine-tuning, and the ways it went sideways

Pretraining gives you a model that completes text. To get something that answers questions, you fine-tune on chat data. This part did not go smoothly, and I'm leaving the mess in because the mess is the honest version.

The first run overfit almost instantly. I trained about 500 examples for 5,000 steps, which at my batch size worked out to something like 160 passes over the same data. The training loss dropped to 0.0004. The model hadn't learned to be an assistant, it had memorized the answer key.

So I split it in two. A first stage on a larger general instruction set of around 10k examples to teach the basic "how to be an assistant" behavior, then a very short second stage on the curated 500 to stamp in identity and tone. Capability is deep and slow to move; style is shallow and latches on fast. This part worked.

Then I spent a genuinely miserable stretch convinced the model was broken, because it responded to "Hello" with what looked like corrupted code. It wasn't broken. The inference script was feeding it prompts formatted differently from how it was trained, so it was completing a string it had literally never seen. The moment the inference prompt matched the training format exactly, token for token, the model started talking.

So what does it actually say?

Here's a real sample once the plumbing was correct:

you> Who are you?
model> <think>User is me. I'll answer directly.</think>
The user is me.

And a slightly better one:

you> Hello! Who are you?
model> <think>Alliance yourself. I'm Loom-1.</think>
I'm Loom-1.

You can see it. The format is perfect. It opens the think tag, closes it, answers, stops cleanly on the right token, knows its own name. The content inside is a model that has read a tiny fraction of the text a modern small model reads, trying its best. It's a concussed parrot that knows its name. And honestly, for what it ran on, I'll take it.

Why it's undertrained

Let's be precise about why Loom-1 is confused, because "it's bad" is not a diagnosis.

The rough rule of thumb for a compute-optimal 1B model puts you around 20B tokens of training data. Loom-1 saw about 11B, already under that line. But that line is about minimizing loss, not about the vibes-level coherence people actually judge a chatbot on. The small models you'd compare it against, things like LFM2.5 350M, Gemma 4 2B, or Qwen3.5 2B, are trained on multiple trillions of tokens. That's a few hundred to a thousand times more data than Loom-1 has seen.

At roughly 11 tokens of training data per parameter, Loom-1 is not a small-but-sharp model. It's a normal-sized model that has barely started reading. The architecture is fine. The training is fine. It simply hasn't seen enough of the world yet, and coherent dialogue is exactly the kind of thing that shows up late, with more data.

Which is why "still descending at step 84k" matters. The model was telling me it wanted more data, and the constraint was never the data itself, which is free, but the time to push it through a single 3090. That's the wall. Not skill, not architecture, just one consumer GPU and a finite number of weeks I was willing to spend.

Would a smaller model be better? No.

I genuinely considered it. Retrain at 275M parameters, feed it the same data, reach a more "saturated" model that's closer to its own ceiling. It's a trap. Coherence scales up with parameters, not down. A 275M on the same data would be nearer its ceiling, but that ceiling sits below where the 1B already is. The thing that actually helps is more training data, and the only cost of that on my setup is patience.

The honest verdict

Loom-1 is a half win, and I mean win as much as I mean half.

The win is a novel three-way hybrid architecture that trains stably from scratch for 84k steps on a single consumer GPU, survives a two-stage fine-tune, and produces format-correct, identity-aware output. Built solo, debugged solo, all the way down from the memory errors to the boring formatting hell.

The half is that it's undertrained by a couple orders of magnitude versus what "good" would require, and no amount of clever fine-tuning fixes a data deficit. It talks like what it is.

I'm not going to pretend a 3090 produced GPT. But I also refuse to file this under failure, because the thing I set out to prove, that this architecture works and that one person can carry a language model end to end on hardware they own, is proven. The rest is just data I didn't have the weeks to feed it.

If nothing else, now the honest account exists. This is what 1B parameters and 11B tokens on a single RTX 3090 actually gets you. Someone starting where I started can read this and know what to expect.

That's it.

See ya