Ready to start training the model!

Technology and Creation MLOps & 推理
1

First, I’m planning to train a 0.5B model to test the waters.

Building on the previous dense decoder + pre-RMSNorm + SwiGLU + RoPE + GQA setup,

The optimizations adopted are:

Muon optimizer (fully validated in production for Kimi K2; requires approximately 2× the compute of AdamW while preventing spikes)

QK-norm (fully used in Qwen3/Gemma3/OLMo2)

WSD scheduling

z-loss

untied embedding

GQA

RoPE-ABF

Late-stage annealing (switch to high-quality math/code during the decay phase).

Rejected:
MLA (no evidence for small models)

MoE (the model is already very small, so it isn’t needed)

256k vocabulary (a 0.5B model can’t afford the embedding proportion)

Fully automated DoReMi mixing (running a search once costs more than the gains; industry priors + greedy fine-tuning are sufficient)

logit softcap (already replaced by QK-norm), full μP migration (adopting its spirit: depth-aware initialization).

Vocabulary decision: research recommends 64k–128k, but a 100k vocabulary on a 0.5B model would consume 300 million parameters (60%), which isn’t cost-effective; GPT-2’s 50k vocabulary requires only 154 million. Also, my Chinese corpus contains only ~4.5 billion tokens (the practical upper limit for Chinese data is simply low), so we’ll keep 50k for this round.

Data: continue not using BAAI (cultivation web novels/elementary-school essays)

Meanwhile, pes2o consists of real papers, code consists of real code, and cosmopedia consists of clean textbooks, so they are adopted.

The practically usable clean corpus is 1749GB ≈ 583B tokens, distributed as follows: pes2o 683G, code 362G, proofpile2 317G, wiki_en 155G, textbooks 80G; the Chinese data totals only ~17.6G.

2

Keep it up, boss—I’m looking forward to seeing how it turns out.
:xhj006:

3

Most likely, it wouldn’t be very good at speaking Chinese.

And its literary abilities were extremely poor (because I forgot at the time… I’ll definitely add philosophy, history and the humanities, long-form literary works, poetry, drama, and so on later):xhj17:

And without SFT and RL, the base model would probably be unusable as well.

4

Is this a plan to enter the web fiction scene?

5

Huh? I just happened to forget to include any literary data this time.

I thought BAAI would probably do the trick, but it was all cultivation web novels and elementary-school compositions (I was worried they’d make the model stupid, so I excluded them).

So the literary material is probably extremely poor. I don’t even know whether the model can still speak normally…:xhj32:

6

The proportion of academic paper data is pretty large. I wonder how well the training will turn out, haha. Keep it up :xhj003: