Radar ·
A diary documents pretraining a MoE model
Published on August 20, 2026, the free book recounts an attempt to pretrain a 1.02-billion-parameter MoE model on an H200.
Published on August 20, 2026, the free book records an attempt to pretrain a 1.02-billion-parameter Mini Kimi K3 on an H200. It covers model design and the data pipeline.
The diary also reports training bugs and attempts to improve throughput. Its records of failures and decisions may help engineers understand practical challenges in pretraining MoE models. Consult the original book to verify its methods and results; the available description does not specify metrics or final outcomes.
Original source: Um diário de treinamento para pré-treinar um Mini Kimi K3