Marah Abdin and Robert McHardy from Pulsar discuss their approach to scaling language models through the integration of modular synthetic data pipelines and rigorous pre-training validation, addressing challenges like data quality, training stability, and model performance. They highlight advancements in their open-weight models, including XGen-2 and the large-scale Laguna S, emphasizing improved data diversity, numerical stability, and community accessibility.
In this talk, Marah Abdin and Robert McHardy from Pulsar discuss the challenges and solutions encountered while scaling their language models, particularly focusing on synthetic data and pre-training techniques. Pulsar recently transitioned from releasing models exclusively for enterprise use to making open-weight models available to the public, including Laguna M and Laguna XS. Marah highlights the importance of synthetic data as a complement to organic data, helping to expose implicit structures and fill gaps in training. They implemented strategies such as auto mixer for efficient data sweeps, improved web data sampling, and incorporated synthetic data—comprising about 13% of the pre-training mix—to enhance model performance and address issues like token repetition.
Marah elaborates on the modular approach to synthetic data pipelines, which consist of components like seeds, metadata, generators, and validators. These pipelines range from simple, scalable rephrasing methods to more complex multi-stage workflows and cross-domain porting, such as converting math problems into code. The team also developed Hive, an infrastructure that orchestrates multiple agents with configurable prompts and parameters, allowing flexible and hierarchical generation processes. This modularity enables Pulsar to break down complex tasks into manageable steps, improving the quality and diversity of synthetic data used in training.
Robert shifts the focus to pre-training challenges encountered at scale, emphasizing the need for holistic treatment of data quality, training code correctness, and architecture. He shares insights from training Laguna M 1, including the use of model replica hashes to detect inconsistencies across distributed training replicas. Examples of issues include broken GPUs causing silent data corruption and exploding gradients due to numerical precision limitations in tensor parallel accumulation. These problems were identified and mitigated through careful monitoring and adjustments, such as switching accumulation to FP32 precision, which restored model convergence.
Building on these lessons, Pulsar developed XGen-2, a new generation open-weight model optimized for coding tasks, incorporating improved data diversity, reduced repetition, and enhanced numerical stability. They also preview Laguna S, a much larger 118 billion parameter model trained on 30 trillion tokens across 4,000 GPUs. Despite encountering a race condition related to FP8 training kernels causing silent gradient corruption, the team identified and is addressing the issue. Early evaluations show Laguna S outperforming previous Pulsar models and competing well against other open-weight models on coding benchmarks, although it lags behind some models on general knowledge tasks due to data differences.
In conclusion, the talk highlights the intricate interplay between synthetic data generation, training infrastructure, and large-scale model pre-training. Pulsar’s approach combines modular synthetic data pipelines with rigorous training validation and architectural improvements to scale models effectively. Both XGen-2 and Laguna S will be made publicly available, supporting the community’s access to competitive open-weight models. The presenters encourage engagement with their work through provided links and express optimism about continuing to scale and refine their models in the future.