# I Trained the TinyStories Model to Do Math

**URL:** <https://www.artofsm.art/t/i-trained-the-tinystories-model-to-do-math/21756>\
**Category:** Content Creators\
**Tags:** function-calling, machine-learning, chris-hay, transformers, open-source, coding\
**Created:** [27 July 2026 17:00 UTC](https://www.artofsm.art/t/i-trained-the-tinystories-model-to-do-math/21756 "2026-07-27T17:00:17Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![merefield](https://www.artofsm.art/user_avatar/www.artofsm.art/merefield/32/74_2.png) [@merefield](https://www.artofsm.art/u/merefield)\
**Post date:** [27 July 2026 17:00 UTC](https://www.artofsm.art/t/i-trained-the-tinystories-model-to-do-math/21756/1 "2026-07-27T17:00:17Z")

</div>

[![](https://www.artofsm.art/uploads/default/original/3X/5/e/5ecfe05c5cec054773694c8c01275c66deb5eb99.jpeg "I Trained the TinyStories Model to Do Math.") ](https://www.youtube.com/watch?v=UINiOMyNI9I)

The video demonstrates how a small language model trained on children’s stories was enhanced to perform arithmetic by integrating external calculator modules through tool calls, enabling accurate math without compromising storytelling quality. This hybrid approach allows the model to combine language understanding with precise computation, showcasing a promising method for multitasking in compact models.

---

<div class="post-metadata">

**Author:** ![merefield](https://www.artofsm.art/user_avatar/www.artofsm.art/merefield/32/74_2.png) [@merefield](https://www.artofsm.art/u/merefield)\
**Post date:** [27 July 2026 17:20 UTC](https://www.artofsm.art/t/i-trained-the-tinystories-model-to-do-math/21756/3 "2026-07-27T17:20:45Z")

</div>

The video explores training a tiny language model, initially trained on the TinyStories dataset, which excels at generating coherent children’s stories but struggles with arithmetic tasks. The model, with only 115 million parameters, was trained on simple narratives involving children and everyday scenarios but could not perform basic math like multiplication or addition when prompted. The presenter explains the model’s architecture, tokenizer design, and training process, emphasizing the simplicity of language modeling as predicting the next token in a sequence. Despite its storytelling capabilities, the model lacked understanding of numbers and arithmetic operations due to the absence of math content in its training data.

To address the model’s inability to do math, the presenter introduced a new training phase incorporating a math-focused dataset alongside the original stories. This mixed dataset contained arithmetic drills and story content to prevent the model from forgetting its storytelling skills while learning math. Early results showed the model beginning to recognize digits and perform simple addition with numbers, but it struggled with more complex calculations and narrative math expressed in words rather than digits. The tokenizer was specially designed to split numbers into individual digits to aid the model’s learning of arithmetic patterns.

However, training the model directly on math tasks caused a decline in its storytelling quality, highlighting the limited capacity of the small model to learn both language and math simultaneously. The presenter observed that the model tended to memorize arithmetic facts or approximate calculations using heuristic methods, which compromised its overall performance. To overcome this, a novel approach was implemented where the model learned to generate “tool calls” or function calls to external calculator-like modules (“cells”) instead of performing the math internally. This method allowed the model to focus on understanding and generating language while delegating precise calculations to specialized components.

This tool-calling approach significantly improved the model’s math accuracy and preserved its storytelling abilities. The model learned to identify when a math operation was needed, format the call correctly with the relevant numbers, and then incorporate the returned result into a coherent narrative. This hybrid system enabled the tiny model to handle complex arithmetic tasks accurately without sacrificing its language generation quality. The presenter demonstrated examples where the model successfully performed addition, multiplication, and division by invoking these tool calls, producing both correct answers and natural language commentary.

In conclusion, the video showcases how a small language model can be enhanced to perform accurate arithmetic by combining language understanding with external computation modules. The approach avoids the pitfalls of trying to embed complex math algorithms directly into the model’s weights, which can degrade language performance. All code, datasets, and model weights are made publicly available for experimentation. This work highlights a promising direction for building efficient, multitasking models capable of both natural language generation and precise mathematical reasoning through modular tool integration.