The video begins with the creator sharing a personal experience of unexpectedly high API bills after upgrading to a newer AI model version, despite no changes in prompts or usage. This led to the discovery that newer tokenizers produce 1 to 1.35 times more tokens for the same text, effectively increasing costs without changing the price per token. The video explains that tokens are a workaround for how AI models process text: instead of reading words or letters directly, a tokenizer breaks text into chunks from a predefined list, which the model then processes. This system, however, is complex, language-dependent, and can unfairly increase costs for certain languages due to varying tokenization rules.
The video then explores the alternative of using bytes instead of tokens. Bytes are a simpler, universal unit of text encoding with exactly 256 possible values, eliminating the need for handcrafted tokenization rules and lookup tables. Recent research, including Meta’s “Breaking the Token Ceiling” paper, investigates byte-level models that read raw bytes rather than tokens. These models show promise in data efficiency, potentially matching token-based models with significantly less training data. However, the practical benefits are limited by increased computational costs during inference, as byte-level models require many more processing steps to generate the same amount of text, leading to slower and more resource-intensive operation.
To address the inference inefficiency of byte-level models, the video highlights two innovative approaches. The first is Meta’s Byte Latent Transformer (BLT), which groups bytes into patches based on predictability, reducing the number of expensive model passes and achieving comparable performance to token models with less compute. The second is Albert Gu’s Hierarchical Network (Hnet), which learns its own chunking dynamically during training, compressing input more effectively than traditional tokenizers and achieving a tenfold data efficiency improvement in training. Both approaches aim to combine the benefits of byte-level input—such as better handling of languages and typos—with computational efficiency, though practical deployment is still limited by software and hardware ecosystem readiness.
The video also emphasizes that despite these advances, the current AI ecosystem still charges and operates primarily on tokens, meaning users are effectively paying “by the byte” without realizing it due to tokenizer changes. Token definitions vary across providers and languages, causing inconsistent costs. The creator advises users to calculate their actual cost per megabyte of text rather than per token to better understand and manage expenses. This approach reveals the hidden impact of tokenizer changes on billing and helps users make more informed decisions when upgrading models or switching providers.
In conclusion, the video argues that while the era of handcrafted tokenizers is likely ending, the fundamental unit of computation—the model’s processing step—remains essential and costly. Byte-level models that process one byte per step are inefficient, but those that learn to group bytes into larger chunks hold promise for the future. However, these innovations are not yet mainstream, and token-based models still dominate current AI services and pricing. The key takeaway is to focus on the real cost drivers—processing steps and data volume—rather than token counts, and to watch emerging research like BLT and Hnet for future improvements in AI model efficiency and fairness across languages.
Useful Links
