Apple’s 2023 research demonstrated streaming AI model weights from storage to run models larger than available RAM, but practical implementation was limited by storage speed. Three years later, the open-source Turbo Fieldfare project combined this streaming approach with Mixture of Experts architectures on Apple Silicon’s unified memory system, achieving a 7x reduction in RAM usage by activating only parts of the model at a time, though performance now depends heavily on SSD speed and thermal management.
In 2023, Apple researchers published a paper titled “LLM in a Flash,” demonstrating that Macs could run AI models twice as large as their available RAM by streaming model weights directly from storage, loading only the necessary parts into memory on the fly. Although the concept was proven mathematically and the hardware was already available, Apple never shipped a product implementing this. The main challenge was that naive loading from storage was too slow for practical use, as drives couldn’t supply data fast enough for real-time AI inference.
Three years later, an open-source project called Turbo Fieldfare successfully implemented Apple’s blueprint on Apple Silicon, running a 26-billion parameter model on a MacBook using only about two gigabytes of active memory. This achievement surpassed Apple’s original 2x memory reduction claim, achieving roughly a 7x reduction. The key innovation was combining Apple’s windowing technique with newer AI model architectures, specifically Mixture of Experts (MoE), where only a small subset of the model’s components are active at any time, allowing most of the model to remain “asleep” and stored on the drive.
Apple Silicon’s unified memory architecture plays a crucial role in this success. Unlike typical PCs where the CPU and GPU have separate memory pools requiring costly data copying, Apple’s design allows the processor and graphics cores to share the same RAM. This zero-copy data handoff enables rapid loading of model fragments from storage directly into memory accessible by the GPU, making the streaming approach viable and efficient on Macs but not on most standard PCs.
While the memory savings are impressive, the trade-off is that the system shifts the bottleneck from RAM to storage speed. The AI’s performance now heavily depends on the SSD’s read speed, which can vary even between identical MacBook models due to differences in NAND chip configurations. Additionally, thermal throttling on fanless laptops can reduce sustained performance during long AI sessions, making memory bandwidth and storage speed critical factors for real-world usability.
Finally, the 7x memory reduction applies mainly to models designed with MoE architectures that allow parts of the network to remain inactive. Traditional dense models, where every parameter is used for every token, still require large amounts of RAM and cannot benefit from this trick. Thus, Apple’s contribution was pioneering the streaming concept and providing hardware that supports it efficiently, while the major memory savings come from evolving AI model designs. The future of local AI on Apple Silicon depends on both hardware innovations and model architectures continuing to evolve in tandem.
