Jack Morris from Engram discusses the challenge of enabling AI models to scale compute on fixed, personalized datasets without retraining from scratch, highlighting limitations of current methods and the potential of self-improving, recursive training approaches to overcome data saturation. He emphasizes the importance of this research for continual, individualized learning and invites collaboration to advance AI’s ability to learn deeply and continuously from proprietary data.
Jack Morris from the startup Engram presents on the concept of “scaling compute on context,” a challenge in AI related to continual learning and personalized knowledge acquisition. He frames the problem by contrasting breadth and depth of knowledge in AI, using mathematician Terence Tao as an example: AI models have broad knowledge of public data but lack the deep, intuitive understanding that comes from focused, long-term study. Morris emphasizes that current AI models, trained on public datasets, cannot acquire personalized or up-to-date knowledge after training, limiting their usefulness in real-world, individualized contexts.
Morris outlines the three traditional axes of scaling AI models: increasing data, increasing compute, and increasing model size. While these have driven progress in AI, they primarily apply to public data sources, leaving a gap in models’ ability to learn from private or proprietary data. He argues that for personalized or company-specific data, the data axis is often fixed or limited, and retraining from scratch is impractical. Therefore, the focus shifts to scaling compute on context—using compute resources to enable models to learn from a fixed dataset without retraining entirely.
He explores several approaches to this problem. One naive method is to simply continue training the model on the new data, but this often leads to overfitting and poor generalization. Another approach is to simulate in-context learning by compressing or distilling the data into a form the model can use during inference, such as key-value compaction or on-policy distillation. Additionally, synthetic data generation and continued pre-training on this data can help models acquire new knowledge, though these methods have limitations, including potential overwriting of prior knowledge and scalability challenges.
Morris highlights a fundamental limitation: any fixed dataset eventually saturates the model’s learning capacity, preventing indefinite improvement through compute alone. To overcome this, he points to the concept of self-improvement, where models generate increasingly challenging training data or tasks for themselves, enabling continuous learning and deeper understanding. This recursive training approach, inspired by successes like AlphaGo, aims to break through the “data wall” and achieve ongoing gains in model capability from a fixed dataset.
In conclusion, Morris presents scaling compute on context as a promising but still nascent research area focused on enabling AI models to learn deeply and continuously from personalized or proprietary data without retraining from scratch. Engram is actively working on these challenges, seeking methods that allow models to improve indefinitely by leveraging compute and self-generated data. He invites interested researchers and engineers to engage with Engram and contribute to advancing this frontier in AI.