In this discussion, Dan Biderman and Jessy Lin, co-founders of Engram, explore the challenges and innovations surrounding memory and continual learning in AI models. They emphasize that unlike traditional views separating pre-training and post-training, their approach treats models as always training, continuously integrating new and evolving context into the model weights. This ongoing learning process aims to make AI systems deeply understand specific domains, such as a company’s internal knowledge, much like a long-term employee would, rather than relying solely on externalized memory or context windows.
Engram’s technology focuses on fine-tuning models within specific workspaces or teams, using techniques like adapter fine-tuning to internalize relevant knowledge efficiently. This approach reduces the need for extensive context retrieval during inference, significantly cutting down token usage and computational costs. The founders highlight the trade-offs between memorization and retrieval, noting that while retrieval systems (like RAG) are useful, internalizing knowledge into model weights enables more intuitive and associative reasoning, which is crucial for complex tasks and personalized workflows.
The conversation also touches on the philosophical and technical aspects of memory in AI, comparing it to human memory’s lossy and selective nature. They argue that AI models need to learn what is important to remember versus what can remain externalized, a problem still open in research. Drawing inspiration from neuroscience, they discuss how biological memory balances compression and abstraction, and how similar principles might guide future AI memory architectures. They also acknowledge the limitations of current transformer architectures in handling long contexts efficiently and the need for breakthroughs in continual learning.
Looking ahead, Biderman and Lin envision a future where everyone has personalized AI models tailored to their unique needs and contexts, distinct from generic foundation models. These models would continuously learn and adapt, improving over time with use, and could serve as neural interfaces to vast data ecosystems. They foresee a shift from one-size-fits-all AI to a diverse ecosystem of specialized models that better capture individual and organizational knowledge, enhancing productivity and innovation across various domains.
Finally, the founders reflect on the surprising dominance of language models over vision models in recent AI progress, offering a speculative theory related to information processing differences between modalities. They stress the importance of integrating research and product development to accelerate breakthroughs in memory and continual learning. Engram aims to be at the forefront of this movement, focusing exclusively on these challenges to unlock new capabilities in AI that go beyond raw intelligence to include dynamic, context-aware learning and memory.
