The video explains that large language models store facts as packed vector representations within their feed-forward networks, reading them through direct address-based lookups rather than unpacking the combined information internally. This approach allows the model to function like a graph database, dynamically retrieving knowledge via layered queries while highlighting memory capacity limits tied to vector dimensionality rather than storage or retrieval ability.
The video explores how large language models, specifically the feed-forward network (FFN) components, function as graph databases by storing facts as weighted edges in their parameters. The presenter demonstrates injecting new, made-up facts into a model (Google Gemma 34B) without any training, showing that the model can natively read these facts during its forward pass. These facts are not stored as words but as numerical coordinates or vectors in specific “slots,” representing relationships like capital, currency, and language. Multiple facts can be packed into a single slot by vector addition, creating a superposition of information that can later be decoded.
To decode these packed facts, the presenter introduces a “match and peel” algorithm that iteratively identifies and subtracts the strongest fact vector from the packed vector until only noise remains. This method effectively unpacks multiple facts from a single vector. However, the model itself does not perform such decoding internally. Instead, it relies on linear lookups at specific layers to read individual facts directly from the packed representations. This means the model stores information in a packed form but reads it by addressing specific vectors rather than unpacking them.
The video further explains that the model builds an “address” over multiple layers to retrieve facts, starting with fuzzy searches and gradually refining to exact matches. This layered approach allows the model to resolve ambiguous or competing facts by combining relation and entity information to pinpoint the correct answer. The model’s memory is thus dynamic, constructing queries and retrieving answers through a process of progressive refinement rather than direct decoding of packed data.
A key insight is that the FFN acts as a key-value memory store, where each row in the input matrix corresponds to an address and the output matrix row corresponds to the stored value. The presenter demonstrates building such an FFN by hand, showing how feeding an address vector through a matrix multiplication yields the stored fact without any complex decoding. This simple lookup mechanism explains how the model can read injected facts natively, as long as the injected data aligns correctly with the model’s known relational structure and addressing scheme.
Finally, the video emphasizes that the model’s limitation lies in the dimensionality and packing capacity of its memory space, not in the ability to store or read facts. While the model packs many facts into limited slots, it reads them through direct addressing rather than unpacking. This understanding opens possibilities for externalizing memory storage beyond the model’s weights, potentially scaling to much larger knowledge bases. Overall, the video clarifies that large language models manage memory through packed storage and address-based retrieval, functioning effectively as graph databases without explicit unpacking mechanisms.