In his talk, George He from LlamaIndex presents a hybrid search approach that combines traditional keyword-based and embedding-based semantic vector search to efficiently manage and retrieve information from large, complex enterprise document collections. He highlights the importance of orchestration tools and pre-indexing strategies to optimize search accuracy, reduce computational costs, and handle multimodal data, demonstrated through a live demo using Alphabet’s financial reports.
In this talk, George He, Head of Engineering at LlamaIndex, discusses advanced approaches to document search within enterprise environments, focusing on how to effectively manage and retrieve information from large and complex company data sets. He contrasts two primary methods: traditional local file search (grep-style) and embedding-based vector search, highlighting their respective strengths and limitations. Local file search works well for smaller, well-structured data like code repositories, where files are concise and hierarchically organized. However, as data scales up to include diverse formats such as PDFs, images, and schematics, and grows into thousands or millions of files, local search becomes inefficient and token-expensive for language models.
George emphasizes the importance of orchestration and context management in search systems, especially as models gain larger context windows but also higher computational costs. To address these challenges, LlamaIndex advocates for a hybrid approach combining keyword-based and semantic vector search, which balances precision and recall better than either method alone. This hybrid method uses a “compass” to roughly locate relevant documents, followed by more detailed semantic filtering and file traversal to ground answers accurately and reduce hallucinations. The talk also covers the necessity of pre-indexing and parsing complex documents into manageable chunks, including multimodal data, to optimize retrieval and reduce repeated processing overhead.
A key part of the solution involves building a harness—a system that equips agents with multiple tools to navigate and query documents effectively. These tools include listing files, filtering by metadata, performing keyword searches within files, and accessing pre-rendered screenshots of documents for multimodal understanding. This layered approach allows agents to interact with remote or distributed file systems without needing to load entire datasets into memory, thus saving tokens and improving efficiency. The harness also supports dynamic decision-making by the agent on whether to use local search, semantic search, or a combination, depending on the query and data scale.
George also addresses practical considerations for deploying such systems in production, including multi-tenancy, permissioning, and data freshness. Maintaining synchronization between the data source and the pre-indexed vector store is challenging, especially when dealing with frequently updated or complex datasets. He outlines a three-step synchronization process: ingestion of documents, parsing into standardized formats (like markdown), and re-bundling into a pre-indexed state. Efficient storage strategies, such as persisting data on disk with a memory cache layer, are recommended to handle millions of documents while supporting metadata-based filtering and access control.
The talk concludes with a live demo showcasing how the hybrid search and harness tools work in practice, using a dataset of Alphabet’s financial reports. The demo illustrates how the agent uses a combination of keyword and semantic search to locate relevant document chunks, accesses metadata and screenshots, and performs targeted reads within files to generate accurate, grounded responses. This approach enables scalable, trustworthy document retrieval that can handle complex, multimodal enterprise data. George invites viewers to explore LlamaIndex’s tools further, emphasizing that effective search combined with a robust orchestration harness is key to unlocking value from large document corpora.