The text discusses the use of sparse and dense vector embeddings in information retrieval, highlighting the limitations and benefits of each type. It introduces Splade as a model that combines sparse lexical models with neural network architectures to improve retrieval accuracy by addressing the vocabulary mismatch problem and enhancing semantic understanding.
The text discusses the use of vector embeddings in information retrieval, where documents and queries are represented in numerical vector format. Two main types of embeddings are highlighted: sparse vectors (e.g. TF IDF, BM25) and dense vectors (e.g. neural network architectures like Transformers). Sparse vectors are high-dimensional with few non-zero values, enabling exact term matching, but they lack generalizability and struggle with abstract concepts. On the other hand, dense vectors compress information into lower dimensions, making them suitable for semantic understanding and multi-modality search, but they require more data and computational resources.
To tackle the limitations of both sparse and dense embeddings, a two-stage retrieval approach is proposed. In this method, a sparse retrieval model is used initially to search through a large candidate document set, followed by a re-ranking stage using a dense embedding model to enhance relevance. While this approach offers benefits such as efficient search and modifiability of stages, it can be slower and more complex to implement, with potential challenges if the initial stage performs poorly.
Recent advancements in single-stage retrieval systems include models like Splade, a sparse lexical and expansion model that leverages pre-trained models like BERT to enhance sparse vector embeddings. Splade introduces term expansion and learnable term expansion capabilities, addressing the vocabulary mismatch problem in information retrieval caused by varying word choices to describe the same concept. By incorporating transformer models and mass language modeling, Splade generates sparse vectors that efficiently capture relevant terms beyond those in the original text, improving retrieval accuracy.
Despite the advantages of Splade in minimizing the vocabulary mismatch problem, challenges remain, such as slower retrieval speed compared to traditional sparse methods. Modifications to Splade have been proposed to optimize query vectors and leverage support from systems like Pinecone for better compatibility. The text provides a practical demonstration of implementing Splade using Hook and Face Transformers and PyTorch, showcasing the creation and comparison of sparse vectors through term expansions. By utilizing Splade and hybrid search techniques, the text envisions improved information retrieval performance with minimal model fine-tuning, making vector search more accessible and efficient.