The video discusses the shift from late fusion to early fusion in multimodal AI models, highlighting how early fusion—processing text and images together from the start—enables more seamless semantic understanding, greater efficiency, and better performance. It also emphasizes the promising integration of mixture of experts (MoE) techniques with early fusion, which naturally specialize in different modalities and may define the future standard for multimodal large language models.
The current landscape of multimodal AI models, such as Claude, Grock, and Llama, suffers from what is described as a “Frankenstein problem.” These models can handle text and images but typically do so through late fusion techniques, where separate components process each modality independently before combining their outputs. This approach leverages pre-trained specialized models for text and vision, making training easier and more efficient. However, it falls short in achieving a genuine semantic understanding that integrates both modalities seamlessly, limiting the models’ ability to reason across text and visual data naturally.
Meta’s research group, Metafare, has pioneered a different approach called early fusion, exemplified by their Chameleon model. Early fusion trains a unified transformer architecture that processes raw text and visual data together from the start, treating images as discrete tokens similar to words. This method allows the model to generate mixed sequences of text and images and reason across modalities without the artificial barriers imposed by separate encoders. Despite initial training challenges, such as instability caused by competing modality signals, these were resolved through normalization techniques, resulting in a model that outperforms late fusion counterparts on various benchmarks.
Recent research, including a comprehensive study by Apple, has further validated the superiority of early fusion models. Analyzing 457 models ranging from 300 million to 4 billion parameters, the study found that early fusion models not only match but often exceed the performance of late fusion models. Contrary to previous beliefs, early fusion models are more efficient, requiring less memory, fewer parameters, and faster training times. This efficiency, combined with simpler architecture, suggests that early fusion could become the new standard for building multimodal AI systems.
A particularly exciting development is the integration of mixture of experts (MoE) techniques with early fusion models. Without explicit guidance, these models naturally develop specialized experts for text and images, enhancing performance while maintaining inference efficiency. Additionally, early fusion models handle higher-resolution images better than late fusion models, challenging the assumption that specialized vision encoders are superior for detailed visual processing. However, late fusion still holds advantages in specific tasks like pure image captioning, indicating that the choice of architecture should depend on the use case and data availability.
Looking ahead, the field may witness a paradigm shift toward native multimodal large language models built on early fusion principles. This unified approach not only simplifies model design but also unlocks richer knowledge by integrating visual and textual information from the ground up. While some concerns remain, such as the appropriateness of tokenizing continuous image data, ongoing research and innovations like multimodal diffusion models may address these challenges. Overall, early fusion combined with MoE represents a promising frontier in AI, with companies like Meta and Apple leading the way, while others may need to catch up to this emerging standard.