Yann LeCun, a pioneer in AI and deep learning, has raised a billion dollars to pursue an alternative AI approach called JEPA (Joint Embedding Predictive Architecture). Unlike large language models (LLMs) that generate text or images by predicting the next token or pixel, JEPA focuses on learning internal representations through joint embedding architectures. Instead of predicting raw outputs, JEPA uses encoders to convert inputs and outputs into embeddings and trains a predictor to forecast the embedding of the next state from the current one. This approach aims to overcome challenges faced by generative models, especially in video prediction, where pixel-level predictions often result in blurry and unrealistic outputs due to the vast complexity and uncertainty in visual data.
LeCun’s journey began with convolutional neural networks in the 1990s and evolved through concerns about the heavy reliance on labeled data in supervised learning. While reinforcement learning and self-supervised learning gained traction, especially in language with models like GPT, generative approaches struggled with vision and video tasks. JEPA builds on joint embedding methods, which LeCun originally helped develop with Siamese networks for signature verification. These networks learn useful representations by comparing embeddings of similar and dissimilar inputs without generating the inputs themselves. However, joint embedding architectures historically faced the problem of representation collapse, where the model outputs trivial identical embeddings for all inputs.
A breakthrough came with the introduction of the Barlow Twins method, inspired by neuroscientific theories to reduce redundancy between neuron outputs in embedding vectors. By encouraging embeddings to be similar for augmented views of the same input while minimizing redundancy across different neurons, Barlow Twins effectively avoided representation collapse and learned powerful image representations. This method and its successors like VICReg and DINO have significantly advanced self-supervised learning in vision, achieving near state-of-the-art results on benchmarks like ImageNet without labeled data. These advances demonstrate that joint embedding architectures can rival supervised models in learning rich, generalizable features.
LeCun’s billion-dollar bet is that the future of AI lies in world models—systems that predict the consequences of actions in the physical world. JEPA is designed to build such models by encoding observations and predicting future states conditioned on actions, enabling planning and control in robotics and other domains. This contrasts with LLMs, which excel at language manipulation but lack the ability to model and predict real-world dynamics. By learning abstract state representations and transition models, JEPA-based systems can simulate outcomes, optimize action sequences, and potentially provide safer, more reliable agentic AI systems.
Looking ahead, JEPA and related world model approaches are being actively developed and tested across various applications, including robotics and vision-language tasks. While LLMs currently dominate language-based AI, LeCun argues that agentic systems require predictive world models to truly understand and interact with their environments. The ongoing research aims to demonstrate how JEPA can complement or even surpass generative models in certain domains. This paradigm shift emphasizes learning internal representations and predictive capabilities over direct generation, potentially marking a new chapter in the development of autonomous, intelligent machines.
