Andrej Karpathy explains that AI models exhibit uneven or “jagged” performance because their training focuses on economically or strategically valuable data distributions, leading to gaps in less prioritized areas like the “car wash problem.” He emphasizes the importance of users staying engaged and understanding these limitations, as model capabilities depend heavily on the specific data they were trained on, requiring fine-tuning for tasks outside their core expertise.
Andrej Karpathy discusses the concept of “verifiability” in AI models, highlighting why their performance can be inconsistent or “jagged.” He explains that this jaggedness partly stems from how labs train these models and, importantly, what data distributions they focus on. Labs tend to prioritize training environments that are economically valuable or strategically important, which means some verifiable environments or tasks might be underrepresented simply because they are not deemed as useful.
Karpathy gives examples to illustrate this point. One is a quirky question about the number of layers in a strawberry, which models historically struggled with, showing their jagged knowledge. Another more practical example is the “car wash problem,” where state-of-the-art models might irrationally suggest walking to a car wash 50 meters away instead of driving. This discrepancy is striking given that these same models can perform highly complex tasks like refactoring massive codebases or identifying security vulnerabilities, underscoring the unevenness in their capabilities.
He emphasizes that this jaggedness signals two things: either the model is slightly off in certain areas, or users need to remain actively involved and treat these models as tools rather than infallible sources. Staying “in the loop” and understanding the model’s limitations is crucial because the model’s performance heavily depends on the data it was trained on and the specific circuits or knowledge areas it has been exposed to during training.
Karpathy also shares an anecdote about the improvement in chess-playing ability from GPT-3.5 to GPT-4. He attributes this leap not just to general capability progression but to the deliberate inclusion of a large amount of chess data in the training set. This example highlights how the presence or absence of specific data in the training distribution can significantly impact the model’s proficiency in certain tasks, reinforcing the idea that model capabilities are shaped by the choices made during training.
In conclusion, Karpathy stresses that users must explore and understand the AI models they work with, recognizing that these models come without manuals and perform well only in certain contexts. If an application falls outside the model’s trained circuits or data distribution, additional fine-tuning or custom work is necessary. This approach ensures that AI tools are used effectively and their limitations are managed appropriately.