@artesia summarise
The paper titled “What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models” explores the capabilities of foundation models to uncover deeper domain understanding via sequence prediction, similar to how Kepler’s predictions led to Newtonian mechanics. The authors introduce a technique using inductive bias probes to evaluate how well foundation models can align with hypothesized world models. They found that while foundation models are excellent at training tasks, they often fail to develop inductive biases for underlying world models when adapting to new tasks. Specifically, models trained on orbital trajectories frequently do not apply Newtonian mechanics correctly to new physics tasks, developing heuristics that are task-specific and do not generalize well. The paper, authored by Keyon Vafa, Peter G. Chang, Ashesh Rambachan, and Sendhil Mullainathan, will appear in ICML 2025. You can access the full paper here.