The speaker stresses the critical role of human prior knowledge in the success of Transformers, noting that these models are built upon a foundation of existing human knowledge. They assert that the intentional integration of extensive human knowledge and structure into Transformers, through meticulous design and refinement, is key to their exceptional performance in complex tasks like language processing.
The speaker highlights the significance of human prior knowledge in the development of Transformers, emphasizing that it plays a crucial role in the success of these models. They point out that Transformers are built upon a foundation of existing human knowledge, which has proven to be effective in enabling these models to perform well. The speaker expresses frustration when people claim that there are no inductive priors involved in Transformers, stating that there is indeed a substantial amount of inductive prior knowledge embedded within the neural network structure of these models.
Furthermore, the speaker suggests that the success of Transformers can be attributed to the extensive effort and resources invested by human beings in creating and refining the underlying design and structure of these models. They imply that the effectiveness of Transformers is a result of the intentional encoding of prior knowledge and structure into the neural networks, which have been fine-tuned through significant investments of time and money. This underscores the idea that the success of Transformers is not merely a matter of chance or luck, but rather a product of deliberate human ingenuity and design.
The speaker also mentions the massive scale at which human knowledge has been integrated into Transformers, pointing out that the billion parameters used in language models are a testament to the depth and breadth of prior knowledge encoded within these models. They emphasize that the effectiveness of Transformers is a direct result of the painstaking efforts made by human beings to imbue these models with the necessary prior knowledge and structure to perform complex tasks such as language processing.
In conclusion, the speaker suggests that the exceptional performance of Transformers can be attributed to the intentional integration of human prior knowledge and structure into these models. They argue that the success of Transformers is not a coincidence but rather a result of the deliberate design choices and investments made by human creators. By acknowledging the pivotal role of human knowledge in shaping the capabilities of Transformers, the speaker highlights the importance of leveraging existing knowledge and expertise in the development of advanced AI models.