The video explains the universal approximation theorem, which states that a neural network with just one hidden layer can approximate any continuous function by combining simple “bump” functions created from neuron outputs. While this theorem guarantees the capability of shallow networks, modern deep networks are preferred for efficiency, and the theorem’s broad applicability underpins the foundation of all AI systems.
The video begins by demonstrating a seemingly impossible feat: a simple neural network with just one hidden layer can perfectly trace any curve, no matter how complex or irregular. This is achieved through basic arithmetic operations—multiplications and additions—within the network. The narrator highlights the mystery behind how such simple math can replicate any shape, setting the stage to explain a fundamental mathematical theorem that underpins this capability.
At the core of the explanation is the behavior of individual neurons, each acting like a switch controlled by two parameters: sharpness and position. By adjusting these parameters, a neuron’s output transitions smoothly from zero to one, or sharply switches on and off at a chosen point. The key insight is that by subtracting the outputs of two such neurons positioned at different points, the network creates a “bump” function—an isolated segment that is zero everywhere except between those two points. This bump serves as the basic building block for approximating any function.
By combining multiple bumps, each controlled independently in terms of width, height, and position, the network can approximate increasingly complex curves. The video shows how adding more bumps—hence more neurons—allows the network to fit the target curve with greater precision. This leads to the statement of the universal approximation theorem: any continuous function can be approximated arbitrarily well by a neural network with just one hidden layer, provided it has enough neurons.
However, the video also clarifies an important caveat: while the theorem guarantees the existence of such a perfect network, it does not guarantee that training algorithms will find it easily. Moreover, it explains why modern deep networks use many layers despite the theorem’s sufficiency of one layer. Deep networks are more efficient because they build complex functions by composing simpler functions layer by layer, whereas shallow networks must approximate the entire complexity at once, requiring exponentially more neurons.
Finally, the video notes that the universal approximation theorem applies broadly across different activation functions, not just the sigmoid used in the demonstration. This universality underpins every neural network and AI system developed over the past decades. The theorem is the mathematical foundation that makes all modern AI possible, explaining how neural networks can learn and replicate any continuous function, whether it represents sound, images, probabilities, or other data forms.