The speaker discusses efforts to enhance the efficiency of general intelligence models by developing hierarchical memory transformers and specialized agent models, alongside leveraging customized silicon platforms like AMD’s FPGAs for optimized deep learning inference. By combining GPUs and FPGAs in a heterogeneous system, these innovations significantly reduce energy consumption and improve performance, enabling practical applications such as accelerated retrieval-augmented generation in enterprise environments.
The speaker begins by emphasizing the importance of efficiency in developing general intelligence models. While recent frontier AI models demonstrate impressive capabilities with billions of parameters, their energy consumption is vastly higher than that of the human brain. For example, some large models require nearly 10 kilowatts of power and hundreds of megajoules of energy over hours of operation, whereas the human cortex operates with only about 500 kilojoules. To bridge this efficiency gap, the speaker highlights ongoing research at UCLA supported by AMD, focusing on two main areas: model innovation and platform customization.
On the model innovation front, the speaker introduces hierarchical memory transformers that mimic human memory structures by combining long-term, short-term, and sensory memory. This approach reduces the quadratic complexity of attention mechanisms by selectively recalling relevant past information rather than attending to all tokens. Experiments show that such models can achieve comparable or better performance than much larger models while significantly reducing memory usage and energy consumption. Additionally, the speaker discusses training small, specialized agent models tailored for specific tasks, which can outperform larger general models in efficiency and effectiveness within narrow domains.
The second major focus is on platform customization, particularly leveraging AMD’s FPGA technology to optimize deep learning inference. The speaker explains that while traditional GPUs excel at parallel computation, certain operations in large language model inference—such as memory processing and attention scoring—benefit from dataflow architectures available in FPGAs. Techniques like quantization with rotation-based transformations improve model compression without sacrificing accuracy, enabling smaller, more efficient hardware implementations. Dynamic reconfiguration of FPGAs allows switching between different inference stages, further enhancing performance and energy efficiency.
A key innovation discussed is the heterogeneous inference system combining GPUs and FPGAs to handle different parts of the model pipeline optimally. For example, GPUs handle dense computations like projecting input tokens into high-dimensional spaces, while FPGAs efficiently manage sparse attention mechanisms and top-k retrieval operations. This fine-grained disaggregation reduces memory bottlenecks and improves throughput. The system achieves measurable speedups and energy savings, with communication latency between devices kept minimal to maintain overall efficiency.
Finally, the speaker illustrates practical applications of these advances, such as retrieval-augmented generation (RAG) systems used in enterprise settings. By offloading inverted-index searches and vector similarity computations to FPGAs, companies can accelerate query processing on large internal datasets. The talk concludes with an invitation for further discussion, highlighting the promise of combining novel model architectures with customized silicon platforms to achieve efficient, scalable general intelligence.
