In this episode of the AI Hardware Podcast, the hosts Sully Ward Foxton and Ian Kutrris discuss the current landscape of AI inference chips in the data center, focusing on several key players including Intel, IBM, Qualcomm, Tensordyne, and Tenstorrent. They begin with Intel’s Gaudi 3 chip, which originated from the Habana Labs acquisition. Despite its competitive pricing and technical advancements like chiplet architecture and HBM2E memory, Gaudi 3 has faced challenges such as slow iteration, lack of integration with Intel’s oneAPI software, and limited customer adoption beyond IBM. Intel’s future AI roadmap, including the mothballed Falcon Shores project, remains uncertain amid company restructuring and shifting priorities.
IBM’s approach to AI acceleration involves their proprietary AI accelerator integrated into their Telum 2 CPU and a data center inference card called Spire. Built on Samsung’s 5nm process, Spire is designed for efficient low-power AI inference with a focus on small, specialized models optimized for tasks like HR and chatbots. IBM combines this hardware with a vertically integrated software stack and consulting services, leveraging their AI accelerators alongside Nvidia and AMD GPUs in their infrastructure. However, IBM’s AI accelerator is not commercially available in the same way as others, reflecting a more controlled deployment strategy.
Qualcomm’s AI 100 series, including the recently announced AI 200 and AI 250 rack-scale designs, represents their ongoing but somewhat quiet presence in the AI inference market. Based on an older Hexagon NPU architecture, Qualcomm’s chips are designed for edge and data center inference with a focus on cost-effectiveness and scalability using LPDDR memory. Despite partnerships and deployments, Qualcomm’s AI hardware lacks clear performance disclosures and a strong enterprise marketing push. The company is also developing ARM-based data center CPUs in collaboration with partners like Humane, signaling a potential expansion of their AI hardware footprint.
Tensodine (formerly Recognize) is highlighted as an innovative startup leveraging logarithmic math for AI inference, a departure from traditional base-10 arithmetic used by most AI chips. This approach aims to achieve the precision of FP16 at significantly lower power consumption, potentially offering substantial efficiency gains. Tensodine’s pivot from automotive vision chips to data center inference reflects a strategic shift to a faster-moving market. While promising, the company faces challenges in building out a comprehensive software stack and system-level solutions to support their novel hardware.
Finally, Tenstorrent is discussed as a rapidly growing company with a unique chip architecture combining AI compute, networking, and memory on a single chip. Their Black Hole product emphasizes scalability through high-bandwidth Ethernet connectivity, enabling large-scale distributed AI workloads without traditional top-of-rack switches. Tenstorrent’s open-source software approach and flexible product offerings aim to attract developers and enterprises alike. With significant recent funding and a growing team, Tenstorrent is positioned to make a notable impact in the AI hardware ecosystem, though performance validation and market reception remain to be seen.
