In this episode of the AI Hardware Podcast, hosts Ian Cutress and S Foxton discuss the current landscape of AI data center training chips, focusing on major players like NVIDIA, AMD, Google, and Huawei. They highlight that while training chips have attracted significant investment and attention, the market is gradually pivoting towards inference chips, which are expected to dominate in volume and revenue in the long term. Training workloads require large, powerful chips with fast interconnects and high memory bandwidth, making the barrier to entry very high. The hosts also touch on the concept of the “training tax,” referring to the additional complexity and requirements training chips must handle compared to inference chips.
NVIDIA remains the dominant force in AI training hardware, having recently surpassed a $4 trillion market cap and generating $100 billion in AI hardware revenue last year. Their Blackwell and Grace Blackwell systems represent the cutting edge of training and inference combined, with a unified architecture that supports both workloads. NVIDIA’s strength lies not only in hardware but also in its extensive software ecosystem and supply chain control, including securing high-bandwidth memory (HBM) supplies. Despite the high cost of development, NVIDIA’s integrated approach and scale make it difficult for competitors to challenge their position in the training market.
AMD is positioning itself as a strong contender with its MI300 series and upcoming MI350 chips, leveraging chiplet technology and HBM integration. While AMD has made significant strides in inference workloads, training remains a more complex challenge due to software stack maturity. The company has been actively acquiring AI and software firms to bolster its capabilities and is focusing on training as a key growth area this year. However, AMD faces challenges in scaling its software ecosystem to match NVIDIA’s dominance, and its market share has been under pressure despite solid revenue growth.
Google continues to develop its own AI training chips, known as TPUs, with the latest generation called Ironwood (TPU v7). Unlike NVIDIA and AMD, Google designs chips specifically tailored to its internal workloads, enabling massive scale with pods containing thousands of chips interconnected by proprietary networking technology. This vertical integration allows Google to optimize performance for its large language models and other AI applications. However, Google’s TPU technology remains largely internal and not commercially available, limiting external visibility into their advancements.
The episode also covers Chinese efforts in AI training chips, focusing on Huawei and other emerging players like Illuvar Corex Tanzhen 100 and Beerren Technologies. Huawei, constrained by U.S. trade restrictions, relies on domestic foundries like SMIC and faces yield challenges but continues to develop its own training chips for local use. Other Chinese companies are pursuing homegrown designs to reduce reliance on foreign technology, though they face significant hurdles in manufacturing and software development. The hosts note that geopolitical factors heavily influence the AI chip market, with companies balancing control, supply chain security, and access to advanced manufacturing as critical considerations in their strategies.
