Google introduced separate TPU chips for training and inference on Wednesday in its AI hardware push. The company said the split arrives with its eighth-generation tensor processing units. Both processors will reach customers later this year.
The update changes a design approach Google had used for years in its AI chips. Earlier TPU generations handled both model training and inference tasks on the same processor. Now, Google will assign those jobs to distinct chips.
Google changes TPU design as AI chip race intensifies
Google said the new TPU lineup separates training workloads from inference workloads. The company described that shift as a response to changing demand in AI infrastructure. It also framed the move as part of its effort to compete with Nvidia.
Vahdat, Google’s senior vice president for AI and infrastructure, explained the change when he said the rise of AI agents and specialized chips now serve the community better. He tied that view to the different needs of training and serving.
Google remains one of Nvidia’s customers, yet it also sells TPUs through Google Cloud. That makes the TPU line an internal tool and a commercial product. Cloud customers can use those processors as an alternative to Nvidia hardware.
Nvidia outlined inference-focused hardware earlier this year. In March, the company promoted silicon for faster model responses to user prompts. Nvidia linked that system to technology from its 20 billion dollar Groq acquisition.
Performance gains and memory design define the new chips
Google said the new training chip delivers 2.8 times Ironwood performance at the same price. Ironwood was the seventh-generation TPU Google announced in November. The company also said the new inference processor improves performance by 80 percent.
Google named the new inference chip TPU 8i. It said the chip uses static random-access memory, or SRAM, in its architecture. Each TPU 8i chip includes 384 megabytes of SRAM, triple Ironwood’s amount.
Nvidia said its coming Groq 3 LPU hardware will also rely on large amounts of SRAM. Google’s design choice places TPU 8i in that same memory direction. Still, Google did not compare its chips directly with Nvidia’s products.
Sundar Pichai described the purpose of the new architecture in a blog post. He said it will deliver throughput and low latency for millions of agents. He added that the design aims to do so cost-effectively.
Google expands TPU adoption across cloud and research users
Google has worked on custom AI processors for years. The company started using its own AI chips in 2015 for internal workloads. Then, it began renting those processors to cloud clients in 2018.
Other companies have followed similar paths with AI chips. Apple has long used neural engine components in iPhone chips. Microsoft announced a second-generation AI chip in January, while Meta works with Broadcom on several processors.
Amazon Web Services built separate AI chips for different jobs. It launched Inferentia in 2018 for AI requests and Trainium in 2020 for training. Those moves show how cloud companies now build silicon for specific uses.
DA Davidson analysts estimated in September that Google’s TPU business and DeepMind together could reach 900 billion dollars. Google also said adoption of its AI chips continues to rise. Citadel Securities, 17 U.S. Energy Department laboratories, and Anthropic use or plan to use Google TPUs.