With Nvidia Groq 3, the Era of AI Inference Is (Probably) Here

This week, over 30,000 people are descending upon San Jose, Calif., to attend Nvidia GTC , the so-called Superbowl of AI—a nickname that may or may not have been coined by Nvidia. At the main event Jensen Huang, Nvidia CEO, took the stage to announce (among other things) a new line of next-generation Vera Rubin chips that represent a first for the GPU giant: a chip designed specifically to handle AI inference. The Nvidia Groq 3 language processing unit (LPU) incorporates intellectual property Nvidia licensed from the startup Groq last Christmas Eve for US $20 billion. “Finally, AI is able to do productive work, and therefore the inflection point of inference has arrived,” Huang told the crowd. “AI now has to think. In order to think, it has to inference. AI now has to do; in order to do, it has to inference.” Training and inference tasks have distinct computational…