While AI GPU giant NVIDIA's chips are widely believed to offer superior total cost of ownership (TCO) compared to custom AI chip alternatives, analysts from Evercore ISI believe that AI engineers are unimpressed by them. NVIDIA CEO Jensen Huang has defended his firm's AI chip price points on multiple occasions by claiming that they offer better performance efficiency compared to peers. However, according to the Evercore report, AI engineers are also focused on other metrics, such as the cost of cooling the chips, when deciding which products to use.
Power Consumption & Cooling Are Important For NVIDIA's AI Chip Costs, Says Bank
Evercore's discussion about the cost of using NVIDIA's AI chips comes soon after a Morgan Stanley note discussed the matter. In its coverage, Morgan Stanley claimed that even though it cost twice as much to build a data center with NVIDIA's Blackwell GPUs over custom AI chips, the performance per watt of the Blackwell GPUs was as much as eight times higher.
However, in its coverage, Evercore points out that AI engineers look at factors other than performance per watt when evaluating AI chips. Quoting AI engineers and others in the hyperscaler industry, the financial firm outlines that users of the chips are looking at other factors as well when using the NVIDIA chips.
Desire To Improve Economics Is Driving Engineers Towards Alternatives To NVIDIA, Says Firm
As per Evercore, the shift to an "inference-led regime" from a "training-led regime" is "increasing focus on cost-per-token, ROI and TCO, which is accelerating hyperscaler interest in homegrown ASICs and alternative accelerators." This claim was mirrored by claims made by an expert from AI computing infrastructure provider Nebius. The expert had remarked that GPUs were being evaluated through metrics such as cost per million tokens generated.
The financial firm also points out that the shift to inferencing is shifting the "buying criteria from max throughput/BW to cost-per-token, power, cooling, utilization, TCO." It adds that NVIDIA's "claims of 35x not resonating with the average AI engineer amidst a belief that 70% gross margins are excessive. As a result, Evercore points out that the average engineer is "willing to use ASICs or 'good enough' alternatives to improve economics."
The Nebius expert had outlined that inference demand was responsible for as much as 95% of the total enterprise workload use cases. The Groq chips were also being preferred due to their higher throughput, according to the expert.
About the author: Ramish is a seasoned technology writer and editor with more than a decade of experience. He specializes in semiconductor fabrication and market analysis. With a background in finance and supply chain management - via his bachelors in Finance and a micromasters in supply chain management from MIT - Ramish combines financial rigor with deep industry insight to deliver accurate and authoritative coverage.
Follow Wccftech on Google to get more of our news coverage in your feeds.
