Key Takeaways:
NVIDIA's software optimization for large language model inference is outpacing AMD's hardware gains, a gap that could determine who captures the next wave of AI spending.
SemiAnalysis, a semiconductor research firm, said NVIDIA's vLLM inference engine delivers more than 12,000 tokens per second per GPU on large mixture-of-experts models, while AMD's MI355X lags due to weaker software integration and rack-level interconnect limits. The firm posted its assessment on X on July 13, following a report two weeks earlier that said NVIDIA's CUDA moat was being "slowly eroded" by hyperscaler custom chips.
NVIDIA's GB200 NVL72 rack system links 72 GPUs through NVLink, enabling wide expert parallelism of 8 to 16 across models like Kimi K2.5, a 100-billion-parameter MoE architecture. The Dynamo distributed inference framework integrates vLLM with disaggregated serving — separating prefill and decode phases — and efficient KV cache transfer across nodes. Each GPU carries reduced expert weights, lowering HBM bandwidth pressure, while all-to-all communication stays within the high-speed NVLink domain rather than traversing slower InfiniBand networks.
AMD's MI355X cannot match the same scale of expert parallelism or rack-level interconnect, according to SemiAnalysis. The company's software stack still relies on standard vLLM and DISAGG versions, without the deep optimizations NVIDIA has built for ultra-large MoE models and wide parallel configurations. The firm described AMD's shortfall as limited to "some models," a qualifier that underscores the gap between partial support and full-coverage optimization.
The finding challenges the narrative that AMD's hardware is closing the gap with NVIDIA. While AMD shares have surged 160 percent year to date to $557.89, and Meta Platforms signed a large procurement order for MI350 chips, the inference software gap remains wide. In AI inference, where enterprise customers run billions of daily calls, microsecond latency differences compound into meaningful cost advantages. Maintaining two software stacks to cover different model types often becomes a decisive factor in procurement decisions.
NVIDIA's software ecosystem rests on three layers: a CUDA tool chain covering roughly 4 million developers accumulated over two decades, priority adaptation across all major machine learning frameworks, and deeply optimized libraries including cuDNN, TensorRT, and NCCL. The switching cost imposed by these layers exceeds any single hardware specification advantage, according to the SemiAnalysis assessment.
The broader competitive picture is more nuanced. Hyperscalers including Google, Amazon, and Microsoft are investing heavily in custom ASICs — Google's TPU and Amazon's Trainium — that undercut NVIDIA on inference cost. Broadcom and Marvell Technology design custom AI accelerators that cost between $6 billion and $11 billion per gigawatt of infrastructure, compared with roughly $19 billion for NVIDIA GB300-based racks, according to Milk Road AI estimates. Morgan Stanley said the four largest hyperscalers will spend over $1 trillion on AI infrastructure next year, adding 19.5 gigawatts of incremental compute capacity.
Yet SemiAnalysis's latest assessment suggests that NVIDIA's software depth in inference has not narrowed in step with the hardware competition. The firm's earlier report noted that Anthropic runs Claude model training on Google TPUs and Claude Code inference on Amazon Trainium, showing that custom chips are gaining share. But the vLLM performance data indicates that for the most demanding MoE inference workloads, NVIDIA's integrated hardware-software stack still holds a lead that AMD and ASIC designers have not yet matched.
NVIDIA shares closed at $210.96 on July 13, down 1.99 percent on the day, tracking broader semiconductor weakness. AMD fell 2.64 percent to $543.14. TSMC, which manufactures both companies' chips, reported a 36 percent jump in quarterly sales with N3 capacity sold out, signaling AI demand remains supply-constrained.
This article is for informational purposes only and does not constitute investment advice.
