Hardware for VERY Large LLM
Nvidia has a reference design for large LLM called HGX, the latest is the HGX B300.
B300 is based on the Blackwell Ultra GPUs which give us up 288 GB RAM per GPU. With 8 of those on one single motherboard, even large LLMs like Kimi K3 with 2.8 trillion parameters can be ran on just one of them (using NVFP4 weights).
Complete Systems
- nvidia - NVIDIA DGX B300
- supermicro - https://www.supermicro.com/en/solutions/dcbbs#ai-compute-systems
- supermicro - https://www.youtube.com/watch?v=fDEqpSKD41U
- asus - https://www.youtube.com/shorts/3Pmp9vL_hik
- gigabyte - https://www.youtube.com/watch?v=uImTUVjGHFA