AI Hardware

Compute Owner campus asset
1

If you are buying new AI hardware for sharing with other Compute Owners, as of 2025-06-01 the following AI hardware are recommended due to their ability to deliver the maximum performance from minimum power consumption:

Ram Size Hardware Model
16 GB Nvidia RTX 5060 Ti
64 GB Apple M4 Pro

Reference low power AI hardware:

AI Hardware RAM Size RAM Width RAM Bandwidth Maths Core Neural Core
RTX 4060 Ti 16 GB 128 288 GB/s 4352 cuda 136 tensor
RTX 5060 Ti 16 GB 128 448 GB/s 4608 cuda 144 tensor
RX 7900 xtx 24 GB 384 960 GB/s 6144 stream 192 ai
M4 Pro 64 GB 256 273 GB/s 2048 alu 16 neural
M4 Max 128 GB 512 546 GB/s 5120 alu 16 neural
DGX Spark 128 GB 256 273GB/s 6144 cuda ?? tensor

Note each Apple M4 GPU Core has 128 alu (arithmetic logic units)

2

AI Cluster

You are welcome to build clusters using any hardware, below are just some that we have experience with.

Existing Frameworks

  • Nvidia CUDA
  • AMD ROCm
  • Apple MLX

VRAM Size

GPUs with 6GB or above is recommended , having at least 16GB is preferred.

VRAM usage estimation for Inference operation:

  1. 32-bit parameters use 4GB of VRAM per billion
  2. 16-bit parameters use 2GB of VRAM per billion
  3. 4-bit parameters use 0.5GB of VRAM per billion