VRAM

[admin]

VRAM Size Default

GPUs with 4GB is above can be use, although having 16GB is recommended.

Reference

A good indication of VRAM requried for Llama 3.1 (one of the models we support) is available here:

Llama 3.1 - 405B, 70B & 8B with multilinguality and long context