AI Performance

[admin]

Distributed

AI workloads can be distributed across many processors.

Distribution

• Distrubuted Llama
  GitHub - b4rtaz/distributed-llama: Tensor parallelism is all you need. Run LLMs on an AI cluster at home using any device. Distribute the workload, divide RAM usage, and increase inference speed.

• Petals
  GitHub - bigscience-workshop/petals: 🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
  Decentralized Distributed LLM- Using AI in Cost Effective and Environment Friendly way | by Vishal Mysore | Medium

• AI Horde
  https://stablehorde.net/

Parallel

Multiple GPUs can be used with llama.cpp now:

• llama : add pipeline parallelism support by slaren · Pull Request #6017 · ggerganov/llama.cpp · GitHub