AI Performance

admin · August 25, 2024, 6:32am

Distributed

AI workloads can be distributed across many processors.

Distribution

Distrubuted Llama
GitHub - b4rtaz/distributed-llama: Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Exolabs
https://exolabs.net/
Petals
GitHub - bigscience-workshop/petals: 🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
https://medium.com/@visrow/decentralized-distributed-llm-using-ai-in-cost-effective-and-environment-friendly-way-8c0a73ee9e6f
AI Horde
https://stablehorde.net/

Parallel

Multiple GPUs can be used with llama.cpp now:

https://github.com/ggerganov/llama.cpp/pull/6017

admin · June 28, 2026, 2:08am

llama.cpp

ik_llama

Merge into llama.cpp is not possible:

github.com/ggml-org/llama.cpp

Port IQ*_K quants from ik_llama.cpp (#19726)

master ← AesSedai:iq-k-ks-quants

opened 06:42AM - 19 Feb 26 UTC

AesSedai

+1847 -10

This PR is an initial effort at porting @ikawrakow's IQ*_K quants from ik_llama.…cpp to mainline llama.cpp. Attribution has been provided for the quantization code, and if additional attribution work is required please let me know. This branch implements the CPU backend for the following quantization types: - IQ2_K, IQ3_K, IQ4_K, IQ5_K, IQ6_K Models quantized with these types from ik_llama.cpp should correctly load in this PR, and llama.cpp can now be used to produce those quantizations. Since this implementation is for the CPU backend only currently, do not expect excellent performance. Further backends will be added in future PRs (CUDA, Vulkan, etc.). I made a few small quality-of-life tweaks to `test-quantize-fns` and it passes for these new quantization types: ``` $ ./build/bin/test-quantize-fns ... Testing iq2_k Testing iq3_k Testing iq4_k Testing iq5_k Testing iq6_k ``` I have also done some initial KLD testing by quantizing a model with ik_llama.cpp, collecting logits for it, then loading that model in llama.cpp and comparing its KLD: ``` # Qwen3-4B-Instruct-2507-IQ6_K.gguf ====== Perplexity statistics ====== Mean PPL(Q) : 10.910113 ± 1.495592 Mean PPL(base) : 10.803911 ± 1.477219 Cor(ln(PPL(Q)), ln(PPL(base))): 99.79% Mean ln(PPL(Q)/PPL(base)) : 0.009782 ± 0.008832 Mean PPL(Q)/PPL(base) : 1.009830 ± 0.008919 Mean PPL(Q)-PPL(base) : 0.106202 ± 0.097559 ====== KL divergence statistics ====== Mean KLD: 0.011825 ± 0.000975 Maximum KLD: 0.270169 99.9% KLD: 0.243841 99.0% KLD: 0.099079 95.0% KLD: 0.032769 90.0% KLD: 0.026773 Median KLD: 0.005709 10.0% KLD: 0.000064 5.0% KLD: 0.000014 1.0% KLD: 0.000000 0.1% KLD: -0.000004 Minimum KLD: -0.000004 ``` Don't expect the PPL and KLD to match up 100% due to backend differences and general noise. However the results should be reasonable and within ballpark range. Disclaimer: AI was used to translate much of the implementation from ik_llama.cpp, especially the quantizations in `ggml/src/ggml-quants.c` and `ggml/src/ggml-cpu/quants.c` I will follow up with further KLD and PPL testing tomorrow to cover models quanted with each of the newly ported types.

admin · July 3, 2026, 10:30am

AI Performance

Distributed

Distribution

Parallel

llama.cpp

ik_llama

Swarm

OptiLLM