# AI Performance **URL:** \ **Category:** model app\ **Tags:** ai\ **Created:** [August 25, 2024, 6:32am UTC](https://discuss.88.io/t/ai-performance/1585 "2024-08-25T06:32:28Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![admin](https://discuss.88.io/user_avatar/discuss.88.io/admin/32/38_2.png) [@admin](https://discuss.88.io/u/admin)\ **Post date:** [August 25, 2024, 6:32am UTC](https://discuss.88.io/t/ai-performance/1585/1 "2024-08-25T06:32:28Z")
# Distributed AI workloads can be distributed across many processors. ## Distribution - Distrubuted Llama [GitHub - b4rtaz/distributed-llama: Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.](https://github.com/b4rtaz/distributed-llama) - Exolabs [https://exolabs.net/](https://exolabs.net/) - Petals [GitHub - bigscience-workshop/petals: 🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading](https://github.com/bigscience-workshop/petals) [https://medium.com/@visrow/decentralized-distributed-llm-using-ai-in-cost-effective-and-environment-friendly-way-8c0a73ee9e6f](https://medium.com/@visrow/decentralized-distributed-llm-using-ai-in-cost-effective-and-environment-friendly-way-8c0a73ee9e6f) - AI Horde [https://stablehorde.net/](https://stablehorde.net/) # Parallel Multiple GPUs can be used with llama.cpp now: - [https://github.com/ggerganov/llama.cpp/pull/6017](https://github.com/ggerganov/llama.cpp/pull/6017) ---
**Author:** ![admin](https://discuss.88.io/user_avatar/discuss.88.io/admin/32/38_2.png) [@admin](https://discuss.88.io/u/admin)\ **Post date:** [June 28, 2026, 2:08am UTC](https://discuss.88.io/t/ai-performance/1585/2 "2026-06-28T02:08:23Z")
# llama.cpp > **[Llama.cpp - Run LLM Inference in C/C++](https://llama-cpp.com/)** > > Llama.cpp (LLaMA C++) allows you to run efficient Large Language Model Inference in pure C/C++. Download llama.cpp for Windows, Linux and Mac. ## ik\_llama > **[GitHub - ikawrakow/ik\_llama.cpp: llama.cpp fork with additional SOTA quants and...](https://github.com/ikawrakow/ik_llama.cpp)** > > llama.cpp fork with additional SOTA quants and improved performance Merge into llama.cpp is not possible: > > > This PR is an initial effort at porting @ikawrakow's IQ\*\_K quants from ik\_llama.…cpp to mainline llama.cpp. Attribution has been provided for the quantization code, and if additional attribution work is required please let me know. > > This branch implements the CPU backend for the following quantization types: > \- IQ2\_K, IQ3\_K, IQ4\_K, IQ5\_K, IQ6\_K > > Models quantized with these types from ik\_llama.cpp should correctly load in this PR, and llama.cpp can now be used to produce those quantizations. > > Since this implementation is for the CPU backend only currently, do not expect excellent performance. Further backends will be added in future PRs (CUDA, Vulkan, etc.). > > I made a few small quality-of-life tweaks to \`test-quantize-fns\` and it passes for these new quantization types: > \`\`\` > $ ./build/bin/test-quantize-fns > ... > Testing iq2\_k > Testing iq3\_k > Testing iq4\_k > Testing iq5\_k > Testing iq6\_k > \`\`\` > > I have also done some initial KLD testing by quantizing a model with ik\_llama.cpp, collecting logits for it, then loading that model in llama.cpp and comparing its KLD: > \`\`\` > \# Qwen3-4B-Instruct-2507-IQ6\_K.gguf > ====== Perplexity statistics ====== > Mean PPL(Q) : 10.910113 ± 1.495592 > Mean PPL(base) : 10.803911 ± 1.477219 > Cor(ln(PPL(Q)), ln(PPL(base))): 99.79% > Mean ln(PPL(Q)/PPL(base)) : 0.009782 ± 0.008832 > Mean PPL(Q)/PPL(base) : 1.009830 ± 0.008919 > Mean PPL(Q)-PPL(base) : 0.106202 ± 0.097559 > > ====== KL divergence statistics ====== > Mean KLD: 0.011825 ± 0.000975 > Maximum KLD: 0.270169 > 99.9% KLD: 0.243841 > 99.0% KLD: 0.099079 > 95.0% KLD: 0.032769 > 90.0% KLD: 0.026773 > Median KLD: 0.005709 > 10.0% KLD: 0.000064 > 5.0% KLD: 0.000014 > 1.0% KLD: 0.000000 > 0.1% KLD: -0.000004 > Minimum KLD: -0.000004 > \`\`\` > > Don't expect the PPL and KLD to match up 100% due to backend differences and general noise. However the results should be reasonable and within ballpark range. > > Disclaimer: AI was used to translate much of the implementation from ik\_llama.cpp, especially the quantizations in \`ggml/src/ggml-quants.c\` and \`ggml/src/ggml-cpu/quants.c\` > > I will follow up with further KLD and PPL testing tomorrow to cover models quanted with each of the newly ported types. ---
**Author:** ![admin](https://discuss.88.io/user_avatar/discuss.88.io/admin/32/38_2.png) [@admin](https://discuss.88.io/u/admin)\ **Post date:** [July 3, 2026, 10:30am UTC](https://discuss.88.io/t/ai-performance/1585/3 "2026-07-03T10:30:42Z")
# Swarm ## OptiLLM - [GitHub - algorithmicsuperintelligence/optillm: Optimizing inference proxy for LLMs · GitHub](https://github.com/algorithmicsuperintelligence/optillm) - [https://medium.com/open-intelligence/think-harder-not-bigger-how-optillm-boosts-llm-accuracy-up-to-10x-at-inference-time-without-79d117de310c](https://medium.com/open-intelligence/think-harder-not-bigger-how-optillm-boosts-llm-accuracy-up-to-10x-at-inference-time-without-79d117de310c)