Ollama

Compute Owner ENTITY AGENT
1

Default Model Provider

There are now more and more high quality open Models available to Compute Owners:

These 2 light weight models are available as base line by default:

  • Llama 3.1 8B Instruct Q4_K_M - 4.9 GB
  • Qwen 2.5 7B Instruct Q4_K_M - 4.7 GB

Notes:

  1. Use of the models with higher number of parameters is recommended if you have more VRAM in your GPU available.
  2. Use language tuned models
    e.g. Chinese tuned version of Llama 3.1 8B above.

Alternative Model Providers

There are other Model Providers

Ollama is the default as of 2024-08-24

Model Interfaces

Must support Retrieval Augmented Generation (RAG) since traditional LLMs are difficult to for non-technical Compute Owners to customise (e.g. fine-tuning GPT-3.5)

Open-WebUI is the default as of 2024-12-10.

2

Mix and Match AI

Below are some preferred AI Model standards:

1. Model File Format

2. Parameters

  • 3B or above
    For general purpose models having at least 3B parameters is necessary for acceptable performance with 2024-10 technologies. Specialised models can have substantially less parameters.

3. Quantization

3

Translation

Collections

Models

BigTranslate

BigTranslate is from Institute of Automation of the Chinese Academy of Sciences (CASIA).

Code:

References:

Llama 3

Llama 3 supports multiple languages:

  • English
  • Spanish
  • French
  • German
  • Italian
  • Portuguese
  • Dutch
  • Russian
  • Chinese
  • Japanese
  • Korean

As of 2024-07-20 it has 3 different sizes: 8 Billion (available), 70 Billion (available), 400 Billion (almost there!) parameters.

References:

4

Open Source Models

Despite its Open AI name, the ChatGPT it developed is not open sourced, but open sourced Large Language Models are being developed quickly by others:

1. Llama

As of 2024-08-12 the default LLM model is Llama 3.1

Data Cut-off Month: 2023-12

1. Stanford Alpaca

Promising for non-commercial applications.

Interesting how they took it down after short time online:

Can be used on less powerful hardware:

2. FLAN UL2

Promising for commercial applications.

20 Billion parameters can be a bit heavy but the gains may be worth it over the older and leaner FLAN-T5 it is based on.

5

Retrieval Augmented Generation

Without Embedding

6

Structured Outputs

7

OpenAI APIs

Ollama has limited support for some OpenAI APIs:

But if that is not absolutely necessary, that just stick to Ollama's own API: