Deployment & Inference ★ 118.0k

Ollama

One command to run an LLM on a laptop — the fastest way to get a model in front of a customer.

Ollama bundles model weights, quantisation and a local server behind a single CLI. It is the standard first step for demos, air-gapped pilots and developer machines, though most teams graduate to a dedicated serving stack for real traffic.

Best for: Local demos, air-gapped pilots, developer machines

Deploy: Self-hostable

#Inference #Quantization #On-Premise #Open Source #Self-Hosted

Related tools

Deployment & Inference ★ 87.0k

llama.cpp

CPU-first inference in plain C/C++ — runs where a GPU cannot go.

CPU-only, edge and embedded inference Read more →

Deployment & Inference

TensorRT-LLM

The last few percent of GPU throughput, paid for in build complexity and hardware lock-in.

High-volume serving of a stable model set on fixed NVIDIA hardware Read more →

Deployment & Inference ★ 152.3k

Open WebUI

The front end you hand to users when the model layer is already done.

Internal chat deployments on top of self-hosted models, with real user accounts Read more →

Deployment & Inference ★ 62.0k

vLLM

High-throughput LLM inference engine with PagedAttention — the default choice when you need to squeeze a GPU.

Self-hosted production inference on NVIDIA/AMD GPUs Read more →