↓ Skip to main content
  1. Posts/

Five open-source LLMs worth running in late 2026

·5 mins
Ai Llm Open-Source Self-Hosted Vps
Table of Contents

The rules
#

A model had to meet four conditions to get on this list:

  1. Open weights you can actually redistribute. Anything under a “research only” licence is out.
  2. It runs on hardware a small team can afford, meaning a consumer GPU or a rented VPS or server.
  3. It’s current, so released or materially updated in the last ~6 months.
  4. It does something specific better than anything else in its class.

The numbers below are wall-clock throughput from our own test rig, which is either a single RTX 4090 (24 GB) or CPU inference on a Zen 4 server with 128 GB of RAM. Every model was served with vllm 0.7 or a recent llama.cpp build. Tokens/s figures count output tokens only, with 1 concurrent user.

1. Llama 3.3 70B: the general-purpose default
#

If you don’t know which model to pick, start with this one. It’s Meta’s late-2025 refresh at the 70B size, and it lands in a useful spot: it comfortably beats GPT-4-class models on most reasoning benchmarks and still fits in 48 GB of VRAM at Q4.

Context is 128k. The Llama 3 Community licence allows commercial use, with some restrictions aimed at big tech.

You’ll need 2x RTX 4090 (48 GB) at Q4_K_M, or a single H100 at bf16. On the two 4090s at Q4 we saw about 28 tok/s. The main weakness is multilingual: quality drops off outside the 8 languages it officially supports. It’s a good fit for general chat, agent planning and long-context analysis, and if you only want to run one model, this is the one.

2. Qwen3-Coder-32B: for code
#

If 80% of your workload is code, run this instead of a bigger generalist. Alibaba’s coder-tuned 32B beats Llama 3.3 70B on nearly every code benchmark we care about (HumanEval+, SWE-Bench Lite, Aider), and it fits on a single 24 GB card.

The context window is 262k, which is unusually large, and the licence is Apache 2.0. One RTX 4090 runs it at Q4, where we measured about 58 tok/s. CPU at Q4 is tolerable rather than pleasant, at around 2 tok/s on a fast desktop.

General chat is fine, but you’ll notice it’s worse than a tuned generalist of the same size. Where it earns its place is code review, PR analysis, agentic coding workflows and in-editor assistants.

3. Mistral-Small-3.1 24B: the one that fits on a VPS
#

When people ask whether they can run an LLM on their VPS, this is the model we point them to. It’s the only one on this list that runs comfortably on CPU alone. At 24B it’s big enough to be useful and small enough that a Q4 quant on a modern desktop CPU produces tokens fast enough that you’d actually sit and read them.

  • 128k context, Apache 2.0
  • 16 GB RAM for Q4 on CPU, or 1x RTX 3090 at Q5
  • about 9 tok/s CPU-only on Zen 4 with DDR5-5600, Q4_K_M

It won’t match 70B+ models on complex reasoning. Use it for chat, summarization, classification, as a RAG backend, or for any high-volume job where latency and cost matter more than peak intelligence.

4. DeepSeek-V3.1: probably not one to self-host
#

It’s on the list because you’ll get asked about it, not because you should run it on your own. DeepSeek-V3.1 is a 671B-parameter MoE with 37B active, and it’s genuinely competitive with closed frontier models on hard reasoning and math.

The problem is hardware. It needs 8x H100 for bf16, or 4x H100 for Q4, which is a datacentre-class GPU budget. Context is 128k, and the DeepSeek Model License allows commercial use with some restrictions.

We have no local numbers for it. We don’t run it ourselves, and our testing went through a cloud endpoint. At this size it’s worth asking whether a self-hosted inference endpoint makes sense at all compared with just calling one. Keep it for the specific tasks where only a frontier model works and the smaller specialists fail, and even then you’ll usually be better off with one of the many providers already hosting it.

5. Pixtral 12B and Qwen2.5-VL 7B: vision
#

This slot is a tie, because the two models win in different situations. Open-source vision-language models are finally usable in 2026, and these two stand out:

  • Pixtral 12B (Mistral, Apache 2.0) is better at OCR and document understanding.
  • Qwen2.5-VL 7B (Alibaba, Apache 2.0) handles charts and UIs better and has a smaller footprint.

Both take 128k of text plus images, and either one runs on a single RTX 3090 at Q5. On a 4090 we measured 70 tok/s for Qwen2.5-VL 7B and 42 tok/s for Pixtral 12B. Typical jobs are PDF extraction, turning screenshots into structured data, document QA and UI test agents.

Summary table
#

Model              Size    HW floor           Best at                     Licence
Llama 3.3 70B       70B    2x 4090            general / default           Llama 3 Community
Qwen3-Coder 32B     32B    1x 4090            code                        Apache 2.0
Mistral-Small 3.1   24B    16 GB RAM (CPU)    VPS / high-volume workload  Apache 2.0
DeepSeek-V3.1      671B    4-8x H100          frontier reasoning          DeepSeek License
Pixtral / Qwen2.5-VL 12/7B 1x 3090            vision                      Apache 2.0

What we left out on purpose
#

Gemma 3 is good, but its edge over Llama 3.3 isn’t big enough to justify learning a second tokenizer. Phi-4 is nice for its size, though we never reach for it when any of the models above will fit. As for anything under ~7B, run them if you must. For most jobs you’d be better off calling a hosted small model than running a weak local one.

A cheap place to start
#

If someone asked us where to start with LLMs on a VPS, we’d suggest this:

  • a VPS with 16 GB RAM, 4 vCPU and NVMe storage
  • Mistral-Small 3.1 24B in Q4_K_M, served by llama.cpp with llama-server
  • Open WebUI in Docker in front of it

Tokens come out slowly, but it’s usable, and you get to learn how the pieces fit together. Scale up once you need more.

Related