This guide sets up Llama 3.1 8B Instruct on an 8 GB VPS with no GPU and serves it through an OpenAI-compatible API on localhost. The model is quantized to Q4_K_M, which is about 4.9 GB on disk and around 6 GB in memory once it’s loaded. On CPU it produces roughly 7-9 tokens per second. That’s enough for interactive chat and automation, with no GPU or external API to pay for. An optional last step adds Open WebUI as a chat interface.
The hardware#
The reference box for this walkthrough:
- 4 vCPU (AMD EPYC Milan)
- 8 GB RAM
- 80 GB NVMe
- Debian 12
Any machine with similar specs will do. What matters most is RAM bandwidth (DDR4-3200 or better) and having enough memory to hold the model plus its context.
Step 1 - Prepare the server#
Start from a fresh Debian 12 install, update it and install the build tools:
apt update && apt upgrade -y
apt install -y build-essential cmake git curl ca-certificates \
python3 python3-venv python3-pipCreate a non-root user if you don’t have one yet:
adduser llm
usermod -aG sudo llm
su - llmEverything after this point runs as llm.
Step 2 - Build llama.cpp#
Building from source takes 2 minutes and means you aren’t stuck on whatever old version a package happens to ship.
cd ~
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=ON
cmake --build build --config Release -j$(nproc)Check that the build worked:
./build/bin/llama-cli --version
./build/bin/llama-server --help | head -5GGML_NATIVE=ON makes the compiler use every SIMD extension your CPU supports, including AVX2 and AVX-512 where available. On our EPYC VPS that made it 2.1x faster than the generic build.
Step 3 - Download the model#
llama.cpp reads GGUF files, so you need the model in that format rather than the raw HuggingFace weights. This pulls Llama 3.1 8B Instruct at Q4_K_M from Bartowski’s semi-official mirror:
mkdir -p ~/models && cd ~/models
curl -L -o llama-3.1-8b-instruct-q4_k_m.gguf \
https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.ggufThe file is about 4.9 GB. On our VPS the download took 90 seconds.
At 8B, Q4_K_M is the quant where quality loss on chat and instruction tasks becomes essentially invisible; the measured perplexity difference against full precision is under 1%. Q5_K_M is slightly better, but once you add context it gets close to the 8 GB RAM ceiling. The Q3 variants save disk space, and their drop in quality is noticeable.
Step 4 - Test it from the CLI#
Run a quick sanity check:
cd ~/llama.cpp
./build/bin/llama-cli \
-m ~/models/llama-3.1-8b-instruct-q4_k_m.gguf \
-p "Explain what a VPS is in three sentences." \
-n 200 \
-t $(nproc)-t $(nproc) tells it to use all 4 vCPU threads, which is nearly always what you want for CPU inference.
The output streams to the terminal, and at the end you get a timing summary that looks roughly like this:
llama_perf_context_print: eval time = 17912.30 ms / 158 runs ( 113.37 ms per token, 8.82 tokens per second)That’s about 8.8 tokens/s, which is fine for interactive use. A medium-length paragraph takes around 15 seconds to generate. That’s on the slow side of human reading speed, but still useful.
Step 5 - Serve it as an API#
llama-cli is only for testing. For real use, run llama-server, which exposes an OpenAI-compatible HTTP API:
./build/bin/llama-server \
-m ~/models/llama-3.1-8b-instruct-q4_k_m.gguf \
--host 127.0.0.1 --port 8080 \
-c 8192 \
-t $(nproc) \
--cache-type-k q8_0 --cache-type-v q8_0What the flags do:
-c 8192sets an 8 k context. 128 k would be nice, but at 128 k the KV cache for an 8B model needs more RAM than this box has.--cache-type-k q8_0 --cache-type-v q8_0quantizes the KV cache. It saves about 40% of the context memory, and the quality loss is negligible.--host 127.0.0.1binds to loopback only. Don’t expose this port directly to the internet unless there’s authentication and TLS in front of it.
Send it a test request:
curl http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "llama-3.1-8b",
"messages": [{"role": "user", "content": "In one sentence, what is KVM?"}]
}'The response is standard OpenAI chat-completion JSON. Any tool that talks OpenAI (your editor, an agent framework, your scripts) now works pointed at http://127.0.0.1:8080/v1.
Step 6 - Run it as a service#
A systemd unit keeps the server running across reboots:
# /etc/systemd/system/llama.service
[Unit]
Description=llama.cpp server
After=network.target
[Service]
Type=simple
User=llm
WorkingDirectory=/home/llm/llama.cpp
ExecStart=/home/llm/llama.cpp/build/bin/llama-server \
-m /home/llm/models/llama-3.1-8b-instruct-q4_k_m.gguf \
--host 127.0.0.1 --port 8080 \
-c 8192 -t 4 \
--cache-type-k q8_0 --cache-type-v q8_0
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable --now llama
sudo systemctl status llamaStep 7 - Add a chat UI (optional)#
For a browser interface, Open WebUI is the usual choice. It runs in Docker:
docker run -d \
--name open-webui \
-p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
-e OPENAI_API_BASE_URL=http://host.docker.internal:8080/v1 \
-e OPENAI_API_KEY=sk-no-key-required \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:mainPut nginx in front of port 3000 with a Let’s Encrypt certificate and HTTP basic auth (or whatever SSO you already use), and you have your own ChatGPT-style chat at https://chat.yourdomain.com/.
Benchmark results#
These were measured on the VPS described above: 4 vCPU EPYC Milan, DDR4-3200, NVMe. Every run used the same prompt, “Write a short paragraph about the history of Linux.”, with 200 output tokens, and the figures are the median of 10 runs.
Config tok/s RAM used First token
Q4_K_M, ctx=8192, kv q8_0 8.9 6.2 GB 240 ms
Q4_K_M, ctx=8192, kv f16 8.6 6.9 GB 230 ms
Q5_K_M, ctx=4096, kv q8_0 7.1 7.4 GB 310 ms
Q4_K_M, ctx=32768, kv q8_0 8.8 7.1 GB 440 ms
Q4_K_M, ctx=8192, kv q8_0, -t 2 (2 threads) 5.3 6.2 GB 260 msDropping from 4 threads to 2 costs a lot of throughput (last row), so use all the cores. Context size barely changes tokens per second, which llama.cpp handles well, but a bigger context does increase first-token latency. Q5 costs 20% of throughput compared with Q4, and at this model size the quality gain is small.
Bigger models#
There isn’t much room for them on this box. An 8 GB VPS limits you to roughly 8B models if you want some headroom. For a 24B model such as Mistral-Small-3.1 (covered in last week’s post), you need at least 16 GB of RAM.
A rough rule for budgeting RAM with CPU inference:
needed RAM ~ model size * 1.1 + 0.5 GB per 2048 tokens of context (kv q8_0)For the 8B Q4 model at 8 k context, that works out to 5.4 GB plus about 1 GB, so roughly 6.4 GB, which fits.
Cost compared with API calls#
A VPS is a flat monthly fee, while an API bills per token. Running flat out, this box generates around 30k tokens per hour (about 22 M a month), and that’s its ceiling.
To compare the two, take the output tokens you’d actually generate in a month, multiply by your API provider’s current output price, and put that next to what you pay for the VPS each month. If you keep the box busy, the flat fee wins. If you only send ten requests a day, the API is cheaper.
There are non-cost reasons too. Your data stays on your own server, latency is predictable, and the model works offline. Those are often the real reasons to run it yourself.
When this isn’t enough#
- If your tasks need the reasoning of a model bigger than the 32B class, rent a GPU box or use a frontier API.
- If you have many concurrent users, run vllm on a GPU instead of llama.cpp on CPU.
- If you need a model that’s good at code, swap in Qwen3-Coder on a GPU server.