↓ Skip to main content
  1. Posts/

Qwen3-Coder 32B on a GPU server vs. GPT-4 on real code reviews

·6 mins
Ai Hosting Llm Qwen Code-Review Gpu Vps Self-Hosted
Table of Contents

Last week we ran Llama 3.1 8B on a plain 8 GB VPS. This time it’s Qwen3-Coder 32B on a GPU server, compared against GPT-4 at reading pull requests and pointing out what’s wrong with them. Of all the jobs people give LLMs, that’s the one where open-source models have been catching up fastest.

The setup
#

Server A, self-hosted on a rented GPU server:

  • 8 vCPU EPYC Milan
  • 32 GB RAM
  • 200 GB NVMe
  • 1x NVIDIA L40S (48 GB VRAM)
  • Debian 12, Docker, NVIDIA driver 560, CUDA 12.6

Server B, GPT-4 through the OpenAI API:

  • Model: gpt-4-turbo-2024-04-09, which is still what most “GPT-4” integrations default to
  • Billed per token at OpenAI’s list price for this model
  • We assumed no system-side prompt caching

The local model is Qwen3-Coder 32B Instruct (Apache 2.0), served with vLLM 0.7 in bf16 with tensor-parallel-size 1 and --max-model-len 65536. It exposes an OpenAI-compatible endpoint on :8000/v1.

Why Qwen3-Coder and not a bigger generalist model
#

In our LLM round-up, Qwen3-Coder 32B beat Llama 3.3 70B on code-specific benchmarks (HumanEval+, SWE-Bench Lite, Aider) at less than half the size. That makes it the obvious self-hosted pick for code review. It also fits on a single 48 GB GPU in bf16 with room to spare, so we didn’t need quantization to keep latency down.

Standing it up in 15 minutes
#

This assumes the box already has the NVIDIA Container Toolkit installed:

docker run -d --name qwen-coder \
  --gpus all \
  -p 8000:8000 \
  -v /srv/hf-cache:/root/.cache/huggingface \
  --ipc=host \
  --restart unless-stopped \
  vllm/vllm-openai:v0.7.3 \
  --model Qwen/Qwen3-Coder-32B-Instruct \
  --max-model-len 65536 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --served-model-name qwen3-coder

The first run pulls about 62 GB of weights into /srv/hf-cache, which takes roughly 15 minutes on a 1 Gbps link. Restarts after that are instant.

To check that it’s up:

curl -s http://localhost:8000/v1/models | jq .

If you need it reachable from outside, put nginx in front of it with HTTPS and a static bearer token header. Don’t expose vLLM directly to the internet, because it has no authentication of its own.

The evaluation
#

The PRs
#

We used six pull requests from one of our own internal services, written in Python and Go, with anywhere from 80 to 1,200 lines changed:

  • 2 bug fixes, one of them with a subtle race condition
  • 2 refactors: one safe, and one broken because it removes a mutex that was actually needed
  • 1 new feature that adds a webhook handler
  • 1 “trap” PR that looks reasonable but introduces an auth bypass

The prompt
#

Both models got the same prompt, kept short and dull on purpose:

You are reviewing a pull request. The diff follows.
List concrete issues: bugs, security problems, missed edge
cases, style problems that matter. Be specific. Quote lines
where relevant. If the PR looks clean, say so.

--- diff ---
{{diff}}

We didn’t add chain-of-thought tricks, agentic loops or self-reflection. We wanted to see what each model does with the kind of prompt a typical developer would actually write.

Grading
#

Three engineers graded each review blind. None of them was me or the author of the PR. For each review they recorded whether it found the primary issue (yes, no or partial), a signal-to-noise score from 1 to 5 (does the review stay focused, or does it drown in nitpicks?), the number of hallucinations, meaning statements that are wrong about the diff, and whether they would accept the review as it stands.

Results
#

                               Qwen3-Coder 32B    GPT-4-turbo
Primary issue caught (6 PRs)     5 / 6             6 / 6
  - missed the auth bypass        no                yes
Signal-to-noise (avg, 1-5)        4.1               3.7
Hallucinations (total)            2                 7
Would-accept (avg across 3)       83 %              78 %

The trap PR is where the two models split. GPT-4 caught the auth bypass and Qwen3-Coder missed it. GPT-4’s review said “this removes the require_auth=True check from the /admin/* route group, which appears unintended given the context”. Qwen3-Coder raised three other small issues on the same PR but never connected any of them to auth.

So for reviews where security impact is the main concern, GPT-4 still has the edge, and that’s the one job we wouldn’t hand over yet. Our guess is that it comes down to training data: GPT-4 has probably seen more real-world CVE write-ups than any open model we can download.

On everything else Qwen3-Coder was the more pleasant reviewer. GPT-4 has a habit of commenting on everything: variable names, docstring phrasing, “you could use a f-string here”. On a 1,200-line diff that gets tiring fast. Qwen3-Coder stays much closer to the things that matter, and our reviewers consistently preferred reading its output.

It also made fewer things up. GPT-4 twice referred to config files that don’t exist, and three times claimed a function behaved differently from what its code shows. Qwen3-Coder had two small hallucinations in total across all six PRs. It was faster too, and the cost works differently, which is covered in the next section.

On the remaining four PRs (routine bugs, refactor safety, edge cases and the new feature) the reviews were about equally good. Either model would have caught the real issues.

Speed and cost per review
#

Average per PR (median across the 6):

                          Qwen3-Coder 32B    GPT-4-turbo
Input tokens (avg)          3,420              3,420
Output tokens (avg)         720                940
Time to first token         0.4 s              0.9 s
Time to full response       8.2 s              22 s
API billing                 none (flat fee)    per token

The cost calculation is simple, and you should run it with your own numbers:

API cost per review   = input_tokens * input_price + output_tokens * output_price
breakeven reviews/mo  = GPU server monthly price / API cost per review

Plug in the token counts above, your API provider’s current list price and a real quote for the GPU server you’d rent. Below breakeven the API is cheaper, and above it the self-hosted server wins. A single engineering team will usually sit below that line, while a platform running automated review for many teams will usually go well past it.

There are reasons to run it locally that have nothing to do with cost. Your code stays on your own network, which some NDAs require outright. Latency is predictable, so you don’t get a tail of 40-second API calls whenever OpenAI has a hiccup. And you control the context window: it’s 64k on this setup and cheap to extend, whereas on the API you pay for every token regardless.

When to pick which
#

  • If you’re below breakeven and don’t mind code leaving your network, use the GPT-4 API.
  • If you can’t send code out, you’re above breakeven, or you have latency SLOs to meet, self-host Qwen3-Coder on a GPU server.
  • For security-critical reviews where auth/authz logic matters, run both in parallel and combine their findings until open models catch up on CVE-style reasoning. We expect them to get there within 12 months.

Related