Local LLM-as-a-Verifier on a 16GB ROCm GPU: Real Benchmark Data

TL;DR: I benchmarked the local LLM-as-a-Verifier setup running Qwen3.5-9B (Q4_K) on a single 16GB AMD GPU via ROCm. The verifier processes a code comparison in ~0.5–2s at 66% cache hit rate, correctly scoring valid code changes at 1.0 and invalid ones at 0.0–0.14. The 16GB card handles the 9B model comfortably, leaving headroom for other services.

This article is a follow-up to the architecture overview and the Docker packaging guide. Those articles described the what and how — this one provides the measured how well.

1. The Setup

The benchmark runs against a production-like deployment on a homelab Linux (CachyOS) machine:

The backend runs on a separate machine connected via a local Gigabit network. The verifier container communicates with the llama.cpp backend over HTTP.

2. What the Verifier Does

The docker-llm-as-a-verifier service wraps the LLM-as-a-Verifier research library. It reads token-level log probabilities for ordered score tokens and turns that distribution into a continuous reward.

The container exposes 7 HTTP endpoints:

The comparison endpoint is the core primitive. Each /v1/compare call makes 3 LLM inference requests: one to score candidate A, one to score candidate B, and one to compare the two scores and produce a verdict.

3. Benchmark Results

3.1 Latency

Measured from the verifier's own log timestamps — each timestamp is the HTTP response from the backend (llama.cpp):

The cache hit rate is 66.4% — 22,417 cached tokens out of 33,768 prompt tokens across 80 requests. The verifier's fine_grained_reward module caches scoring results on disk (the verifier-cache Docker volume), so repeated comparisons of the same code snippets bypass the LLM entirely.

3.2 Scoring Quality

Across the benchmark session, the verifier produced consistent scores:

With MIN_SCORE=0.8, even the best incorrect candidate (0.14) is correctly rejected. The score distribution shows a clean separation between correct and incorrect — no ambiguous cases near the boundary.

3.3 Token Usage

Over 80 verifier requests (the benchmark session):

The 0 reasoning tokens confirm the model is Qwen3.5-9B (a dense model, not a reasoning/R1 model). The verifier uses the model's direct log-probability output, not chain-of-thought.

3.4 VRAM & GPU Utilization

On the 16GB AMD GPU:

The 9B model at Q4_K fits comfortably with ~5.7 GB headroom for KV cache and other workloads. The 131K context window is a llama.cpp default — the verifier's actual prompt sizes are much smaller (~400–800 tokens per comparison).

4. Architecture: What Actually Runs

The production pipeline is simpler than the full architecture diagram in the first article:

Agent (OpenCode)
  │ Submit change proposal
  ▼
guardrail-mcp (MCP server)
  │ commit_token → validate → consume
  │ apply_patch (cargo check / tsc syntax validation)
  │ softguard (HTTP → verifier)
  ▼
llm-verifier (Docker :8010)
  │ POST /v1/directed (compare old vs new code)
  ▼
llama.cpp (ROCm, cybertron :8081)
  │ Qwen3.5-9B Q4_K inference

The guardrail-mcp provides:

• commit_token: 2PC lifecycle (create → validate → consume → revoke), SHA-256 proposal binding, 30-min TTL, git revision binding
• apply_patch: Patch application with compiler verification (cargo check / tsc) before write
• inspect_context: Tree-sitter structural extraction for context-aware patches
• softguard: HTTP verifier runner with configurable endpoints, timeout, and API key

The key insight: the LLM verifier runs as a soft guard — it catches semantic bugs that syntax checking misses. Syntax validation (cargo check / tsc) runs first as a cheap deterministic gate. Only proposals that pass syntax get sent to the LLM verifier for semantic review.

5. Limitations

These numbers are from a single-session benchmark, not a formal evaluation suite:

• No batch throughput test: The 80 requests were sequential, not concurrent. True TPS under load would be lower.
• No cloud baseline: The OpenRouter free route was not benchmarked. A direct comparison between local ROCm and cloud API remains to be done.
• No VL data: The multimodal vision endpoint (9B VL) is still in development. These numbers are for the text-only code comparison path.
• No Track 1 short-circuit rate: The AST-level structural check described in the first article is aspirational. The current guardrail-mcp uses syntax validation (cargo check / tsc) as its deterministic gate, not a dedicated AST structural checker.
• No cost-per-request: The 0.5–2s per comparison at 78% GPU utilization implies ~0.2–0.5 cents per request in electricity (at 250W system power), but this is a rough estimate, not a metered measurement.

6. Conclusion

The local LLM-as-a-Verifier setup works on a single 16GB AMD GPU:

• Qwen3.5-9B Q4_K fits comfortably with headroom (10.2 GB / 15.9 GB)
• Compare latency: ~0.5–2s — fast enough for interactive guardrail use
• Cache hit rate: 66% — repeated comparisons are nearly free
• Score quality: Clean separation between correct (1.0) and incorrect (0.0–0.14)

The setup costs under $500 (the GPU) and runs on a standard Linux machine with ROCm. No cloud API keys, no monthly bills, no data leakage.

The next step is to benchmark the OpenRouter free route as a cloud baseline and measure concurrent throughput under load. But even without that comparison, the local path is already practical for daily use.

Repositories:

• Verifier Docker packaging: github.com/cawa0505/docker-llm-as-a-verifier
• 2PC guard gate: github.com/cawa0505/guardrail-mcp
• Architecture overview: Poor Man's LLM-as-a-Verifier