What does it actually take to ship an LLM review layer?

1. Question

For this exploration, the practical question was that if I put an LLM reviewer in front of support replies, what has to support at mimimum it so it behaves like a useful teammate instead of a very confident autocomplete box?

Why now

Two easy answers usually appear when discussing solutions with colleages: “give it documents” or “fine-tune it.” But those are names, not plans. So I built a small environments where I could watch each idea work on a product. NimbusMail has realistic support rules: refunds, billing disputes, deliverability problems, account deletion, and countries where service is restricted.

While review-screen demo is the visible artifact. The real product here is the measurement trail of approaches used which could hopefully provide a way to make the same decisions again for a different workflow.

2. Experiment setup

I chose Tinker Labs to provide model inference and training and settled on 3 models due to their parameter count and performance. In LLM work, an eval is just a test set. “Locked evals” means I do not keep changing the test until the model looks good. So in order measure our LLMs's work, I generated a 150-ticket exam set based off the policy/KB and locked it before looking at the final answers. The exam was balanced across four support decisions: approve, edit, escalate, or reject. Then I decided on four test environments for our LLMs: R0 is where LLMs gets only the ticket, R1 gets relevant policy snippets, R2 adds a hard rule checker, and R3 adds a small trained adapter called LoRA. Think of LoRA as a clip-on memory layer: it nudges the model toward the behavior it practiced earlier without rebuilding the whole model.

3. Findings

R0 — no evidence

R0 is the model walking into the exam with no handbook. It ended up looking safe because it barely says yes. Llama made only 4 approvals out of the 40 tickets that were safe to approve; Qwen and Nemotron make zero. This is known as conservative collapse: where the LLM tries to avoid danger by refusing to help.

R1 — retrieved evidence (with a random-evidence control)

R1 gives the model retrieved evidence, meaning the ticket plus the handbook pages (in embedded chunks) that seem relevant. For Llama, accuracy rose to 66%, and approvals return to a manageable territory. I also tried giving random policy snippets which made performance dropped by more than 10 points. This tells tells us the content of the evidence mattered, not just the structure of a citation block.

R2 — deterministic gate

R2 adds a deterministic gate (just means it follows fixed rules). If a reply lacks a citation, trips a risk flag, or conflicts with policy, it's blocked. This is useful but here it became too strict. It removed unsafe approvals by blocking every approval, including the good ones. A gate can help catch clear violations but often cannot turn a confused answer into a good one.

R3 — process-supervised LoRA

R3 is where the system started to look like a trained reviewer. I did not only train it on final answers but on process as well. Like, which decision to make, which claim was supported by policy, which citation belonged to that claim, and which risks should block approval. Llama in R3 reached 95.3% accuracy, kept all 40 safe approvals, made no unsafe approvals, and returned clean JSON every time. Qwen showed a strong research signal but broke JSON on 7 rows. Unfortunately a broken output is still broken product.

Paired bootstrap, 5000 resamples

Decision-accuracy improvement with 95% CI. Positive = candidate wins. Dots right of 0 = reliably better.

Final method matrix (static)

Same results as the interactive view, frozen as a paper-style reference.

4. Surprises

Three things changed the plan. 1) the original eval rewarded doing nothing, so I had to rebuild. 2) more training was not automatically better. A checkpoint is a saved version of the model after some training; Llama’s second checkpoint ended up the best best. Later checkpoints kept studying the examples but drifted toward unsafe approvals. 3) output format matters. If the app expects JSON and the model returns something else often after sevearl evals. It is a broken handoff.

Training loss is the model’s practice-score while it studies. It tells us whether training is happening; it does not tell us whether customers are safer. I picked checkpoints on a separate 50-ticket dev set using product metrics: accuracy, unsafe approval, useful approvals, parse errors, and citations that stayed inside the provided evidence.

5. Production implications

On Tinker, Llama R3 costed about 0.13¢ per review, and retraining the LoRA adapter is roughly $0.60 per refresh. Even at 1M reviews/month, the model bill would sit in the low hundreds. The real cost is mantainance: updating policy chunks, refreshing the eval when support policies change, re-auditing labels, and deciding who owns an unsafe approval.

Platform options and list prices (orientation only)

All prices are per 1M tokens, using list prices from 2026-04. They are orientation numbers, not procurement quotes. Source links are in the appendix.

Maintenance cost at three volumes

6. Recommendation

In this short set of experiments, Llama R3 stood our, keeping R1 as the simpler fallback. But I would not treat this demo as proof on its own. Before making external claims, the eval still needs a full human audit. Still, I beleive it makes sense to add complexity only when the simpler layer has failed in a measurable way - which we have tested.

Appendix

Confusion matrices (Llama, all 5 rungs)