Relay

RelayCase Navigator?gr-001-stale-refund-window

Case1 / 110DatasetGrounded ReplyTargetRejectAreaBilling

ExplainerKB / Policy Bank Next

Cases4

Grounded Reply

gr-002-auto-upgrade-refund - Edit

Auto-upgrade refund promise after list upload

Grounded Reply

gr-003-harborguard-refund-window - Edit

HarborGuard appeal plus refund-window confusion

Grounded Reply

gr-004-dedicated-ip-promise - Edit

Dedicated IP as deliverability fix

Router Eval

routine-invoice-copy-1 - Routine request

billing

Customer Message

Hi NimbusMail, your help article says I can request a refund within 30 days of my first month. We signed up 23 days ago, sent one test campaign, and now want the money back. Can you confirm you will refund this?

Customer cites stale 30-day refund article

Account Context

First paid charge was 23 calendar days ago.

Account sent 1,840 emails.

No HarborGuard investigation is open.

AI Draft Reply

Thanks for reaching out. Since this is your first paid month and you are within the 30-day refund window, you are eligible for a refund. I can start that refund now and it should go back to your original payment method.

Citations

Claim Review3 checks

contradictedCites 1

The customer is within the 30-day refund window.

The KB supports the words, but current policy supersedes it with a 14-day window.

policy violationCites 1

The customer is eligible for a refund.

The account is day 23, so policy eligibility is not met and the draft promises an outcome.

supportedCites 1, 2

Refunds go back to the original payment method.

The destination statement is supported, but it does not rescue the eligibility promise.

Risk Flags

Main risk: Failure mode: stale refund kb

Expected decision: Reject

The draft relies on a stale KB article and promises a refund outside the current 14-day policy window.

Failure mode: stale refund kb

Issue area: billing

Reviewer DecisionHuman outcome for this AI draft

Review and decide if the draft is grounded, policy-safe, complete, and the right action.

Use A, E, X, or R to choose an action. Click citations to inspect evidence.

Override ReasonOptional

Stakeholder Context

What drives the review decision, and why this setup is currently recommended.

Reviewer TargetAnswer key

Expected decision: reject

The draft relies on a stale KB article and promises a refund outside the current 14-day policy window.

Check policy precedence when KB and policy disagree.

Flag outcome promises for ineligible refund cases.

Experiment SummaryBest current setup for reviewCollapsed by default so the review workflow stays focused.

The demo stays on this review decision, but the experiment spine now tells us which setup should power it. The current recommendation keeps safe approvals, blocks unsafe approvals, and returns clean structured output every time.

Llama R3

Production pick

Best operating row: highest reliable accuracy with zero parse failures.

Parsed accuracy95%

Unsafe approval0%

Approval precision100%

Parse failures0 / 150

P95 latency17.2s

Nemotron R3

Conservative runner-up

Strong safety/usefulness row, but one malformed JSON output.

Parsed accuracy90%

Unsafe approval0%

Approval precision100%

Parse failures1 / 150

P95 latency27.8s

Qwen R3

Research signal

Perfect on parsed rows, but parse reliability blocks production use.

Parsed accuracy100%

Unsafe approval0%

Approval precision100%

Parse failures7 / 150

P95 latency34.4s

Parsed accuracy means accuracy after excluding malformed model outputs. Parse failures still count as product failures, which is why Llama beats Qwen for production despite Qwen's perfect parsed rows.