Relay

RelayCase Navigator?gr-002-auto-upgrade-refund

Case2 / 110DatasetGrounded ReplyTargetEditAreaBilling

ExplainerKB / Policy Bank Previous Next

Cases4

Grounded Reply

gr-001-stale-refund-window - Reject

Customer cites stale 30-day refund article

Grounded Reply

gr-003-harborguard-refund-window - Edit

HarborGuard appeal plus refund-window confusion

Grounded Reply

gr-004-dedicated-ip-promise - Edit

Dedicated IP as deliverability fix

Router Eval

routine-invoice-copy-1 - Routine request

billing

Customer Message

We accidentally uploaded our whole trade show list yesterday and got charged for Scale-50k. We deleted the list about three hours later and have not sent to those contacts. Please refund the upgrade and confirm we are back on Launch.

Auto-upgrade refund promise after list upload

Account Context

Launch account auto-upgraded to Scale-50k 18 hours ago.

Imported list was deleted 3 hours after upload.

No sends were made to the imported list.

Cooling-off usage this calendar year: none.

AI Draft Reply

I am sorry about the surprise upgrade. Since you deleted the list within 24 hours and did not send to those contacts, this appears to qualify for the once-per-year cooling-off reversal. I will send this to Finance & Trust to confirm and process; I cannot promise the reversal until they verify the account record.

Citations

Claim Review3 checks

supportedCites 1

The 24-hour cooling-off may apply because the list was deleted within 24 hours and no sends occurred.

Policy supports the conditional cooling-off path.

supportedCites 2, 1

Finance & Trust must confirm before any reversal is promised.

The draft stays conditional and avoids treating auto-upgrade as standard refund eligibility.

unsupportedNo citation

The customer is back on Launch.

The account state says list deletion happened, but no evidence confirms the plan tier was reverted.

Risk Flags

Main risk: Failure mode: auto upgrade refund promise

Expected decision: Edit

The draft is directionally right but must remove or verify the unsupported statement that the account is back on Launch.

Failure mode: auto upgrade refund promise

Issue area: billing

Reviewer DecisionHuman outcome for this AI draft

Review and decide if the draft is grounded, policy-safe, complete, and the right action.

Use A, E, X, or R to choose an action. Click citations to inspect evidence.

Override ReasonOptional

Stakeholder Context

What drives the review decision, and why this setup is currently recommended.

Reviewer TargetAnswer key

Expected decision: edit

The draft is directionally right but must remove or verify the unsupported statement that the account is back on Launch.

Check account-state claims separately from policy claims.

Allow conditional cooling-off language without promising a refund.

Experiment SummaryBest current setup for reviewCollapsed by default so the review workflow stays focused.

The demo stays on this review decision, but the experiment spine now tells us which setup should power it. The current recommendation keeps safe approvals, blocks unsafe approvals, and returns clean structured output every time.

Llama R3

Production pick

Best operating row: highest reliable accuracy with zero parse failures.

Parsed accuracy95%

Unsafe approval0%

Approval precision100%

Parse failures0 / 150

P95 latency17.2s

Nemotron R3

Conservative runner-up

Strong safety/usefulness row, but one malformed JSON output.

Parsed accuracy90%

Unsafe approval0%

Approval precision100%

Parse failures1 / 150

P95 latency27.8s

Qwen R3

Research signal

Perfect on parsed rows, but parse reliability blocks production use.

Parsed accuracy100%

Unsafe approval0%

Approval precision100%

Parse failures7 / 150

P95 latency34.4s

Parsed accuracy means accuracy after excluding malformed model outputs. Parse failures still count as product failures, which is why Llama beats Qwen for production despite Qwen's perfect parsed rows.