Relay

RelayCase Navigator?mid-duplicate-charge-1

Case41 / 110DatasetRouter EvalTargetBorderline review requiredAreaMid-risk

ExplainerPrevious Next

Cases4

Router Eval

routine-invoice-copy-1 - Routine request

billing

Router Eval

routine-invoice-copy-2 - Routine request

billing

Router Eval

routine-invoice-copy-3 - Routine request

billing

Grounded Reply

gr-001-stale-refund-window - Reject

Customer cites stale 30-day refund article

Customer Message

Hi, this is Ava from Northstar Labs. I think we were charged twice for the same monthly subscription on 11 March. I attached what finance saw in billing and the invoice that came through by email. Can you refund one of them?

mid-duplicate-charge-1

Attachments

Billing charge screenshot

Shows two charges for the same plan and billing period on the customer account.

Open asset

Invoice PDF

Invoice references the same billing cycle and reflects one duplicate line item.

Open asset

AI Draft Reply

Hi Ava, thanks for flagging this. I reviewed the case details and supporting evidence for duplicate-charge refund. The draft is ready, but a human reviewer should confirm the action before anything is sent or applied.

Citations

Risk Flags

Main risk: Reviewer authority check

Reviewer authority check

A human should confirm that the proposed action stays within the support authority band.

Evidence quality

The case is supportable, but one or more inputs still benefit from a quick human verification pass.

Borderline outcome

The model can draft confidently, but the send/no-send decision should remain human-owned.

Reviewer DecisionHuman outcome for this AI draft

Review and decide if the draft is grounded, policy-safe, complete, and the right action.

Use A, E, X, or R to choose an action. Click citations to inspect evidence.

Override ReasonOptional

Stakeholder Context

What drives the review decision, and why this setup is currently recommended.

Experiment SummaryBest current setup for reviewCollapsed by default so the review workflow stays focused.

The demo stays on this review decision, but the experiment spine now tells us which setup should power it. The current recommendation keeps safe approvals, blocks unsafe approvals, and returns clean structured output every time.

Llama R3

Production pick

Best operating row: highest reliable accuracy with zero parse failures.

Parsed accuracy95%

Unsafe approval0%

Approval precision100%

Parse failures0 / 150

P95 latency17.2s

Nemotron R3

Conservative runner-up

Strong safety/usefulness row, but one malformed JSON output.

Parsed accuracy90%

Unsafe approval0%

Approval precision100%

Parse failures1 / 150

P95 latency27.8s

Qwen R3

Research signal

Perfect on parsed rows, but parse reliability blocks production use.

Parsed accuracy100%

Unsafe approval0%

Approval precision100%

Parse failures7 / 150

P95 latency34.4s

Parsed accuracy means accuracy after excluding malformed model outputs. Parse failures still count as product failures, which is why Llama beats Qwen for production despite Qwen's perfect parsed rows.