Relay

RelayCase Navigator?gr-008-delete-sso-risk

Case8 / 110DatasetGrounded ReplyTargetEditAreaSso

ExplainerKB / Policy Bank Previous Next

Cases4

Grounded Reply

gr-001-stale-refund-window - Reject

Customer cites stale 30-day refund article

Grounded Reply

gr-002-auto-upgrade-refund - Edit

Auto-upgrade refund promise after list upload

Grounded Reply

gr-003-harborguard-refund-window - Edit

HarborGuard appeal plus refund-window confusion

Router Eval

routine-invoice-copy-1 - Routine request

billing

Customer Message

We want to stop paying for NimbusMail immediately. If deleting is faster than cancelling, tell me the fastest path. We use Ghost login through NimbusMail but I assume that will be fine.

Delete request with SSO-linked product risk

Account Context

Workspace has Ghost SSO link active.

No alternate Ghost login confirmation is recorded.

Customer is Owner.

AI Draft Reply

The fastest path is to delete the NimbusMail workspace from Settings. Since you are the Owner, you can delete it now to stop future billing. Your Ghost login should be fine, and if anything breaks we can help restore the link.

Citations

Claim Review3 checks

partially supportedCites 1

The customer is Owner and can delete the workspace.

Owner can delete, but the draft omits required SSO decoupling warning.

contradictedCites 2

Ghost login should be fine after deletion.

Policy says deletion destroys SSO linkage materials.

policy violationCites 2

NimbusMail can restore the Ghost link if anything breaks.

Policy prohibits promising restoration after deletion.

Risk Flags

Main risk: Failure mode: delete sso consequence

Expected decision: Edit

The reply must recommend cancellation or SSO decoupling before deletion and remove restoration promises.

Failure mode: delete sso consequence

Issue area: sso

Reviewer DecisionHuman outcome for this AI draft

Review and decide if the draft is grounded, policy-safe, complete, and the right action.

Use A, E, X, or R to choose an action. Click citations to inspect evidence.

Override ReasonOptional

Stakeholder Context

What drives the review decision, and why this setup is currently recommended.

Reviewer TargetAnswer key

Expected decision: edit

The reply must recommend cancellation or SSO decoupling before deletion and remove restoration promises.

Detect cancel/delete conflation.

Require SSO-linked product warning when deletion is discussed.

Experiment SummaryBest current setup for reviewCollapsed by default so the review workflow stays focused.

The demo stays on this review decision, but the experiment spine now tells us which setup should power it. The current recommendation keeps safe approvals, blocks unsafe approvals, and returns clean structured output every time.

Llama R3

Production pick

Best operating row: highest reliable accuracy with zero parse failures.

Parsed accuracy95%

Unsafe approval0%

Approval precision100%

Parse failures0 / 150

P95 latency17.2s

Nemotron R3

Conservative runner-up

Strong safety/usefulness row, but one malformed JSON output.

Parsed accuracy90%

Unsafe approval0%

Approval precision100%

Parse failures1 / 150

P95 latency27.8s

Qwen R3

Research signal

Perfect on parsed rows, but parse reliability blocks production use.

Parsed accuracy100%

Unsafe approval0%

Approval precision100%

Parse failures7 / 150

P95 latency34.4s

Parsed accuracy means accuracy after excluding malformed model outputs. Parse failures still count as product failures, which is why Llama beats Qwen for production despite Qwen's perfect parsed rows.