Skip to content

Agent reply scorecard before a release

Runs a fixed set of test conversations against your customer facing agent and grades each reply, so a prompt or model change cannot quietly change what it says. A scorecard with a ready or not ready verdict goes to the team.

EngineeringWorks with n8n · Claude Code+1Community example
Build your promptSee requirements

What it does

  • Sends each test case to the agent's webhook
  • Checks replies in code for required phrases, forbidden phrases and expected tool calls
  • Has a second model grade each rubric with a strict pass or fail and one reason
  • Compares with the last run to show what now fails or now passes
  • Treats a timeout or a bad response as a failed case, not a crashed run

Your approval boundaries

Approval required

These actions always wait for your yes:

  • Calling the agent ready for release
  • Changing the test cases or the pass threshold
  • Posting the scorecard to a shared channel

You decide what happens next.

More about this workflow

This setup is for developers and small engineering teams who want fewer chores. It takes a repeat chore off your plate and hands back something you can check quickly.

First, it sends each test case to the agent's webhook. Next, it checks replies in code for required phrases, forbidden phrases and expected tool calls. Then it has a second model grade each rubric with a strict pass or fail and one reason. After that, it compares with the last run to show what now fails or now passes. Finally, it treats a timeout or a bad response as a failed case, not a crashed run. It works with your automation platform, chat and spreadsheets, each on its own login with limited access.

You get the prep work done and waiting for you. The agent stops and asks for your yes before calling the agent ready for release, changing the test cases or the pass threshold and posting the scorecard to a shared channel, and says plainly if a step cannot be undone.

“tell you when your automations break, before a customer does”
romielwillautomate romielwillautomate on GitHub · Oct 26, 2026
Before you startAutomation platform · Chat accessScoped permissionsn8n or Claude Code
Explore required tools and model recommendations

Tools and access

  • n8n

    Handles the automation platform part of this job. Grant create workflows, left inactive until reviewed.

    API required: Yes. Needs an n8n API key from settings. New workflows should stay inactive until you review them.

    Sign up: n8n

  • Slack

    Handles the chat part of this job. Grant a bot token in the channels it needs.

    API required: Yes. Needs a Slack app created in your workspace. It gets a bot token with the scopes you approve.

    Sign up: Create a Slack app

  • Google Sheets

    Handles the spreadsheets part of this job. Grant access to the single sheet it needs.

    API required: Yes. Needs a Google Cloud OAuth app or a service account with the Sheets API turned on.

    Sign up: Google Workspace

Official links only. Not sponsored, no affiliate links.

Model recommendations

Best pick

Claude Opus 5.5

Its maker points to it for long running coding work.

$4 per million input tokens, $20 per million output tokens.

Anthropic docs as of October 11, 2026

Cheaper backup

Claude Sonnet 5.5

Half the price per token and still strong for reviews and docs.

$2 per million input tokens, $10 per million output tokens.

Anthropic docs as of October 11, 2026

Our own picks. Not sponsored, no affiliate links.

Build your prompt

Add your details, then copy the prompt into your tool.

1Your details

Fill these in and the prompt updates as you type.

json or a sheet

percent

any failure means not ready

2Your prompt

4 of 4 still empty
Fill these in before you run it: [AGENT WEBHOOK URL] [TEST CASES: json or a sheet] [PASS THRESHOLD: percent] [CRITICAL CASES: any failure means not ready] You are the Agent reply scorecard before a release agent. Runs a fixed set of test conversations against your customer facing agent and grades each reply, so a prompt or model change cannot quietly change what it says. A scorecard with a ready or not ready verdict goes to the team. Use only the details filled in above. If any [BRACKETED] item is still blank or unclear, stop and ask for it first. Do not guess. What you do on your own: - Sends each test case to the agent's webhook - Checks replies in code for required phrases, forbidden phrases and expected tool calls - Has a second model grade each rubric with a strict pass or fail and one reason - Compares with the last run to show what now fails or now passes - Treats a timeout or a bad response as a failed case, not a crashed run What you must never do without a yes from a person: - Calling the agent ready for release - Changing the test cases or the pass threshold - Posting the scorecard to a shared channel Rules: - Before any step on the list above, stop and ask. Say in plain words what you want to do, who it affects, and whether it can be undone. - If something cannot be undone, say so first, in one short line. - Write drafts, not final actions. A person sends, pays, posts, or deletes. - Keep a short log of every action you took and every source you read. - If you are unsure, ask instead of guessing.
View run instructionsPaste into: n8n, Claude Code, Make

Your answers stay in this browser only.

Run your workflow

Connect your tools with only the access this setup needs, then follow these steps.

Steps

Do these in order. Test on old data before it touches live work.

  1. 1Connect n8n with its own login, not yours. Grant create workflows, left inactive until reviewed.
  2. 2Connect Slack with its own login, not yours. Grant a bot token in the channels it needs.
  3. 3Connect Google Sheets with its own login, not yours. Grant access to the single sheet it needs.
  4. 4Paste the prompt where your harness takes instructions. See the note under the Copy button.
  5. 5Do a dry run on old or test data first. Check every draft and every stop point by hand before you let it near live work.
  6. 6Read its log after the first real run. Only widen access once a week of runs looks clean.

Safeguards and trust score

The score describes this example setup’s access, approval boundaries, ability to undo changes, and scope.

Access18 / 30

Can draft things that send or spend, so it sits close to risky actions.

Human approval30 / 30

Waits for a yes on: calling the agent ready for release, changing the test cases or the pass threshold, posting the scorecard to a shared channel.

Undo and logs14 / 20

Proposes changes to live records, so the setup needs a change log and a way back.

Scope16 / 20

Runs on its own login across 3 tools. Fewer connections means a smaller blast radius.

How to raise it

  • Access: Take away its ability to send, post or spend. Let it only write drafts, and have a person press send or pay from their own account +6
  • Undo and logs: Stop it from editing live records. Have it write the change as a proposal you apply, and keep a log of every change with a way to roll it back +3
  • Scope: Cut it down from 3 connected tools to the 2 it needs most, each with its own limited login instead of your full account +2

Do all of these to reach 89, up from 78.

Want help getting there? Get an AI Trust Audit from Pixeldesigns

Reviewed by Trustcase (example setup). How scoring works

Embed badge

Show this trust score on your site or README. It links back here.

Trustcase trust score badge
Markdown
[![Trustcase trust score for Agent reply scorecard before a release](https://trustcase.dev/badge/agent-reply-scorecard-before-a-release.svg)](https://trustcase.dev/use-cases/agent-reply-scorecard-before-a-release)
HTML
<a href="https://trustcase.dev/use-cases/agent-reply-scorecard-before-a-release"><img src="https://trustcase.dev/badge/agent-reply-scorecard-before-a-release.svg" alt="Trustcase trust score for Agent reply scorecard before a release" height="20"></a>

More engineering workflows

Other workflows that use the same tools

  • Spoken site update to daily report

    Operations. Turns a short spoken end of day update from a crew lead into a structured daily report. Plain rules, not the AI, decide who hears about each issue.

  • Safety stock review with planner approval

    Operations. Recalculates safety stock and reorder points for a product category in plain code, then has the model explain each change and the working capital it frees or adds. A planner approves or rejects every decision by email before it reaches the purchase order queue.

  • App ratings drop war room

    Support. When an app's store rating dips, an agent team finds the release that caused it, backs the verdict with clustered review evidence and a confidence score computed in code, files deduplicated bugs and drafts support replies. Replies sit as pending until someone approves them in Slack, and nothing is ever published to a store.