Nightly flaky test fix verified by stress runs
Once a night a bot picks one test that has flaked in two places, measures how often it fails on main, and lets a coding session try a fix on a runner that holds only read credentials. A separate job stress-runs the result in a clean checkout and opens a draft pull request for a person to review.
What it does
- Picks one flaky test with failures in two distinct places and records the pick at once
- Measures the failure rate on main to size the number of verification runs
- Lets a coding session attempt a fix with read scopes only
- Stress-runs the candidate in a second clean checkout and judges it from test output, not the session's claims
- Opens a draft pull request, or comments on the issue when nothing was pushed
Your approval boundaries
Creates draftsThese actions always wait for your yes:
- Merging any fix pull request
- Turning the nightly job on at all
- Any change that touches workflow files
You decide what happens next.
More about this workflow
This setup is for developers and small engineering teams who want fewer chores. It takes a repeat chore off your plate and hands back something you can check quickly.
First, it picks one flaky test with failures in two distinct places and records the pick at once. Next, it measures the failure rate on main to size the number of verification runs. Then it lets a coding session attempt a fix with read scopes only. After that, it stress-runs the candidate in a second clean checkout and judges it from test output, not the session's claims. Finally, it opens a draft pull request, or comments on the issue when nothing was pushed. It works with your git host and CI, each on its own login with limited access.
You get the prep work done and waiting for you. The agent stops and asks for your yes before merging any fix pull request, turning the nightly job on at all and any change that touches workflow files, and says plainly if a step cannot be undone.
“The model never holds a write credential.”
Explore required tools and model recommendations
Tools and access
GitHub
Handles the git host and ci part of this job. Grant a fine-grained token, read code and write comments.
API required: Yes. Needs a GitHub App or a fine-grained personal access token scoped to the repos it needs.
Official links only. Not sponsored, no affiliate links.
Model recommendations
Best pick
Claude Opus 5.5
Its maker points to it for long running coding work.
$4 per million input tokens, $20 per million output tokens.
Anthropic docs as of October 11, 2026
Cheaper backup
Claude Sonnet 5.5
Half the price per token and still strong for reviews and docs.
$2 per million input tokens, $10 per million output tokens.
Anthropic docs as of October 11, 2026
Our own picks. Not sponsored, no affiliate links.
Build your prompt
Add your details, then copy the prompt into your tool.
2Your prompt
4 of 4 still emptyYour answers stay in this browser only.
Run your workflow
Connect your tools with only the access this setup needs, then follow these steps.
Steps
Do these in order. Test on old data before it touches live work.
- 1Connect GitHub with its own login, not yours. Grant a fine-grained token, read code and write comments.
- 2Paste the prompt where your harness takes instructions. See the note under the Copy button.
- 3Do a dry run on old or test data first. Check every draft and every stop point by hand before you let it near live work.
- 4Read its log after the first real run. Only widen access once a week of runs looks clean.
Safeguards and trust score
The score describes this example setup’s access, approval boundaries, ability to undo changes, and scope.
Writes drafts and notes inside your tools, nothing leaves on its own.
Waits for a yes on: merging any fix pull request, turning the nightly job on at all, any change that touches workflow files.
Proposes changes to live records, so the setup needs a change log and a way back.
Runs on its own login across 2 tools. Fewer connections means a smaller blast radius.
How to raise it
- Undo and logs: Stop it from editing live records. Have it write the change as a proposal you apply, and keep a log of every change with a way to roll it back +3
- Scope: Run it on one tool only, with its own login that can see just what this job needs +2
Do all of these to reach 91, up from 86.
Want help getting there? Get an AI Trust Audit from Pixeldesigns
Reviewed by Trustcase (example setup). How scoring works
Embed badge
Show this trust score on your site or README. It links back here.
[](https://trustcase.dev/use-cases/nightly-flaky-test-fix-verified-by-stress-runs)<a href="https://trustcase.dev/use-cases/nightly-flaky-test-fix-verified-by-stress-runs"><img src="https://trustcase.dev/badge/nightly-flaky-test-fix-verified-by-stress-runs.svg" alt="Trustcase trust score for Nightly flaky test fix verified by stress runs" height="20"></a>More engineering workflows
- Nightly flake hunt that files one issue per test
Each night every app's test suite runs 100 times with random seeds on spare machines, outside the normal CI path. Any test that fails some runs and passes the rest gets an issue with its failure rate, the failing seeds, the messages and the commands to reproduce it.
- Security incident triage that recommends and records approvals
A tier one security analyst pastes an alert or log and gets back a structured triage: incident type, severity with confidence, missing information and ordered first actions. Each action is marked safe or needing a specialist's approval, and approvals are only recorded, never carried out by the agent.
- Ticket to pull request with approval of every write
Takes one tracker ticket and decides whether it is ready, too big or too vague. Ready tickets get a fix and tests in a sandbox, a self review and a pull request. Too big tickets are split, vague ones get questions posted back. Every write to the tracker or the code host stops for a person to approve, edit or reject the exact arguments.
- Query and migration cost review before every push
Before a branch is pushed, a reviewer looks at every changed database query and migration and asks what it does at production size. It measures plans on a local copy and writes a findings table into the pull request.
Other workflows that use the same tools
- Site change tester
Marketing. Tries small edits to your site, watches whether more visitors stay, and keeps only the changes that help. Good for steady small wins on a landing page.
- Daily kids lesson builder
Personal. Builds a short interactive lesson for your kids each night and has a second agent check it. You read a review email and approve it before the kids get the link.
- Weekly delivery status report
Operations. Pulls your team's work items once a week and builds a status report grouped by initiative. It puts at risk items first and flags anything with no update in two weeks.