Skip to content

Nightly flake hunt that files one issue per test

Each night every app's test suite runs 100 times with random seeds on spare machines, outside the normal CI path. Any test that fails some runs and passes the rest gets an issue with its failure rate, the failing seeds, the messages and the commands to reproduce it.

EngineeringWorks with Claude Code · Cursor+1Community example
Build your promptSee requirements

What it does

  • Runs each suite 100 times in shards with random seeds
  • Finds tests that fail some runs and pass the rest
  • Files one issue per flaky test with failure rate, seeds, messages and reproduce commands
  • Adds a model written diagnosis only when a token is configured
  • Skips a night quietly if the machines are down, since nothing waits on it

Your approval boundaries

Creates drafts

These actions always wait for your yes:

  • Fixing or quarantining any test
  • Changing the run count or shard size

You decide what happens next.

More about this workflow

This setup is for developers and small engineering teams who want fewer chores. It takes a repeat chore off your plate and hands back something you can check quickly.

First, it runs each suite 100 times in shards with random seeds. Next, it finds tests that fail some runs and pass the rest. Then it files one issue per flaky test with failure rate, seeds, messages and reproduce commands. After that, it adds a model written diagnosis only when a token is configured. Finally, it skips a night quietly if the machines are down, since nothing waits on it. It works with your git host and CI, each on its own login with limited access.

You get the prep work done and waiting for you. The agent stops and asks for your yes before fixing or quarantining any test and changing the run count or shard size, and says plainly if a step cannot be undone.

“if Crew is down the hunt misses a night”
Nate Todd ntodd on GitHub · Nov 7, 2026
Before you startGit host · CI accessScoped permissionsClaude Code or Cursor
Explore required tools and model recommendations

Tools and access

  • GitHub

    Handles the git host and ci part of this job. Grant a fine-grained token, read code and write comments.

    API required: Yes. Needs a GitHub App or a fine-grained personal access token scoped to the repos it needs.

    Sign up: Create a GitHub account

Official links only. Not sponsored, no affiliate links.

Model recommendations

Best pick

Claude Opus 5.5

Its maker points to it for long running coding work.

$4 per million input tokens, $20 per million output tokens.

Anthropic docs as of October 11, 2026

Cheaper backup

Claude Sonnet 5.5

Half the price per token and still strong for reviews and docs.

$2 per million input tokens, $10 per million output tokens.

Anthropic docs as of October 11, 2026

Our own picks. Not sponsored, no affiliate links.

Build your prompt

Add your details, then copy the prompt into your tool.

1Your details

Fill these in and the prompt updates as you type.

e.g. 100 in 4 shards

2Your prompt

4 of 4 still empty
Fill these in before you run it: [REPO] [APPS OR SUITES TO HUNT] [RUNS PER NIGHT: e.g. 100 in 4 shards] [ISSUE LABEL] You are the Nightly flake hunt that files one issue per test agent. Each night every app's test suite runs 100 times with random seeds on spare machines, outside the normal CI path. Any test that fails some runs and passes the rest gets an issue with its failure rate, the failing seeds, the messages and the commands to reproduce it. Use only the details filled in above. If any [BRACKETED] item is still blank or unclear, stop and ask for it first. Do not guess. What you do on your own: - Runs each suite 100 times in shards with random seeds - Finds tests that fail some runs and pass the rest - Files one issue per flaky test with failure rate, seeds, messages and reproduce commands - Adds a model written diagnosis only when a token is configured - Skips a night quietly if the machines are down, since nothing waits on it What you must never do without a yes from a person: - Fixing or quarantining any test - Changing the run count or shard size Rules: - Before any step on the list above, stop and ask. Say in plain words what you want to do, who it affects, and whether it can be undone. - If something cannot be undone, say so first, in one short line. - Write drafts, not final actions. A person sends, pays, posts, or deletes. - Keep a short log of every action you took and every source you read. - If you are unsure, ask instead of guessing.
View run instructionsPaste into: Claude Code, Cursor, LangGraph

Your answers stay in this browser only.

Run your workflow

Connect your tools with only the access this setup needs, then follow these steps.

Steps

Do these in order. Test on old data before it touches live work.

  1. 1Connect GitHub with its own login, not yours. Grant a fine-grained token, read code and write comments.
  2. 2Paste the prompt where your harness takes instructions. See the note under the Copy button.
  3. 3Do a dry run on old or test data first. Check every draft and every stop point by hand before you let it near live work.
  4. 4Read its log after the first real run. Only widen access once a week of runs looks clean.

Safeguards and trust score

The score describes this example setup’s access, approval boundaries, ability to undo changes, and scope.

Access24 / 30

Writes drafts and notes inside your tools, nothing leaves on its own.

Human approval30 / 30

Waits for a yes on: fixing or quarantining any test, changing the run count or shard size.

Undo and logs14 / 20

Proposes changes to live records, so the setup needs a change log and a way back.

Scope18 / 20

Runs on its own login across 2 tools. Fewer connections means a smaller blast radius.

How to raise it

  • Undo and logs: Stop it from editing live records. Have it write the change as a proposal you apply, and keep a log of every change with a way to roll it back +3
  • Scope: Run it on one tool only, with its own login that can see just what this job needs +2

Do all of these to reach 91, up from 86.

Want help getting there? Get an AI Trust Audit from Pixeldesigns

Reviewed by Trustcase (example setup). How scoring works

Embed badge

Show this trust score on your site or README. It links back here.

Trustcase trust score badge
Markdown
[![Trustcase trust score for Nightly flake hunt that files one issue per test](https://trustcase.dev/badge/nightly-flake-hunt-that-files-one-issue-per-test.svg)](https://trustcase.dev/use-cases/nightly-flake-hunt-that-files-one-issue-per-test)
HTML
<a href="https://trustcase.dev/use-cases/nightly-flake-hunt-that-files-one-issue-per-test"><img src="https://trustcase.dev/badge/nightly-flake-hunt-that-files-one-issue-per-test.svg" alt="Trustcase trust score for Nightly flake hunt that files one issue per test" height="20"></a>

More engineering workflows

  • Security incident triage that recommends and records approvals

    A tier one security analyst pastes an alert or log and gets back a structured triage: incident type, severity with confidence, missing information and ordered first actions. Each action is marked safe or needing a specialist's approval, and approvals are only recorded, never carried out by the agent.

  • Ticket to pull request with approval of every write

    Takes one tracker ticket and decides whether it is ready, too big or too vague. Ready tickets get a fix and tests in a sandbox, a self review and a pull request. Too big tickets are split, vague ones get questions posted back. Every write to the tracker or the code host stops for a person to approve, edit or reject the exact arguments.

  • Query and migration cost review before every push

    Before a branch is pushed, a reviewer looks at every changed database query and migration and asks what it does at production size. It measures plans on a local copy and writes a findings table into the pull request.

  • Code review thread replies checked against the branch

    Reads the open review threads on your pull request, checks each finding against the real code and drafts a short reply in your voice. You post, skip or rewrite each one, and only approved replies go up.

Other workflows that use the same tools

  • Site change tester

    Marketing. Tries small edits to your site, watches whether more visitors stay, and keeps only the changes that help. Good for steady small wins on a landing page.

  • Daily kids lesson builder

    Personal. Builds a short interactive lesson for your kids each night and has a second agent check it. You read a review email and approve it before the kids get the link.

  • Weekly delivery status report

    Operations. Pulls your team's work items once a week and builds a status report grouped by initiative. It puts at risk items first and flags anything with no update in two weeks.