A verdict you can open, not a claim.

The problem

Looked fine is not evidence

An agent can click through an app and report that it looked fine. That report is a sentence. It names no build, holds no recording, and gives a second person nothing to open or to run again next week.

“Claude clicked through it and it looked fine”

QA-Pilot treats that sentence as the failure it is built to remove. An agentic browser session can help write a test, but it produces no trace and no machine-readable result, so it has nothing to publish with. What QA-Pilot leaves behind instead:

• A committed Playwright spec per case, which produces the verdict today and regresses the feature in CI later.
• A trace.zip per case per run: video, console, DOM snapshots and network log in one file.
• A deploy SHA read from the running environment before the run and again after it.

The pipeline

Six commands, with QA between design and execution

Each command is a Claude Code skill. The deterministic work, such as parsing the report and validating it, sits in zero-dependency Node scripts that a model does not get to reinterpret.

1. /qa-pilot:qa-init

   Describe the project

   Scans the repo, asks for what it cannot discover (environment URLs, where the deploy SHA is served, the tracker), and writes a schema-valid qa-pilot.config.yaml. Once per project.

2. /qa-pilot:setup-profiles

   Save a login once

   Gives you one command that opens a real browser for a single sign-in per account, then saves cookies, localStorage and IndexedDB so specs reuse the session. Once per developer.

3. /qa-pilot:generate-tests

   Author the cases

   Reads the tracker task and the code, then writes cases.yaml: at most 25 cases, a required scenario mix, and an assertion that could become a Playwright check in every case. Per feature.

4. QA approves the scenario matrix, the write policy and the cases
5. /qa-pilot:run-tests

   Run against a deployed build

   Turns approved cases into committed Playwright specs and runs them with a trace per case, reading the deploy SHA before and after. A feature that may not write freely runs under the write guard.

6. /qa-pilot:publish-results

   Validate, then publish

   Builds report.json from the Playwright report and runs the gate. Only a report that passes reaches the tracker.

7. /qa-pilot:qa-review

   Review as QA

   Sorts the queue with failures first, tracks the sampling quotas, and records approve, reject with a reason, or retest. Every confirmed defect becomes a linked bug.

What it refuses to publish

Six things it will not publish

The publish step is a script, not a judgement call. It rejects each of these outright. The lines below are the gate’s own output, as printed under REFUSED. This report cannot be published.

1. A verdict without evidence

   Every executed case needs a Playwright trace and a deploy SHA read from the running environment. A missing, empty or unreadable artifact fails the report.

   cases[CHECKOUT-ORDER-001].trace: required for a pass verdict. An unevidenced verdict is exactly the unverifiable claim this pipeline exists to prevent.

2. A verdict from a local or sandbox environment

   Mock backends are seeded and always succeed. Runs against them are useful for stabilising specs and are stamped as such, but they cannot carry a verdict.

   api_mode: "mocks" is this host's sandbox mode. Sandbox backends are seeded and always succeed, so their results are structurally false passes and cannot be published.

3. A pass that was really a retry

   Pass-on-retry is recorded flaky, always, derived from the attempt results and not from Playwright’s own status field, so no config change can turn it green. A failure in any browser is a failure, never a flake. The gate recomputes every verdict from results.json.

   rule 1b: cases[CHECKOUT-ORDER-002].verdict says pass but results.json gives flaky

4. A verdict for a case QA never approved

   Checked against the case’s recorded status at the time of the run, not against the case map. Every generated case is in the map from the moment it is created, so membership proves nothing about approval.

   cases[CHECKOUT-ORDER-004]: was "Case Review" (case_review) when the run happened, which is not an approved state. Only cases QA approved may be published.

5. A run that lost its ground truth

   If the deployed build changes mid-run, every case is blocked, because half tested one build and half another. Past 10% blocked the run halts: the environment failed, not the feature.

   sha_mismatch: the deployed build changed mid-run (a1f9c3e -> 7d02be4) but 12 case(s) still carry a verdict; every case must be blocked

6. A run that broke its write policy

   Each feature declares whether its specs may write: read-only, scoped-write with a name prefix, or unrestricted. The write guard records every write control and write request it saw. A read-only run that recorded one, or a run whose guard never reported in, does not publish.

   rule 3: read-only run recorded 2 write(s) in cases[CHECKOUT-ORDER-003]

Evidence

The evidence is one file

testing/checkout/runs/20261001T092200Z-checkout-a1f9c3e/
  meta.json
  spec-shas.json
  results.json
  statuses.json
  report.json
  fixtures/
  test-results/
    checkout-order-001/
      trace.zip
      writes.json
    checkout-order-002/
      trace.zip
      writes.json

Production

Production, without the damage

An environment declares what it is: kind: qa | staging | production, never guessed from the hostname. Production is allowed, under rules that are refusals, not advice. The first is a restricted account: a production environment must name its test_account, the account the runs sign in as and what restricts it (its own tenant, no admin rights, no billing). It is an attestation nothing verifies, so QA reviews it.

{
  "mode": "local",
  "cases": [
    {
      "case_id": "CHECKOUT-ORDER-001",
      "verdict": "pass",
      "target_status": "Under Review",
      "env_name": "production",
      "build_id": "a1f9c3e",
      "trace_path": "test-results/checkout-order-001/trace.zip",
      "trace_sha256": "26f9dc66c7273...a1b2"
    }
  ],
  "summary": {
    "counts": { "pass": 1, "fail": 0, "flaky": 0, "blocked": 0 },
    "blocked_pct": 0,
    "confidence": { "score": 0, "label": "Not Ready" },
    "ready": false
  }
}

Portability

One profile per project, owned by QA

project: shop
tracker: clickup

environments:
  staging:
    kind: staging
    apps:
      storefront: https://staging.example.com
    sha_source:
      url: https://staging.example.com/version.json
      json_path: commit
  production:
    kind: production
    test_account: qa-runner@example.com, its own tenant, no admin rights, no billing
    apps:
      storefront: https://app.example.com
    sha_source:
      url: https://app.example.com/version.json
      json_path: commit

evidence:
  capture: always

selectors:
  testid_attribute: data-testid

Honest limits

How strong these gates actually are

The checks are deterministic scripts, so a rushed developer cannot skip them by accident, and that is the failure this system is built for. What they verify is that a report is internally consistent and that its evidence exists on disk.

They do not cryptographically bind a report to a Playwright run that happened. Someone determined to fabricate a green result could hand-write the inputs, and nothing here stops that. QA-Pilot makes the evidenced path the path of least resistance, not the only conceivable one. The environment name in a report is taken on trust, like the deploy SHA, and the write guard’s ceilings are listed in its source.

Sampling does not close that gap. QA opens a share of passing evidence (all of P0, at least 30% of P1 and 10% of P2) and every failure, which checks that the evidence shows what the case claims. A pass that rests on a weak assertion looks green in its trace. The spec lint catches a missing assertion, a fixed sleep or a skipped test, and the case lint rejects vague expected outcomes; how strong an assertion is stays with QA’s design review, not sampling.

Install

Two commands

Requires Node 20 or newer (the scripts are zero-dependency ESM) and, in the host repo, Playwright 1.51 or newer. That is the release where storageState({ indexedDB: true }) landed, and below it IndexedDB-persisted auth such as Firebase silently fails to restore.

Setting up a project QA-Pilot on GitHub