QA-Pilot0.4

Project

Changelog

0.4.3 (2026-10-24) #

Pilot readiness. A full-cycle review of 0.4.2 found ways for CI to go green on cases that never ran, for the published score to count approvals the same publish took away, for a merge conflict to erase every approval, and for writes to slip past the guard. This release closes them, lets QA work from their own machine, and adds a report of the numbers a pilot is judged on.

Upgrading from 0.4.2 #

  1. Every approval resets once. The approval hash now covers the spec, the local files it imports and the case's steps, expected outcomes, preconditions and fixture, with line endings normalised. Every entry in approved.json stops matching, so CI skips every case until QA re-approves it with /qa-pilot:qa-review after the next published run, and fails with REFUSED: no approved spec is runnable until at least one is re-approved. Disable the workflow until then, and set MAX_SKIPPED empty while the rest are pending.
  2. Replace the CI workflow with the new template and pin v0.4.3. select now requires --cases, and the template passes the selection to verdict --expect. An old workflow against the new plugin fails at selection.
  3. CI is stricter. Any blocked case, failed fixture spec, Playwright error outside a test, or selected case missing from the report fails the job (ENVIRONMENT FAILED or INCOMPLETE). Before 0.4.3 up to a tenth of the cases could be blocked on a green job.
  4. Spec paths must not contain (, ), [ or ]. Playwright reads them as patterns and runs nothing for them; CI now fails instead of passing.
  5. Lint fixture specs and selectors. Copy the new templates/eslint.qa-pilot.config.mjs (or add its FIXTURE-*.ts glob and the no-raw-locators, no-nth-methods and prefer-native-locators rules to yours). Put a component library's overlay root in no-raw-locators' allowed list. Fix what it reports without weakening an assertion; every fix changes a hash, which item 1 already requires re-approving.
  6. Commit the new per-feature directories: testing/<feature>/verdicts/ (one record per published run) and testing/<feature>/reviews/ (one per review session). Add .qa-pilot/ and the storage-state directory to .gitignore if they are not; /qa-pilot:run-tests now checks both, and test-results/, on every run.
  7. Decide where QA opens traces. Approvals now travel through git, but under the default reference the traces stay on the executor's machine. A team whose QA reviews elsewhere sets evidence_upload: tracker on the QA environment, with Private Attachment Links on.

Fixed #

  • CI was green on unproved results. It passed with up to 10% of cases blocked, with a failed fixture spec, after a global timeout, and when a selected spec matched nothing because Playwright reads file arguments as patterns.
  • The published score counted approvals the publish was removing. Confidence was scored on the statuses before the publish; case-status --apply-transitions scores them after, and publish-results now computes transitions first.
  • A merge conflict in approved.json erased the ledger, and one in bugs.json refiled every open bug. Both are refused now, naming the conflict markers.
  • An approval survived edits it should not have: to a helper the spec imports, to the case's expected outcome, and to the spec between the run and the publish. Hashes are taken when the run starts (spec-shas.json) and cover all three.
  • CI kept running a case deleted from cases.yaml; it is now skipped and named for removal from the ledger.
  • A run in more than one browser could not publish. Playwright 1.62 writes one entry per project, which read as duplicate cases; entries are grouped by case id first.
  • The write guard missed a keepalive fetch (an autosave on unload reached the server under read-only) and a SharedWorker's requests, over-blocked inside dialogs (a dialog's tabindex made its Save button the label of every click inside it), and did not judge a spec handler's route.continue overrides. All four are closed, each with a real-browser case that fails on 0.4.2.
  • The gate could publish a stabilization run, and a run id with : could not name a directory on Windows. Runs now record run_kind (verdict, stabilization or ci); only verdict publishes, and run ids are letters, digits, ., _ and -.
  • Tracker-mode trace paths were not confined to the run directory.

Added #

  • QA works from their own machine. Each publish commits testing/<feature>/verdicts/<run_id>.json; --record-approvals and bug-report accept it, so approvals, and bugs where the tracker receives failure text, no longer need the executor's report.json. Where evidence stays local the record carries no failure text, URL or executor, and bugs are still filed on the executor's machine.
  • pilot-report.mjs testing/<feature> prints flaky and blocked rates, review minutes per decision, bugs filed, and the approved specs that have dropped out of CI, from committed files.
  • A session check before every run, so an expired login stops the run instead of failing every case.

Changed #

  • /qa-pilot:setup-profiles never opens the saved login file; it judges the capture from the script's printed counts.
  • Bug reports and run summaries say to open traces with npx playwright show-trace.
  • The CI template's Node default is 22, with a note to follow the host's version file.

0.4.2 (2026-10-23) #

Documentation only; no behaviour changes and nothing to do when upgrading.

  • Shorter skills and schemas. What an agent reads to run QA-Pilot is about 30% smaller. Validator detail the scripts already print, history and rationale are gone; every command, STOP condition and rule a script does not enforce stays.
  • ClickUp steps load only when used. Each skill keeps its ClickUp-only steps in a references/ file it reads when the profile's tracker is clickup.
  • One home per rule. The evidence settings, the refresh-token risk, Private Attachment Links and test_account are explained once and linked from elsewhere.
  • Claims corrected. Sampling passing traces is no longer described as the backstop against a false pass (weak assertions are caught by the spec lint and QA's review of the cases); onboarding a project is described as writing its profile, write signatures included; an unmeasured reviewer throughput figure and "CI regression comes free" are gone.

0.4.1 (2026-10-09) #

A patch release that fixes five flaws found in 0.4.0 before any host relies on it. QA approvals never carried forward, so CI selected nothing; traces, which carry the session and can carry a long-lived refresh token, went to the tracker by default off production; CI stayed green while approved specs were skipped; nothing checked that a generated spec asserts anything; and a few settings were dead or too strict. Nothing here changes a verdict, the publish gate's other rules, the write guard or the confidence score.

Upgrading from 0.4.0 #

Several of these change behaviour without failing loudly, so read all of them before moving the CI pin.

  1. The default off production is now evidence_upload: reference. The tracker still gets every per-case field and the failure text, and no longer gets the trace file; each case carries run_id, trace_path and trace_sha256 instead, and the trace stays on the machine that ran it. In ClickUp, create the three text fields Run ID, Trace Path and Trace SHA256 if you have not (SETUP-CLICKUP.md). Add testing/*/runs/ to .gitignore in the host repo.

  2. To keep attachments, opt in per environment. Set evidence_upload: tracker on that environment, after turning on Private Attachment Links in ClickUp (Settings, Advanced Permissions) and using a restricted account. Production still refuses both tracker and reference: its evidence is local.

  3. /qa-pilot:run-tests now stops on a non-production run until testing/*/runs/ is gitignored. In 0.4.0 only production checked. The check now covers every run whose evidence stays on the machine, which is reference and local.

  4. Rule 6 now applies off production. validate-report.mjs pins evidence for reference as it did for local, so a report whose traces sit outside the run directory or lack trace_sha256 is refused, where 0.4.0 accepted it under tracker. Parse reports from the run directory with parse-report.mjs as usual and nothing changes.

  5. bug-report.mjs without --profile now files qa and staging bugs as reference: the failure text and the trace's path and sha256, no attachment. It was tracker. Production without a profile is still refused.

  6. Cases approved under 0.4.0 have no approval ledger entry, so none of them run in CI. For each case that is approved now, record its hash once, using the report.json of the last run QA reviewed. That file sits in the gitignored run directory on the machine that ran it, and it must have been parsed with --specs so it carries spec_sha:

    node case-status.mjs --record-approvals --verdicts testing/<feature>/runs/<run_id>/report.json --approved testing/<feature>/approved.json --ids <A,B,...>

    Write the approved_ledger it prints to testing/<feature>/approved.json and commit that file. --approved is required every time, so an earlier ledger is never overwritten by mistake; a path that does not exist yet starts a new ledger with a warning. It records the hash of that run's spec, so a case whose last reviewed run was not a pass records nothing and an approved failure is removed from the ledger. Or skip the command and re-approve each case in /qa-pilot:qa-review, which now records the hash itself. Cases whose report is no longer on any machine stay approved in the tracker but do not run in CI until recorded.

  7. An edited spec now fails CI until QA re-approves it. Its hash no longer matches the ledger, so select skips it and the strict template fails on the skip. To retire a case, delete its key from approved.json; a case dropped from the ledger is not a skip.

  8. Lint and fix every approved spec before switching to the strict CI template. Existing specs may fail the new rules (expect-expect, no-wait-for-timeout, no-skipped-test, no-focused-test), and fixing one changes its hash, so lint them all once, fix what it reports without weakening an assertion, and re-approve, all before you copy the template in. Keep each spec's assertions in the test body: a helper that holds the only expect() hides it from expect-expect.

  9. Re-copy templates/qa-pilot-ci.yml and add the lint setup. The template now pins ref: v0.4.1, sets MAX_SKIPPED: '0', lints the selected specs and uploads the run directory as an artifact. Re-copy it rather than patching your copy. Then add the four dev dependencies eslint@9, eslint-plugin-playwright, typescript-eslint and typescript, and commit templates/eslint.qa-pilot.config.mjs at the root of the host repo as eslint.qa-pilot.config.mjs. /qa-pilot:run-tests copies it there if it is missing.

  10. Edit the ESLint config's files list to cover every app's spec_dir. It holds one <spec_dir>/**/*.spec.ts glob per app and ships with e2e/**/*.spec.ts. CI passes file paths, and ESLint ignores a file no glob covers and still exits 0, so the template fails the lint step when it prints File ignored. /qa-pilot:run-tests passes directories instead, where a glob that matches nothing exits 2: list only directories that exist.

  11. CI artifacts upload only on a private repository. The template reads the repository's visibility and skips the upload when the repository is public or the visibility cannot be read, since anyone could download the artifact and a trace can carry a refresh token. A public repository that relied on 0.4.0's tracker-mode upload gets no artifact now; the step summary says so.

  12. clickup.plan_tier is optional and the model list only warns. A profile that still sets plan_tier stays valid, and a value outside free | unlimited | business | enterprise is still an error. A case stamped with a model_version outside models.generation_approved now produces a warning instead of an error; a missing model_version is still an error.

Everything else is backward compatible: a host that sets nothing new gets the safer defaults above, and --max-skipped is optional on ci-gate.mjs select (absent, it behaves as in 0.4.0). A CI job still pinned below 0.4.1 silently ignores --max-skipped, since the older ci-gate.mjs only reads the flags it knows, so move the pin together with the template.

Approvals now carry forward #

0.4.0 never wrote a case's first ledger entry, so no approval survived to CI and select chose nothing. /qa-pilot:qa-review now records the spec hash when QA approves a result: after Approve decisions on a result review (never on a design review) it runs case-status.mjs --record-approvals once per run, against that run's report.json, and commits the updated testing/<feature>/approved.json. A pass records the hash; approving a confirmed failure records the verdict and removes the entry rather than baselining a regression. A case missing from the report, or a pass with no spec_sha, is refused with a message naming it, and a ledger file that is not valid JSON or not a plain object is refused instead of replaced. --transitions at publish is unchanged.

Evidence stays off the tracker by default #

evidence_upload is now tracker | reference | local. reference, the default off production, sends the tracker the same fields and run summary as tracker but never the trace, plus each case's run_id, trace_path and trace_sha256; the summary's reviewer note says to verify the hash and open the trace with npx playwright show-trace <path>. QA reviews reference and local traces on the machine that ran them, verified by sha256, and prefers show-trace to the hosted viewer for any trace that carries credentials. tracker remains as an opt-in. Under tracker: none, any explicit evidence_upload other than local draws the warning environments.<env>.evidence_upload: <v> is ignored under tracker: none; evidence stays local. The setup guides now state the real risk: long-lived refresh tokens stay valid until the account is disabled, deleted or changed, so the old "short-lived" advice is gone.

CI fails on skipped specs and lints the specs it runs #

  • ci-gate.mjs select --max-skipped <n> refuses, exit 1 and no run list on stdout, when more than n approved specs were skipped, because the job would otherwise report green on part of the suite. A value that is not a whole number, 0 or more, throws. Cases the priority filter drops are not skips. The template sets MAX_SKIPPED: '0'; raise it deliberately to tolerate drift. The step summary is still written when select fails.
  • A spec lint. templates/eslint.qa-pilot.config.mjs adopts eslint-plugin-playwright with expect-expect, no-wait-for-timeout, no-skipped-test and no-focused-test as errors. /qa-pilot:run-tests lints the feature's specs before a verdict run and fixes any error without weakening an assertion; the CI template lints the selected specs after select. SETUP-CI.md also recommends a pull request check that flags weakened or deleted tests.

Relaxed configuration #

  • clickup.plan_tier is optional. Nothing ever read it; /qa-pilot:qa-init no longer asks for it.
  • The model gate warns. A new model release no longer blocks generation: a model_version not in models.generation_approved yields model_version: "<m>" is not in models.generation_approved (<list>); review these cases with that in mind, and add the model to the list once QA trusts it. The profile still requires a non-empty list.
  • Dead rules removed. run-tests no longer writes run-log.json (nothing read it; the hand-off now says which new specs had three consecutive green runs), and the undefined "false-pass threshold" is gone from qa-review and setup-profiles.

0.4.0 (2026-10-07) #

Upgrading from 0.3.1 #

One change will stop an existing setup working until you act on it; a second can leave a CI job quietly doing the wrong thing.

  1. A production environment must declare test_account. The field is a string of at least 20 characters naming the account the runs use and how it is restricted. It is a production environment without it that invalidates the profile, and an invalid profile fails every command and every CI job once its pin is v0.4.0, CI against qa or staging included: the template validates the profile first, and a profile with a production environment is invalid as a whole whichever environment a job targets. A job still pinned below 0.4.0 keeps working, which is why item 2 says to add the account before moving the pin. The error reads:

    environments.<env>.test_account: required on a production environment, at least 20 characters. Name the account the runs use and how it is restricted (its own tenant, no admin rights, no billing). The write guard is the second layer, not the boundary.

    An example value: test_account: "qa-runner@example.com, own tenant, no admin rights, no billing". On qa and staging the field is optional, and when present it must also be at least 20 characters.

  2. Move a CI pin to v0.4.0 once the account is in place. A job pinned below 0.4.0 silently ignores --priority, since the older ci-gate.mjs only reads the flags it knows, and runs every approved spec: a job that sets PRIORITY: P0 looks like a P0 job and is not. The template now pins ref: v0.4.0; re-copy it, and add test_account first or the pinned job fails on the profile.

Everything else in this release is backward compatible: requirements and covers are optional, and a cases file without them lints, reports and scores exactly as under 0.3.1; PRIORITY is empty by default, which runs every approved spec as before.

Restricted accounts on production #

The write guard is the second layer; the account is the boundary. A restricted account (its own tenant, no admin rights, no billing) cannot change data however a click is judged, whereas a signature list is only as complete as the host's inventory. test_account is an attestation: nothing checks it against the login a profile saved, so QA owns keeping the account restricted.

Requirement coverage #

  • requirements and covers in cases.yaml. A feature may declare its requirements, each with acceptance criteria, independently of its cases, and a case claims the criteria it tests with covers: [<requirement id>/<criterion id>]. Both keys are optional.
  • A lint. validate-cases.mjs reports a malformed block, a dangling covers entry and covers with no requirements as errors, and a criterion that no case covers as a warning, since that is a design gap QA decides on, not a mistake.
  • Four states per criterion, decided in this order: uncovered (no case covers it), failing (a covering case failed or was flaky), proved (a covering case passed and QA approved the verdict), and unproved (everything else, a blocked case included). A failure is never hidden by another case's pass.
  • requirement_coverage in the run summary, in every publishing mode (tracker, local and none): the criterion counts per state plus not_proved, the ids of the requirements with a criterion not yet proved. Ids and counts only, never criterion text or requirement titles, so it is safe on production. It is absent when the cases file declares no requirements or its block is malformed.
  • The confidence score is unchanged. Coverage is reported beside it and does not feed it.

Priority runs #

  • ci-gate.mjs select --cases <cases.yaml> --priority <P0[,P1[,P2]]> keeps only the approved specs whose case has one of those priorities. Priority lives in cases.yaml, not in the approval ledger, so --priority without --cases is refused, as are an unknown priority, a missing cases file and an empty list element.
  • It refuses to go green on nothing. If no approved spec at that priority is runnable, select exits 1 rather than letting the job report success having proved nothing.
  • The CI template gains a PRIORITY setting. Empty runs everything approved; P0 runs the P0 cases. The recommended split is P0 on every deploy and everything on the nightly schedule; promoting a case to P0 moves it into the deploy job with no workflow change. See SETUP-CI.md.

0.3.1 (2026-10-06) #

Fixes #

  • The ClickUp guard now blocks only writes to QA-Pilot's own case tasks. In 0.3.0 it blocked every ClickUp write in a repo that has a host profile, so ordinary work there (bug tickets, comments, attachments, status changes on unrelated tasks) needed the skills' write flag. It now blocks a write only when the tool input names a task ID recorded in one of the repo's testing/*/clickup-map.json files, wherever in the input the ID appears (task_id, task_ids, links, a task URL, #id). Creating tasks, editing lists and touching any other task always pass. The skills still open the guard with the write flag before changing their own tasks, so nothing about publishing or review changes. A missing or malformed case map owns nothing and is skipped.

0.3.0 (2026-10-01) #

Upgrading from 0.2.0 #

These changes will stop an existing setup working until you act on them. Each one closes a gap, mostly around running against production safely.

  1. Every environment must declare kind: qa | staging | production. The profile no longer guesses production from the hostname: the guess missed app.<domain>.com, the commonest production shape. A production-looking URL on a non-production kind is now a warning, never an error.
  2. allow_production is retired. A profile that still sets it, to anything, fails with "retired in 0.3.0; declare kind: production".
  3. A production environment brings rules with it. evidence.capture must be always; evidence_upload: tracker on it is refused, since its traces stay local; a mutation block with at least one write_signatures entry is required; and /run-tests refuses a production run unless testing/*/runs/ is gitignored and the feature declares a mutation.policy other than unrestricted.
  4. validate-report.mjs now requires --cases <cases.yaml>, on every environment. It is the only source of the feature's mutation policy and fixtures, and the gate takes neither from the report. A report with no env_kind, which every 0.2.0 run has, is refused: re-run with 0.3.0 rather than re-publishing an old run.
  5. The CI template exports QA_PILOT_MUTATION from testing/$FEATURE/cases.yaml, plus QA_PILOT_MUTATION_CONFIG from the profile, reads evidence.capture from the profile instead of hard-coding on-failure, gives each run its own directory, and refuses an environment whose kind is production. Re-copy it rather than patching your copy.
  6. Node 20 or newer. Node 18 is end-of-life.
  7. Move the reporters out of your Playwright config. /run-tests now passes --reporter=json,html on the command line with per-run output paths (PLAYWRIGHT_JSON_OUTPUT_NAME, PLAYWRIGHT_HTML_OUTPUT_DIR, --output, all absolute), and a JSON outputFile in the config would override them and send every run to the same file. If you declare shared fixtures, scope the fixture projects' testMatch to `**/${process.env.QA_PILOT_FEATURE}/FIXTURE-*.setup.ts` (and .teardown.ts), as the run-tests skill shows: dependency and teardown projects ignore the command line's spec list, so unscoped they would run every feature's fixtures on every run.
  8. parse-report.mjs writes report.json into the run directory by default, and the trace paths in it are relative to that directory. It no longer prints the report to stdout when -o is absent.

Nothing else requires action. Hosts that keep evidence local on any environment add three text fields in ClickUp (Run ID, Trace Path, Trace SHA256; see SETUP-CLICKUP.md).

Production, safely #

  • The write guard. A feature declares mutation.policy: read-only, scoped-write with a name prefix, or unrestricted (the default, so 0.2.0 case files stay valid off production). templates/write-guard.fixture.ts enforces it with no help from the specs: capture-phase listeners injected before any app script judge clicks, submits and Enter/Space in the page; the network route judges every request against the host's write_signatures, which can match the request body, so a GraphQL mutation is told apart from a query on the same URL. Specs' own route handlers are wrapped so the guard always decides first, and unroute / unrouteAll remove only the spec's own handlers. Under any policy but unrestricted every request path a spec would reach for is refused: the request fixture, page.request and context.request, request.newContext() from @playwright/test, Node's global fetch(), browser.newContext() and browser.newPage() (also from beforeAll and beforeEach), and every browser type's launch and connect. newGuardedContext() replaces browser.newContext() for cross-app specs. The page cannot widen the scoped-write scope or hide a blocked click. What spec authors will notice: every guarded test gets a browser context even if it never asks for a page, and a page made in beforeAll can no longer be shared across serial tests; use the per-test page. Node's node:http, node:https and node:net are not seen by the guard and must not be used from a guarded spec.
  • The gate re-reads everything. Each attempt leaves a writes.json; the publish gate rebuilds every case and fixture from the Playwright report and those records on disk, sweeps the run directory for records the report does not account for, and refuses any field that differs. A read-only run that recorded a write, a scoped-write run that was blocked, or a guarded run whose guard never reported in does not publish.
  • Local evidence. environments.<env>.evidence_upload: tracker | local, local by default on production. Traces stay on the executor's machine and are pinned by sha256; the tracker receives only ids, verdicts, build, run and the trace's path and hash, computed by the new scripts/publish-payload.mjs, which the skill posts verbatim. Bugs from such a run carry no failure text.

Also new #

  • tracker: none. The whole pipeline on local files: approvals in testing/<feature>/statuses.json, evidence local, bugs as markdown in the run directory.
  • Shared fixtures. cases.yaml fixtures and cases[].fixture: one entity built by a FIXTURE <name> setup spec and worked inside by many cases, with an optional teardown.
  • Concurrent runs. Every output lands under testing/<feature>/runs/<run_id>/, so two runs of the same feature no longer overwrite each other.
  • Documentation sources. context.sources names docs or commands /generate-tests reads first; commands are confirmed with the user before their first run in a session. /generate-tests can author several features at once, one subagent each.
  • Stabilization off the sandbox. stabilization.env names a non-production deployed environment where new specs may earn their three greens. Those runs are never published.
  • The ClickUp guard only acts in repos that use QA-Pilot. Installed user-scope, it used to block ClickUp writes everywhere; it now acts only where a host profile exists.
  • qa-init never opens .env* files. It finds variable names in the code that reads them and asks for the value.

Fixes #

  • The bundled yaml is ISC licensed, not MIT; THIRD_PARTY_NOTICES.md carries its text.
  • spec-conventions.md no longer claims the gate requires a separate console log (the trace satisfies it), and run-tests no longer contradicts its own capture table.

0.2.0 (2026-09-23) #

Upgrading from 0.1.0 #

Three changes will stop an existing setup working until you act on them. All three are deliberate: each one closes a gate that could previously be walked past.

  1. validate-report.mjs now requires --statuses. Pass the statuses file recorded at the start of the run. Without it the gate cannot tell an approved case from an unapproved one, and a gate that can be skipped by omitting a flag is not a gate.
  2. A profile registering a production-looking URL no longer validates. Point the environment at QA or staging, or set allow_production: true on it to accept that runs will mutate real records and that traces carrying live session tokens will be uploaded to your tracker.
  3. The confidence score reads Unknown until you pass --verdicts. Approval alone never meant the feature worked, and the old number said it did.

Nothing else requires action. approved.json, specs.json and bugs.json are new files the skills write as you go.

The defect path #

A confirmed failure now becomes a bug. Previously the pipeline produced a fully evidenced failure and dropped it: qa-review said "record it as such against the feature", which is prose with no mechanism behind it.

  • scripts/bug-report.mjs assembles each bug from report.json and cases.yaml. The failure text is Playwright's, verbatim. A model paraphrasing an error into a ticket reintroduces the unverifiable claim this pipeline exists to remove, and a developer who cannot trust the error in the ticket opens the trace instead, which buys nothing.
  • Deduplicated through a testing/<feature>/bugs.json ledger, keyed by case ID plus a normalized failure signature. Digits are stripped before hashing, because Playwright errors carry timeouts and element counts that vary run to run while naming one defect. Without this, a weekly regression run refiles every open bug every week.
  • Bugs link to their case through a native ClickUp task relationship rather than a twelfth custom field: a bug's lifecycle belongs to the developers, not to QA-Pilot.
  • clickup.bug_list is optional. Unset, bugs land in the feature list and the script says so, so a first pilot is not blocked on ClickUp admin.
  • flaky verdicts can be filed too, tagged intermittent. An intermittent product bug is real, and a ticket that hides the intermittency costs the developer their first hour.
  • Removed the feature-actually-broken rejection tag, which contradicted the rule beneath it. Rejection means the test was wrong; a real break is an approved verdict plus a bug.

CI runs the approved suite #

The committed specs were only ever run by a human typing /run-tests, so the claim that regression coverage accumulates as a side effect of normal work had nothing behind it.

  • scripts/ci-gate.mjs select decides what CI runs from approved.json alone. CI cannot reach ClickUp (the lifecycle is driven over MCP by a model, and a build runner is neither) and now does not need to: the ledger is committed, names every case QA accepted, and carries the hash of the spec they accepted.
  • A spec that no longer matches its approved hash is excluded rather than run. A green CI run against an unreviewed spec claims a human stands behind something no human read.
  • select exits nonzero when nothing is runnable. A suite that passes having executed zero tests is the most expensive kind of false confidence.
  • ci-gate.mjs verdict separates a regression from a broken environment, since paging the feature team because the QA box is down burns the credibility of the whole signal. flaky fails the build, like everywhere else here.
  • templates/qa-pilot-ci.yml plus SETUP-CI.md, which is explicit that CI does not publish, that this is a post-deploy and nightly check rather than a PR check, and that a non-interactive login is host-specific work the plugin cannot do for you.
  • specs.json moved from per-run to testing/<feature>/specs.json: the mapping does not vary per run, and CI reads it from the repo.
  • read-env-sha.mjs --sha-only, so a CI step stays one line.

Gates measure what they claim to measure #

  • Confidence requires an approved pass, not just approval, and reads Unknown rather than a number when no verdicts are in hand.
  • An unchanged spec that passes again keeps its approval, judged by a per-verdict spec hash and an approved.json ledger.
  • The publish gate checks recorded status instead of case-map membership, and refuses outright without --statuses.
  • The assertion lint no longer loses to padding, and no longer flags "request form".
  • Production environments are refused unless allow_production is set.

Committing QA artifacts is a choice #

Nothing is required to be committed, and no skill refuses over it. Keeping specs and testing/ out of a repo is a legitimate decision, so /qa-pilot:qa-init now prices each file rather than forbidding it, and names the two coherent modes:

  • Local-only. Approved cases, evidenced verdicts and the ClickUp record still work. CI, cross-machine reproduction and approval carry-forward do not.
  • Committed. Ignore only testing/*/runs/, and everything works.

CI is the one part with a hard requirement, because it reads the specs and ledgers out of the checkout and has no other source.

0.1.0 (2026-08-30) #

First build. Implements PRD v1.0.

  • qa-init, generate-tests, run-tests, publish-results, qa-review, setup-profiles skills
  • Deterministic validators: host profile, cases, deploy SHA, report parse, publish gate
  • ClickUp write guard (PreToolUse hook), so the scripted path is the only write path
  • No third-party code except scripts/lib/yaml.mjs, a bundled copy of yaml@2 (MIT)

Evidence is hosted in ClickUp as a single trace.zip per case per run, rather than a video attachment plus a linked trace plus a console log. Verified by inspecting a real Playwright trace: it contains the video byte-for-byte, the console output and the screenshot film-strip, so the three-file scheme stored the same bytes twice and split one investigation across three places. Reviewers open traces at trace.playwright.dev, which runs client-side and uploads nothing.

Deviations from the PRD, each forced by a verified constraint:

  • Scripts are zero-dependency Node ESM (.mjs), not TypeScript, because a plugin must not require a build step or an npm install in the host repo.
  • The qa-skills fork was dropped. The PRD called for forking and vendoring neonwatty/qa-skills, on the assumption we would adopt its converters, generators, playwright-runner and CI scaffolding. All of those were skipped for good reasons, leaving only four reference documents: nothing referenced them, they were largely mobile and iOS guidance for a desktop-web pipeline, and they carried no network-assertion ban plus a hardcoded sync timeout that contradicts our profile-driven ceiling. Our own spec-conventions.md and case-style.md cover the relevant ground, profile-aware.
  • /setup-profiles keeps the upstream approach (headed browser, human logs in once, session reused) but is written from scratch. Upstream captures via playwright-cli state-save, which omits IndexedDB and so loses Firebase auth. Replaced with save-storage-state.mjs using storageState({ indexedDB: true }). The capture is run by the user, in their own terminal, and the skill hands them a fully-resolved command rather than running it. An agent's Bash tool has no interactive terminal, so the sign-in prompt could never resolve; the script now detects that before launching a browser it would orphan, and exits with the command to run.
  • publish-clickup.ts is not a script. ClickUp writes are driven by skill instructions over MCP, with the deterministic work (parse, validate) staying in scripts.
  • The write guard scopes by flag file rather than by ClickUp space; checking the target space would require an authenticated API call from inside a hook.