CI Site QA on Agent PRs

CI Site QA on Agent PRs

After a preview deploy succeeds, point Boidwatch at the preview URL with the PR’s design intent—via the CLI in GitHub Actions, the HTTPS API, or the Boidwatch GitHub Action surface. Census-derived personas drive real Chromium sessions and return ranked gaps you can post to the PR; a capped peek keeps the gate cheap (3 personas, 15 steps, 5-minute stop) while --quick or a full flock produces a richer report. Treat critical entry-block or top-severity gaps as fail-the-job, and softer findings as PR comments with the report link. This complements Playwright or Cypress CI: those assert code paths; the flock asserts visitor experience on the built preview.

Why agent PRs need a preview UX gate

Agent-authored pull requests change UI faster than reviewers can click. A Cursor or Claude Code PR can ship a new pricing table, move the primary CTA, or rewrite a form in one push—then the preview deploys green while a cold visitor still cannot complete the job.

Unit tests and E2E suites answer “did this code path work?” They do not answer “would a stranger finish the design intent on this preview?” Reviewers skim diffs; they cannot exercise every marketing path on every agent PR. The missing gate is a persona UX check against the public preview URL after deploy succeeds—cheap enough to run often, ranked enough that a tech director can fail the job or leave a comment without reading a JSON dump.

That is the job Boidwatch owns on agent PRs: flock (or peek) on the preview, design-intent gaps, client-ready report link. Morph, TestSprite, and QA.tech own E2E / autonomous functional PR tests on preview URLs. Keep those. Add a persona layer when visitor experience—not only DOM assertions—is the risk. Adjacent framing: Agent-native shipping proof (peek before client link) and Test the site before client handoff (checklist + persona layers).

Event shape that works

Do not fire the flock on pull_request open alone. The preview may not exist yet. The pattern that works across preview-test CI (and what Morph / TestSprite / QA.tech document in their own Actions docs) is:

1. Wait for a successful preview deployment.

2. Read the public preview URL from the deployment (commonly deployment_status → environment_url, or your host’s equivalent status + URL field).

3. Only then call Boidwatch with that URL and a one-line design intent from the PR.

Prove the URL is publicly reachable before you spend credits. Password walls, IP allowlists, and bot challenges will block headless Chromium; Boidwatch exits 2 with REACHABILITY_ERROR, creates no run, and charges no credits (see llms.txt). Fix reachability, then re-run.

In Actions terms: either trigger on deployment_status when state == success and pass environment_url, or have your deploy job write PREVIEW_URL to an output / artifact and start a dependent job that receives it. The sketch below assumes the second shape so the YAML stays honest as an example job, not a proprietary Action marketplace claim.

Minimal Action using the CLI

Product surfaces include CLI, HTTPS API, MCP, and a GitHub Action. For a first workflow you can implement today without undocumented Action YAML, install the CLI in the job, export the API key secret, and run create. That is the primary pattern this page recommends.

Secrets: store BOIDWATCH_API_KEY in the repo or org secrets (create a key). New accounts get 30 free credits; 1 credit = 1 persona.

Install (CLI v1.1.0+; check with boidwatch version --json):

curl -fsSL https://boidwatch.com/install.sh | sh
# or: brew install boidwatch/tap/boidwatch

Cost preview, then a quick flock:

export BOIDWATCH_API_KEY="${{ secrets.BOIDWATCH_API_KEY }}"
export PREVIEW_URL="${{ needs.deploy.outputs.preview_url }}"
export INTENT="Understand pricing and start a trial"

# Preview credits only — no flock
boidwatch --json run create \
  --url "$PREVIEW_URL" \
  --design-intent "$INTENT" \
  --quick --dry-run

# 5 personas, wait for compact JSON summary
boidwatch --json run create \
  --url "$PREVIEW_URL" \
  --design-intent "$INTENT" \
  --quick --wait --wait-timeout 10m

Drop --quick for a full flock (default 20 personas). Agents and CI scripts can introspect flags and exit codes with boidwatch commands (pipe to jq .exit_codes for the full table).

Example workflow job (receives PREVIEW_URL)

# Example only — wire PREVIEW_URL from your deploy job or deployment_status.
name: preview-ux-gate
on:
  workflow_call:
    inputs:
      preview_url:
        required: true
        type: string
      design_intent:
        required: true
        type: string

jobs:
  boidwatch-quick:
    runs-on: ubuntu-latest
    steps:
      - name: Install Boidwatch CLI
        run: curl -fsSL https://boidwatch.com/install.sh | sh

      - name: Dry-run cost preview
        env:
          BOIDWATCH_API_KEY: ${{ secrets.BOIDWATCH_API_KEY }}
        run: |
          boidwatch --json run create \
            --url "${{ inputs.preview_url }}" \
            --design-intent "${{ inputs.design_intent }}" \
            --quick --dry-run

      - name: Quick flock on preview
        id: flock
        env:
          BOIDWATCH_API_KEY: ${{ secrets.BOIDWATCH_API_KEY }}
        run: |
          set +e
          OUT=$(boidwatch --json run create \
            --url "${{ inputs.preview_url }}" \
            --design-intent "${{ inputs.design_intent }}" \
            --quick --wait --wait-timeout 10m)
          CODE=$?
          echo "$OUT" > flock.json
          echo "exit_code=$CODE" >> "$GITHUB_OUTPUT"
          # Policy A — fail the job on entry-block / check failed
          if [ "$CODE" -eq 9 ]; then exit 9; fi
          # Reachability: fail so the team fixes the preview, not the flock
          if [ "$CODE" -eq 2 ]; then exit 2; fi
          # Wait timeout: run continues server-side; decide whether to soft-fail
          if [ "$CODE" -eq 7 ]; then exit 0; fi
          exit "$CODE"

      - name: Comment strongest gap + report link
        if: always() && hashFiles('flock.json') != ''
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          # Prefer: parse top gap + report URL from flock.json; post a short PR comment.
          # Do not paste the raw JSON dump into the PR.
          true

Exit codes → job outcomes

Pick one policy per repo and document it. Fail on exit 9 when entry-block or top-severity gaps are ship blockers. Use comment-only while you tune design intents and preview reachability. Exit 9 is a check result, not a transport error—read stdout; do not treat it as a retryable crash.

Peek vs quick vs full in CI

Escalate by branch protection: peek or --quick on feature PRs; full flock on main / release / client-facing UAT. MCP (claude mcp add boidwatch -- boidwatch mcp) is the agent-IDE surface—see MCP site QA for Cursor and Claude Code—this page stays on the CI job.

What to post back on the PR

Reviewers need a decision, not a log.

Post:

• One-line design intent used for the run
• Strongest gap (severity + short description)
• Link to the Boidwatch report / results view
• Whether the job failed or comment-only (your policy)

Do not post:

• Raw --json stdout dumps
• Full persona transcripts
• Credit math unless finance asked

A good comment reads like: “Preview UX gate (--quick): personas never reached the trial CTA (entry friction on mobile nav). Report: \. Job failed per entry-block policy.” Humans fix; agents can re-run after the next preview deploy.

Limits

Reachability and bot protection. If the preview serves a challenge page or is not publicly reachable, expect exit 2 / REACHABILITY_ERROR and zero credit spend. Open the preview for the flock (or allowlist), then re-run. Do not interpret exit 2 as “the UX failed.”

Not a replacement for unit, E2E, or security scanners. Playwright / Cypress / Morph / TestSprite / QA.tech assert functional paths and regressions. Dependabot, SAST, and secret scanners own security. Boidwatch owns persona UX flock / peek on the preview URL with a report link on the PR. Complementary, not a swap. Optional deeper product framing: AI website QA with real browser sessions.

No guarantee of every UX issue. The flock ranks gaps against the design intent you pass. Bad intents produce weak reports. Tune the one-liner to the primary conversion on that preview.

Add the gate on one agent-heavy repo tomorrow

Pick the repo where agents open the most PRs. Wire a job that receives the public preview URL after deploy success. Run --dry-run once to confirm credits, then --quick --wait --json. Map exit 2 and 9 to your fail-vs-comment policy. Post the strongest gap and report link on the PR—not the raw JSON.

boidwatch --json run create \
  --url "$PREVIEW_URL" \
  --design-intent "Understand pricing and start a trial" \
  --quick --dry-run

boidwatch --json run create \
  --url "$PREVIEW_URL" \
  --design-intent "Understand pricing and start a trial" \
  --quick --wait --wait-timeout 10m

Product: boidwatch.com · agent docs: llms.txt · web app: app.boidwatch.com · keys: API keys settings.