Boidwatch ← All posts

AI Website QA That Runs Real Browser Sessions

AI website QA means checking whether visitors can complete the job the page was built for — not only whether selectors and assertions pass. Boidwatch does this by flying Census-derived personas through the URL in real Chromium sessions, recording intent, perception, affect, and memory at every step, then composing a client-ready report with a public share link. It is complementary to E2E suites: automation proves the build; a persona flock proves the experience. Agencies and technical directors can run it from the web app, CLI, HTTPS API, or MCP.

What “AI website QA” usually means in 2026

Search for “AI website QA” or “AI testing tools” and the SERP is dominated by autonomous E2E, self-healing locators, visual regression, and managed Playwright services. Roundups named in that space — QA.tech’s AI testing tools lists, BrowserStack’s AI testing comparisons, and similar “best AI website testing” pages — mostly optimize for engineering pass/fail: did the suite stay green after the last deploy?

That job matters. It is not the same job an agency technical director has when a founder asks for a forwardable UX packet. Most “AI testing” pages do not center Census personas, a client-ready narrative, or an agent-callable flock. The category cut below is the filter: buy E2E when you need assertions; buy persona + real-browser audits when you need experience proof you can hand a client.

Two different jobs — pass assertions vs prove the visit

Conflating them causes wrong buys. Agencies need both.

E2E / self-healing tools (what they own)

Playwright, Cypress, and AI-assisted wrappers around them own:

They answer: Did the build still do what we asserted? They do not mint a client narrative about how strangers behave against design intent.

Persona + real-browser UX audits (what they own)

Boidwatch owns the complementary layer (boidwatch.com):

They answer: Would visitors complete the design intent? Keep Playwright/Cypress for the suite. Add a flock when the deliverable is experience proof, not another flaky test file to maintain.

What a real-browser persona flock actually records

Every step writes four fields — behavioral traces, not heatmap guesses. Homepage framing:

| Field | Example |

|---|---|

| Intent | “I'm trying to find pricing before I sign up.” |

| Perception | “Top nav has five links. None of them say Pricing.” |

| Affect | confidence 0.4 · frustration 0.7 · trust 0.5 |

| Memory | “The hero mentioned a 30-day trial — try clicking the signup CTA.” |

One run × N personas × every step is the report substrate. Rows map to navigation archetypes sampled from the Census-weighted population; dashed cells are where an archetype broke off. That is why the artifact reads like a visit log, not an LLM summary of the HTML.

What the deliverable looks like

The report is composed for handoff, not for a data dump:

Example-report framing on the homepage covers a landing-page exploration and a web-app POC — use those as shape references, not as invented outcomes for your URL. Specialist UX audits still have a place; product positioning on boidwatch.com contrasts a four-figure / three-week specialist pass with a flock you can run before kickoff ends. Phrase that as positioning, not a market study.

How to run a first check today

Authenticate once (BOIDWATCH_API_KEY from API keys settings). New accounts get 30 free credits, no card; 1 credit = 1 persona.

Primary path (documented in llms.txt) — CLI --quick:

curl -fsSL https://boidwatch.com/install.sh | sh
# or: brew install boidwatch/tap/boidwatch

export BOIDWATCH_API_KEY=fg_live_...

boidwatch --json run create \
  --url https://STAGING-URL \
  --design-intent "Understand pricing and start a trial" \
  --quick --wait

--quick runs 5 personas / 5 credits. Drop --quick for a default full flock (20 personas) when the client packet needs the complete report.

The 5- and 20-persona counts and ungated complete reports require wallet top-up / a paid run outside the free cap. Without top-up, ordinary runs remain free-capped at 3 personas with gated results, even with signup credits. Peek remains the force-free capped adjacent gate, even on paid accounts: 3 personas, 15 steps, a 5-minute hard stop → one strongest finding; full report gated.

MCP — after CLI ≥ v1.1.0:

claude mcp add boidwatch -- boidwatch mcp

Call boidwatch_quick_check (returns within about 45s, or a run_id you resume with boidwatch_get_results — do not call quick_check again). Cursor and other stdio MCP clients point at the same local boidwatch mcp server. Wiring detail: MCP site QA for Cursor and Claude Code.

Adjacent shipping gate — authenticated POST /v1/peek on https://api.boidwatch.com (3 personas, 15 steps, 5-minute hard stop → one strongest finding; full report gated). Prefer documented CLI --quick / MCP quick_check for the first flock until peek appears in public llms.txt; treat peek as the pre-client yes/no gate described in Agent-native shipping proof.

Web app: app.boidwatch.com. CLI releases: github.com/boidwatch/cli.

Where AI website QA does not replace other work

Persona flocks are shipping / engagement proof for the primary visitor job. They are not:

Keep those as separate owners. Conflating “AI website QA” with accessibility audits or human studies is the same category error as conflating it with Playwright.

FAQ

What is AI website QA?

Checking whether visitors can complete the job the page was built for — using AI-driven methods — rather than only whether automated assertions pass. In Boidwatch that means Census-derived personas driving real Chromium sessions against a design intent, then a client-ready report with a public link.

How is AI website QA different from traditional E2E test automation?

E2E (Playwright, Cypress, self-healing suites) proves the build against known assertions in CI. Persona-based AI website QA proves the experience: would different visitor types pursue and complete the design intent? Complementary. Wrong buy: replacing your suite with a flock, or expecting a flock to be your regression gate.

What’s the difference between an LLM reading a page and personas driving a real browser?

An LLM summary of HTML guesses how a visit might go. Boidwatch personas each drive a real headless Chromium session — click, scroll, get stuck — and record intent, perception, affect, and memory at every step. Traces are behavioral and replayable, not heatmap guesses or static page reads.

When should an agency use AI website QA vs writing Playwright/Cypress tests?

Write or keep Playwright/Cypress when you need stable CI assertions on known flows. Run AI website QA via ordinary CLI/MCP runs (--quick / full flock) when you need a forwardable experience packet for a founder or client — ranked friction, segment breakdown, public report link — especially before kickoff ends or before staging handoff. Peek is the capped adjacent HTTPS gate for one strongest finding. Most engagements need both layers.

Run the first check on staging

On the staging URL you are about to show the client, run a documented quick flock (or capped peek as the adjacent gate); open the public report link (or strongest finding) and decide whether the top gap blocks handoff.

boidwatch --json run create \
  --url https://YOUR-STAGING-URL \
  --design-intent "YOUR ONE-LINE INTENT" \
  --quick --wait

Product & examples: boidwatch.com · agent docs: llms.txt · keys: API keys settings.