Shopify AI Storefront QA Before Client Launch
Shopify AI Storefront QA Before Client Launch
Shopify AI storefront QA usually means theme/Liquid checks (Theme Check, TaskerArmy-style automation), staging-preview checklists, or App-snippet visual QA (Superflow on the storefront). Those jobs catch theme hygiene and launch-list items; they do not run Census-derived personas through the shopping path (collection → PDP → variant → add to cart). Boidwatch does the visitor-proof job: a capped peek or --quick flock in real Chromium against a reachable unpublished theme preview or password-openable storefront URL, with design intent set to browse → pick variant → add to cart (stop before Checkout payment), then a shareable report with a public link—no client account. Use it after theme checks and the launch checklist and before “Publish,” especially when an AI theme or agency build just landed on the preview URL.
What Shopify theme checklists / Superflow / TaskerArmy already cover vs visitor proof
Shopify delivery leads keep getting pitched “AI QA” as one product. On a Shopify project it is several jobs that look similar in a sales deck and fail differently on the day the client opens the storefront.
Theme and Liquid hygiene. Theme Check, TaskerArmy-style Liquid and JSON automation, and storebuilt-style theme release checklists inspect the theme: broken Liquid, schema, and staging-preview hygiene. The output stays with the theme. It never opens the storefront URL the client will click as a stranger.
Agency launch lists. xigmapro- and thankik-style lists walk the purchase journey: accessibility, SEO, links, forms, and collection, product, and cart basics. DFY theme review in the cartproof, ybuild, codifiedcommerce, and kumocart class is a publish gate of the same kind. That is a checklist. It is not an accessibility certification, and it does not fly personas through collection → PDP → variant → add to cart.
App-snippet visual QA. Superflow’s Shopify integration puts checklist agents and a snippet on the storefront. The artifact is pinned visual review on the page. It is not a Census-derived flock, and it is not Checkout automation.
Visitor proof. Boidwatch is platform-agnostic: a reachable URL in, a flock of Census-derived personas in real headless Chromium, a public report out. It is not a Shopify App Store app, and it is not an official Shopify Partner badge. No native Shopify App is required. Design intent is the shopping path the client hired you to ship—find a product, pick a variant, add to cart, open the cart drawer—and the flock stops before Checkout payment. Theme-editor hygiene is not visitor proof.
Theme checks prove Liquid and schema are intact. Launch lists prove the launch items were ticked. Personas prove strangers can finish the shopping path on the URL the client will open. Keep the owners separate in the SOW. The same split, on other stacks, is Webflow AI QA before client sign-off and Framer AI QA before client sign-off. The generic gate is Test the site before client handoff; the category cut is AI website QA with real browser sessions.
Unpublished theme preview vs live storefront vs password wall
Flock the URL the client will open. Match theme, seed products, and feature state to that URL. A pass on a different host, or a different theme, is a different storefront.
Sign-off on an unpublished theme preview. If Publish is still ahead and the client opens an unpublished theme preview, run the flock on that preview. A green pass on yesterday’s live domain does not prove today’s preview.
Sign-off on a password-openable storefront. If the client opens a storefront behind a password you can open for the flock, use that reachable URL. Match the theme, seed products, and feature state they will see.
Sign-off on the live published domain. If the client signs off only after you publish to the live domain, flock that published URL. Do not substitute the unpublished preview and call it the same proof.
Reachability. Storefront password walls, bot protection, or an unpublished preview the flock cannot reach: the CLI exits 2 with error.code REACHABILITY_ERROR. The run is not created and no credits are charged. error.message says why and how to fix it. Open the password, allowlist the flock, or point at a reachable preview that matches what the client will see, then re-run. Do not treat a blocked load as visitor proof.
A page that loads but stops every persona at the door is a different result. Exit 9 is the always-on entry-block check (or a failed --assert-* flag): the personas never got past the entry URL. With --json, stdout is {results, assertions}. A password wall that still serves a page lands here when the flock loads the URL and never gets past it—not on exit 2, because the host responded. Fix access before you treat the run as visitor proof. Exit 2 is reachability. Exit 9 is a check on a run that started. Neither result means the client will Publish, and neither result means the launch will be clean.
Preview the credit math first with --dry-run if you want the cost before the flock flies.
The three-layer gate before client Publish
Run the layers in order. Do not collapse them into one “AI QA passed” line.
| Layer | Who owns it | What it proves |
|---|---|---|
| 1. Theme / Liquid checks | Theme Check or TaskerArmy-style automation | Broken Liquid, schema, and staging-preview hygiene |
| 2. Launch checklist | Your agency Shopify launch list | Accessibility, SEO, links, forms, and collection/PDP/cart basics—as a checklist, not an accessibility certification |
3. Persona peek or --quick flock | Delivery lead or the agent in the loop | Strangers can attempt collection → PDP → variant → add to cart on the reachable Shopify URL; the flock stops before Checkout payment; public report link for the client note |
Layer 3 sets design intent to the shopping path. It sits after theme checks and the launch checklist, and before Publish. If an AI theme or an agency build just landed on the preview URL, run the capped pass the same day—do not wait until the night before the call.
Depth scales with the note you have to send. A peek (3 personas, 15 steps, 5-minute hard stop) is the strongest blocker before the call. --quick is a small flock when the note needs ranked gaps. The full flock is the teardown packet. The peek gate itself is Agent-native shipping proof. The billable packet is Client-ready UX teardown for agencies.
How to run Boidwatch on a Shopify URL (shopping path, stop before payment)
Boidwatch needs a reachable Shopify URL: the unpublished theme preview, a password-openable storefront, or the published domain. No Shopify App Store install and no theme-editor embed. Authenticate with an API key and point the run at that URL.
Install and auth (new accounts: 30 free credits; 1 credit = 1 persona; no card for that pool):
curl -fsSL https://boidwatch.com/install.sh | sh
# or: brew install boidwatch/tap/boidwatch
export BOIDWATCH_API_KEY=fg_live_...
# key from https://app.boidwatch.com/settings/api-keys
CLI — Shopify preview or storefront, shopping path as design intent:
boidwatch --json run create \
--url https://SHOPIFY-PREVIEW-OR-STOREFRONT \
--design-intent "client path: collection → PDP → pick variant → add to cart → open cart drawer" \
--quick --wait
--quick runs 5 personas for 5 credits. Swap the design intent for the shopping path you actually shipped. Stop before Checkout payment. Do not encode “complete Checkout,” “enter card,” “place test order,” tax, or shipping-rate assertions. Boidwatch does not claim those jobs.
| Shopify build | Example design intent |
|---|---|
| Collection into the cart drawer | “Find a product from the collection, pick a variant, add to cart, open the cart drawer” |
| Homepage into Add to cart | “Land homepage → collection → PDP → Add to cart” |
| Search into the cart | “Search → PDP → add to cart” |
Drop --quick for the full client packet (default 20 personas, 20 credits). Use that only when the note needs the teardown, not as the default before every Publish call.
Cap choice
| Depth | Cost shape | Use before Publish |
|---|---|---|
| Peek | 3 personas, 15 steps, 5-minute hard stop → one strongest finding | Strongest blocker before the call |
--quick | 5 personas, 5 credits | Small flock when the client note needs ranked gaps |
| Full flock | 20 personas, 20 credits | Teardown packet only |
Peek is an adjacent shipping gate: authenticated POST /v1/peek on https://api.boidwatch.com. The full report stays gated. Prefer the documented CLI and MCP until peek appears in public llms.txt. CLI and MCP create ordinary runs, not peeks. The peek gate is Agent-native shipping proof. The billable packet is Client-ready UX teardown for agencies.
MCP (CLI v1.1.0 or later), when the agent is already in the delivery loop:
claude mcp add boidwatch -- boidwatch mcp
Call boidwatch_quick_check. It returns within about 45 seconds, or a run_id you resume with boidwatch_get_results—do not call quick_check again for the same run. Use boidwatch_teardown only for the thorough client packet. Wiring: MCP site QA for Cursor and Claude Code and Agent-native shipping proof.
What goes in the client note at launch
The client-facing artifact is a public report link. No Boidwatch account, no invite, no second login. Copy the link into the Publish note with one line: the strongest shopping-path finding—a variant picker miss, add-to-cart friction, or a cart-drawer fail.
Narrate it as visitor proof on their URL: “A persona flock on the Shopify preview URL you’ll open found X on collection → cart; we fixed Y before Publish.” Name the finding and the fix or the accepted exception. Say the scope in the same note: the flock stopped before Checkout and payment. That is theme and cart proof, not payment proof. Do not promise that Publish will go through, and do not promise the launch will be clean.
Keep the tool jobs in separate sentences. Do not paste raw JSON, credit burn, Theme Check logs, or Superflow screenshots into the same paragraph as the flock proof. Theme hygiene, snippet review, and visitor proof are different evidence. The handoff shape is Test the site before client handoff. The platform parallels for keeping review tools and the flock apart are Webflow AI QA before client sign-off and Framer AI QA before client sign-off.
What the report carries, enough for a cold open:
- The design intent you set on the run
- A Census-derived persona flock (sampled from 36,000+ profiles) driving real Chromium
- Ranked gaps against that intent, with recommendations
- Funnel and friction framing a stakeholder can skim
Honest comparison posture
Choose by job. Boidwatch does not replace Theme Check, Superflow, TaskerArmy, test orders, or Checkout QA.
| Tool | Job it owns on Shopify projects | Shape |
|---|---|---|
| Theme Check / TaskerArmy / theme-release checklists | Liquid and theme hygiene, plus staging preview | Theme Check, TaskerArmy-style Liquid and JSON automation, storebuilt-style theme release checklists |
| Superflow / DFY theme review / agency launch lists | App-snippet visual QA and purchase-journey checklists | Superflow on the storefront; cartproof, ybuild, codifiedcommerce, and kumocart-class review; xigmapro- and thankik-style launch lists. Not an accessibility certification |
| Boidwatch | Pre-Publish persona Chromium flock on the reachable Shopify preview or storefront URL | Public report link as the client-ready proof artifact. Design intent is browse → variant → add to cart, not Checkout. Not a Shopify App Store app. Not an official Shopify Partner badge |
This is the same delivery-gate job as Webflow AI QA and Framer AI QA, on the Shopify storefront journey. Theme tools, snippet QA, and launch checklists keep their jobs. The flock covers the visitor-proof gap those tools do not run. Pair them. Choose by job.
FAQ
How should an agency QA an AI-built Shopify storefront before the client publishes it?
Finish the theme and Liquid checks (Theme Check or TaskerArmy-style automation) and the launch checklist (accessibility, SEO, links, forms, collection/PDP/cart basics). Then run a persona peek or a --quick flock on the reachable unpublished theme preview, password-openable storefront, or published URL the client will open. Set design intent to collection → PDP → pick a variant → add to cart, and stop before Checkout payment. Put the strongest finding and the public report link in the client note. A green theme check is not visitor proof.
What’s the difference between Shopify theme / Liquid checklists (or Superflow / TaskerArmy) and a persona visitor proof pass?
Theme Check, TaskerArmy-style automation, and theme-release checklists inspect Liquid, schema, and staging-preview hygiene. Superflow puts checklist agents and a snippet on the storefront for visual QA. Agency launch lists tick the purchase-journey checklist. A persona visitor proof pass flies Census-derived personas through real Chromium on the client-facing preview or storefront URL and ranks friction against design intent. Different evidence, different owners. Boidwatch does the visitor-proof job. It does not replace Theme Check, Superflow, or TaskerArmy, and theme-editor hygiene is not the same evidence as the flock.
Should I run Boidwatch on an unpublished theme preview, a password-protected storefront, or the live published domain?
Run it on the URL the client will open, and match theme, seed products, and feature state. If sign-off is on an unpublished theme preview, flock that preview. If the storefront is password-protected, open the password so the URL is reachable, or point at another reachable URL that matches what the client will see. If sign-off is on the live domain after Publish, flock that domain. If the URL cannot be reached, the password wall blocks the load, or bot protection challenges the request, the CLI exits 2 (REACHABILITY_ERROR): the run is not created and no credits are charged. Fix access, then re-run. If the page loads but no persona gets past the entry, the CLI exits 9 (entry-block, or a failed --assert-* flag). A password wall that still serves a page can land on exit 9, because the host responded. Exit 2 is reachability. Exit 9 is a check on a run that started.
What should go in the client note at Shopify launch—and where does the flock stop relative to Checkout?
The public report link, one line for the strongest shopping-path finding, and what you fixed or accepted before Publish. A usable sentence: “A persona flock on the Shopify preview URL you’ll open found X on collection → cart; we fixed Y before Publish.” The flock stops before Checkout and payment: theme and cart proof, not payment proof. Boidwatch does not complete Checkout, take a card, place a test order, or assert tax or shipping rates. Keep Theme Check logs, Superflow screenshots, raw JSON, and credit burn out of that paragraph.
Paste the Shopify URL into a peek before Publish
Before the client Publish call, paste the reachable Shopify theme preview or storefront URL into a capped peek with design intent set to collection → PDP → add to cart (stop before payment). Drop the strongest finding and the public report link into the client note. Until peek is in your loop, the documented flock is --quick:
boidwatch --json run create \
--url https://SHOPIFY-PREVIEW-OR-STOREFRONT \
--design-intent "client path: collection → PDP → pick variant → add to cart → open cart drawer" \
--quick --wait
Product and example reports: boidwatch.com · agent docs: llms.txt · web app: app.boidwatch.com · keys: API keys settings.