Triage the first red run

The first pyro run of a migrated suite usually fails in a handful of ways, each repeated across many tests. This page is a checklist: catch what you can before running, run once and group the errors, then fix each kind with the pattern below. Every failure here was reproduced on a small suite whose tests all pass on Chromium, and every fix was run on both Pyro and Chromium (Playwright 1.59.1), with Bun 1.4 on macOS.

Before the first run

These checks take minutes and find failures without running anything.

1. Type-check against Pyro's types. The selector-first page methods and the deprecated APIs Pyro leaves out are type errors:

npx tsc --noEmit

e2e/first-run.spec.ts(5,14): error TS2339: Property 'fill' does not exist on type 'Page'.

Unsupported toHaveScreenshot options (animations, mask, ...) are in the types, so tsc does not catch them. Search for them:

grep -rnE "animations|mask|maskColor|caret|omitBackground|scale|stylePath" e2e --include='*.ts' | grep -i screenshot

e2e/first-run.spec.ts:27:  await expect(page).toHaveScreenshot('login-still.png', { animations: 'disabled' });

The search only finds options written on the same line as the call.

2. Search for runner features that are not there.

grep -rn "dependencies" playwright.config.ts                 # setup projects: they do not run first
grep -rn "testInfo.status\|testInfo.error" e2e --include='*.ts'  # stays 'passed' in afterEach
grep -rn "\-\-shard\|show-report\|--ui\|--trace" package.json .github

• A setup project with dependencies does not run first. Move it to globalSetup: see Sign in once with a setup project's storageState.
• testInfo.status and testInfo.error are not updated while a test runs, so an afterEach that acts on failures never fires. See Behaviour differences.
• --shard, --trace, --ui and report scripts fail or do nothing. See CI without sharding.

3. Check the browser starts. Before the first @povio/pyro release, a package installed from a checkout has no bundled browser. The first run then fails in global setup or in the first test with:

error: Missing bundled browser: /Users/xban1x/Povio/pyro-rename/packages/pyro/native/darwin-arm64/pyro. Reinstall @povio/pyro or set PYRO_BIN to a built browser.

Set PYRO_BIN to the browser you built. See Before the first release.

Run once and group the errors

Run the whole suite once, keep the log, and count the error lines. Fix the biggest group first.

bunx pyro run --reporter dot > run.txt 2>&1
grep -E '^    [A-Za-z]*Error' run.txt | cut -c1-120 | sort | uniq -c | sort -rn

On the example suite below, with its six typical failures:

   1     TypeError: page.fill is not a function. (In 'page.fill("input[name=text]", "Call mum")', 'page.fill' is undefined)
   1     TimeoutError: locator.click: Timeout 2000ms exceeded.
   1     Error: expect(received).toBe(expected)
   1     Error: expect(locator).toHaveText(expected) failed
   1     Error: Pyro.run: step 0: unknown field `animations`, expected one of `name`, `baseline`, `sceneHash`, `threshold`, `
   1     Error: Pyro.run: step 0 (matchScreenshot /private/tmp/docs-r2-pw/r4/e2e/first-run.spec.ts-snapshots/login-chromium-d

On a 400-test suite, the counts tell you which fix pays most. To check one test against Chromium, run the same suite on both: see Side by side.

While you fix, rerun only what you touch: bunx pyro run e2e/checkout (a file path regex) or bunx pyro run -g "adds a todo" (a title regex). -x stops at the first failure.

The example suite

All six tests pass on Chromium with stock Playwright, with baselines it wrote. Under pyro run all six fail.

The suite: app, sign-in helpers and global setup (as in the sign-in recipe), config

// app/server.ts
// A small app to test: sign in, a list of todos, a form that adds one.
const port = Number(process.env.PORT ?? 4517);
const sessions = new Map<string, string[]>(); // session id -> todos

const page = (title: string, body: string) =>
  new Response(
    `<!doctype html><html><head><meta charset="utf-8"><title>${title}</title>
<style>body{font:16px system-ui;margin:2rem}li{margin:.25rem 0}</style></head>
<body>${body}</body></html>`,
    { headers: { 'content-type': 'text/html' } },
  );

const sid = (req: Request) => /(?:^|; )sid=([^;]+)/.exec(req.headers.get('cookie') ?? '')?.[1];

Bun.serve({
  port,
  async fetch(req) {
    const url = new URL(req.url);
    const id = sid(req);
    const signedIn = id !== undefined && sessions.has(id);

    if (url.pathname === '/login') {
      return page('Sign in', `
<h1>Sign in</h1>
<form id="f">
  <label>Email <input name="email" type="email"></label>
  <label>Password <input name="password" type="password"></label>
  <button>Sign in</button>
</form>
<p role="alert" id="err"></p>
<script>
  f.onsubmit = async (e) => {
    e.preventDefault();
    const r = await fetch('/api/login', { method: 'POST', body: new FormData(f) });
    if (r.ok) location.href = '/todos'; else err.textContent = 'Wrong email or password';
  };
</script>`);
    }

    if (url.pathname === '/api/login' && req.method === 'POST') {
      const form = await req.formData();
      if (form.get('password') !== 'secret') return new Response('no', { status: 401 });
      const newId = crypto.randomUUID();
      sessions.set(newId, ['Buy milk', 'Walk the dog']);
      return new Response('ok', { headers: { 'set-cookie': `sid=${newId}; Path=/; HttpOnly` } });
    }

    if (url.pathname === '/api/todos') {
      if (!signedIn) return new Response('[]', { status: 401 });
      const todos = sessions.get(id!)!;
      if (req.method === 'POST') {
        const { title } = (await req.json()) as { title: string };
        await Bun.sleep(150); // a slow save, so the app shows "Saving..."
        todos.push(title);
        return Response.json(todos, { status: 201 });
      }
      return Response.json(todos);
    }

    if (url.pathname === '/todos') {
      if (!signedIn) return Response.redirect('/login', 302);
      return page('Todos', `
<h1>Todos</h1>
<user-badge></user-badge>
<ul id="list"></ul>
<p id="empty" hidden>Nothing to do</p>
<form id="add">
  <label>New todo <input name="text"></label>
  <button>Add</button>
  <span id="state" role="status"></span>
</form>
<script>
  customElements.define('user-badge', class extends HTMLElement {
    connectedCallback() {
      this.attachShadow({ mode: 'open' }).innerHTML = '<button>Sign out</button>';
    }
  });
  const render = (todos) => {
    list.replaceChildren(...todos.map((t) => Object.assign(document.createElement('li'), { textContent: t })));
    empty.hidden = todos.length > 0;
  };
  fetch('/api/todos').then((r) => r.json()).then(render);
  add.onsubmit = async (e) => {
    e.preventDefault();
    state.textContent = 'Saving...';
    const r = await fetch('/api/todos', {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ title: add.text.value }),
    });
    render(await r.json());
    add.reset();
    state.textContent = 'Saved';
  };
</script>`);
    }

    return Response.redirect('/login', 302);
  },
});
console.log(`app on http://localhost:${port}`);

// e2e/auth.ts
import type { Page } from '@povio/pyro/wright';

export const port = Number(process.env.PORT ?? 4517);
export const baseURL = `http://localhost:${port}`;
export const authFile = 'e2e/.auth/user.json';

/** The sign-in steps, shared by global setup and by tests that sign in themselves. */
export async function signIn(page: Page, email = 'ana@example.com') {
  await page.goto('/login');
  await page.getByLabel('Email').fill(email);
  await page.getByLabel('Password').fill('secret');
  await page.getByRole('button', { name: 'Sign in' }).click();
  await page.waitForURL('/todos');
}

// e2e/global-setup.ts
import { mkdirSync } from 'node:fs';
import { dirname } from 'node:path';
import { chromium } from '@povio/pyro/wright';
import { authFile, baseURL, signIn } from './auth';

export default async function globalSetup() {
  const browser = await chromium.launch();
  const context = await browser.newContext({ baseURL });
  const page = await context.newPage();
  await signIn(page);
  mkdirSync(dirname(authFile), { recursive: true }); // storageState({ path }) does not create it
  await context.storageState({ path: authFile });
  await browser.close();
}

// playwright.config.ts
import { defineConfig } from '@povio/pyro/wright';
import { authFile, baseURL, port } from './e2e/auth';

export default defineConfig({
  testDir: './e2e',
  globalSetup: './e2e/global-setup.ts',
  use: { baseURL, storageState: authFile },
  projects: [{ name: 'chromium' }],
  webServer: { command: 'bun app/server.ts', url: `${baseURL}/login`, env: { PORT: String(port) } },
});

// e2e/first-run.spec.ts
import { test, expect } from '@povio/pyro/wright';

test('selector-first fill', async ({ page }) => {
  await page.goto('/todos');
  await page.fill('input[name=text]', 'Call mum');
});

test('sign out button in a shadow root', async ({ page }) => {
  await page.goto('/todos');
  await page.getByRole('button', { name: 'Sign out' }).click({ timeout: 2000 });
});

test('transient "Saving..."', async ({ page }) => {
  await page.goto('/todos');
  await page.getByLabel('New todo').fill('Water plants');
  await page.getByRole('button', { name: 'Add' }).click();
  await expect(page.getByRole('status')).toHaveText('Saving...');
});

test('login page looks right', async ({ page }) => {
  await page.goto('/login');
  await expect(page).toHaveScreenshot('login.png');
});

test('login page, animations off', async ({ page }) => {
  await page.goto('/login');
  await expect(page).toHaveScreenshot('login-still.png', { animations: 'disabled' });
});

test('goto response headers', async ({ page }) => {
  const response = await page.goto('/todos');
  expect(response!.headers()['content-type']).toBe('text/html');
});

bunx pyro run first-run (stack frames left out):

Running 1 file using 5 workers

  ✘   1 e2e/first-run.spec.ts › selector-first fill (44ms)
  ✘   2 e2e/first-run.spec.ts › sign out button in a shadow root (2009ms)
  ✘   3 e2e/first-run.spec.ts › transient "Saving..." (5186ms)
  ✘   4 e2e/first-run.spec.ts › login page looks right (23ms)
  ✘   5 e2e/first-run.spec.ts › login page, animations off (4ms)
  ✘   6 e2e/first-run.spec.ts › goto response headers (4ms)

The failures, one by one

Selector-first page methods

TypeError: page.fill is not a function. (In 'page.fill("input[name=text]", "Call mum")', 'page.fill' is undefined)

Spot it: is not a function on page. or frame., or tsc errors. The same goes for page.click, page.check, page.hover, page.press, page.textContent(selector), page.getAttribute, page.isVisible(selector), page.selectOption, page.dispatchEvent and the rest of the selector-first forms.

Fix: call the same method on a locator. page.$(), page.$$(), page.$eval() and page.waitForSelector() exist and need no change.

await page.locator('input[name=text]').fill('Call mum');

Elements inside a shadow root

TimeoutError: locator.click: Timeout 2000ms exceeded.
Call log:
  - waiting for getByRole('button', { name: 'Sign out' })
  - {"name":"Sign out","role":"button"} matched no element

Spot it: matched no element for an element you can see, inside a web component. Playwright finds it because its CSS and getByRole locators enter open shadow roots. Pyro's CSS, getByRole, getByLabel and getByPlaceholder locators do not; getByText does.

Fix: locate it by text.

await page.getByText('Sign out').click();

Transient states

Error: expect(locator).toHaveText(expected) failed

Locator: getByRole('status')
Expected: "Saving..."
Received: Saved
Timeout: 5000ms

Spot it: the received value is the state after the one the test expects: "Saved" for "Saving...", a list for a spinner. The test takes the full expect timeout (5 s) to fail. In Pyro, click() returns after the page settles, including the request the click started, so the in-between state is already gone.

Fix: assert the final state, and check the request if it matters:

const saved = page.waitForResponse((r) => r.url().endsWith('/api/todos') && r.request().method() === 'POST');
await page.getByRole('button', { name: 'Add' }).click();
expect((await saved).status()).toBe(201);
await expect(page.getByRole('status')).toHaveText('Saved');

If the in-between state is what the test is about, hold the request in a route. The click then returns while the request waits, and the state is there to see:

await page.route('**/api/todos', async (route) => {
  if (route.request().method() === 'POST') await new Promise((r) => setTimeout(r, 1000));
  await route.continue();
});

Screenshot options Pyro rejects

Error: Pyro.run: step 0: unknown field `animations`, expected one of `name`, `baseline`, `sceneHash`, `threshold`, `maxDiffPixels`, `maxDiffPixelRatio`, `clip`, `fullPage`, `locator`, `update`, `captureOnly`, `as`, `timeout` at line 1 column 132

Spot it: unknown field in a toHaveScreenshot call. Pyro accepts threshold, maxDiffPixels, maxDiffPixelRatio, clip, fullPage and timeout.

Fix: remove the option. Pyro renders a still frame, so there is no animation or caret to turn off. mask has no replacement; clip the screenshot to a stable region, or screenshot a locator.

Screenshot baselines from Chromium

Error: Pyro.run: step 0 (matchScreenshot /private/tmp/docs-r2-pw/r4/e2e/first-run.spec.ts-snapshots/login-chromium-darwin.png): matchScreenshot '/private/tmp/docs-r2-pw/r4/e2e/first-run.spec.ts-snapshots/login-chromium-darwin.png': 3311 pixels differ (1280x720); the actual image and the diff are in error.data
  actual: /private/tmp/docs-r2-pw/r4/test-results/e2e-first-run.spec.ts-login-page-looks-right/login-actual.png
  diff:   /private/tmp/docs-r2-pw/r4/test-results/e2e-first-run.spec.ts-login-page-looks-right/login-diff.png

Spot it: pixels differ on every toHaveScreenshot with a baseline Chromium wrote. Pyro renders with its own CPU renderer, so they never match. login-expected.png is written beside the two images named in the message.

Fix: remove rejected options first (above). Then look at the -actual images: a page Pyro renders wrongly would otherwise become the new baseline. Then rewrite the baselines. Do it on the platform CI uses, since the file name carries the platform:

bunx pyro run -u all -g "login page"

Running 2 files using 5 workers

  ✓   1 e2e/first-run.spec.ts › login page looks right (79ms)
  ✓   2 e2e/first-run.spec.ts › login page, animations off (8ms)

  2 passed (0.1s)

Each baseline gets a .json file beside it (login-chromium-darwin.json). Commit both. See Screenshots and snapshots.

Response headers from goto

Error: expect(received).toBe(expected)

Expected: "text/html"
Received: undefined

Spot it: a check on (await page.goto(url)).headers() or .body() fails. The response goto returns carries the URL and status only. On a booted page goto returns null.

Fix: wait for the response instead:

const doc = page.waitForResponse('**/todos');
await page.goto('/todos');
expect((await doc).headers()['content-type']).toBe('text/html');

The fixed tests

// e2e/fixed.spec.ts
import { test, expect } from '@povio/pyro/wright';

test('locator fill', async ({ page }) => {
  await page.goto('/todos');
  await page.locator('input[name=text]').fill('Call mum');
  await expect(page.getByLabel('New todo')).toHaveValue('Call mum');
});

test('sign out button by text', async ({ page }) => {
  await page.goto('/todos');
  await expect(page.getByText('Sign out')).toBeVisible();
  await page.getByText('Sign out').click();
});

test('final state after saving', async ({ page }) => {
  await page.goto('/todos');
  const saved = page.waitForResponse((r) => r.url().endsWith('/api/todos') && r.request().method() === 'POST');
  await page.getByLabel('New todo').fill('Water plants');
  await page.getByRole('button', { name: 'Add' }).click();
  expect((await saved).status()).toBe(201);
  await expect(page.getByRole('status')).toHaveText('Saved');
});

test('"Saving..." with a held request', async ({ page }) => {
  await page.route('**/api/todos', async (route) => {
    if (route.request().method() === 'POST') await new Promise((r) => setTimeout(r, 1000));
    await route.continue();
  });
  await page.goto('/todos');
  await page.getByLabel('New todo').fill('Water plants');
  await page.getByRole('button', { name: 'Add' }).click();
  await expect(page.getByRole('status')).toHaveText('Saving...');
});

test('response headers', async ({ page }) => {
  const doc = page.waitForResponse('**/todos');
  await page.goto('/todos');
  expect((await doc).headers()['content-type']).toBe('text/html');
});

On Pyro:

Running 1 file using 5 workers

  ✓   1 e2e/fixed.spec.ts › locator fill (85ms)
  ✓   2 e2e/fixed.spec.ts › sign out button by text (6ms)
  ✓   3 e2e/fixed.spec.ts › final state after saving (163ms)
  ✓   4 e2e/fixed.spec.ts › "Saving..." with a held request (14ms)
  ✓   5 e2e/fixed.spec.ts › response headers (23ms)

  5 passed (0.3s)

The same file, with its import changed to @playwright/test, on Chromium:

  ✓  1 [chromium] › e2e/fixed.spec.ts:3:1 › locator fill (107ms)
  ✓  2 [chromium] › e2e/fixed.spec.ts:9:1 › sign out button by text (85ms)
  ✓  3 [chromium] › e2e/fixed.spec.ts:15:1 › final state after saving (247ms)
  ✓  4 [chromium] › e2e/fixed.spec.ts:24:1 › "Saving..." with a held request (128ms)
  ✓  5 [chromium] › e2e/fixed.spec.ts:35:1 › response headers (54ms)

  5 passed (2.3s)

Each fix keeps the test valid on Chromium, so you can make them while both browsers still run the suite.

Other failures to look for

These did not occur in the example suite, but show up in real ones. Each has a row in Typical first failures:

• A confirm() dialog acts as if Cancel was pressed: no page.on('dialog') listener was registered before the action.
• toHaveRole('button') fails on a <button>: Pyro's toHaveRole reads the role attribute only.
• getByTestId(/regex/) or click({ clickCount: 3 }) throws "not supported yet".
• The app shows an "unsupported browser" page, or a web API is undefined: see Web platform support.
• test.skip(isMobile) skips everything: unset options are undefined, not false.
• Code that parses V8 stack traces gets JavaScriptCore's format.

Next

• Compare every result with Chromium automatically: Side by side.
• The full list of differences: Behaviour differences.
• Check one API: API coverage.