Why Playwright Tests Pass Locally and Fail in CI: Our Field Notes on Auto-Waiting, Test Isolation, and Trace-Driven Debugging

Playwright is Microsoft's browser automation and end-to-end testing framework that drives Chromium, Firefox, and WebKit through one API. We run it against Next.js App Router applications, and on more than one project the suite was green on developer machines and red on roughly one CI run in five. These are our notes on what those failures actually were.

Key takeaways

A flaky Playwright test is a race condition in the test, not randomness in the browser. Playwright's auto-waiting covers element actionability; it does not wait for application business state.

Our teams never call page.waitForTimeout() in a test. Web-first assertions such as expect(locator).toBeVisible() retry until expect.timeout (5 seconds by default) and pass the instant the condition holds.

Parallel workers need separate data. Playwright runs multiple worker processes by default, so two tests can mutate the same database row at the same moment.

CI is a slower machine with different fonts, timezone, and network. We pin timezoneId and locale in playwright.config.ts and stub third-party requests with page.route().

Setting trace: 'on-first-retry' is the single highest-value configuration line. It records DOM snapshots, network, and console output for exactly the runs that failed, at no cost on green runs.

What actually makes a Playwright test flaky?

A flaky Playwright test is one whose result depends on timing rather than on application behaviour. Playwright already performs actionability checks before every action: before locator.click() it waits for the element to be attached, visible, stable, enabled, and able to receive pointer events. That covers the DOM but not the state that matters — whether the mutation committed, whether React finished hydrating, whether a toast that will cover the button has appeared.

The three races we hit most often are clicking a button before Next.js hydration attaches its handler, asserting on a list before a Server Action revalidated it, and a cookie banner intercepting a click. Playwright strict mode catches a fourth class: a locator resolving to more than one element throws instead of silently picking the first. That is a feature, not flake.

The fix pattern is consistent. Assert an interactive signal such as expect(save).toBeEnabled(), perform the action, then assert the consequence with expect(page.getByRole('status')).toHaveText('Saved'). Each step is a retrying assertion rather than a guess about timing.

When should we use web-first assertions instead of waitForTimeout?

Always. A web-first assertion is a Playwright assertion that retries its condition until it passes or the timeout expires — toBeVisible(), toHaveText(), toHaveCount(), toHaveURL(). page.waitForTimeout(2000) does the opposite: it is always exactly as slow as the guessed number, and still too short on the day the runner is loaded.

Our two rules: assert the consequence rather than the delay, and prefer role-based locators such as getByRole('button', { name: 'Save' }) over CSS selectors. Role-based locators are re-resolved on every retry and survive markup refactors. Playwright locators are lazy — page.locator() stores a query, not an element — so a stale element reference is impossible by construction. For conditions with no built-in matcher, expect(async () => { ... }).toPass() retries an entire block instead of sleeping.

How do we keep parallel Playwright workers from corrupting each other's data?

Give each worker its own data namespace. Playwright isolates browser state for free — every test receives a fresh BrowserContext with its own cookies and storage — but it does not isolate the database. Two workers signing in as the same user and editing the same project fail in ways that look random.

Our standard solution is a worker-scoped fixture created with base.extend and { scope: 'worker' }, which provisions one account per worker keyed by workerInfo.parallelIndex and tears it down afterwards. The account is created once per worker process, not once per test, so the cost stays low.

Authentication is the other large win. We sign in once in a setup project, save the session with storageState, and let every test start already logged in. That removes a login flow from every test, which is the most common source of both flake and wasted runtime in an end-to-end suite.

Why does the same test pass locally and fail in CI?

Because CI is a different machine with a different clock, a different font set, and fewer cores. Playwright derives its worker count from available CPUs, so a 2-core runner executing a config tuned on a 10-core laptop is far more contended; we set workers to 2 on CI and leave it automatic locally.

Locale and timezone are the next offender. A date rendered as 19/09/2026 locally and 9/19/2026 on the runner fails a text assertion, so we pin timezoneId: 'UTC' and locale: 'en-US' in the config. Missing fonts shift layout and break screenshot comparison, which is why we run in the official mcr.microsoft.com/playwright image rather than a bare Node image.

Third-party scripts are the last one. Analytics and payment widgets are a network dependency nobody controls, so we abort them in tests with page.route('**/analytics/**', route => route.abort()).

How do we debug a flake we cannot reproduce locally?

Capture a trace of the failing run instead of trying to re-create it. Setting trace: 'on-first-retry' writes a zip containing a DOM snapshot per action, the full network log, and console output. Running npx playwright show-trace trace.zip opens a time-travel viewer where any step can be inspected against the exact DOM the click landed on.

When local reproduction is wanted, npx playwright test tests/checkout.spec.ts --repeat-each=50 --workers=4 runs the test fifty times under contention, which surfaces ordering races far faster than re-running CI. The --last-failed flag re-runs only what failed previously. On sharded pipelines, the blob reporter plus npx playwright merge-reports produces one HTML report across all shards instead of several partial ones.

Should we retry a flaky test, quarantine it, or delete it?

Retries are a reporting tool, not a fix. Setting retries to 2 on CI keeps the pipeline usable and marks the test as flaky in the report so it stays visible. If retries hide a failure permanently, a failing test has been traded for a test that no longer means anything.

Retry when the suite is mostly stable and CI needs to stay usable; the cost is hiding intermittent product bugs if nobody reads the flaky report.

Quarantine with test.fixme or a @flaky grep tag when a test fails daily and blocks merges; the cost is that the user path is untested until someone returns to it.

Fix the race when the root cause is a missing assertion or shared data; it usually takes an hour and typically repairs a whole family of tests.

Delete the test when it asserts nothing a user would notice; the cost is lost coverage, but the report becomes honest.

FAQ

Q: Playwright auto-waits, so are explicit waits ever needed?

A: Playwright auto-waits for element actionability before each action, but not for application state such as a completed server mutation. Assert the consequence with a web-first assertion like expect(locator).toHaveCount(4) rather than adding a wait.

Q: What is the difference between page.waitForTimeout() and expect().toPass()?

A: waitForTimeout() sleeps a fixed duration regardless of state. expect().toPass() retries a block of assertions until they all pass or the timeout expires, so it finishes as soon as the condition becomes true.

Q: Should end-to-end tests run against a real database?

A: Yes, for confidence, but with per-worker isolation. Use a worker-scoped fixture keyed by testInfo.parallelIndex so two parallel workers never touch the same rows.

Q: How many end-to-end tests should a project have?

A: Few enough to run on every pull request. Cover authentication and revenue paths end to end, and push everything else down to component and unit tests, which are faster and far less prone to flake.

Q: Does trace: 'on' slow the suite down?

A: Yes, tracing every run adds runtime overhead and disk usage. trace: 'on-first-retry' gives identical debugging power but only on runs that already failed.