AI-generated Playwright tests: how to avoid false positives

What exactly turned green? A Playwright test can click the right button, find a success toast, and still leave an order unpersisted. The code passes. The business rule does not.

That is the most deceptive risk in AI-generated tests: they look complete because they reproduce the visible sequence. Validation starts when you replace "did the test run?" with "what state did it prove?".

Practical summary

• Treat a generated test as a draft, not as automatic proof.
• Write the business consequence before accepting locators and clicks.
• Run the browser and check observable state beyond temporary messages.
• Keep the trace, report, and repair diff before putting the test in CI.

Green can be misleading

An end-to-end test is useful when it connects a user action to a consequence that matters to the product. Playwright documentation recommends testing visible behavior and using web-first assertions, but an agent can still choose the cheapest signal that produces green output. The runner has to review that choice.

The false positive appears when the assertion confirms an intermediate detail. A toast may say that a request started. A redirect may happen before the backend saves data. A locator may find a hidden element. In each case, the sequence looks correct without closing the flow's contract.

Compare the two tests:

test("completes checkout", async ({ page }) => {
  await page.getByRole("button", { name: "Pay" }).click();
  await expect(page.getByRole("status")).toHaveText("Order created");
});

The first test checks only a message. A stronger version checks the confirmation page and, when the product allows it, reads the state through a test-only API:

test("completes checkout", async ({ page, request }) => {
  await page.getByRole("button", { name: "Pay" }).click();

  await expect(page).toHaveURL(/\/orders\/[a-z0-9-]+/);
  await expect(page.getByRole("heading", { name: "Order confirmed" })).toBeVisible();

  const orderId = await page.getByTestId("order-id").textContent();
  const response = await request.get(`/api/test/orders/${orderId}`);

  expect(response.ok()).toBe(true);
  expect(response.status()).toBe(200);
  expect(await response.json()).toMatchObject({ state: "created" });
});

The read endpoint is only an example. Do not expose internal data to the user's browser just to make a test easier. If the API does not exist, verify the consequence through the UI, a test queue, or another safe artifact the system already produces.

Citation capsule: An AI-generated Playwright test can turn green by checking only a toast, a redirect, or a present locator. Playwright documentation recommends assertions that wait for the expected state. To avoid a false positive, the test must also confirm the business consequence the user should observe.

The contract comes before the .spec.ts file

Start with a sentence that describes the rule, not the implementation. The contract should say who starts the flow, which state must exist first, what action happens, and what evidence confirms the outcome. That sentence becomes the standard for reviewing the .spec.ts file generated by the agent.

## Approved checkout

- Given: a cart with an available product and an authenticated user.
- When: the user confirms an approved payment.
- Then: the order appears as created, receives an identifier, and the cart is empty.
- Does not prove: only the presence of a toast or a URL change.

Playwright provides planner, generator, and healer agents to explore an application, create tests, and propose repairs. The documentation also keeps a seed test, a readable plan, and auditable files in the flow. The agent can fill the skeleton, but the team still decides whether the assertion represents the rule.

Use this list before reviewing style details:

The last item deserves attention. Fixing a locator is different from changing the expected result. The first can follow an interface change. The second can hide a regression. That difference should appear in the pull request and CI report.

The contract also connects to PR evals that keep coding agents honest. An E2E test covers the complete flow, but the business rule may also need an integration test or a controlled service read.

Locators, assertions, and state form three checks

Verification needs three layers. First, confirm that the locator describes the element as the user finds it. Next, use an assertion that waits for the state. Finally, check the effect that exists beyond the element used to start the action.

Playwright recommends user-facing locators such as getByRole, getByLabel, and getByTestId instead of fragile CSS or XPath chains (Playwright, best practices, retrieved July 28, 2026). That does not make a test automatically good. A semantic locator can still point at the wrong element if the assertion is weak.

test("updates the address shown in the profile", async ({ page }) => {
  await page.getByLabel("Street").fill("10 Flower Street");
  await page.getByRole("button", { name: "Save address" }).click();

  await expect(page.getByRole("status")).toHaveText("Address saved");
  await expect(page.getByLabel("Street")).toHaveValue("10 Flower Street");
  await expect(page.getByTestId("profile-address")).toContainText("10 Flower Street");
});

Here, the status says that the operation was accepted, the field shows the local value, and the profile summary confirms the visible state. In an asynchronously persisted system, add a safe read or reopen the page in a new context so the test does not check browser memory only.

Do not turn every detail into an assertion. The goal is to prove the contract, not freeze the entire HTML structure. Also avoid waitForTimeout as an automatic repair. If the agent cannot explain which condition must be awaited, the problem is in the test or the product, not in the absence of an arbitrary pause.

Citation capsule: User-facing locators make a test more resistant to internal changes, but they do not eliminate false positives. Full verification combines an action, a web-first assertion, and an observable business consequence. If the healer changes the expectation, the diff needs human review before the test enters the stable suite.

Evidence needs to travel with the test

The minimum is a readable test file, the command that ran, the result, a trace when it fails, and the diff proposed by the agent. This collection answers whether the test failed because of a regression, environment, stale locator, or wrong expectation. Without it, CI returns only a color.

Configure the runner to keep a trace on the first retry in CI:

import { defineConfig, devices } from "@playwright/test";

export default defineConfig({
  testDir: "./tests",
  forbidOnly: Boolean(process.env.CI),
  retries: process.env.CI ? 1 : 0,
  reporter: [["html", { open: "never" }]],
  use: {
    baseURL: "http://127.0.0.1:3000",
    trace: "on-first-retry",
    ...devices["Desktop Chrome"],
  },
});

Playwright documents on-first-retry as a way to collect a trace when a failure repeats, without recording that heavy artifact on every run (Playwright, trace viewer, retrieved July 28, 2026). The trace shows actions, DOM snapshots, network, console, and errors. It does not replace rule review, but it reduces guesswork about what happened.

In long loops that switch between an agent, a browser, logs, and review, I use RemoteCode to take Claude Code and Codex further with less repeated context. It is my own tool, so the recommendation is editorial and tied to the cost of carrying evidence, not a promise that it validates the test.

CI judges the artifact, not the story

The shortest path is to treat the generated test as pull request code. The agent opens the file, CI runs the flow, and a job publishes the artifacts. Approval happens only when the test proves the contract and the agent diff does not change the expectation without review.

A simple sequence for a Node.js project is:

npm ci
npx playwright install --with-deps chromium
npx playwright test tests/checkout.spec.ts --reporter=line
npx playwright show-report

Verify locally with a deliberately broken path. Temporarily remove data-testid="order-id" or make the test API return state: "pending". The test should fail for the expected reason. Restore the behavior and confirm that the run passes again. This small mutation check tells you more than looking only at the first green result.

In CI, store playwright-report and test-results when the run fails. For an agent flow, also record the original plan, commit hash, model or tool name when available, and repair diff. Do not send credentials, cookies, or sensitive payloads to the report.

This artifact complements regression evals for coding agents: the eval asks whether the patch preserved behavior, while the browser test provides concrete proof for that flow.

Citation capsule: CI should judge a generated file by the evidence it produces, not by the agent's prose. Playwright provides reports and traces for debugging failures. A reliable gate runs the flow, checks the expected result, preserves artifacts, and blocks silent changes to the expectation.

Common mistakes when reviewing agent tests

A common mistake is accepting an assertion that only confirms that the interface reacted. Replace temporary messages with persisted state, confirmation on an independent screen, or a controlled service read. Also do not let the healer edit the expectation without review. A mechanical locator fix is one thing; changing what counts as success is another.

It is easy to confuse a retry with reliability. Playwright labels tests that pass only after a retry as flaky; repeating the run helps diagnosis, but it does not turn instability into proof (Playwright, retries, retrieved July 28, 2026). A bounded retry should create a signal for the team, not hide the cause.

Isolation is often forgotten when the agent receives only the screen and requirement. Playwright's best-practices documentation recommends separating test data and context to improve reproduction and avoid cascading failures. Without clear fixtures and preconditions, the file may pass only because another run left state in the environment. The full reference appears in the source list.

Line coverage can mislead as well. A test may cross many functions without checking the relevant decision. Tie the case to the requirement, risk, and observable result. Coverage helps locate untested code; it does not decide whether an assertion proves the rule.

When the suite grows, record this evidence in coding agent observability in CI. Test name, commit, attempt, and artifact form a short trail for separating product, environment, and automatic repair failures.

Limits of this approach

No checklist prevents every defect. An E2E test does not prove every data combination, replace unit tests, or guarantee that an external service is available. Validation also cannot detect a wrong business rule by itself. It only makes the mistake harder to hide.

For payments, permissions, migrations, personal data, and integrations with external effects, require contract tests, a disposable environment, and human approval. An agent can suggest the case and write the skeleton, but the team must decide whether the scenario is safe, deterministic, and representative.

Playwright Test Agents are useful for exploring, generating, and repairing tests, but the documentation describes a flow that still produces reviewable plans and files. Use that separation as the principle: automate production, preserve proof, and make changes to intent explicit.

That is the same boundary defended by a verification harness for coding agent pull requests: each step can be fast, but the final change must arrive small, traceable, and verifiable.

Frequently asked questions about AI-generated Playwright tests

Does an AI-generated Playwright test replace human review?

No. An agent can create valid locators and assertions, but it should not decide alone whether the case proves the product rule. The reviewer must compare the requirement, flow, final state, test data, and repair diff before adding the test to the stable suite.

How can I tell whether a Playwright assertion is weak?

It is weak when it can pass without confirming the main consequence. A toast, URL, or visible element can be useful evidence, but it is not enough when the requirement involves persistence, calculation, permission, or delivery. Add an independent check and run a path that should fail.

Can the healer change a test automatically?

It can propose a mechanical fix, such as updating a changed locator, as long as the diff remains visible. Do not automatically accept changes to the expected result, business rule, error condition, or scenario scope. Those changes need human review and new evidence.

Should I use tracing on every run?

No. In CI, on-first-retry keeps evidence when a test fails again, while continuous recording increases artifact cost. During development, turn tracing on for a specific failure and turn it off after you understand the cause.

Sources consulted

• Playwright, "Playwright Test Agents", retrieved July 28, 2026, https://playwright.dev/docs/test-agents
• Playwright, "Best Practices", retrieved July 28, 2026, https://playwright.dev/docs/best-practices
• Playwright, "Assertions", retrieved July 28, 2026, https://playwright.dev/docs/test-assertions
• Playwright, "Retries", retrieved July 28, 2026, https://playwright.dev/docs/test-retries
• Playwright, "Trace viewer", retrieved July 28, 2026, https://playwright.dev/docs/trace-viewer-intro
• Playwright, "Release notes", retrieved July 28, 2026, https://playwright.dev/docs/release-notes
• GitHub, "Building and testing Node.js", retrieved July 28, 2026, https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing/nodejs