Playwright and flaky tests

What are Flaky Tests?

Flaky test cases are tests that show unpredictable behavior during execution.

They:

• Pass sometimes
• Fail sometimes
• And this happens without any code changes

Common Causes of Flakiness

Flaky tests are usually caused by:

• Timing issues (async operations not completed)
• Test data problems
• Environment differences
• External dependencies (APIs, network, services)

👉 Key idea: You often don’t fix the test — you fix the environment or conditions around the test.

Debugging Strategies for Flaky Tests

1. Identify the Pattern

First, understand how the test is failing. Run the test multiple times and observe:

• Always fails at the same step → Likely a bug in test or app
• Fails randomly → Timing / async issue
• Fails in CI but passes locally → Environment issue

2. Controlled Retry Mechanism

Most frameworks (including Playwright) support retries.

• If a test fails, it is automatically retried
• Helps identify whether the failure is flaky

👉 Track:

• Retry success rate
• If test passes on retry → likely flaky

⚠️ Important: Retries are for diagnosis, not a solution

3. Analyse Execution Artifacts

Without changing test code, use artifacts generated during execution:

Logs:

• Console logs
• Network logs
• Backend logs

Playwright Debug Tools:

• Trace Viewer
• Video recordings
• Screenshots

Run:

npx playwright show-trace trace.zip

What to check in trace:

• Element visibility & state
• Timing gaps
• API delays

4. Network-Level Debugging

Many flaky tests are actually API-related issues.

Example:

• You click Save
• Button is clickable and action is triggered
• But dialog doesn’t close

👉 Reason:

• API response is delayed
• UI is ready, but data is not updated yet

5. Stabilize Test Environment

a. Data Stability

• Use fixed and reliable test data
• Avoid using data generated by other tests
• Reset database before test runs

b. Environment Consistency

Ensure consistency across executions:

• Same browser version
• Same OS configuration
• Same Node.js version

Final Takeaway

Flaky tests are not random — they are signals of instability in:

• Timing
• Data
• Environment
• External systems

👉 Your job is to find the pattern and stabilize the system, not blindly fix the test.