Analysis
The Analysis page provides test health analytics across all your test runs, helping you identify trends, flaky tests, and consistently failing tests.
Overview
After running tests, the Analysis page aggregates results to show:
- Overall pass rate trends
- Which tests are flaky (intermittently passing/failing)
- Which tests are consistently failing
- Performance metrics across test plans
- Individual test health history
Summary Cards
At the top of the page, five summary cards show key metrics:
| Card | What It Shows |
|---|---|
| Completed Runs | Total number of completed test plan runs |
| Unique Tests | Number of distinct tests executed across all runs |
| Avg Pass Rate | Average pass rate across all runs (color-coded) |
| Flaky Tests | Tests that pass sometimes and fail other times |
| Consistently Failing | Tests failing 3+ consecutive runs |
Pass Rate Trend
A bar chart showing pass rates for recent runs:
- Green bars: 80%+ pass rate
- Yellow bars: 50-79% pass rate
- Red bars: Below 50% pass rate
- Run names/numbers appear below each bar
This helps you spot quality trends — are things getting better or worse?
Test Plan Performance
A table comparing performance across your test plans:
| Column | Description |
|---|---|
| Test Plan | Plan name |
| Total Runs | How many times this plan has been executed |
| Avg Pass Rate | Average across all runs of this plan |
| Latest Pass Rate | Most recent run's pass rate |
| Results | Passed/Failed/Total breakdown |
| Avg Duration | Average execution time |
| Last Run | When this plan was last executed |
Click a test plan row to filter the entire Analysis page to that plan's data only.
Filtering
By Test Plan
Use the dropdown at the top to filter analytics to a specific test plan, or select "All Test Plans" to see aggregate data.
By Test Type
Filter the test health table to show:
- All: Every test
- Auto: Only automated scenario tests
- Manual: Only manual tests
- Flaky: Tests flagged as flaky
- Failing: Tests currently failing
Test Health Table
The main table showing individual test health:
| Column | Description |
|---|---|
| Status | Mini sparkline dots showing recent pass/fail history |
| Test Name | Test name with "flaky" badge if applicable |
| Page | Source page path |
| Type | Auto or Manual badge |
| Runs | Total number of runs this test has been part of |
| Pass Rate | Percentage with progress bar |
| Failures | Total failure count |
| Avg Duration | Average execution time |
| Last Run | When this test was last executed |
Click column headers to sort ascending/descending.
Problematic Tests
Two dedicated sections at the bottom highlight tests that need attention:
Flaky Tests
Tests that inconsistently pass or fail. These are unreliable and reduce confidence in your test suite. Common causes:
- Timing-dependent assertions
- Tests affected by dynamic content (dates, counters)
- Race conditions in the application
Consistently Failing Tests
Tests failing 3+ consecutive runs. These typically indicate:
- A real bug in the application
- A broken test that needs updating
- A page that has changed since the test was created
Using Analysis Effectively
- After each test run: Check if new flaky or failing tests appeared
- Weekly review: Look at pass rate trends — is quality improving?
- Before releases: Ensure consistently failing tests are addressed
- For prioritization: Focus on fixing consistently failing tests first, then address flaky tests
Related Topics
- Test Runs — Execute test plans to generate analysis data
- Actions — Get AI recommendations based on analysis results
- Automated Tests — Fix or regenerate failing tests
- Dashboard — High-level project health overview