Diff Analysis
The Diff Analysis step compares snapshots of your website over time to track changes.
Overview
Diff Analysis helps you understand how your website has changed:
- Which pages were added or removed
- Which elements changed on each page
- Whether test failures are due to real bugs or expected changes
- What new testing opportunities exist
Prerequisites
Diff Analysis requires at least 2 snapshots to compare. Snapshots are created each time you run element extraction.
Starting a Comparison
Selecting Snapshots
- Navigate to the Diff Analysis step
- Select the Base Snapshot (earlier version)
- Select the Compare Snapshot (newer version, or "Current")
- Click "Compare"
Snapshot Options
- Historical Snapshots: Previous extraction runs with timestamps
- Current: The most recent state of your site
Understanding the Results
Changes Tab
Shows DOM and element differences between snapshots:
Added Pages Pages that exist in the compare snapshot but not in the base:
- New pages added to your site
- Pages that were inaccessible before
Removed Pages Pages that existed in the base but not in the compare:
- Deleted pages
- Pages that became inaccessible
Modified Pages Pages that exist in both but have element changes:
- Elements added to the page
- Elements removed from the page
- Elements with changed attributes/selectors
Element Changes Detail
Click a modified page to see specific changes:
| Change Type | Icon | Description |
|---|---|---|
| Added | + (green) | New element on the page |
| Removed | - (red) | Element no longer exists |
| Modified | ~ (yellow) | Element properties changed |
For each changed element:
- Element type and selector
- What specifically changed
- Old value → New value (for modifications)
Analysis Tab
AI-powered analysis of what the changes mean:
Real Bugs Changes that appear to be actual problems:
- Broken functionality
- Missing content
- Error states
Expected Failures Test failures that are actually correct behavior:
- Intentional UI changes
- Feature updates
- Content updates
New Coverage Opportunities Areas that need new tests:
- New pages without tests
- New elements added
- New user flows
Running Intelligent Diff Analysis
For deeper insights:
- Click "Run Analysis" in the Analysis tab
- QAID uses AI to:
- Classify each change
- Identify test impact
- Suggest actions
Analysis Output
The analysis provides:
- Summary of changes
- Test results interpretation
- Recommendations for next steps
Use Cases
After Deployment
- Run element extraction after deploying
- Compare against pre-deployment snapshot
- Verify changes match what you expected
- Identify unintended changes
Investigating Test Failures
- Compare snapshot from when tests passed vs. now
- See what changed on failing test pages
- Determine if failure is due to code change or real bug
Tracking Site Evolution
- Keep snapshots at regular intervals
- Compare across longer periods
- Understand how your site has evolved
Managing Snapshots
Automatic Snapshots
Each element extraction creates a new snapshot automatically.
Snapshot Retention
Snapshots are kept indefinitely. You can delete old ones if needed.
Comparing Non-Adjacent Snapshots
You can compare any two snapshots, not just consecutive ones:
- Compare last week to today
- Compare before/after a major release
- Compare seasonal variations
Best Practices
Regular Snapshots
- Extract elements on a schedule (weekly, after deploys)
- Keep key snapshots (releases, milestones)
Meaningful Comparisons
- Compare against the last known "good" state
- Use deployment boundaries as comparison points
- Document what each snapshot represents
Acting on Results
- Update tests for expected changes
- Fix real bugs promptly
- Create tests for new coverage opportunities
Troubleshooting
No Snapshots Available
Cause: Element extraction hasn't been run enough times.
Solution: Run element extraction at least twice to have two snapshots.
Too Many Changes
Possible causes:
- Major site redesign
- Different authentication state
- Crawl parameters changed
Solutions:
- Focus on specific pages
- Verify crawl settings are consistent
- Compare smaller time windows
Changes Not Detected
Possible causes:
- Changes are in areas not crawled
- Element selectors remained stable
- Change is visual-only (use Visual Regression)
Solutions:
- Ensure all relevant pages are discovered
- Use Visual Regression for visual changes
- Check element extraction completed fully
Related Topics
- Element Extraction - Creates snapshots
- Visual Regression - For visual changes
- Bug Detection - Find bugs in current state