This is an independent editorial comparison. No sponsor paid for inclusion. The aim is to help teams choose between two open-source accessibility scanners for CI integration.
Methodology
We ran axe-core 4.10 (@axe-core/playwright) and Pa11y 8 against three
fixtures: a static marketing landing page, a logged-in dashboard, and a
PDF report viewer embedded in an iframe. Each tool was given default rule
sets without per-rule tuning.
Rule coverage
| Rule family | axe-core 4.10 | Pa11y 8 | Notes |
|---|---|---|---|
| WCAG 2.1 AA | full | full | Both implement the same normative rules |
| WCAG 2.2 AA | full | partial | Pa11y trails on 2.4.11/2.4.12 target spacing |
| Best-practice | extensive | minimal | axe-core ships ~30 best-practice rules |
| Color contrast | strict | lenient | axe-core flags more borderline cases |
| Live regions | full | partial | axe-core checks aria-live patterns more deeply |
| Forms | full | full | Equivalent |
axe-core’s broader rule set yields more findings, including some that require human judgement to triage. Pa11y is more conservative and produces fewer false positives in our fixtures.
CI ergonomics
| Concern | axe-core | Pa11y |
|---|---|---|
| Test runner | Playwright, Cypress, Selenium adapters | Standalone CLI + JSON output |
| Baseline support | via custom matcher, not built-in | built-in --threshold |
| Output formats | JSON, HTML reporter | JSON, CSV, JUnit, HTML |
| Headless | yes (Playwright) | yes |
| Authenticated routes | via Playwright fixtures | via custom action scripts |
Both integrate cleanly into GitHub Actions. axe-core wins on test-runner breadth; Pa11y wins on baseline ergonomics out of the box.
False-positive rate
Across our fixtures, axe-core surfaced 14 findings on the dashboard; manual review confirmed 11 as genuine, 3 as ambiguous. Pa11y surfaced 7; all 7 were genuine. Both rates are reasonable; axe-core’s broader rule set trades signal for additional triage.
Documentation
axe-core’s documentation is more comprehensive (rule descriptions, WCAG mapping, remediation hints inline in JSON output). Pa11y’s documentation is concise and adequate for CI integration but thinner on remediation guidance.
Recommendation
For most teams: run axe-core as the primary CI gate (zero violations on new routes; baselined legacy) and Pa11y as a secondary advisory check. This matches the recommendation in our axe-core CI guide.
For PDF-heavy compliance work, supplement with IBM Equal Access, which has stronger PDF/UA coverage than either tool above.
Cross-links
- How to wire axe-core into CI
- How to test with NVDA for the manual layer that automation cannot replace
- axe-core source; Pa11y source
- IBM Equal Access