Automate Accessibility Testing with Pa11y and Bitbucket Pipelines

To ensure our websites meet accessibility (a11y) standards, we automated testing using pa11y. The tool can be installed and used on the command line, so we could run it that way, and had in the past. But our marketing team can change content at any time on any of our sites and we want to be sure the sites remain accessible over time.

So we built a simple NodeJS project to run a11y tests on our websites every week, scheduled and executed using Bitbucket Pipelines. It’s really simple, I can’t believe we didn’t do this earlier!

Photo by Denys Nevozhai on Unsplash

First, we created a new NodeJS project and installed pa11y:

npm install pa11y --save

This is the only direct dependency in the project.

We then created a simple library with a couple functions to run tests:

const pa11y = require('pa11y');
// There are HTML and other reporters but we use simple text for the console
const cliReporter = require('pa11y/lib/reporters/cli');

// These are just some base options that can be overridden when running tests
const baseOptions = {
    runners: ['axe'],
    chromeLaunchConfig: {
        // Needed to run headless chrome.
        // Normally this wouldn't be secure but we're running tests inside
        // a bitbucket pipeline against our own websites, so no real risk.
        args: ['--no-sandbox']
    }
};

// This just runs tests on each URL in the array using the options and
// returns the total number of errors
async function runTests(urls, options) {
    let totalErrors = 0;
    for (const url of urls) {
        totalErrors += await testUrlAndLogResults(url, options);
    }
    return totalErrors;
}

// This runs a test on a single URL, logs the results, and returns the
// number of errors on that page
async function testUrlAndLogResults(url, options) {
    console.log(`Testing ${url}`);

    const results = await pa11y(url, {...baseOptions, ...options});

    console.log(await cliReporter.results(results));

    let totalErrors = 0;
    for (const issue of results.issues) {
        if (issue.type === 'error') totalErrors++;
    }
    return totalErrors;
}

module.exports = { runTests }

Now all we have to do is put together a quick script with the URLs we want to test. Our sites are pretty small and static. For larger sites this could easily be adapted to loop through an XML site map or some other dynamically generated list.

const lib = require('./lib');

const root = 'https://maxifiplanner.com';

const urls = [
    `${root}`,
    `${root}/how-it-works`,
    `${root}/how-monte-carlo-analysis-works`,
    // More URLs...
]

const options = {
    // pa11y lets you ignore certain rules that might not be relevant
    ignore: [
        'frame-title', // Frames are 3rd party, out of scope
        'frame-tested'
    ],
    // You can also exclude certain elements that may be out of your control
    // or for any reason
    hideElements: [
        '#some-element'
    ]
};

lib.runTests(urls, options)
    .then((numErrors) => {
        if (numErrors > 0) {
            console.error(`${numErrors} Total Errors Found!`);
            process.exit(1);
        }
    })
    .catch((error) => {
        console.log('Unexpected error encountered while running tests');
        console.error(error.message);
        process.exit(1);
    });

As you can see, if any a11y errors are detected, we exit with code 1 so our pipeline step will fail. The dev team gets notified of the pipeline failure in Slack and we can look into the a11y problem.

This script is named src/maxifi-tests.js and I can easily run it locally with node src/maxifi-tests

That’s great to clean up any errors that might exist. But to automate it, we need to create a simple pipeline script. The script below runs our MaxiFi site tests, along with similar test scripts for two other sites. The steps are set up to run in parallel so all three sites are tested quickly and we can easily see which site has a problem and fix it without re-running the other two.

image: node:18

pipelines:
  default:
    - parallel:
        steps:
          - step:
              name: Company Site Tests
              script:
                - apt-get update && apt-get install -y chromium
                - npm ci
                - node src/esp-tests
          - step:
              name: Maximize My Social Security Tests
              script:
                - apt-get update && apt-get install -y chromium
                - npm ci
                - node src/mmss-tests
          - step:
              name: MaxiFi Tests
              script:
                - apt-get update && apt-get install -y chromium
                - npm ci
                - node src/maxifi-tests

We turned on Pipelines in our Bitbucket repository for this project, committed the file, and boom! All sites are automatically checked for a11y errors.

Scheduling the pipeline to run every day or week takes just a few clicks.

So, what are you waiting for? I encourage you to go automate your own a11y tests this afternoon! Automated a11y scans like this of course don’t substitute for manual review and testing, but it’s a great additional check to help make sure issues don’t creep into your sites over time.

Thanks for reading, let me know if you have any comments or questions.