QA metrics from scratch (p.1)

How do you measure quality?

Most people think “quality” is something abstract, hard or impossible to measure. Some companies have huge QA teams, some outsource, and some don’t have QA at all. There isn’t one single metric that proves that the presence or absence of QA can drastically improve the quality of your app or speed up your release process. However, sometimes QA presence is absolutely indispensable, depending on the type of product, functionality, the company and the release process.

For companies that choose to have QA, it’s never been easy to figure out a way to consistently measure the impact testing has on the app’s quality or processes. Most companies, with or without QA, have a basic bug management process and something revolving around assessing users’ feedback. But there’s also the usual set of worries, like “ do we have too many bugs?” or “is testing slowing down the releases too much?”. And most of the time, there’s little understanding of how to measure what we’re doing and if we’re doing it the right way.

So, do you want more visibility on your QA work? Or do you want to know if you need QA at all? That’s the spirit! Let’s build some dashboards 🛠️

In this first part, I’ll cover the very first dashboard you could build and where you could “store it”. In the (hopefully) next part, we’ll get to which data we want to gather going forward (don’t build dashboards just because they’re pretty, sometimes the less the better!).

The first dashboard

The very first dashboard we’ll try to build will show us how many bugs we find in a pre-production environment (during testing) vs. production. In my opinion, this is the most important one, and this is why we’re building it before everything else. One of the first things you learn as a QA is illustrated by this iconic image:

Bugs! Bugs!! Bugs!!!

It costs exponentially more to fix bugs which are found later in the release process (the least expensive bug would be the one “found” during the project creation and specs, and the most expensive one is the one in production). Here, if the total number of bugs is 100%, the percentage of “testing bugs” to “production bugs” will show us the QA prevention rate. A graph will look something like this:

QA prevention rate — the GOAT

To be able to build this, a team should have a system to distinguish a production bug from a “testing” bug. It could be a label, a tag or, even better, a ticket type in your software tracking tool (like Jira). Jira provides us with ticket types for this: a “bug” ticket type, which can be created without a relation to any Epic (it just has to be in your team’s project), and a “defect” ticket type, which is easy to use during testing because it has to be linked to an existing task. Your team can completely personalise the description or keep it standard; the most important thing here is having a clear distinction between an issue which was found during testing and an issue which escaped. If your tracking tool hasn’t been used in this way and any issue created is of the same type, you can start adding a label “testing” or “production” to your tickets, and if you haven’t done any of this work yet, just take a batch of issues and tag them manually. Trust me, it’s worth it.

Implementation

The second important step of implementation of any type of metrics is “Where should I store/ publish them? Who needs to see them?”.

1. The “keep it on Jira” option — the easiest

The fastest and easiest option here is just to keep them on Jira (or other software tracking tool). In this case, first you need to create a filter:

• Search this: project = “YOUR PROJECT” AND “Team[Team]” in (YOUR TEAM) AND issueType in (Bug, Defect) AND created >= -90d ORDER BY created DESC (it might be slightly different based on your company’s workflow)
• Save this filter as “My QA Dashboard”, with privacy settings “visible to anyone from ‘my organisation’
• go to ‘Dashboards’ -> ‘new Dashboard’ -> ‘add gadget’ -> pie chart -> grab your saved filter -> select ‘issue type’ in ‘statistic type’ -> Publish

Congrats! You’ve just published your first QA Dashboard! 🎉

Why I’m not a fan of the Jira option

I knew straight away that I wasn’t going to stay on Jira 💁 As of now, the metrics it can display are very limited. Yes, on the plus side, you have a dashboard that updates in real time, with the information from the tool that you’re already using, so you won’t need to configure hooks or do any other setup options. But, on the downside, you will only be able to create 2 or 3 very basic dashboards that don’t show evolution in time, can’t use keywords or compare metrics. The only thing you’ll easily customise is colours 🤡

2. Google Looker Studio — good for MVP

I never used Looker Studio before this, but it turned out to be the perfect option to get the MVP. I knew I wouldn’t use it long-term because it wouldn’t easily get data in real time. It’s just a static dashboard displaying data from a CSV. But it was perfect for wooing the stakeholders. With its very intuitive design, similar to the other Google products, you can easily set up several graphs, show them to your team and say something like “if we want this data in real time, I’m going to have to look into proper implementation options” (you need to remember to look professional while saying this, otherwise it won’t work).

To start, you’d need to download data from your software tracking tool (Jira) into CSVs and then plug them into Looker Studio. To do this, create a filter in Jira (as described in the previous option). Then, export it as a CSV file. There are a lot of export options in Jira, and trust me, I’m yet to understand the difference.

If you got to this step, you can expect a surprise in this section. No matter how organised your colleagues are and how good your processes with the software tracking tool, you can expect a huge, messy CSV document with a lot of custom fields that nobody’s using. Don’t let it stop you! You can manually clean up the fields that you don’t need (which is what I did 😇), or you can ignore this and just add the document as is in Looker. I didn’t manage to properly “plug it” from my laptop into Looker, so I created an online version of the CSV and connected it to Looker Studio. It could also benefit you in the long term, as you can build automation that exports stuff once a day from Jira to the doc and then into Looker, and this way your dashboards are almost “real time” ✨

In Jira, you often get held back by your vision being incompatible with how their Dashboard page is “supposed to work” (the easiest explanation would be: their values seem to be hardcoded in the design, and it takes a lot of time and mental gymnastics to figure out if you’re able to display something you want). But Looker Studio seems to be easier to negotiate with and to conform to your vision. Remember: you’re the boss!

For example, how many revenue-critical bugs did we get in a given period? I’m going to set this up just based on the keywords used in the ticket titles. This could be a good example of some data you need to get quickly, but without a proper setup (no labels or tags have been added to track this, and this is why we’re using keywords). I’m going to start with the assumption that every ticket that contains the words ‘purchase’, ’payment’, ‘charge’ or ‘refund’ is revenue-critical, because people love money and hate to lose it.

The Looker Studio Dashboards work based on the fields found in a CSV and filters that you define yourself. This Dashboard, for example, works on the filter that I created with a Regex and added as a ‘dimension’ in the Chart Setup.

I’m painfully aware that there’s a risk of missing out on issues that don’t match this filter, but as a quick way to view your data, it can be very powerful.

3. Your own webpage using Jira API

This option is only worth exploring if your company has an app or hosts a website which is used for internal tooling or dashboards. Creating something completely from scratch would be overdoing it. Unless you’re really serious and passionate about metrics!

In my case, I created a subpage in our internal application, using Jira API. Here’s the tech stack:

• Backend: Node.js/Express API server
• Frontend: Vanilla JavaScript with Chart.js for visualisations
• Database: PostgreSQL
• Authentication: Passport.js with GitHub OAuth (can bypass for local dev)

Where do you start?

We’re going to start with deciding what we want our very first Dashboard to be. Here, I still want it to be ‘defects vs bugs’, but now I want 3 dashboards, side by side, for each of the product squads in the company.

First, we need to get the API token for Jira, or another Software Tracking tool you’re using. The “cleanest” way to do it would be to ask IT for a new Jira account, for which you’d generate the token. If IT is not keen on doing it or if you’re paying on a per-account basis, you can generate the token for yourself and then add it to the secrets vault. The obvious downside of that is that it’s linked to your company account and its permissions, so in case of change, it needs to be re-configured (trying to be vague here, but hope you understand what I’m getting at 💀).

Steps to get a Jira API token:

• go to https://id.atlassian.com/manage-profile/security/api-tokens
• click ‘create API token’ -> give it a name -> copy it straight away -> save it somewhere
• test it (there’s a very simple JS script that lists all of your company projects and keys if you run it locally — this will be useful later)

Just like with the Jira-generated CSV file, you’re more than likely to find out that your company has an incredibly twisted project structure. But don’t give up, just remember how your torturers (aka people who created and decided on those projects) were nice to you in real life.

Once you’ve figured out the IDs of the teams you’d like to see in your dashboard, you can build your page, referencing the teams in the code. I’ll spare you the details of my struggle with html, I reused the same design that was used everywhere else, but the backend API I created was doing the following:

• connecting to Jira Cloud REST API v3
• fetching bugs and defects for 3 teams using Team IDs
• calculating QA prevention rate: defects / (bugs + defects) * 100

The function fetching the metrics was declared this way:

Backend API

I didn’t want it to be slow so I decided to take results from the past 3 months (hence (daysBack = 90)). I also had to provide the script with all the details of the project, so make sure you get the ID of the project and IDs of teams as well!

In the end, the Dashboard looked like this:

A real-time QA metrics dashboard

Obviously, you can see a little issue there — the third team wasn’t using the same process as everyone else, and this is why we needed to wait for another 3 months to get the real picture (for some reason, nobody volunteered to edit tickets manually 😫).

Conclusion

Perhaps the most important part of this work is presenting it to people outside of your team. Most people wouldn’t know if 50% or 60% is good — they probably think 99% is good, and “we need to aim at 100%”. This is why this dashboard cannot exist on its own; it needs to be presented with explanations, and not just any explanations. They can’t be defensive or weak; they need to come from a place of pride and confidence. They can even state that it’s your logging that needs to be improved, not the prevention rate. A lot of bugs might be discussed over messages, without proper tracking (which often happens specifically during testing). This needs to be presented with all the context surrounding the work, but it should never look like it’s diminishing someone’s work or calling for competition. Sometimes, to understand a dashboard, you need to build another one. And sometimes, you need to scrap everything and start from scratch. In any case, if you’ve read this article until the end, it means you’re one step closer to understanding Quality in your company!