The satisfaction of engaging with users — our private beta experience

Hive IT have been working with the Department for Education (DfE) to redevelop the way they disseminate their statistical releases. The releases are a public asset; they inform, monitor progress and provide transparency over the education system. Currently they are released as PDFs on .gov.uk. Through our previous discovery and alpha projects we’ve worked to develop a new platform to make those statistical series, and their underlying data, easier to find, navigate and engage with to support their fundamental purpose. This new service will be called ‘Explore Education Statistics’.

the homepage for the new Explore Education Statistics service

It’s been a long process of user research and development, and we’re only just nearing the end. But I wanted to share our experience of the private beta phase for anyone looking for some inspiration, information or who just has a general interest, because it’s been an enjoyable eye opener for me.

“Private beta involves inviting a limited number of people to use your service so you can get feedback and improve it.” — GDS Service Manual: How the Beta Phase Works

Our system has a front end public facing service, and a back end administrators’ service where Statisticians can create their online releases. I’ll talk about how we’re undertaking our private beta phase for the public facing website in this blog. There will be a follow up blog at a later date about the administrators’ service, as we approached both slightly differently.

Our public facing private beta

Once we were fairly happy with the build of the public facing website (populated with some selective data sets), we released this to around 400 users covering all of our identified user types. Some of the users the DfE have engaged with from the very beginning of their discovery project, some are just interested parties who have expressed curiosity at various talks and across various platforms. Throughout the lifecycle of the project, with users’ consent, the DfE have been building up a list of people to invite to the private beta.

Thinking about private beta from the very start of discovery has meant that we had a strong database of users. Had we left it until the beginning of the beta project to start sourcing users to engage with, I think we’d have struggled to meet the number of users that GDS recommends.

“Hint 1: Think about private beta from the very beginning of any project. With users’ consent, start building up a database of people who are keen to be involved. That way, you aren’t scrabbling around to find people when you’re ready to move into this phase. Simple, but very helpful and timesaving!”

Kiah Peters, our user researcher, had a very clear plan of how we should be gathering feedback and engaging with users during private beta. We wanted to make sure that we allowed people to provide feedback in the way that best suited them. We know that across our very varied personas, some people would only want to give short feedback, while others would want to provide very detailed analysis. This meant we set up various feedback mechanisms.

example showing hotjar pop up on beta service

We’ve implemented Hotjar polls and surveys to gather quick, insightful feedback from users. This uses unobtrusive pop up panels on the site with parameters defining when to appear. Hotjar is also supporting our research by recording users sessions and providing heat map analysis. (It’s important to note that users were made aware of this software and how it was analysing their behaviours.)

We’ve implemented the GDS beta banner, allowing users to provide feedback through a more in-depth, but still short, survey via smartsurvey.

We also set up a very in-depth Google form, and in our initial instructions to users invited them to complete this after visiting the site. There is no restriction on the number of times they can complete the survey as we want to continually gather data as we make iterations.

We also have Google Analytics installed on the site, to measure against our defined Key Performance Indicators.

Finally, we have a dedicated mailbox here at Hive IT which the whole team are subscribed to. This allows for a free form feedback mechanism, with no set questions or format. This mailbox was given out to users in the initial invitations to the private beta, and updates are also sent from here on a regular basis.

“Hint 2: Multiple feedback mechanisms with varying degrees of requested detail means that users can engage at a level to suit them, making it more likely that they will provide feedback.”

During the private beta phase, we have regularly analysed the feedback we have received. This has been THE MOST FUN! It has been eye opening and really helped us understand our user needs in greater depth. Don’t get me wrong, we have engaged and tested and met with users throughout the phases of this project, but to have so many people jumping on the system, with no guidance or set tasks has really been exciting.

We’ve had really positive feedback:

“I thought it was great, it was very intuitive and the process and tables produced were immensely easier to understand than the Excel tables we currently produce.”

and really negative feedback:

“Poor and I cannot really see the point of a complicated interface to get slices of a table. I would much prefer to just have an index of tables and choose the data I wanted.”

And feedback asking for things that the service is not currently planning to provide due to business constraints or data protection issues. What I’ve found to be absolutely key, is responding to that feedback individually wherever possible. We’ve phoned or emailed people (where consent has been given), sometimes just to understand their feedback a little more, often to let them know that we’re addressing their concerns. And what I’ve found is that even the people giving negative feedback just appreciate the recognition that you’ve listened.

It’s a mammoth task, constantly reviewing and analysing the feedback that you’re receiving and you shouldn’t underestimate the amount of time you’ll need to dedicate to this. It’s exactly why having a multidisciplinary team with defined roles is so key. Our user researchers’ incredible planning around the private beta phase, the feedback mechanisms and the analysis process has been essential to making this possible.

“Hint 3: Engage individually with the users who’ve given you feedback — no matter what that feedback was. Users appreciate that you’re listening to them and in most cases, really do just want to help you build the best product, platform or service that you can.”

All the feedback we receive is combined into a centralised system (fancy word for a spreadsheet!) and we have weekly meetings to run through that feedback, make decisions and prioritise any resulting iterations. As a team, what we’ve come to realise is that in some cases, the feedback has to be viewed in a wider context. The temptation is to jump on every piece of feedback raised and think — “well, we need to change that.’’ But remembering that the feedback is from a single person among many is key. It will stop you instantly implementing changes, and instead thinking — “ok, how can we measure whether that’s a common feeling or issue”. We do that by undertaking spikes of guided user testing sessions that focus around one particular area. If the findings support the feedback, then we’ll look to introduce an iteration.

There are times when the same feedback is received from numerous users, in which case the data supports the change that’s being requested. In this instance, we’ll raise an item in our backlog and push it forwards as a change, then measure the impact through all the metrics we have in place. We make sure that all changes being implemented due to feedback are marked as such in our backlog against an epic ‘Iterations’ so that we can demonstrate our process when the time comes for our GDS assessment.

how our iterations are marked in Jira to keep track of them

“Hint 4: Make sure you’re measuring the impact of the changes you’re making based on user feedback. And make sure where possible you go back and ask the users who provided the feedback “Is this better — why?”

“Hint 5: Make sure you’re tracking the changes you’re implementing for GDS assessment evidence!”

Our private beta testers are integral to our service improvements. We need them to be engaged but we don’t want to overburden them. So rather than group emailing them every time we release an iteration, we inform them about a group of changes instead. And, if a month’s gone by when we’ve not managed to release any changes to the environment they’re testing on, we drop them an update email anyway, so that they know we haven’t dropped off the face of the earth! For example, we’ve spent some considerable time implementing changes that most users might not recognise following our accessibility audit and with the release of the new GDS design patterns. Whilst they might not be obvious changes, it’s still really beneficial for our testers to understand that we’re working to build an accessible system that’s inline with government standards.

“Hint 6: Regular group communication is useful, but be careful not to overburden your users with information.”

I mentioned that the analysis of the feedback has been really enjoyable, and thats because its just great to see how a service that you’re really proud of can always be made better with the help from a wider team — aka the users. The iterative design and development process means that you have the freedom to keep exploring, to keep developing and to keep trying different things. As a team we have always embraced this approach, as any agile team should, because one of our core values is understanding that “nothing is ever perfect” — there’s always room for improvement. I think to succeed in agile, you need to be brave enough to accept and enjoy that.

As we near the end of this phase, we have a backlog of post MVP development features that we know can enhance the service. But moving in to public beta isn’t about giving the users everything they WANT just yet — it’s about giving them what they NEED, then looking at how we can improve servicing those needs. I for one am excited to see where the users will tell us this service needs to go next. :)