Can AI improve NGO Programme Processes? We put it to the Test.

Ross Angus, with Megan Cruickshank & Grace Thuo

Image: AI generated

AI constantly grabs the headlines. Those headlines tend to fall into two camps: the existential (job losses, cognitive decline, data centre projections), or the persona (tracking which tech mogul is riding high this week). But beneath the noise, nearly every organisation is quietly working out where AI can genuinely make a difference — can it have a positive impact on some challenging problems? Can it remove some of the drudgery from daily work routines?

These questions apply to INGOs as much as to the private sector, though likely with fewer resources to deploy. Over the last year, GAIN has worked on how AI can move the dial to advance nutrition outcomes by improving the consumption of nutritious and safe food for all people, especially the most vulnerable to malnutrition. Alongside our programmatic work (which you can read about in a previous blog here), we’ve been running a series of operational experiments and this blog shares what we’ve learnt from those.

Our first step was to recognise that doing AI ‘well’ would be as much about the people at GAIN, as much as the technology itself. We therefore started with external training for all our staff in 2024, led by Innovation Specialist and AI Trainer, Ross Angus of Innovatable.

Our second step was to create an AI policy to guide staff in basic do’s and don’ts and essential issues of data security. We found that staff were in very different places when it comes to AI: confidence, knowledge and interest varied widely. There was a real appetite to experiment, but also a nervousness about diving in without guidance, expert input, or an agreed way of working.

Our third step was to explore AI solutions to a specific set of challenges. Innovatable ran a series of workshops with five Operations Teams: Finance, Legal, Comms, IT and HR. The overarching question we wanted to answer was simple: can AI meaningfully support our Ops teams in addressing real, day-to-day challenges.

This is the focus of this blog.

From the outset, we were deliberate about where we focused: we wanted use-cases that genuinely affected daily work, where an AI-based approach could make a real difference. For each experiment, we defined a threshold of ‘good enough’ and that, as with any innovation project, we were open to the fact that the answer to that question might be ‘no’. We call this ‘invalidating our hypothesis’; a valid outcome that still offers useful learning.

The five Ops Teams used an overarching framework for experimenting with AI, and we’d recommend this to others.

1. Start with a problem: the first workshops gathered problems from the team before ranking them based on how many people were inconvenienced by each, ensuring we were testing something that would have a genuine impact if validated, rather than just throwing AI at anything.
2. Plan metrics before you build anything: We used the ‘Build, Measure, Learn’ framework when creating prototypes. It ensures that you are only putting energy and time into building prototypes for which you can objectively measure the outcomes.
3. Learn quickly, cheaply and safely: The more specific your prototype is, the faster it is to create. ‘Cheaply’ can mean two things: financial outlay and time. We tested using pre-existing Copilot tools, acknowledging that whilst some specialist AI tools might be more effective, it’s important to test with tools to which everyone already has access. On the safety point, we used fictional data in early tests, or created isolated test areas, such as specifically permissioned Sharepoint sites, never moving into real data before initial validation.

With that framework in hand, here is what the five teams went and tested — and what happened when they did.

Finance

● Could an AI agent uncover misnamed files based on natural language (for example: “I’m looking for a file that is the most up to date, signed contract with our partner X on the Y project”).

IT

● How to measure the efficacy of an existing IT chatbot, designed to solve routine IT queries all around the organisation before escalating to our Managed Service Provider (MSP). We created clear metrics for what we considered a helpful answer and therefore were able to measure how often the chatbot delivered value.

Legal

● Could we minimise the error rate of internally created contracts, up-skilling the team through tutelage. (Note — we were not testing AI within the contract creation itself but instead experimenting whether it could guide us to create a more robust system).

HR

● Could we could build a prototype to track rapid changes around the organisation, allowing everyone on the HR team to follow updates.

These four had a range of outcomes. The least successful gave us 58% accuracy with fictional data. The AI Agent capably offered ‘mostly’ correct answers almost every time, but unhelpfully, 42% of the time sprinkled in additional inaccuracies. We deemed that not good enough and discontinued this work.

The fifth use case was the most promising: whether we could use AI to improve our storytelling: Communications.

The Comms team is very aware of how many impactful stories are witnessed every day in GAIN’s work across 16 countries, but many of these stories never get retold. This matters for organisational narrative, to share and spread good practice and to influence the broader conversation around nutrition. So, we set about to see whether we could change that.

There were two working assumptions here:

● Colleagues don’t always realise that the incredible work they just witnessed (or indeed helped to create) would be inspiring to others, if shared.

● Colleagues don’t feel as though they have a large enough contribution to share with the Comms Team — but these nuggets are exactly what the Comms Team would like to hear.

To test these assumptions, we built an agent that guided users through a brief series of questions. Behind the scenes, the Comms Team’s existing storytelling guidelines were the knowledge base providing the framework for responses. This meant that anyone around the organisation could quickly bring their story to life. Launching initially with a select number of colleagues we gathered feedback before making tweaks and sharing more broadly.

This test is still ongoing, but our ultimate hypothesis is that stories discovered and shaped through the agent will lead to better external engagement with GAIN’s work in these areas — because that’s the power of a good story, but only if it gets told.

Building a thriving AI framework

At GAIN we are continuing to explore how AI could support our operational work. Working with Ross, we have identified the following ‘necessary conditions’ for AI work to thrive in our Operations teams;

· Technical expertise — who in your organisation is best placed to support AI? It may or may not be your IT team. The problem ‘owner’ matters for problem solving.

· Manage the ‘feeding frenzy’ — AI has the capacity to generate an appetite for activity everywhere all at once. But before you get started, ask yourself, can we? And then, should we? Avoid a free-for-all, not least because of time​ ​and data security. You will need some guidelines and standards to test success with rigour.

· Innovation as a way to de-risk development — an effective AI innovation process will test small scale assumptions in an iterative way. You learn what is working and what is not, so that when you scale you have the data to show that the product is filling a gap.

· Start with a clearly articulated problem then generate ideas and test the most promising, then decide whether to ​​shelve​​ or scale. In so doing, be led by your colleagues — AI will add value where it addresses real problems.

· Consider cost implications. During small scale experiments you are unlikely to use many tokens or need to invest in multiple licenses. However once rolled out to entire organisations, the cost of licences, tokens and staff time can rise.

A caveat: AI development moves fast, and some of the experiments we have touched on here would have different outcomes if we, or you, were to re-test them today. That’s why it’s so important to have metrics in place allowing you to objectively measure the outcome of your experiments.

We hope that, by reading this, you’ll not only consider what is possible, but that you will also be inspired to identify problems in your own teams and organisations that, with thoughtful experimentation, could possibly be solved with AI tools.

We are grateful to the Gates Foundation for their support for this work. The Foundation has no responsibility for the findings.

Bio: Ross Angus, founder of Innovatable, has spent the last two years guiding non-profits around the world to understand how AI may be able to support their mission. With a healthy dose of scepticism and innovation frameworks, Ross helps teams build confidence and understanding through thoughtful experimentation. If you’d like to discuss any of what you’ve read with Ross, he’s happy to receive messages here: ross.angus@innovatable.co.uk