Navigating Complexity:Combining Theories of Change, Typologies, and Rubrics for Effective Evaluation

Rubrics Are All the Rage

In the world of adaptive and/or politically informed Monitoring, Evaluation, and Learning (MEL), rubrics have gained a reputation as versatile and effective tools for tracking progress. By providing clear, consistent criteria for assessing performance levels, rubrics allow practitioners to systematically evaluate outcomes, even in portfolios where change unfolds gradually and unpredictably.

In a conversation with MEL practitioners who champion Thinking and Working Politically (TWP), rubrics emerged as a central tool. Their familiarity and adaptability make them valuable across contexts. For example, teachers can use rubrics for grading, while parents and students can assess school services through them.

Rubrics also help practitioners capture the nuanced ways political change unfolds, acknowledging that transformation rarely happens overnight. In a recent MEL project with Alix Wadeson, we designed a rubric to assess whether social accountability projects were progressing toward sustainability at scale. This rubric broke down a complex scaling causal pathway (resonance) into manageable levels, identifying key milestones along the way. We could map a complex portfolio of interventions across stages — think “wrong direction of travel,” the “right direction of travel but not quite there yet,” and “they’ve made it.” Especially when built on top of theories of change, rubrics can provide structure without being overly rigid, allowing for iterative adaptation as conditions evolve.

Sometimes Rubrics Are Not Enough

While rubrics are incredibly useful, they don’t fit every MEL task. Rubrics work best when the pathway to an outcome is (or can be made) “relatively” easy to aggregate, where “good” can be defined and progress measured in discrete, observable steps.

However, many portfolios don’t quite fit this mold. For example, I recently evaluated a portfolio operating in different organizational ecosystems, each requiring a unique approach within the collaboration spectrum. In some contexts, catalyzing tight partner collaboration was a priority, while in others, looser coordination proved more politically wise. These differing strategies — shaped by political realities — meant a single rubric couldn’t adequately capture progress.

Source: Adapted from The Collaboration Spectrum. Adapted from Tamarak Institute (2017). The Collaboration Spectrum. https://www.tamarackcommunity.ca/interactive-tools/collaboration-spectrum-tool. Image adapted by The Center for Implementation, © 2023. Version: V2024.01. https://thecenterforimplementation.com/toolbox/the-collaboration-spectrum

Moreover, portfolios often pursue multiple outcomes that interact in different ways. Imagine an organization or program simultaneously working toward preserving gains in one area, achieving fast positive change in another, and pursuing incremental progress elsewhere where the previous two strategies might interact negatively with the system. In such cases, relying solely on rubrics feels inadequate to tell the story of the portfolio — like using a screwdriver when you need a Swiss army knife.

When We Need Typologies

Enter typologies: tools for organizing and categorizing complexity. A typology groups cases or contexts into distinct categories based on shared characteristics, providing a descriptive framework for understanding complexity. Unlike rubrics, which measure performance, typologies aim to uncover patterns and classify phenomena according to conceptual dimensions to guide customized decision-making for subsets of challenges, such as:

• Classify types of performance or outcomes;
• Capture diversity across cases or contexts;
• Identify patterns or variations in responses, and;
• Evaluate interventions across different dimensions of complexity.

Example 1: Coe and Schlangen argue that ‘there is not a single path to Rome’ in advocacy campaigns. Instead, different actors play diverse roles, and these varying starting points can significantly influence their contributions. For instance, a lone advocate pursuing change independently will likely have different goals and expectations compared to an organization actively collaborating with others. It’s not about judging who ‘did better’ but about recognizing that their contributions differ qualitatively. Evaluating a lone advocate based on the standards of a collaborative team player could be inherently unfair, as their success is defined by different criteria. By understanding the unique role and context of each actor, evaluators can move beyond simply measuring the ‘amount’ of contribution and gain a more nuanced understanding of their impact on the overall change process.

Example 2: Deepak Bhargava and Stephanie Luce, in their book Practical Radicals, identify seven distinct strategies employed by progressive grassroots movements in the United States. These strategies, such as base-building, disruption, insider-outsider, electoral change and narrative shifts, each have unique characteristics and require different approaches to success.

Bhargava and Luce emphasize that ‘winning’ looks different for each strategy. For example, success for a base-building campaign might be demands are wholly or partly achieved and the organization gets credit for it, while success for electoral change may be about capturing governing power. These varying definitions of ‘success’ highlight the limitations of a one-size-fits-all approach to MEL.

Furthermore, these strategies can have unintended consequences or even interact negatively with each other. For example, a disruptive campaign might alienate potential allies, hindering the success of a parallel insider-outsider effort. This complexity underscores the need for assessment frameworks that go beyond one-size-fits all rubrics and consider the unique characteristics and interdependencies of different strategies.

Example 3: One can also use typologies of contexts in which a strategy is expected to work. In ‘Six Models for Understanding Impact,’ Liz Ruedy emphasizes that the nature of social change efforts varies significantly depending on the context. She introduces the ‘holding the line’ model, which stands apart from other models like ‘expanding opportunity,’ or ‘transforming systems.’

While other models prioritize achieving positive gains, ‘holding the line’ focuses on preventing negative change or maintaining existing positive conditions (i.e. keeping change at bay). For example, in repressive environments, ‘holding the line’ for an education intervention might involve ensuring that access to essential services for marginalized communities does not diminish.

Imagine a rubric design to assess a portfolio that includes both types of contexts. It may include criteria such as outcomes achieved and have predefined performance levels (e.g., excellent, good, fair, poor) with specific descriptors outlining what each level entails. But within this portfolio,

• Different Models Have Different Goals: “Expanding Opportunity” prioritizes creating new opportunities, while “Holding the Line” focuses on preventing negative change. A single rubric might struggle to capture these diverse goals effectively.
• Context Matters: The “best” performance level for a specific criterion might vary depending on the context. For example, an intervention using a “good” strategy for “Expanding Opportunity” in a closed society might achieve a different outcome compared to the same strategy in a more open society.
• Qualitative Differences: Success can manifest differently across models. “Excellent outcomes” in “Holding the Line” might involve a stable number of people accessing service delivery of poor quality, while that would be a poor outcome for “expanding opportunities”. An intervention that successfully prevents the erosion of access and quality of care in a restrictive civic environment might appear ‘unsuccessful’ if evaluated against metrics designed for a model focused on achieving rapid expansion of rights in a more open context. A single rubric might struggle to differentiate these qualitative differences.

A single rubric might help when the intervention seeks to move people from one type of state to another, but in many cases changing the context is not the goal being assessed. In many cases, the assessment is about crafting savvy strategies for each type of context to attain qualitatively different results. A single evaluation rubric cannot adequately assess the success of both ‘holding the line’ (defensive advocacy MEL is getting more attention since the last American Presidential election) and ‘expanding opportunity’ strategies. Attempting to do so risks misjudging the effectiveness of interventions.

Typologies and Theories of Change

Savvy readers will have noted the “easy” link between typologies and theories of change. By pairing typologies with theories of change, evaluators can articulate not just how to classify a situation but also why those classifications matter, linking them to specific strategies and outcomes. I was recently in a discussion with a service-focused donor agency that works in a broad range of countries. Considering different types of contexts, the team’s assumptions and expectations for the next strategy cycle included countries where they aimed for “keeping change at bay” and countries where they hoped to “shoot for the stars” in terms of access to quality care for underrepresented groups. In other words, their theory of change included types that made MEL aggregation across context, less than productive for accountability, learning and storytelling. Their assumptions grouped countries into different categories (e.g. according to different types of civic space) and those types where given. By surfacing these assumptions about how change is expected to happen, the typology can can enable MEL practitioners to tailor assessments to context-specific challenges while avoiding one-size-fits-all approaches.

As Collier et al. (2012) argue, typologies are especially useful for identifying configurations, connections, and patterns that might otherwise go unnoticed. And when a case does not fit a category, the typology might be iterated. Imagine most cases in a portfolio are neither fully open nor fully closed, but they are trending towards each extreme. A politically minded team may revisit its categorization and associated assumptions and aim for incremental progress toward greater desired change (or conditions for change) or interventions to slow or reverse harmful trends, respectively. In these cases, setting a MEL system sets the same benchmark across these contexts is not very informative. Averaging results out is not particularly insightful for assessing results nor for informing action. It is more useful to look into each category in its own terms. In fact, expecting positive change when the only relevant outcome is keeping change at bay may end up doing harm.

Why Combine Typologies and Rubrics?

What happens when a portfolio pursues multiple causal pathways and outcomes simultaneously? In such cases, a good approach may involve combining typologies and rubrics as well as a theory of change.

Here’s how it might work:

1. Use a typology to meaningfully categorize contexts into distinct groups (e.g., closed, closing, open, and opening civic spaces), guided by your assumptions or theories of change.
2. Apply a rubric to evaluate progress within each category, tailoring the criteria to the specific causal pathways and goals for that context.

For example, in a restrictive civic space, the rubric might assess whether the program has successfully “held the line”, ie. defended against negative changes. In an open civic space, the rubric might focus on progress toward fast, transformative outcomes. The descriptors in each rubric would be different. By combining theory-inform typologies and rubrics, MEL practitioners can account for the nuances of different contexts while still measuring progress along causal pathways toward multiple outcomes, as discussed above.

Going back to and building the Coe and Shlangan’s roads to advocacy typology example, to effectively evaluate these diverse contributions, one might take sequential approach:

1. Determine the actor’s role: What specific role did this organization play in the advocacy effort?
2. Assess the intentionality and quality of their pathways: Did they actively seek to pursue a pathway to change, or were their actions more incidental? What pathway was that?
3. Evaluate their performance within that role: A rubric can then be used to assess how effectively they fulfilled their intended role and pathway. However, this rubric must be tailored to the specific expectations and challenges associated with that particular role change-making road.

This approach allows for a dual focus: typologies help MEL teams understand the broader context and causal pathways, while rubrics provide a structured way to assess performance within those pathways.

The Right Mix of Tools for the Job

In politically informed MEL, the complexity of change demands tools that balance rigor with adaptability. As Apgar and Aston argue, no single tool can address the full spectrum of challenges MEL practitioners face when working across dynamic contexts with diverse stakeholders and shifting power dynamics. Bricolage is also everywhere.

In portfolios with multiple pathways, outcomes, or contexts, rubrics alone might not always be sufficient for a comprehensive evaluation. While rubrics can incorporate some contextual factors, they can become unwieldy when dealing with a high degree of complexity. For instance, if a rubric attempts to account for all possible variations in context and strategy, it may become overly complex and difficult to use effectively.

Typologies, on the other hand, can offer a more nuanced approach by grouping similar situations or interventions. This allows MEL to identify patterns and relationships that might be obscured by a single, overarching rubric. For example, a typology might categorize programs based on their target population, geographic location, or the political context in which they operate. This categorization can then guide the development of more specific rubrics tailored to each category.

Furthermore, typologies can be particularly useful when there is no single “best” outcome or when the ideal approach varies significantly across contexts. Rubrics often imply a hierarchical structure, with some levels of performance deemed “better” than others. However, in some situations, there may be multiple valid and equally effective approaches, depending on the specific context and the goals of the intervention. Typologies can help to identify and appreciate this diversity of approaches, recognizing that “success” can manifest in different ways depending on the circumstances.

An integrated approach between rubrics, typologies and theories of change can enable MEL practitioners to address the “what,” “how,” “under what conditions” and “why” of change in complex systems. It can provide flexibility for adaptation while maintaining a structured framework for accountability and learning. Ultimately, effective MEL requires a mix of science and craft. Reflecting on how we experiment and learn from bricolaging tools is key to navigating MEL’s own complexity.