Indicators are not the be-all and end-all of program performance measurement

Note: This article draws heavily on content from FSDA’s ‘Compendium of FSD indicators: Measuring the outcomes and impacts of financial sector deepening programmes’ (Shejavali, 2017).

Got (only) indicators in mind for program results measurement?

A question I often get, in one form or another, from leaders of nonprofit or international development programs is:

• “What indicators should I use to measure my program’s performance in [such-and-such] result area?”

The assumption underlying this frequently-asked question is that indicators are always the most appropriate – perhaps the only – means to measure progress or success in the given program result area. Given how useful and ubiquitous indicators are, it’s no surprise that many social impact program leaders automatically turn to them as the go-to monitoring and evaluation (M&E) tool for measuring their programs’ results.

But are indicators always the most appropriate tool?

I’ll cut to the chase: No.

This article covers what indicators are and aren’t good for. Let’s dive in.

Indicators are oh-so-useful…

Indicators: the handyperson of M&E

The dictionary definition of an indicator is “a measurement or value which gives you an idea of what something is like” (Collins English Dictionary, 2024). Building on this while taking a perspective that is more specific to measuring the performance of social impact programs, we can define an indicator as follows:

• “a variable that provides a reliable means to measure achievement, to reflect changes connected to an intervention, or to help assess the performance of an intervention” (Shejavali, 2017, p. 23).

As such, an indicator is an extremely useful tool in the M&E toolkit, viz.:

• “Indicators, if they are well formulated, can be viewed as key navigation tools that are used to indicate whether interventions are on track…; to assess whether targets are being met; to retain focus on intended results; and to hold implementers accountable. The values that indicators have at a specific point in time provide a snapshot of an intervention’s progress along its results chain up to that point, and they provide useful information on which to base learning and decisions.” (Shejavali, 2017, p. 23)

Thus, it’s understandable that nonprofit program executives, international development program team leaders, and leaders of other types of social impact programs might automatically ask which indicators are best suited when it comes to measuring the results that their programs aim to achieve.

Sidenote on frameworks for selecting and documenting indicators

A framework for selecting indicators

When indicators are the most fitting tool to use and when related data are available, I help my clients apply a structured approach to selecting appropriate indicators. Using my method, the indicators measuring my clients’ program performance are always aligned with the results areas in the program’s theory of change (ToC) (which forms the core of the M&E system). Furthermore, they are not only SMART, but also wise (e.g., respecting the principle of parsimony, seeking to avoid the creation of perverse incentives, etc.).

A framework for documenting indicators

A critical step following – or done in parallel with – the selection of indicators is their documentation. Fully documenting indicators is one of the most practical ways to enhance the robustness of a social impact program’s performance measurement, reporting, and improvement system while also enhancing accountability and transparency. If you’re curious about what exactly to document about each indicator, I’ve written about that elsewhere – see, for example, this LinkedIn Pulse article via which you can also download a handy template for documenting social impact program indicators.

[RELATED RESOURCE: Read the article ‘Is anything missing from your M&E indicator documentation?’, available here.]

…but indicators are not appropriate in every circumstance

But, whether fully documented or not, “[d]espite their usefulness, indicators are not a panacea” (Shejavali, 2017, p. 23).

Lack of related data

Sometimes indicators aren’t appropriate for the simple reason that related data aren’t available in the context of the specific social impact program in question.

• For example, the formal employment rate may be the most appropriate indicator to measure the impact of an initiative aimed at increasing contractually agreed work that meets specified labor standards in a certain zone. Yet, it could be that data on the formal employment rate in that zone are not gathered through the official statistics system or by other data producers, and the program may not have the means to gather that information itself. Alternative options, such as citizen-generated data, may not be feasible, either. And if data on substitute indicators also aren’t available, the program may have to drop the idea of using indicators to measure success in that particular result area.

No indicator data = no indicator.

“[N]ot all results are amenable to being captured in the form of indicators”

Where data are available, caution should be exercised when adopting indicators.

Indicators – quantitative indicators in particular (on which there is a tendency to over-rely) – can all too often be overly reductionist (Shejavali, 2022). Furthermore, measurement and analysis by means of quantitative indicators is largely focused on ‘going in with the intellect’. We have to accept that this means dissecting things and focusing only on what can be comprehended by the head, which is only one instrument of finding the truth. It, thus, reveals only a partial truth and, at least from my perspective, it’s important to remain open to other instruments of knowing or truth-finding, including more heart-focused approaches, which may be more intuitive, more natural to being human, and more reflective of our humanity (inspired by Sadhguru, 2014).

In short, “not all results are amenable to being captured in the form of indicators, which are challenged when it comes to reflecting [the] nuance and complexity” that is often prevalent at the outcome and impact levels of social impact programs (Shejavali, 2017, p. 23) (emphasis added).

• Picking up on the formal employment example used earlier, the formal employment rate, if it was available at an acceptable level of data quality, would provide insight on the “ratio of the [formally] employed to the working age population” (OECD, 2024). However, by itself, it would not tell us much about the characteristics of the employment (beyond that it is formal), the nature and quality of employed people’s experiences, the perspectives of employers, and other such details – and we would not expect it to.

Therefore, it’s important to keep in mind that indicators typically do not capture the full depth of the story.

Indicators are just that – indications of reality, not reality itself

“Furthermore, by their nature, indicators are only ‘representatives’ of [an underlying phenomenon]. As such, changes in the data reported against them will not always be exactly proportionate to the actual situation.” (Shejavali, 2017, p. 23)

Indicators often leave the “why did this happen?” question unanswered

“In addition, indicators [typically] only answer the ‘what happened?’ question, not the ‘why did it happen?’ question. … However, what…indicators can do with respect to the ‘why’ is help provide robust evidence for [social impact programs] when they carry out evaluations as part of their documented plan for appropriately assessing and estimating the attribution of observed changes to [the program]….” (Shejavali, 2017, p. 23 and p. 12)

Key takeaway on the limitations of indicators

The considerations above highlight why indicators “should not be used in a vacuum; they have to be reported on in the context of a larger results narrative that uses other forms of evidence gathered as part of [a broader, ToC-grounded, impact-oriented M&E system].” (Shejavali, 2017, p. 23) (emphasis changed)

A measurement framework that goes beyond indicators

Given the limitations of indicators, performance measurement systems for nonprofit, international development, and other social impact programs should be set up to account for evidence gaps that may need to be filled via non-indicator measures.

To help identify those gaps, a program’s measurement framework should specify the means of measurement and analysis in each of the program’s key result areas. A holistic measurement framework, like the ones I advise my clients to set up, will be aligned with the levels of the program’s ToC and will lay out not only the means but also, importantly, the purpose of measurement and analysis in each of the program’s key result areas as well as the main audience(s) of the evidence.

The process of developing a holistic measurement framework itself serves as an opportunity to consider the full range of measurement options in the different result areas. Depending on the nature and context of the program, the level and nature of the result area, the questions that M&E system-generated evidence is meant to help answer, data availability, and other considerations, indicators may or may not be the most appropriate measurement tools – and, even where they are appropriate, they may need to be supplemented by other sources of evidence.

• Outcome harvesting, for example, “has proven to be especially useful in complex situations when it is not possible to define concretely most of what an intervention aims to achieve, or even, what specific actions will be taken over a multi-year period” (Wilson-Grau, 2021).

In short, M&E methodologies (in particular, the tools and approaches used) will – and should – vary depending on the circumstances of each program. Different types of result areas, analytical questions, or contexts will require different methodologies, and that’s perfectly okay.

Putting measurement and analysis into perspective

Lastly, despite the usefulness of both indicator and non-indicator tools of measurement and analysis, keep in mind that measurement and analysis do not a comprehensive M&E system make! To wield the full power of M&E, the evidence that those activities generate has to be used to fully deliver on the M&E system’s objectives (and those objectives should, at a minimum, should include supporting accountability, informing program steering decisions, and adding credibility to external communications).

Recap – and then it’s over to you!

Indicators are not be the be-all and end-all of nonprofit or international development program’s performance measurement systems. Where feasible, they should be supplemented with evidence generated via other tools of measurement and analysis.

Take a look at your program’s measurement framework: Does it include a sufficiently holistic range of measurement tools given the context of your program? If not, what indicator and/or non-indicator measures might you include? (And, while you’re at it, consider which of the existing measures could be dropped, e.g., if they are not serving any specific measurement objective and/or if they are not able to generate quality data.)

And, if you’re ready to go even further to deliver on the promises of your program through robust M&E, share your needs with me on a NO-SALES research call. I’m running these calls for a limited time and would love to learn about your program performance measurement challenges, frustrations, dreams, and goals, so book now!

References:

Collins English Dictionary (2024) “indicator”. Web page. Collins. Available at: https://www.collinsdictionary.com/dictionary/english/indicator#:~:text=indicator%20in%20American%20English,-(%CB%88%C9%AAnd%C9%AA%CB%8Ckeit%C9%99r)&text=noun-,1.,speed%2C%20volume%2C%20or%20the%20like (accessed: 22 April 2024).

Organization for Economic Co-operation and Development (OECD) (undated) “Employment rate”. Web page. Available at: https://data.oecd.org/emp/employment-rate.htm (accessed: 23 April 2024).

Sadhguru (2014, February) Comment s’échapper du piège de l’intellect. Video. Isha Foundation. Available at: https://youtu.be/vADOTQXNbqo (accessed: 23 April 2024).

Shejavali, K (2017, November 2) Compendium of FSD indicators: Measuring the outcomes and impacts of financial sector deepening programmes. FSD Africa. Available at: https://www.fsdafrica.org/wp-content/uploads/2019/10/Compendium-of-Indicators-FSD-Outcomes-Impacts.pdf (accessed: 22 April 2024).

Shejavali, K. (2022, April 11). We can do better than this: why the overreliance on quantitative indicators needs to end. Blog article. RM3 Consulting. Available at: https://rm3resources.com/blog/we-can-do-better-than-overrelying-on-quant-indicators (accessed: 22 April 2024).

Wilson-Grau, R. (2021, November 2). Outcome Harvesting. Web article. BetterEvaluation, https://www.betterevaluation.org/methods-approaches/approaches/outcome-harvesting (accessed: 23 April 2024).

Photo credit:

Isaac Smith on Unsplash

Suggestion for how to cite this article (using APA 7 style):

Shejavali, K. (2024, May 6). Indicators are not the be-all and end-all of program performance measurement. Blog article. RM3 Consulting. Available at: https://rm3resources.com/blog/indics-not-be-all-end-all (accessed: [insert the date that you last accessed this article at the link]).