We can do better than this: why the overreliance on quantitative indicators needs to end

Asking for a friend: are indicators really all that?

Just like I’ve yet to come across a chocolate cake that I didn’t like, I have yet to come across a monitoring and evaluation (M&E) system that does not feature indicators. Indicators are widely used in M&E as a handy tool to measure different parts of a project’s results framework. Some M&E systems even feature very long lists of indicators, seemingly measuring everything that is measurable by means of an indicator.

And indeed indicators, when well formulated, provide a useful and reliable means to measure project achievement (or lack thereof), to reflect changes connected to a project, and to help assess the performance of an intervention.

But as handy as indicators are, are they really all that?

Let’s discuss.

[RELATED RESOURCE: Speaking of handy, here’s a free template for documenting your M&E indicators.]

Benefits of indicators

Indicators are key navigation tools that are used to provide insight on whether interventions are on track. In results-focused M&E systems, indicators enable project implementers to answer performance-related questions regarding progress and success.

• Example of a progress question: Are we moving towards the attainment of desired results?
• Example of a success question: Have the intended results been attained?

The values that indicators have at a specific point in time provide a snapshot of an intervention’s progress up to that point. Not only do they provide useful information on which to base program decisions, but indicator data can also support learning by illuminating how lessons are showing up in the data.

Indicators are also useful beyond results-focused systems, employed as they are to measure the project’s finances and other operations. Data related to these indicators can feed into the M&E system when it comes, for example, to assessing efficiency and other dimensions of value-for-money (VfM).

So indicators are extremely useful. So much so that they are easily overused.

Yet indicators not a panacea, and their limitations must be considered.

Notable limitations of indicators – and potential solutions to address them

Despite their great utility and widespread use, indicators are not the be-all end-all. Here are three key reasons for why this is the case:

1. Not all results are amenable to being captured in the form of an indicator. For example, it is sometimes difficult to capture nuanced and complex concepts by this means.
2. Furthermore, by their nature, indicators are only “representatives” of the phenomena in question; as such, changes in the data reported against them will not necessarily always be exactly proportionate to the true extent of movement.
3. In addition, unless specially formulated, indicators can only answer the “what happened?” question, not the “why did it happen?” question.

Furthermore, we must consider that the way indicators are typically employed represents a reductionist way of thinking that tends to be valued in the Western world but that is not always entirely aligned with indigenous and other non-dominant ways of knowing. This highlights the importance of meaningful, broad participation, a principle that should form part of any M&E system.

[RELATED RESOURCE: For more on the core principles that should form part of an M&E system, see step 2 in my free, downloadable 3 no-hassle steps towards a practical and effective M&E system guide. The guide is available via the TolaData site here (just look for the word “gift” in the first paragraph).]

Meaningful, broad participation helps ensure that the most appropriate measures for the project’s M&E system are selected.

Indicators could still feature in the menu of options, but consideration could be given to how to incorporate within them more nuanced phenomena and understandings. One example of a reasonable approach by which to do this is using rubrics that allow for a score to be assigned to a qualitative description of a phenomenon. The value of the score would vary according to the degree to intensity of the phenomenon in question, thus incorporating the more subtle distinctions that come with a qualitative evidence base within a numeric indicator.

Any remaining measurement gaps could then be filled by including non-indicator measures in the M&E system. You know that saying that goes “never say never”? Well, I’m going to ignore it and say that it is basically never a bad idea to supplement indicators with qualitative evidence and other complementary analyses.

Conclusion

It is said to be good practice to start off with a measurement system that is simple and quantitative in nature. However, robust M&E systems accommodate a broader range of measures, allowing for quantitative and qualitative indicators as well as for non-indicator measures.

Where indicators are truly appropriate, avoid the temptation of going overboard trying to measure everything that is easy to measure; rather, adopt the principle of parsimony. Then ensure that those indicators that do end up being selected meet the SMART criteria, which full indicator documentation can help assure.

[RELATED RESOURCE: Download my free template for documenting your M&E indicators.]

Then, where feasible quantitative indicators cannot be established, consider capturing phenomena of interest through more qualitative measures that reflect meaningful, broad participation.

[RELATED RESOURCE: See step 2 in the downloadable 3 no-hassle steps towards a practical and effective M&E system guide, which is available via the TolaData site here (just look for the word “gift” in the first paragraph).]

Taken together, this will help ensure the effective achievement of M&E system – and, more importantly, of overall project – objectives and ultimately enhance the prospects for people and the planet... the prospect for Life to be experienced in abundance.

The bottom line is this: indicators should never (oops, I said it again!) be employed and reported on in a vacuum but rather in the context of a larger results narrative that uses other forms of evidence gathered as part of a broader, ToC-grounded M&E system.

Over to you!

PRACTICAL TIP #1: Review your project’s measurement framework and assess whether there may be an overreliance on quantitative indicators. (A good rule of thumb is to ask yourself this: “Would our results story be significantly enriched with a broader range of measures?” If the answer is “yes”, that could be a sign of overreliance on indicators.)

PRACTICAL TIP #2: Adopting the core principles suggested in the 3 no-hassle steps towards a practical and effective M&E system guide, select the most critical indicators to retain in the M&E system. Then consider what additional measures could be used to complement these indicators in the telling of your project’s results story.

PRACTICAL TIP #3: Ensure that each of the indicators that remain in your M&E system are fully documented, including all the elements provided in my free template for documenting your M&E indicators.

Update of 28 April 2022: Please share your experience by commenting or leaving an emoticon reaction at the bottom of this LinkedIn post.

Photo credit:

Stevepb on Pixabay

Suggestion for how to cite this article:

Shejavali, K. (2022, April 11). We can do better than this: why the overreliance on quantitative indicators needs to end. Blog post. RM3 Consulting. Available at: https://rm3resources.com/blog/we-can-do-better-than-overrelying-on-quant-indicators (accessed: [insert the date that you last accessed this article at the provided link]).

Updated on 14 July 2022