Transforming Monitoring, Evaluation and Learning Systems: A Call to Action

by Yulye Jessica Romo Ramos

March 19, 2024

Jessica uses a systems change approach to better understand evaluation systems and name the lack of equity, diversity and inclusiveness (EDI) in evaluation; the need to decolonise evaluation and learning while expanding its current focus on indigenous people and international development; and the culture and leadership gap. This thought piece is an output of the MEL in Systems Change Inquiry Group, hosted by the School of System Change in 2023.

Between July and November 2023, I was part of an international group convened by the School of System Change. It was a group inquiry on Monitoring, Evaluation and Learning (from now on referred to as “evaluation”) in systems change, with all of us co-facilitating the sessions.

My contribution was to use a systems change approach to better understand evaluation systems: its patterns and how structures, mental models and organisational cultures/capabilities influence how evaluation is practiced and its impact. Three main issues emerged for me as follows:

The lack of equity, diversity and inclusiveness (EDI) in evaluation
The need to decolonise evaluation and learning while expanding its current focus on indigenous people and international development.
And the culture and leadership gap

All of the above are interconnected. I unpack each issue below and hope to share thought provoking ideas that could transform evaluation systems. By doing so, I believe we can ensure evaluation adds real value to decision-making, learning, adaptation and system change efforts.

1. The lack of equity, diversity and inclusiveness (EDI) in evaluation

Systems change leads one to recognise indigenous knowledge as a source of holistic, relational and systemic practice¹. And once in this space, it is hard to ignore the Euro-Western supremacy over knowledge and consequently evaluation practice². Current structures, behaviours and mindsets have led to institutional and systemic racism and discrimination of non-white people and other minority groups³. Yet there is little recognition and action about these issues and the institutions that could otherwise be part of its transformation. This includes professional bodies like evaluation societies and the most reputable evaluation journals worldwide.

Take for example the Evaluation journal - an international journal whose mission is to promote exchange between European, North American, Asian and Australasian voices within the evaluation community. However, there is a remarkable absence of non-white, Asian or LGBTQ+ voices in its publications. When I analysed all 84 authors that published in this journal in 2022, I found that 93% of all of them are white, and that 67% of them were affiliated to a UK, USA/Canada or Australian-based organisation⁴.

Eight out of the top 10 most read evaluation websites belong to white males, and eight of those websites are affiliated to US-based organisations⁵. But change is possible, take for example the “Decolonising Evaluation” issue by the Journal of Multidisciplinary Evaluation, where approx. 20% of authors are from a Black heritage or minority group and 73% are women - sadly none of them were from Global South organisations.

2. The need to decolonise evaluation and learning while expanding its current focus on indigenous people and international development.

Decolonising evaluation discourse needs to go beyond its current focus on indigenous peoples and international development⁶ if it is to be localised and reflective of the minority groups found in the Global North. For example, in the UK, 9.3% of the total population are from Asian ethnic groups, followed by Black people with African and Caribbean ancestry⁷. And Roma are the largest ethnic minority in Europe⁸.

Decolonising evaluation is about:

Deconstructing the superiority and privilege of dominant groups, particularly Euro-Western knowledge systems and practice⁹, and addressing its inherent biases
Recognising and valuing diverse forms of knowledge and experiences, particularly from non-dominant groups
And restructuring evaluation values, approaches and tools based on local conditions and diverse cultural nuances¹⁰.

The UK and European Evaluation societies could help transform the sector by:

Developing an equity, diversity and inclusiveness (EDI) policy with accompanying actionable and resourced plans, aiming to lead by example and supporting the sector to change mindsets, behaviours and culture towards more EDI evaluation practice. This will need to focus on:

Ensuring its executive and management groups are more diverse as well as speakers they engage in events and publications. This should help improve representation and mainstream values and lived experience from non-dominant groups - including minority ethnicities, races, genders and those with disabilities and from less privileged socio-economic backgrounds.
Using power and convening spaces in ways that are more inclusive and equitable for non-dominant groups. This should help safeguard and support minority people and groups once spaces become more diverse.
Implementing accountability and response mechanisms (e.g. professional and effective ways to report and deal with discriminatory, racist and non EDI practices in the spaces they own and convene) to adequately address issues and drive real transformation.
Learning from other evaluation societies that have implemented EDI policies and practices such as the Canadian, American and some Australasian evaluation societies – with a focus on understand what has worked well and less well, and why.

Helping decolonise evaluation, with a focus on:

Localising the discourse, making it more reflective of the minority ethnic groups in the UK and Europe – effectively moving the conversation beyond indigenous knowledge and international development evaluation practice. This should also reflect lived experience from evaluators with disabilities, different genders and those from less privileged socio-economic backgrounds.
Designing appropriate professional development offers that go beyond the dominant norm. Examples include arts-based evaluation methods or how to value/use different knowledge systems.
Convening meaningful conversations across the sector – see table below for a range of questions that could enable group inquiry, collective action and learning – while also documenting and helping disseminate emergent knowledge and good practice in ways that are EDI.

Advocate for a revision of the current focus on rigid publication standards and privilege of Euro-Western academic rigour and experts, which fails to include and limit access for the majority of world practitioners with a different set of values, knowledge and approaches. Good examples include the Roots and Relations initiative by the Canadian Journal of Program Evaluation; Evaluation Matters—He Take To Te Aromatawai by the New Zealand Council for Educational Research (NZCER), in conjunction with the Aotearoa New Zealand Evaluation Association (ANZEA); and a new framework to reconceptualise rigour.

Identifying, mentoring and sponsoring minority evaluators, to help build the next generation of leaders that could disrupt and transform the evaluation sector.

At the collective and individual level, we can be part of the transformation by asking ourselves, and encouraging others to reflect on, the following questions:

View a larger version of the above tables

3. The culture and leadership gap

Transforming evaluation will go a long way but it is also important to address the cultural and leadership gap within organisations.

"Our current processes, incentives, and ways of working go against the grain of a stronger focus on systems, learning, and adaptation" - UNDP Strategic Innovation blog (May 20, 2022)

It is widely acknowledged that most organisations still lack a culture of learning and capabilities to recognise what good evidence looks like. This results in poor quality evidence and siloed, results-base management and accountability functions that do not support decision-making, learning nor adaptive management. All these contribute to a long-held perception that evaluation is not useful. Moreover, too often evaluators are asked to build this culture and practice, yet they often lack the seniority and power to do so, and I think it is time to ask more of our leaders and the organisational cultures they create and maintain.

We lack an overwhelming number of leaders who create demand for evidence and learning, even though they are best placed to design and implement organisational change plans, performance management models and bonus schemes that:

Create and sustain the enabling environments for more equitable, inclusive and diverse work practices.
Incentivise single, double or triple learning.
Reward building on good practice and on scaling up promising solutions.
Redefine reputational and financial risk to allow for more collaborative, contribution-focused and systems change practice to flourish – deemphasising success framed around an individual/team/organisation.
Allocate adequate resources and time to engage in systems change evaluation.
Invest in the professional development of evaluation staff to keep them up to date with the latest thinking and practice, or even ahead of the curve and innovating.
Create safe spaces where innovation, and learning from failure, can happen without negative impact to individuals and their ability to operate effectively.

Please help share this more broadly if anything resonated with you, and reach out if you want to add to this conversation or if you would like to arrange a chat with me and you/your team to reflect and discuss any of the contents in this blog.

Finally, I would like to thank Niki Wood, Julian King, Thomas Aston and Alan Hudson for the candid conversations we had in the lead up to me writing this paper and for the time spent reviewing an earlier draft. Your generosity was much appreciated!