Unearthing the Inquiries to Monitoring, Evaluation and Learning as Systemic Practice

Having worked in change making spaces for twenty years, and being passionate about learning, I have always been interested and enjoyed the work of monitoring and evaluation; bringing cycles of learning to projects, teams, designing learning frameworks and doing my postgraduate doctorate in action inquiry.  It also however fills me with dread–the trepidation of funders asking for reports that seem to have limited usefulness for the work and it taking up so much time and resources; the perpetual feeling that you have to justify your work, like you are back at school getting marked if you have been good or not, even if you are the ones that know the context, the approach and the work.

At the School of System Change we are all about systems change learning of individuals, organisations and wider ecosystems. We seek to contribute to wider fields and the system of change making, not only helping to cultivate practice but also to raise questions that we might explore together to help evolve our practices. Monitoring, evaluation and learning (MEL), is one of those areas that seems to become a barrier to taking a systemic approach to change but therefore could also have the potential to unlock multiple other approaches and ways to do systemic practice. This is why we set out to convene a collective inquiry into MEL and systems change, to not only understand what our approach might be as a School but also inquire into the wider field.

We sent an invite out to the field to find a group of practitioners, we got over 100 applications - which in itself says what an interesting question this is to people. We found a group of six diverse practitioners, who purposefully spanned from different positions, from inside large donors, evaluation consultants, those closer to academia and those coming from a deeper practitioners lens, paying attention to questions of equity and power, those who start from systems and systems change approaches and those seeking to bring systems approaches into their MEL methods.

Barbara Schmidt-Abbey: PhD Researcher passionate about systems thinking and practice, evaluation, systemic policymaking, and learning and innovation. Learn more.

Brenna Atnikov: Principal, Reos Partners. Learn more.

Calin Brown: Impact Measurement and Management Lead, World Economic Forum. Learn more.

Fodé Beaudet: Principal, Learning Partner Edge. Learn more.

Yulye Jessica Romo Ramos: Founding Director, Nexus Evaluation LTD. Learn more.

Jewlya Lynn: Systems change facilitator. Learn more.

We came together to explore inquiry questions that we felt would help us take forward our own work.  Knowing that we were trying to balance where we are today with what was needed as well as exploring and stretching the edges or questions.

The purpose of this article is to get under the surface and to discover what the narrative is, from the starting questions to what emerged and what it tells us about the field and what we might consider doing next. I also acknowledge that the map is not the territory, that this article comes from the mappers, my vantage point, influenced by my inner landscape, perspectives and experiences as well as what I feel might be interesting to highlight, questions for the field.

The questions that came from the applications and the group started with everything from what is MEL in the context of systems change, what is its role, how might it support systems change and how do we bring a systemic approach to the work of MEL? 

As you can imagine we did not cover all of these topics or areas in our four few inquiry sessions, however by picking a few of these questions, driven by the participants' inquiries, based on their practice, some interesting conversations (and articles) emerged. The territory we did cover in the end took us under the surface, where we talked about what might be hidden or not addressed in MEL, or how it might open up learning rather than conclude it, questioning the purposes of MEL and touching into deeper worldviews and different perspectives of ourselves as practitioners.

In systems work getting under the surface, or questioning where a structure or pattern comes from, what mental model, perspectives or assumptions informs the strategies or actions we might take are fundamental systemic practices, drawing on frameworks such as the iceberg model or learning models that encourages triple loop learning, learning that questions the assumptions of why we do things.

Across our conversations, what perspective we bring to our MEL work kept on coming up - in a couple of places we started to map out the differences in these perspectives or paradigms and how they relate to systems practice. 

Perspective: A worldview, a particular philosophy of life, paradigm, mind-set or set of core assumptions about how the world works, how we make meaning of our lives. A perspective informs how we act. (Birney, 2015: 14)

Although I believe that there are multiple perspectives and paradigms, all with their nuances and differences, it can also be said there might be two framings that might help to see how we perceive the world, and what this means for systems approach, the ways we know and do MEL. The table below seeks to summarise or archetype some of these polarities* and differences.

*These are words drawn from our conversations and purposefully put in polarity of mental models to demonstrate the tensions. It can sometimes seem like a paradox to represent things as binary when seeking to embrace complexity - when actually these are just representations of extremes on a continuum, where there is blurring between them.  However if we seek to dance and embrace these differences it helps us to bring awareness to these extremes, that often drive conflict or false choices, and move beyond these helpful dynamics, to a “yes and” and find the source of integration and wholeness. Although Fodé also challenges the need to represent binaries as a way to surface tensions, is there a different way to get into the dance?

As we are beginning to see from this table, which is by no means complete, what perspective we might bring starts to raise questions for the orientation of MEL, its purpose, who’s involved and why.  Also it starts to raise the question about where we put the emphasis across MEL, is the focus on the monitoring, the evaluation or the learning.

To understand where these tensions have originated, and why they may need to be questioned, we might need to bring in another systemic practice, to not only look at the patterns of today but to see what patterns might perpetuate from the past and why they may need to be questioned. The field of monitoring and evaluation has some of its origins in the introduction of logical frameworks for project management, with the identification of outcomes that we can expect to see. It was and is still used as a method to support those who were funding or asking for the work to hold those accountable to the deliverables they said would result. With that also comes power dynamics of control, the extreme cases of which can be seen through colonial patterns of aid and development and across philanthropic systems, and - as Jessica highlights in her article - the supremacy, racism and discrimination that can be perpetuated in the system, where the ripples can still be felt across the field and why the call for decolonisation and DEI (diversity, equity and inclusion) in MEL is so critical. 

By looking at it from this perspective it helps us ask questions about the purposes of MEL, which Fodé also raises in his article - the difference between MEL (or even game playing with his sons) as seeking a finite answer instead of an infinite process of learning.  We might also see the purpose of evaluation being about making judgements, which seeks to be done from a perspective of professional, academic, objective-seeking, trying to fix the system rather than an emphasis on the adaptation, driving a feedback process, knowing how to shift and respond differently, bringing of awareness and perhaps inner judgement or reflection on what has been happening.  

Seeing “MEL as the beginning of a conversation rather than the conclusion of a process evaluating predetermined outcomes,” and goes on to conclude, “maybe it’s less about choosing finite vs infinite games. It’s about recognizing that we all interact in finite games one way or another - the key is to keep in mind the bigger game. Life as an Infinite Game." - Fodé 

The implications of this is it also starts to raise questions about who it is for–the funder or management team or those who are on the ground acting and learning. Whose voices and who is involved in the work of MEL? Are we just perpetuating dynamics of the past or wanting to shift to a new paradigm? What power dynamics are we perpetuating?

Although the dominant paradigm and therefore where resources flow is often seen as the first column, due to its connection to these historical patterns, it can be easy to critique it and throw it out. However these two archetypes are only a polarity and as such seeing if we need to choose which one is also a false dichotomy. There is obviously value in both orientations and perspectives and we need to find and look at what these are. Jewlya in her article and in the discussion got me thinking about how we cannot throw the baby out with the bathwater. Robust learning processes (and ideally deeply participatory approaches to learning) are critical, yet may not be enough. We need to ask how a learning process helps us to “make visible the most hidden parts of the system,” which may include soliciting insights using evaluative approaches.

The concept of taking a bricolage approach appeals to me here (and was a new term), mixing and matching of multiple methods, and triangulation across questions at multiple levels in the systems we are looking at. Participants of the group helped show where they were doing this in their practice, as Jewlya at the end of her article invites in her framework  - or the need to also span boundaries as Fodé suggests. There is a need for us to both monitor rigorously, draw evaluative conclusions and insights from the patterns we collectively see, beyond those who are holding (and acting on) a shared set of assumptions about how change happens and take a continuous collective learning approach. All of this whilst questioning the power dynamics and the perspectives we bring.

Brenna’s article calls for us to really value this collective learning in systems change, to “harness the full potential….in the face of the challenges we collectively confront” whilst Calin’s question and practice is grappling with how to bring in more of a learning paradigm and way of doing MEL into an organisation that (I assume) values accountability and reporting more , and resonates for me with many systems practitioners in organisations I work with trying to operate from a systems perspective.

How do we not get trapped by a more traditional paradigm (on the left hand side of the table) but use its value and recognise we live in a world that is complex and unknown? Here we talked about how we might shift the soil, or the deeper perspective from which we do all MEL so when you are looking at accountability, analysing you are doing that with this bigger awareness. How do we value both and not see it as an argument to win between which approach is best but acknowledge the value and role each has to say

We might also need to acknowledge that investment is needed in the more learning oriented approach, as it is not about short term reporting but about embedding the capacity across the systems to have agency and accountability to the learning. Resources, time and money, in our current organisations and change systems need to be placed here.

As someone who is interested in building the capacity of individuals, I think one of the most poignant points was raised in our last session - through exploring Barbara’s inquiry- “you can only evaluate from the perspective you are in”, the person really does matter and the position they are taking in the work.

How do we bring awareness and reflexivity about who is holding the power to learn, monitor and evaluate? Who decides what we are valuing and can we step back and see our own bias, positionality, assumptions and at least make them explicit whilst also acknowledging we are all caught in these systems. For me it also raises the question and a potential for an unlock: How might evaluators and learning facilitators hold multiple perspectives and worldviews? And how might holding multiple worldviews bound or expand collective learning?

So through being a part of this group, listening, contributing and learning myself and attempting to weave a narrative where does it leave me? It leaves me reflecting on again the extent to which opening up and supporting those people and organisations grappling with questions of bringing a systemic approach to MEL, or looking at how we evaluate systems change, could really unlock the whole change systems from the trap of our current dominant paradigms? How do we continue the conversation, the action inquiry with others, in this space?

We are committing ourselves at the School to evolving how we monitor, evaluate and learn, taking the learnings from this inquiry into our own practice including understanding equity and decolonisation. We commit to having reflexivity in who we are and how we perceive the learning and evaluation, ensuring we are uncovering what is both hidden and looking at zooming out to understanding the context of our system and weaving and spanning learning across the field, learning collectively when we think of systems change and MEL.

We are interested continuing the questions this group has started as well as those that you hold:

If you would like to join a group or stay updated please do express your interest here. 

This work was supported by our partnership with the Johnson & Johnson Foundation, to help understand our own approach to MEL and to contribute to the field. I am deeply grateful for those who participated, those who also applied and also for what might come next. I acknowledge that this article is only that of my reflections and not a full representation of the group - as such I encourage you to read and explore the other articles - as well as the resources we shared between us as we inquired.

Centre for Public Impact's work on Human Learning Systems can be an example of a resource to deepen the role of learning to reflect your thinking, whereby the key is not to scale ‘impact’ but learning - the capacity to learn is what contributes to embed/influence meaningful change. Available at: https://www.centreforpublicimpact.org/insights/human-learning-systems-a-practical-guide-to-doing-public-management-differently

Aston, T. and Apgar, M. (2022) The Art and Craft of Bricolage in Evaluation. CDI Practice Paper 24. Brighton: Institute of Development Studies. DOI: 10.19088/IDS.2022.068. Available at: https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/17709

How do we measure systems change? Available at: https://medium.com/@undp.innovation/how-do-we-measure-systems-change-26af77dc4fb8 

How to do M&E when you’re working with complex problems. Available at: https://medium.com/@undp.innovation/how-to-do-m-e-when-youre-working-with-complex-problems-b91622ebc9b2 

Towards systemic evaluation in turbulent times – Second-order practice shift. Available at: https://journals.sagepub.com/doi/full/10.1177/1356389020909120 

TWG – Systems Approaches in Evaluation. Available at: https://europeanevaluation.org/twg-8-systems-approaches-in-evaluation/

Journal of MultiDisciplinary Evaluation – Special Issue: Decolonizing Evaluation: Towards a Fifth Paradigm. Available at: https://journals.sfu.ca/jmde/index.php/jmde_1/issue/view/77