Sharing her reflections about the inquiry group and her role as a MEL practitioner, Barbara explores what it could mean to apply second-, and third-order shifts to the evaluation (MEL) ‘system’ itself. This thought piece is an output of the MEL in Systems Change Inquiry Group, hosted by the School of System Change in 2023.
During our joint inquiry into Monitoring, Evaluation and Learning in Systems Change, convened by the School of Systems Change during 2023, a diverse group of seasoned MEL practitioners came together to jointly explore some of the many questions that were curated as part of the process. The collective inquiry provided an opportunity for us that participated in this process to deep-dive into just a few of the range of topics that we and others that were part of this journey had identified.
This blog covers two parts: first I share my own reflections about the process of our joint inquiry, and I then write about my own personal inquiry within this overall inquiry, where I look at the centrality of the MEL practitioner herself - so please read on till the end!
What I took away from this process
What surprised me personally throughout the entire process spanning several months during 2023 was the broad-ranging diversity of experiences, perspectives and traditions the practitioners involved are coming from into this shared space, which exceeded my imagination! Put on an (imagined) continuum, the experiences ranged from those coming into this journey from a traditional Monitoring, Evaluation and Learning (MEL) context, for example, people like myself working in internal evaluation functions within organisations that are involved in various operations that are subject to monitoring and evaluation (and learning), ranging from research to philanthropy and advocacy, to practitioners in systems change on the other end of the spectrum.
These "traditional" practitioners (like myself) are deeply rooted in MEL practices and assumptions that come from a conventional project and programme M&E tradition moulded by conventional ‘Results-Based Management’ cultures, guidelines and standards that underpin the evaluation practices in the organisational contexts they operate in.
Like in my own case, many practitioners who ‘grew up’ in these linear, reductionist RBM infused cultures have grown increasingly uncomfortable with the underlying mental models predicated on ‘measuring impact’, oftentimes with a heavy bias towards quantitative Key Performance Indicators (KPIs), and traditional ‘logic models’.
In my own journey, I have become increasingly uncomfortable with these approaches over time, and striving ‘beyond’ these limiting framings.
Encountering systems thinking and complexity sciences constituted a welcome relief and alternative framing for me on the one hand and a growing frustration on the other hand why such approaches are still not sufficiently well known and adopted in mainstream M&E practices - despite espoused interest and curiosity about such approaches, the adoption of such ways of working in MEL practice seems to lag woefully behind the rhetoric advocating for such approaches.
Others joined this quest from a very different starting point, such as those working with communities and beneficiaries of systems change interventions (for want of a better word) directly.
To my pleasant surprise, I learned from other practitioners in the MEL Inquiry Group coming from a quite different tradition and understanding of what is coined as ‘systems change’ - at the opposite end of the spectrum. I learned from them that quite a few systemic MEL practices exist in these constellations, bringing a very different repertoire of practice to the inquiry. In these practice constellations - often using participatory, community based ethnographic approaches (in contrast to the RBM-based practices dominant in my own professional context), learning cultures appear to be deeply rooted and more mature in such learning communities - whereas in my previous experience, the L in MEL was often an afterthought, and even E (evaluation) often playing second fiddle to the overpowering M (Monitoring with quantitative indicators).
Depending on the initial starting conditions and personal / professional trajectories, the assumptions, worldviews and contexts, and therefore experiences of those involved in the collective inquiry proved to be quite different from each other, and this was thought-provoking and assumption-busting. Thanks to the joint inquiry process and experience, we developed a certain shared repertoire of concepts and understandings, and the sharing of these different experiences and worldviews contributed to a more common and appreciative inquiry stance to jointly explore the inquiry questions each of us had chosen to focus on.
My own inquiry within this inquiry: zooming into the centrality of the MEL practitioner
My own inquiry focused on the individuals involved in MEL practices themselves, guided by the question of my previous inquiry I held for some time, which gave rise to my still ongoing PhD research: what does it take for MEL practitioners to BE and ACT in a volatile and uncertain world, when confronted with ‘wicked problems’, messy systems, systems change and complexity?
This inquiry was initially expressed in the article Schmidt-Abbey et al. (2020). By this proposition, our authoring team (myself and co-authors Martin Reynolds and Ray Ison) suggested that those involved in MEL practices need to be able to make a ‘second-order practice shift’ - evaluation practitioners need to become more systemically aware and complexity-sensitive when conducting evaluations, by moving towards a ‘second-order’ evaluation praxis.
What is meant by ‘second-order practice shift’, and why does this matter?
In the 2020 article, we explain this as follows:
As MEL practitioners, we can choose to approach thinking about our practices in either ‘first-order mode’, or we can decide to take a ‘second-order’ approach to viewing ourselves as practitioners in relation to the situations we are engaging in, and are inevitably part of.
What is the difference, and why does this matter?
- In first-order mode, a practitioner can approach an evaluation assignment based on the widely accepted theory that performance of an intervention in a situation can be measured with (quantitative) KPIs, for example to ‘measure the impact’ of an intervention, to evaluate it as being successful or having ‘worked’. She may therefore decide to use the same standard methods and tools that appear to have worked successfully in previous situations, are considered as ‘good’ (or even ‘best’ practice), and help to ‘get the evaluation job done’, i.e. to deliver an evaluation report that is accepted by the evaluation commissioners / donors, and subsequently paid for. This might be viewed as a systematic, first-order approach.
- Alternatively, in second-order mode, the practitioner might approach a situation as one where the MEL practitioner recognizes that they are themselves involved in the situation they evaluate or monitor: this is an intervention in its own right which changes the situation, and has consequences for it. This implies an ethical and moral responsibility to other people or ‘stake’-’holders’ (even those that are not directly involved or represented) that are affected by any intervention. The practitioner may therefore opt for a ‘systemic’ frame: to enter the evaluation from an ethical stance, acknowledging the wider consequences of the work that is done (or what is omitted), and viewing the role of the evaluator as one of a craft artisan / bricoleur, rather than a technician.
- In a first-order tradition, we tend to conceive ‘objects’ of study as fixed entities which we can study and measure objectively (fixed objectives, targets, goals, linear causation (e.g in a logframe).
- In a second-order tradition, we emphasise the experiential and relational understanding of practitioners engaging in situations we ourselves part of (not as distant observers).
- We engage in reflexivity about a reality that includes the evaluator. (Ison, 2017: 278-282; Schmidt-Abbey et al., 2020: 215).
The Practitioner-Framework-Methods-Situation (PFMS) framework (Ison, 2017:50) offers a useful heuristic for this distinction, as depicted in this diagram:
In first order mode, the practitioner (P) (in the inner ‘bubble’) is shown with her own mental models (here expressed as ‘framework (F) of ideas / theories’ - shaped by her tradition of understanding - which constitute the framing choice for MEL the practitioner applies to a given situation (S), which in turn influences which the methods (M) and tools are being considered to be appropriate by P to draw on for engaging in evaluating a situation (S).
In my continuing PhD research, and also explored in this present inquiry, we jointly engaged with some of these specific questions:
1. What DO evaluation practitioners actually DO when they DO what they DO? (a second-order question (as proposed by Ison (2017:5, based on second-order cybernetician Humberto Maturana’s proposition for the need of human beings to live life to braid language and emotion). - How do we actually engage with complex situations of change and uncertainty in their practices?
2. How do MEL practitioners reflect on the choices and use of approaches and methods in these situations? (taking a first-order, or second-order approach?)
3. And what opportunities do actually exist for MEL practitioners to make a ‘second-order’ practice shift in their own practices?
My conclusions of the broader discussion in the group
The discussion in the inquiry group revealed a certain amount of resonance with the second-order shift, and it was compared with the related social learning theories of ‘single-loop’, double-loop’, and ‘triple-loop’ learning. (Argyris and Schön, 1974) and Gregory Bateson's Learning levels 0, I, II and III (Bateson, (1972), and discussed by Tosey, P. (2006).
The discussion in this group opens up some interesting further questions for me, that deserve further exploration for my own inquiry:
- Is it actually sufficient to ‘stop’ at making second-order shifts as practitioners? Is this necessary, but not sufficient, to bring about systemic changes?
- Should we rather strive towards ‘third-order shifts’, that reverberate with Argyris and Schön’s, and Bateson’s, three-levels of learning? From first-order learning (are we doing things right), towards second-order learning (are we doing the right things) towards third-order learning: is the system within which we operate the ‘right’ one, given the ethical implications and practical consequences for those that are affected, but not involved in situations? All while being conscious that learning at level II and particularly III is very hard, and being aware of the traps that lurk within the mental gymnastics when approaching learning Level III (as discussed in Tosey (2006)).
- Recognising these difficulties, what does it take to apply this second-, and third-order shift to the evaluation (MEL) ‘system’ itself, and the conditions under which MEL practitioners - in the widest sense - including commissioners and donors - engage in situations and their performance assessment?
Possible scope for future actions
It strikes me that these are very important questions we need to continue to grapple with, jointly with other MEL practitioners from different traditions of understanding.
There are opportunities we can identity and capture:
- Continuing exchanges in this MEL Inquiry Group, perhaps reframed as a ‘community of practice’, that may explore these and other of the many questions we touched during the course of this collective inquiry - perhaps in topical practice groups (Wenger-Trayner, 2023);
- Similar questions and concerns are currently expressed by other practitioners, in the MEL field and also beyond.
- For example, the European Evaluation Society (EES) has a dedicated Thematic Working Group (TWG8) on ‘systems approaches in evaluation’, which aims to develop its activities as a community of practice for those EES members interested in these questions, and the TWG also connects with other ‘systems’ working groups in evaluation societies (such as the American Evaluation Association SETIG group)
- At recent evaluation conferences (UKES (October 2023), and EES Online Conference (November 2023), similar questions were discussed in a number of contributions, such as the panel discussion “Systems And Complexity-informed Approaches To Evaluation: A Contribution To ‘Alternative Futures’”.
In light of the increasingly perceived urgency for action for systemic change in a whole raft of policy fields and indeed for humanity as a whole, has the time finally arrived to join forces by combining our efforts and pool our shared questions and capacities for joint inquiry, beyond the already established fora, to establish networks of networks seeking to make a difference to these enduring and challenging questions?
What are the opportunities to overcome the existing and enduring silos, be they geographic, disciplinary, paradigmatic, partisan, or ideological?
And what role can MEL practitioners capable of second- and third-order shifts play in “making the difference that makes a difference?” (Bateson 1972:453).
References
Argyris, C. and Schön, D. (1974) Theory in Practice. Increasing professional effectiveness, San Francisco: Jossey-Bass.
Bateson, G. (1972) Steps to an Ecology of Mind. Collected Essays in Anthropology, Psychiatry, Evolution, and Epistemology. University of Chicago Press.
European Evaluation Society (EES), Thematic Working Group 8 “Systems approaches in Evaluation”. TWG 8 - Systems Approaches in Evaluation - EES (europeanevaluation.org)
Ison, R. (2017) Systems Practice: How to Act in situations of uncertainty and complexity in a climate-change world. 2nd ed. London: Springer.
Schmidt-Abbey, B., Reynolds, M., & Ison, R. (2020) Towards systemic evaluation in turbulent times – Second-order practice shift. Evaluation, 26(2), 205-226. https://doi.org/10.1177/1356389020909120
Tosey, P. (2006) Bateson’s Levels Of Learning: a Framework for Transformative Learning? Paper presented at Universities’ Forum for Human Resource Development conference, University of Tilburg, May 2006. https://www.avannistelrooij.nl/website/files/BatesonLevels2006.pdf
Wenger-Trayner, E and B. (2023) Communities of practice within and across organizations.
A guidebook. Sesimbra, Social Learning Lab. https://www.wenger-trayner.com/wp-content/uploads/2023/06/23-06-08-Community-Of-Practice-digital.pdf
