Evaluating complex development programs: Integrating complexity thinking and systems analysis
Michael Bamberger and Aaron Zazueta
This is Chapter 18 of a forthcoming Research Handbook on Program Evaluation edited by Katherine Newcomer and Steve Mumford (scheduled for publication in late 2024)
Abstract
This chapter provides an introduction to the evaluation of complex programs. We discuss the importance of a complexity focus, and then review elements of complexity and systems thinking. We introduce an Integrated Complexity-Responsive and Systems (ICS) evaluation approach, integrating compatible concepts from both systems thinking and the complexity literatures. We use ICS as a generic term that covers all of the of the evaluation tools and techniques that can be used, singly or in combination, to evaluate the outcomes and impacts of complex interventions operating in complex contexts. The steps of the approach include: (a) understanding complexity in programs and in the systems in which they are embedded, (b) developing a theory-based evaluation framework, (c) selecting from among ICS approaches, (d) designing the evaluation for each approach, and (e) assessing the program’s contribution to broader development objectives. We also offer case studies illustrating how to implement the ICS approach.
Links to the two annexes included in the chapter
| Annex 1 Evaluation tools and techniques for each stage of the complexity-responsive evaluation frameworkFor descriptions of each of the 5 steps see Section 2 | ||
| Stage of the evaluation | Topics covered | References [the full references are given in the reference list at the end of the text] |
| Step 1 Understanding the complexity scenario | Understanding the system | |
| Conceptualizing systems | Davis and Guevara (2016)Raimondo et al (2016) p.34 Innovation network system mappingZazueta and Bahramalian et al 2021 | |
| Understanding interactions among large numbers of stakeholders and organizations | UNIDO 2019Zazueta, Le et al 2021UNIDO 2020 | |
| Defining system boundaries | Williams and Hummelbrunner 2029 Chapter 19 | |
| Understanding dynamic processes of change | ||
| Describing and measuring interactions and change within and between project components | Burke (2007)Williams and Hummelbrunner 2029 Chapter 1 | |
| Describing the processes through which a project is implemented | Beach and Pedersen (2013) | |
| Describing and monitoring interactions among agents involved in a project | Morrell (YouTube reference)Brown, Page, Riolo, and Rand (2004) GEF/IEO 2012UNIDO 2019 | |
| Identifying short and long-term change indicators (DEMANTEL model) | Si et al (2018) | |
| How processes of change are influenced by initial conditions, traces left by actions of other actors, and resistance to change | Morell (2021) | |
| Step 2: Theory-based evaluation frameworks | Complex adaptive system (CAS) framework | Zazueta and Bahramalian (2021)GEF, IEO (2012) and (2016)Green Climate Fund (2022) |
| Theory of change | Weiss (1995) | |
| Realist evaluation | Pawson and Tilley (1997) | |
| Contribution analysis | Mayne (1999) | |
| Critical systems practice | Jackson (2020) | |
| Blue marble evaluation | Patton (2020) | |
| Purposeful program theory | Funnell and Rogers (2011) | |
| Step 3: Identifying components to be analyzed and selecting the unit of analysis | Defining the unit of analysis | Gertler et al (2011) Chapter 2Bamberger, Raimondo and Vaessen (2016) Chapter 7 |
| Step 4: Defining the methodology for the component evaluations | Tracking and managing outcomes | |
| Tracking stakeholder opinions and perceptions on outcomes | Williams and Hummelbrunner (2009) Chapter 4 | |
| Understanding power relations and who can influence decisions | Si et al (2018)Zazueta, Bahramalian et al (2012) | |
| Measuring changes over time and evaluating impacts | ||
| Identifying the necessary and sufficient conditions of attributes required to achieve an outcome in groups | Lam and Ostram (2010)GEF/IEO (2016) | |
| Assessing the contribution of different organizations to the achievement of outcomes – when it is not possible to use quasi-experimental designs (contribution analysis) | Mayne (2012) | |
| Assessing project outcomes and impacts using conventional experimental, quasi-experimental designs and case-based comparisons | Vaessen (2016)Vaessen, Raimondo and Bamberger (2016) Chapter 4 | |
| QCA | Lam and Ostram (2010) | |
| Contribution analysis | Mayne (2012)GEF/IEO (2016) | |
| Randomized control trials and quasi-experimental designs | Gertler et al (2011)Vaessen, Raimondo and Bamberger (2016) Chapter 4 | |
| Interpretative, qualitative approaches (naturalistic enquiry, grounded theory). Structured approaches: configurational, within and between case comparisons, Narrative designs: participatory and empowerment evaluations | Vaessen, Raimondo and Bamberger (2016) Chapter 4 | |
| Agency designs: learning by doing, policy dialogue, collaborative and action research | UNIDO (2019) | |
| Step 5: Reassembling the different components of a program to assess the big picture | Systems modellingDescriptive and inferential statisticsComparative case study approachesPortfolio analysisReview and synthesis approachesRating scales | Bamberger, Raimondo and Vaessen (2016) Chapter 7 |
| Articulating the program theory | Case study No. 2 Part 2 Case study No. 1 Part 1 Zazueta, B and Bahramalian 2021GEF, IEO 2012 and 2016Green Climate Fund 2022 | |
| Describing and measuring interactions and change within and between project components | Burke (2007) in Williams and Imam Chapter 3Williams and Hummelbrunner 2029 Chapter 1 | |
| Describing the processes through which a project is implemented | Beach and Pedersen 2013 | |
| Identifying short and long-term change indicators | Si et al 2018GEF 2022GEF 2012GEF/IEO 2016Green Climate Fund 2022Si et al 2018UNIDO 2019UNIDO 2021Zazueta, Bahramalian 2021Green climate fund 2021 | |
| Capturing stakeholders opinions and perceptions on outcomes | Williams and Hummelbrunner 2009 Chapter 4Wilson-Grau and Britt (2012)SMARTFish | |
| Understanding power relations and who can influence decisions and outcomes | Si et al 2018Zazueta, Bahramalian et al 2021 | |
| Identifying the necessary and sufficient contribution of attributes require to achieve an outcome in groups | Lam and Ostram (2010) GEF/IEO 2016 | |
| Assessing the contribution of different organizations to the achievement of outcomes – when it is not possible to use a quasi-experimental design | Mayne (2012) | |
| Assessing project outcomes and impacts using conventional experimental and quasi-experimental designs | Vaessen (2016) | |
| Annex 2 Recommended readings | |
| Bamberger, M; Vaessen, J & Raimondo, E (editors) (2016) Dealing with complexity in development evaluation: A practical approach. Sage Publications. | This edited volume provides an overview of current thinking on complexity. It contains 5 parts: (1) dealing with complexity in development evaluation: framework, (2) methodological approaches, (3) Emerging data and innovative approaches to deal with complexity in development evaluation, (4) the institutional challenges, (5) case studies. |
| Beach, D and Pedersen,R,B (2013) Process-tracing methods: Foundations and guidelines. University of Michigan | Modeling and tracing the stages in the process of project implementation. A widely-used approach for defining and tracking the how projects are designed and implemented |
| Byrne, D & Callaghan, G (2015) Complexity theory and the social sciences: The state of the art. Routeledge. | Discussion of the contribution of case based methods to the analysis of complexity with chapters illustrating a wide range of case-based approaches, including Qualitative Comparative Analysis (QCA) |
| Byrne, D & Ragin, C (eds) (2009). The Sage Handbook of case-based methods. Sage | Overview of the theory and practice of research into complexity |
| Forss, K, Marra, M,& Schwarz,R (eds) (2011) Evaluating the complex: attribution, contribution and beyond. Transaction. | Readings on complexity-focused evaluations of development programs. |
| Funnell, S and Rogers, P (2011) Purposeful program theory: Effective use of theories of change and logic models. Jossey-Bass | The book explains all of the stages in the design and use of program theory, including the theory of change, and with a specific focus on complexity. They also introduce the widely-used framework of simple, complicated and complex evaluations. |
| Furubo, J, Rist, R, Speer, s (editors) (2013) Evaluation and turbulent times: Reflections on a discipline in disarray. Transaction. | Readings on how evaluation is applied in situations of conflict and uncertainty. The focus is not directly on complexity, but many of the situations where the evaluations are applied are complex. Interesting discussion of the differences between political and economic turbulence and how this affects the evaluation approach. |
| Gallager, L et al 2020 | Combines transdisciplinary research design with participatory causal loop diagramming processes, scenario modeling, and a new resilience analysis method to identify and test anticipated water-energy-food risks in Cambodia. Combines modeling techniques and participatory processes |
| Gertler, P et al (2011) Impact evaluation in practice. World Bank | Review the economic approach to quantitative impact evaluations, including: causal inference and counterfactuals, randomized designs, regression discontinuity, difference-in-difference, and propensity score matching and other matching techniques |
| Global Evaluation Initiative. https://www.betterevaluation.org | A global resource site with material on evaluation courses and reference material |
| Jackson, M (2019) Critical systems thinking and the management of complexity. Wiley | Provides a framework to assist decision makers and other stakeholders to understand “volatile, uncertain and complex and ambiguous world of general complexity”. The focus is on managing, rather than evaluating, complexity, but the approach provides guidance on how to articulate complexity frameworks that is very helpful for step 1 on the approach discussed in this chapter. |
| Masset, E, Shreshta, S & Juden, M (2021) Evaluating complex interventions in international development. CEDIL Methods working paper No. 6. Available online: https://doi.org/10.51744/CMWP6. | “ reviews promising methods for the evaluation of complex interventions that are new or have been used in a limited way. It offers a taxonomy of complex interventions in international development and draws on literature to discuss several methods that can be used to evaluate these interventions” |
| Mayne, J (2012) Contribution analysis: coming of age? Evaluation, 18(3), 270-280 | The approach provides practical guidance on how to analyze cause and effect in the many contexts where it is not possible to use experimental or quasi-experimental designs. The approach is widely used in situations where there are multiple agencies involved in a program, and the challenge is to assess the contribution of a particular agency. |
| Morrell, J (2012) Evaluation in the face of uncertainty: Anticipating surprise and responding to the inevitable. Guildford | This is a useful resource for addressing the often-overlooked issues of uncertainty, surprise and unintended outcomes. The approach addresses many of the issues that complexity theorists often refer to “emergence”. A strength of the book is that includes 18 case studies that help understand the concepts and the evaluation approaches. |
| Patton, M (2015) Qualitative research and evaluation methods. Sage | Broad review of participatory, observational and interview techniques and the rationale for their use. |
| Patton, M (2020) Blue Marble Evaluation | Blue Marble evaluation addresses global, multidisciplinary issues that are beyond the scope of most evaluation approaches. The focus is on the Anthropocene, and the recognition that these issues are complex. While the issues are global, Patton emphasizes that they must build on the integration of local issues. Transboundary issues must be addressed and silos broken down. |
| Pawson, R and Tilley, N (1997) Realistic evaluation Sage Publications; and Pawson, R (2013) The science of evaluation: A realist manifesto. Sage Publications. | Outcomes are the result of the interactions between context and the mechanisms developed by the project. The focus is on outcome configurations, rather than single outcomes. Programs have different outcomes in different contexts and for different people, and asks questions such as “Who benefits and who does not?”, “How?”, “where”, why?” and “when?” |
| Tashakkori[MB1] [MB2] , A; Johnson, B and Teddlie, C (2021) Foundations of Mixed Methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage. | Comprehensive coverage of mixed methods designs which is an essential element of the complexity approach proposed in this chapter. |
| (Van der Merwe et al., 2019) | “ The Cognitive Edge SenseMaker® presents a method “for capturing and making sense of people’s attitudes, perceptions, and experiences. It is used for monitoring and evaluation” mapping ideas, mind-sets, and attitudes; and detecting trends and weak signals. Various mapping techniques |
| Williams, B and Hummelbrunner, R 2011 Systems concepts in Action: A practitioners toolkit. Standford Business Books. | A comprehensive, but not too technical overview of a wide range of systems analysis approaches that are applicable to the approach discussed in this chapter. |
| Williams, B and Imam, I (2007) Systems concepts in Evaluation: An expert anthology. (2007) American evaluation Association. | This anthology includes case studies illustrating all of the techniques described in the Williams and Hummelbrunner text. |
| Wilson Grau and Britt (2012) Outcome harvesting | Using outcome harvesting to capture the views of different stakeholders on the significant project outcomes |
| Zazueta, A and Bahramalian et al 2021 | Modeling of complex systems , Influential Relation Map and use of visualization and participation to assess contributions to long term goals. |
Books and book-length publications
Deprez, S., Majale, M and Bamberger, M (2021) Impact Evaluation of UN-Habitat Housing Approach to Adequate, Affordable Housing and Poverty Reduction 2008 – 2019. Nairobi, UN-Habitat.
York, P and Bamberger, M (2021) Measuring Results and Impact in the Age of Big Data: The nexus of evaluation, analytics and digital technology. New York: The Rockefeller Foundation
Bamberger, M and Mabry, L (2020 Third edition) Realworld Evaluation: Working Under Budget, Time, Data and Political Constraints. Thousand Oaks, CA: Sage Publications
Bamberger, M (2020) “Big data analytics and development evaluation: Optimism and caution” in Garcia, O and Kotturi, P (eds) Information and communication technologies for development evaluation. Chapter 3: 76-110. New York: Routledge
Bamberger, M (2017) Integrating gender into project-level evaluations. Economic Cooperation Group. Published by the African Development Bank. Abidjan. Ivory Coast.
Bamberger, M., Vaessen, J and Raimondo, E (2016) Dealing with Complexity in Development Evaluation: A practical Approach. Thousand Oaks, CA: Sage Publications
Bamberger,M., Segone, M and Tateossian, F. (2016) Evaluating the Sustainable Development Goals: With a “No-one Left Behind” lens through equity-focused and gender-responsive evaluations. New York: UN Women.
Bamberger, M (2016) Integrating Big Data into the Monitoring and Evaluation of Development Projects. New York: UN Global Pulse.
Bamberger, M (2014) Measuring resilience: Lessons from the design and implementation of monitoring and evaluation methodologies of the Horn of Africa Risk Transfer for Adaptation (HARITA/R4) Rural Resilience Initiative. A case study. Oxfam America and the Rockefeller Foundation. June 2014 available at: http://www.oxfamamerica.org/static/media/files/Measuring_Resilience_-_R4_Case_Study.pdf and at: Policy & Practice http://policy-practice.oxfamamerica.org/publications/measuring-resilience-r4-case-study/
Bamberger, M., Segone, M and Reddy, S (2014) National evaluation policies for sustainable and equitable development: How to integrate gender equality and social equity in national evaluation policies and systems. EvalPartners, UNWomen and IOCE. http://www.mymande.org/sites/default/files/files/NationalEvaluationPolicies_web-single-color(1).pdf 2013
Raftree, R and Bamberger, M (2014) Emerging Opportunities: Monitoring and Evaluation in a Tech-enabled World. Discussion Paper. New York: The Rockefeller Foundation
Bamberger, M (2013) Engendering M&E Special Series on “The Nuts & Bolts of Monitoring and Evaluation Systems” No. 27. Poverty Reduction and Economic Management Department. Washington, DC: World BankWorld Bank
Bamberger, M (2012) Introduction to mixed methods in impact evaluation. Impact Evaluation Notes. No. 3. Washington, DC: InterAction.
Bamberger, M., Rugh, J and Mabry, L (2012 Second edition) Realworld Evaluation: Working under budget, time, data and political constraints. Thousand Oaks, CA: Sage Publications
Bamberger, M and Segone, M (2011) How to Design and Manage Equity-focused evaluations. New York: UNICEF
Bamberger, M (2009) Institutionalizing Impact Evaluation with the Framework of a Monitoring and Evaluation System. Washington, DC: Independent Evaluation Group, The World Bank.
Bamberger, M (2009) “Enhancing the utilization of evaluation for evidence-based policymaking” in Marco Segone (editor) Bridging the Gap: the role of monitoring and evaluation in evidence-based policymaking. UNICEF. http://www.unicef.org/ceecis/evidence_based_policy_making.pdf
Bamberger, M., Rugh, J and Mabry, L (2012 First edition) Realworld Evaluation: Working under budget, time, data and political constraints. Thousand Oaks, CA: Sage Publications
Bamberger, M (2006)“Evaluation Capacity Building” in Marco Segone and Ada Ocampo (editors) Creating and Developing Evaluation Organizations: Lessons Learned from Africa, Americas, Asia, Australasia and Europe. International Organization for Cooperation in Evaluation.(Published by UNICEF). http://www.unicef.org/ceecis/07_IOCE_EvalOrg.pdf
Bamberger, M (2006) Conducting Quality Impact Evaluations Under Budget, Time and Data Constraints. Washington, DC: Independent Evaluation Group, The World Bank
Bamberger, M and Ooi, E (editors)(2005) Influential Evaluations: Detailed case Studies. Washington, DC: Operations Evaluation Department, World Bank
Bamberger, M with Ayata, A., Greenberg, M., Lingen, A., Sen, K.M., Pearson, R and Sarr, Fatou (2006) Evaluation of gender mainstreaming in UNDP. Evaluation Office. New York: UNDP
Bamberger, M and Ooi, E (2004) Influential Evaluations: Evaluations that improved performance and impacts of development programs. Washington, DC: Operations Evaluation Department, The World Bank. http://lnweb18.worldbank.org/oed/oeddoclib.nsf/24cc3bb1f94ae11c85256808006a0046/67433ec6c181c22385256e7f0073ba1c/$FILE/influential_evaluations_ecd.pdf
Bamberger, M with Kulessa,M., Alailima, P., Flaman,R and Forrester, A. (2004) Evaluation of the Second Global Cooperation framework of UNDP. Evaluation Office. New York: United Nations Development Programme.
Bamberger, M and Fujita, N (2003) Impact Evaluation of Development Assistance: A Practical Handbook for Designing Methodologically Sound Impact Evaluations under Budget, Time and Data Constraints. Foundation for Advanced Studies in International Development. Tokyo. Japan
Bamberger, M and Maramba, P (2001). A Gender-Responsive Monitoring And Evaluation System For Rural Travel And Transport Programs In Africa. Africa Rural Travel and Transport Program. World Bank.
Bamberger, M (editor) (2000) Integrating Quantitative and Qualitative Research in Development Projects. Directions in Dev elopement. Washington, DC: World Bank.
Valadez, J and Bamberger, M (1994) Monitoring and Evaluating Social Programs in Developing Countries. A Handbook for Policymakers, Managers, and Researchers EDI Development Series. Washington D.C. The World Bank. url: http://www-wds.worldbank.org/servlet/WDS_IBank_Servlet?pcont=details&eid=000009265_3970716143510
Book chapters
Bamberger, M (2020) “Big data analytics and development evaluation: Optimism and caution” in Garcia, O and Kotturi, P (eds) Information and communication technologies for development evaluation. Chapter 3: 76-110. New York: Routledge
Bamberger, M (2012) “Reconstructing baseline data for impact evaluation and results management”. In Lopez-Acevedo, Krause and MacKay (editors) Building better policies: The Nuts and Bolts of Monitoring and Evaluation Systems. Washington, DC: World Bank
Bamberger, M (2011) “Reconstructing baseline data for impact evaluation and results management”. In Lopez-Acevedo, Krause and MacKay (editors) Building better policies: The Nuts and Bolts of Monitoring and Evaluation Systems. World Bank
Bamberger, M., Rao, V and Woolcock, M (2010) Using mixed methods in monitoring and evaluation: Experiences from international development. in Tashakkori and Teddlie (editors) Sage Handbook of Mixed Methods in Social and Behavioral Research. Second Edition (Chapter 24 pp. 613-642)
Bamberger, M., Rao, V and Woolcock, M (2010) t(February 1, 2010). BWPI Working Paper No. 107. Available at SSRN: http://ssrn.com/abstract=1545914
Bamberger, M and Rugh, J (2009) “RealWorld Evaluation: Conducting evaluations under budget, time, data and political constraints” in Marco Segone (Editor) Country-led Monitoring and Evaluation Systems: Better Evidence, Better Policies and Better Development Results. New York: UNICEF http://www.ceecis.org/remf/Country-ledMEsystems.pdf
Bamberger, M and Podems, D (2002) “Feminist Evaluation in the International Development Context” in Feminist Evaluation. New Directions for Evaluation No. 96 Winter 2002
Bamberger, M, Blackden,M, Fort, L and Manoukian, V (2002) “Gender” chapter of A Sourcebook for Poverty Reduction Strategies. World Bank. Washington D.C.
Articles
Bamberger, M (2018) “New challenges and opportunities for Multi and Mixed Methods Research (MMMR) in the Evaluation of International Development Programs”. International journal of multiple research approaches, 2018 vol. 1, no. 1, 168–182
https://doi.org/10.29034/ijmra.v10n1a11
Raftree, L, Bamberger, M and Olazabal, V (2016) “The role of new information and communication technologies in equity-focused evaluation: Opportunities and challenges”. Evaluation 2016, Vol 22(2) 228-244.
Bamberger, M (2015) “Innovations in the use of mixed methods in real-world evaluation”. Journal of Development EffectivenessVol 7 Issue 3 Sept 2015 317-326
Bamberger, M (2009) “Strengthening the evaluation of development effectiveness through reconstructing baseline data” Journal of Development Effectiveness. Volume 1 No. 1 March 2009.
Bamberger, M and White, H (2008). “Impact Evaluation in Official Development Agencies.” IDS Bulletin Volume 39 No. 1 March 2008.
Bamberger, M and White, H (2007) “Using Strong Evaluation Designs in Developing Countries: Experience and Challenges” Journal of MultiDisciplinary Evaluation. October 2007 Vol 4 No. 8 pp. 58-73 [http://survey.ate.wmich.edu/jmde/index.php/jmde_1/article/view/31/78]
Bamberger, M (2006) Book Review “Removing Unfreedoms: Citizens as Agents of Change in Urban Development” (Edited by Jane Samuels. ITDG Publications). American Journal of Evaluation. Vol 27. No. 4 December 2006 498-502.
Bamberger, M, Rugh,J, Fort, L and and Church, M (2004) “Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints” American Journal of Evaluation 25(1), 5-38.
Bamberger, M (2001). “Evaluation in developing countries: experience with agricultural research and development and the Annotated Bibliography of International Program Evaluation” Book Review Article. American Journal of Evaluation. 22(1), 117-122.
Bamberger, M, Kaufmann, D and Velez, E (2000) Interhousehold transfers: using research to inform policy. PREMNotes. Development Economics Department. Washington, D.C.: The World Bank.
Bamberger, M (2000) “The Evaluation of International Development Programs: A View from the Front” American Journal of Evaluation. Vol. 21 No. 1 Winter 2000.
evaluationperspectives.org 