Pete York and Michael Bamberger 2021
This introduction draws heavily on contributions from Pete York (BCT Partners) and Veronica Olazabal (The Rockefeller Foundation)
The attached paper presents the case for the importance of a convergence between big data and data science and the field of program evaluation. This convergence can potentially produce significant benefits for the future of equity oriented social and economic development in both developing and industrialized countries. However, there are also technical, political, economic, organizational and even philosophical factors that have slowed the achievement of this convergence and its multiple benefits.
We are living in a world that is increasingly dependent on big data and data science in every aspect of our personal lives and our economic, political and social systems. While many of these trends began in industrialized nations, they are expanding at an exponential rate in middle and low income countries. For better or worse, more people now have access to cell phones than to potable water.
For a number of reasons, discussed in the report, the agencies responsible for evaluating social programs, have been slower to adopt data science approaches than have been their colleagues working in research and planning (see Section 3). Data science and program evaluation are built on different traditions and use different tools and techniques, so that working together requires both groups to move out of their comfort zones.
Some of the promising areas where data science can make the greatest potential contributions to evaluation include:
- Reducing the time and cost of data collection so that evaluators can focus on the key evaluation tasks of defining the key evaluation questions, developing a theoretical framework for the evaluation and the analysis and interpretation of the findings. Many evaluators have to spend so much time and effort on the collection and analysis of data that they have very little time or resources to focus on the critical elements of the evaluation process. Freeing up time will also allow evaluators to focus on the areas of data quality (spending more time with the communities being studied, triangulation, ground truthing, mixed methods) – how many evaluations lament not having the time to properly address these questions?.
- Dramatically expanding the kinds of data that can be collected and analyzed. This includes access to Artificial Intelligence (AI) (making it possible to identify patterns in huge volumes of multiple kinds of data), a range of predictive analytics tools, also making it possible to develop models and analytical tools making it possible to evaluate complex programs. Another major advance is the possibility of studying longitudinal trends, in some cases over periods of as long as 20 years possibility. This makes it possible to both observe historical trends before a program is launched and to track sustainability of program induced changes, maintenance of program infrastructure and continued delivery of services. All of these are virtually impossible with conventional evaluations that have a defined start and end-date.
- Another very powerful set of tools for evaluation managers and policy-makers are the many kinds of algorithms, using artificial intelligence and data mining, that can process huge volumes of data to help improve decision-making and prediction of the best treatments for different groups affected by a program. The ability to analyze the factors affecting outcomes for individuals or small groups and to provide specific real-time recommendations on the best treatment or combination of treatments to provide for each small group or individual, contrasts with conventional evaluation designs that usually only make recommendations on how to improve the average outcome for the whole population. However, many of these algorithms are based on complex predictive models which are usually not well understood by most users (both because they are complex and because they are proprietary and not usually made available to clients). Consequently, there is the danger that some algorithms can have unintended negative outcomes that clients may not even be aware of.
- Although these have received less attention, and appear less exciting than the changes described above, one very important development concerns the ability of AI to combine multiple data sources into a single integrated data platform making it possible to explore relationships between the different data sets that was not previously possible. The program to combat modern slavery (see Section 2 Box 4) provides an example of the great potential of integrated data platform.
Although most of the discussion in the literature has concerned how data science (which is seen as the exciting new frontier) can assist evaluators who are often portrayed as having fallen behind the times with respect to the use of new technology; it is important to recognize that there are some potential weaknesses in data science approaches. This is particularly true as many data science approaches were originally developed in much simpler and less demanding environments such as marketing analysis and on-line advertising. In many of these areas, an on-line advertiser is only interested in correlations (if the font size and color of the ad is changed more users of the site will click on the ad); or men who purchase diapers in the supermarket are likely to also purchase beer. In these cases the client does not need to know why this relationship exists. Because of the limited demands of clients, many data scientists do not have to develop the kinds of theoretical frameworks and theories of change used by most evaluators.
So for all of these reasons, when data scientists and app developers venture into the new world of community development, designing complex programs for disadvantaged communities, and trying to explain why a program produces certain outcomes for some groups and not for others – there are many lessons that data scientists can learn from their evaluation colleagues. Some of these lessons include:
- greater concern about the quality and validity of data
- understanding the importance of construct validity (how to interpret indicators extracted from social media, phone call records or satellite images). How can changes in the number of references to hunger or sickness be used as an indicator of changes in short-term poverty levels? What do satellite counts of the proportion of roofs constructed or straw compared to zinc, tell us about trends in poverty?
- addressing issues of social exclusion and sample bias
- rethinking the role of theory and the need to base an evaluation of a theory of change
- the importance of ground-truthing (checking on the ground hypotheses generated from the analysis of remote, non-reactive data.
All of these issues are discussed in the attached paper.
evaluationperspectives.org 