6 Theory of Change
Underneath any good claim of causal inference is a theoretical model of how the researcher thinks X actually causes Y. Few studies set out to test a specific mechanism of impact, but most impact evaluations are designed around a theory about how the world works. This is called a theory of change.
A theory of change articulates how an intervention—or a policy, program, or treatment—is expected to impact an outcome. It explains how X causes Y, and what is needed for this to happen. This concept may be referred to as a theory of change, a logic model, a logical framework, a causal model, a results chain, a pipeline model, a results framework, a program theory, or one of several other combinations of these terms. This chapter presents strategies for creating a theory of change and logic model as a precursor to thinking about study measurement, the topic of the following chapter.
6.1 All Models are Wrong, But Some Are Useful
A theory of change falls under the larger umbrella of conceptual models. Of course, a model is a simplified representation of a more complex reality, like a plastic replica of the human heart or an epidemic model of Ebola transmission. Neither model is perfect, but both are valuable teaching tools. As the mathematician and statistician George Box famously wrote, “Essentially, all models are wrong, but some are useful” (Box and Draper 1987).
Conceptual models are used to represent the ways in which constructs are related without necessarily claiming that the relationships are causal. Many different types of conceptual models permeate the literature. Here are a few examples.
6.1.2 IN EPIDEMIOLOGY
Directed Acyclic Graphs
A related type of causal diagram is the directed acyclic graph, or DAG. This idea from mathematics and computer science has been applied to epidemiologic (observational) research to identify potential confounding variables that need to be addressed to make valid causal inferences (Greenland, Pearl, and Robins 1999). Figure 6.4 shows two DAGs. In this example, Suttorp et al. (2015) shows that, at a minimum, it is necessary to control for age when estimating the relationship between chronic kidney disease and mortality (a). If cancer is directly related to kidney disease (b), it is also important to control for cancer. Drawing out these relationships helps clarify that it may not be necessary to control for dementia because there is no direct relationship between dementia and kidney disease.
Epidemic models are used to explain or predict the spread of an epidemic. Kermack and McKendrick (1927) proposed a deterministic compartmental model called SIR that consists of the number of uninfected people susceptible to the disease (S), the number of infected (I), and the number of people removed (R) through death or immunization. Rivers et al. (2014) used this basic framework to create a compartmental model of the 2014 Ebola outbreak in Liberia and Sierra Leone (Figure 6.5):
- Susceptible (S)
- Exposed (E)
- Infectious (I)
- Hospitalized (H)
- Funeral (F; handling bodies)
- Recovered/Removed (R)
Rivers et al. (2014) published this model in November 2014, concluding:
The forecasts for both Liberia and Sierra Leone in the absence of any major effort to contain the epidemic paint a bleak picture of its future progress, which suggests that we are in the opening phase of the epidemic, rather than near its peak.
The key phrase is in the absence of any major effort to contain the epidemic. It turned out that the peak of new cases actually occurred in November 2014, thanks to a coordinated response to the outbreak. As the authors wrote at the time:
Of the modeled interventions applied to the epidemic, the most effective by far is a combined strategy of intensifying contact tracing to remove infected individuals from the general population and placing them in a setting that can provide both isolation and dedicated care. This intervention requires that clinics have the necessary supplies, training and personnel to follow infection control practices. Although both of these interventions in isolation also have an impact on the epidemic, they are much more effective in parallel.
6.1.3 IN STATISTICS
Statistical models are nondeterministic and thus incorporate stochastic variables. In other words, some of the variables being modeled have probability distributions rather than constant inputs like those in physics or mathematics. Statistical models are typically communicated as a set of equations and are visualized as a set of results. An exception is the path diagram represented visually in Figure 6.3.
6.1.4 IN ECONOMICS
Economic models can be stochastic or nonstochastic. The field of econometrics shares much in common with statistics, including a focus on stochastic models. Economics more broadly, however, also uses nonstochastic models. For instance, Figure 6.6 shows the hypothesized mathematical relationship between mortality and income (Wildman and Shen 2014). This graph is based on theory and does not plot actual empirical data collected in a particular setting.
6.2 Theory of Change
Theory of change diagrams can be designed in various ways. There is no RIGHT WAY™ to create one, as long as the fundamentals of how X leads to Y are conveyed. If the diagram is easy to understand, it is a good diagram.
Most theory of change diagrams include a few common elements. The United Kingdom’s Department for International Development, commonly known as DFID or UK Aid, commissioned a report on the uses of theories of change in international development that identified several common components:36
- Influence of context
- Discussion of long-term change
- Process/sequence of change explained
- Underlying assumptions
- Presented as a diagram and narrative summary
Start with the (research) problem statement (Box 1). This box gets at the heart of the reason the intervention exists. What is the problem to solve? Although this seems obvious, too often there is a disconnect between the primary aim—solving a problem or answering a research question—and the intervention strategy.
Next, take and inventory of the assets that already exist and the needs that remain (Box 2). There is always something to build upon, so it is important to look for strengths in addition to challenges. This process is best conducted in collaboration with people impacted by the problem so that the proposed solution is grounded in their reality. If available, descriptive data sources like the DHS can help to outline boxes 1 and 2. For a more local perspective, it is often beneficial to conduct a brief needs assessment in partnership with representatives from the local community if resources permit.
Box 3 jumps to the desired results. If the program works, what will change? With the results articulated, reconsider the factors that might affect the program’s success positively or negatively (Box 4).
The next step is to outline strategies for achieving the desired results, accounting for potential barriers and facilitators (Box 5). At this point, what the program will actually do should be stated clearly.
Finally, Box 6 reminds us that every theory of change is built on a set of assumptions. It is important to be thorough and transparent when considering the hidden beliefs that underlie the ideas about how this program will achieve results.
6.2.2 EXAMPLE: KENYAN “SUGAR DADDIES”
In 2014, an estimated 1.4 million people in Kenya were living with HIV, a prevalence rate of 5.3% among adults aged 15 to 49. Without a cure for AIDS, prevention remains critical to ending the epidemic.
Starting in 2001, Kenya integrated HIV/AIDS education into the primary school curriculum as a new prevention strategy (JPAL 2007). At the time, the focus of this program—and many other programs across sub-Saharan Africa—was complete risk avoidance (Dupas 2011): abstinence. Information on risk reduction was limited. Specifically, students were not learning about the differential prevalence of HIV infection by age and gender. Girls were not learning that the older “sugar daddies” who provide nice things like phones and airtime in return for sex are more likely than the girls’ “goofy” age mates to be infected.
An organization called International Child Support (ICS) Africa set out to change this by rolling out a “Relative Risk Information Campaign” in Kenya. The intervention was brilliant in its simplicity. A program staffer would talk with students for 40 minutes. During this time, the staffer showed the class a 10-minute video on sugar daddies and led a discussion about cross-generational sex. During the session, the staffer reviewed results of recent studies and wrote facts about the distribution of HIV prevalence on the chalk board.
Researchers from the Abdul Latif Jameel Poverty Action Lab (JPAL) tested ICS Africa’s risk reduction strategy in a randomized experiment in Western Kenya. In the first phase (2003), 328 schools were randomized to teacher training on the national HIV prevention curriculum (Duflo et al. 2006). In the second phase (2004), 71 of these schools were stratified and randomized to receive the sugar daddy intervention (Dupas 2011). In total, there were 4 study arms: (1) teacher training only, (2) sugar daddy only, (3) teacher training and sugar daddy, and (4) nothing.
The results were shocking. Teacher training was a bust. Although the training led to a change in teaching practices—notably that trained teachers mentioned HIV in class more often than nontrained teachers—it had little effect on HIV knowledge or childbearing rates.
In contrast, the 40-minute sugar daddy discussion and video reduced childbearing with men at least 5 years older by 65%—and not because girls started having babies with males their own age. The overall incidence of childbearing fell by 28%. With a cost of $28.20 USD per school and $0.80 per student, the cost per childbirth averted was $91 (JPAL 2007).
Returning to the template, the theory of change for the relative risk reduction program can now be understood more readily.
Increasing knowledge about HIV makes intuitive sense as an outcome for a study about HIV prevention. But why childbearing?
Because it is harder to lie about having a baby than it is to lie about private sexual behavior, childbearing is considered a more objective measure of unprotected sex. Unprotected sex is also a main driver of HIV transmission, so childbearing serves as a proxy for HIV risk from unprotected sex.
6.3 Logic Model
While a theory of change tends to be a high-level depiction of the “why”, a logic model—or logframe—is more detailed and focuses on the “how.” Logical models are useful tools for program planning, monitoring program implementation, and program evaluation and reporting. They are often presented in the “results chain” or “pipeline” format shown in Figure 6.9. In a logic model, inputs and activities represent the planned work. Outputs, outcomes, and impact are the intended results.
|Input||The resources needed to implement the program. People, money, time, etc.|
|Activities||What the program will do. Trainings, events, distribution of goods, etc.|
|Outputs||What the program did. Number of people trained, number of events held, number of goods delivered and number of people who benefitted.|
|Outcomes||Short- and medium-term results of the program. Increased knowledge, decreased risky behavior, improved functioning, etc.|
|Impacts||Long-term effects of the program outcomes. Lower HIV prevalence, reduced morbidity and mortality, etc.|
Figure 6.10 shows what a logic model might look like for the Kenyan relative risk reduction program introduced earlier.
As with a theory of change, there is no RIGHT WAY™ to create a logic model, but the same basic components (inputs, activities, outputs, outcomes, impacts) are often included. Figure 6.11 shows a logic model for evaluating reproductive health programs.
The next step is to carefully consider how to measure key variables throughout the project’s conceptual model, theory of change, or logic model.
Donors like USAID commonly require project staff to create results frameworks to monitor progress toward achieving the stated goals. The following example of a results framework is from the Feed the Future strategic plan for Kenya.
This diagram reads best from the top down. The overall goal reflects the mission of the broader FTF program: reduce poverty and hunger. The plan for attaining this goal is to reach 2 objectives: (1) inclusive agricultural sector growth and (2) improved nutritional status.
If the goal represents the ultimate desired impact, objectives are long-term strategies. The achievement of intermediate results, or IRs, indicate that the project is on track. For instance, IR6, “Improved utilization of MCH and nutrition services,” is expected to improve the nutritional status of women and children. This IR has 3 sub-IRs, including 6.3 “Strengthened MCH nutrition surveillance.” The rationale for sub-IR 6.3 is that better monitoring and data will enable earlier identification of at-risk individuals, which in turn should mean earlier initiation of nutrition interventions.
Every IR has a set of indicators for measuring progress. In the case of IR6, FTF lists the following illustrative indicators:
- Prevalence of maternal anemia
- Number of children under five years of age who received vitamin A from USG-supported programs
- Number of people trained in child health and nutrition through USG-supported health area programs (disaggregated by gender)
- Number of clients who received food and/or nutrition services (PEPFAR indicator)
Box, G. E. P., and N. R. Draper. 1987. Empirical Model Building and Response Surfaces. New York, NY: John Wiley & Sons.
Rosenstock, Irwin M. 1974. “Historical Origins of the Health Belief Model.” Health Education & Behavior 2 (4):328–35.
Fisher, Jeffrey D, and William A Fisher. 1992. “Changing Aids-Risk Behavior.” Psychological Bulletin 111 (3):455.
Zhang, Hua, Meizhen Liao, Xijuan Nie, Rongjian Pan, Chuangxin Wang, Shiman Ruan, Changqing Zhang, Xiaorun Tao, Dianmin Kang, and Baofa Jiang. 2011. “Predictors of Consistent Condom Use Based on the Information-Motivation-Behavioral Skills (Imb) Model Among Female Sex Workers in Jinan, China.” BMC Public Health 11 (1):1.
Greenland, Sander, Judea Pearl, and James M Robins. 1999. “Causal Diagrams for Epidemiologic Research.” Epidemiology 10 (1):37–48.
Suttorp, Marit M, Bob Siegerink, Kitty J Jager, Carmine Zoccali, and Friedo W Dekker. 2015. “Graphical Presentation of Confounding in Directed Acyclic Graphs.” Nephrology Dialysis Transplantation 30 (9):1418–23.
Kermack, William O, and Anderson G McKendrick. 1927. “A Contribution to the Mathematical Theory of Epidemics.” In Proceedings of the Royal Society of London a: Mathematical, Physical and Engineering Sciences, 115:700–721. 772. The Royal Society.
Rivers, Caitlin M, Eric T Lofgren, Madhav Marathe, Stephen Eubank, and Bryan L Lewis. 2014. “Modeling the Impact of Interventions on an Epidemic of Ebola in Sierra Leone and Liberia.” PLOS Currents Outbreaks.
Wildman, J, and J Shen. 2014. “Encyclopedia of Health Economics.” In, edited by AJ Culyer. Vol. 2. Amsterdam: Elsevier.
JPAL. 2007. “Cheap and Effective Ways to Change Adolescents’ Sexual Behavior.” http://www.povertyactionlab.org/publication/cheap-and-effective-ways-change-adolescents-sexual-behavior.
Dupas, Pascaline. 2011. “Do Teenagers Respond to Hiv Risk Information? Evidence from a Field Experiment in Kenya.” American Economic Journal: Applied Economics 3 (1):1.
Duflo, E, P Dupas, M Kremer, and S Sinei. 2006. “Education and Hiv/Aids Prevention: Evidence from a Randomized Evaluation in Western Kenya.” 4024. World Bank Policy Research Working Paper. http://ssrn.com/abstract=935173.
Consensus on the definition of ‘theory of change’ remains elusive.↩