Implementation capacity matters when scaling up

Paul Niehaus was presenting at USF's seminar yesterday and mentioned the following working paper by Bold, Kimenyi, Mwabu, Ng'ang'a and Sandefur:

Interventions & Institutions: Experimental Evidence on Scaling up Education Reforms in Kenya

Abstract: The recent wave of randomized trials in development economics has provoked criticisms regarding external validity and the neglect of political economy. We investigate these concerns in a randomized trial designed to assess the prospects for scaling-up a contract teacher intervention in Kenya, previously shown to raise test scores for primary students in Western Kenya and various locations in India. The intervention was implemented in parallel in all eight Kenyan provinces by a non-governmental organization (NGO) and the Kenyan government. Institutional di erences had large e ffects on contract teacher performance. We find a signifi cant, positive eff ect of 0.19 standard deviations on math and English scores in schools randomly assigned to NGO implementation, and zero eff ect in schools receiving contract teachers from the Ministry of Education. We discuss political economy factors underlying this disparity, and suggest the need for future work on scaling up proven interventions to work within public sector institutions.

 The paper is openly hosted here, and the Center for Global Development has an interview with Sandefur here. The introduction to the paper gives a particular succinct summary of its motivation:

The recent wave of randomized trials in development economics has catalogued a number of cost-eff ective, small-scale interventions proven to improve learning, health, and other welfare outcomes. This methodology has also provoked a number of criticisms regarding the generalizability of experimental findings, including concerns about external validity, general equilibrium eff ects, and the neglect of political economy in much of the evaluation literature (Acemoglu 2010, Deaton 2010, Heckman 1991, Rodrik 2009). These criticisms are particularly relevant when randomized trials of pilot projects run by well-organized and monitored NGOs are used as the basis for policy prescriptions at the national or global level. As noted by Banerjee and Duflo (2008), what distinguishes possible partners for randomized evaluations is competence and a willingness to implement projects as planned. These may be lost when the project scales up. [. . . ] Not enough e ffort has taken place so far in trying `medium scale' evaluation of programs that have been successful on a small scale, where these implementation issues would become evident." 

In this paper we employ the methodology of randomized trials to assess these substantive concerns about political and institutional constraints and measure precisely how treatment eff ects change when scaling up...