Evaluating the Impact of Intervention Programs in Agricultural Sector
Keywords:
Agricultural sector, Counter factual, Impact evaluation, Intervention program, Source of bias
Abstract
Interventions in the agricultural sector need to be evaluated whether they are really successful and have the expected impact on beneficiaries, such as farmers. However, the definition of impact evaluation has varied considerably in the last two decades. Some empirical studies are also still biased in evaluating the impact, and this is related to the method used. This research has three objectives. First, to review the definition of impact evaluation according to the literature. Second, to identify sources of bias in impact evaluation studies. Third, to review empirical studies on impact evaluation, which have different results, especially from the methodology used. Empirical studies that serve as 'cases' in the agricultural sector are studies on the impact of coffee standards and certification. Thus, this research method is a theoretical literature review supported by empirical research results. The literature review reveals that one of the most common definitions of impact evaluation is that evaluation relates to counterfactual, which is a comparison between what actually happened and what would have happened in the absence of the intervention. Inappropriately designed counterfactuals are a major source of bias in impact evaluation, in addition to validity threats. Based on a review of empirical studies on the impact of coffee standards and certification, it is concluded that some studies have more reliable results than others. Studies with more reliable results use reliable counterfactuals, so as to minimize impact evaluation bias through an ‘apple-to-apple’ comparison between the intervention group and the control group.
Downloads
Download data is not yet available.
References
Bacon C. 2005. Confronting the Coffee Crisis: Can Fair Trade, Organic, and Specialty Coffees Reduce Small-Scale Farmer Vulnerability in Northern Nicaragua? World Dev. 33(3):497–511.
Bacon CM, Ernesto Mendez V, Gómez MEF, Stuart D, Flores SRD. 2008. Are sustainable coffee certifications enough to secure farmer livelihoods? The millenium development goals and Nicaragua’s Fair Trade cooperatives. Globalizations. 5(2):259–274.
Blackman A, Naranjo MA. 2012. Does eco-certification have environmental benefits? Organic coffee in Costa Rica. Ecol. Econ. 83:58–66.
Blackman A, Rivera J. 2011. Producer-Level Benefits of Sustainability Certification. Conserv. Biol. 25(6):1176–1185.doi:10.1111/j.1523-1739.2011.01774.x.
Bolwig S, Gibbon P, Jones S. 2009. The economics of smallholder organic contract farming in tropical Africa. World Dev. 37(6):1094–1104.
Calo M, Wise TA. 2005. Revaluing peasant coffee production: Organic and fair trade markets in Mexico. Glob. Dev. Environ. Institute, Tufts Univ.
Crato N, Paruolo P. 2018. Data-driven policy impact evaluation: How access to microdata is transforming policy design. Springer Nature.
Dietz T. 2021. Social impact assessment as a tool for rangeland management. Di dalam: Developing Strategies for Rangeland Management. CRC Press. hlm. 1613–1634.
Dietz T, Estrella Chong A, Grabs J, Kilian B. 2020. How effective is multiple certification in improving the economic conditions of smallholder farmers? Evidence from an impact evaluation in Colombia’s Coffee Belt. J. Dev. Stud. 56(6):1141–1160.
Ferraro PJ. 2009. Counterfactual thinking and impact evaluation in environmental policy. New Dir. Eval. 2009(122):75–84.doi:10.1002/ev.297.
Gertler PJ, Martinez S, Premand P, Rawlings LB, Vermeersch CMJ. 2016. Impact evaluation in practice. World Bank Publications.
Glasbergen P. 2018. Smallholders do not eat certificates. Ecol. Econ. 147:243–252.
Grabs J. 2020. Assessing the institutionalization of private sustainability governance in a changing coffee sector. Regul. Gov. 14(2):362–387.
International Initiative for Impact Evaluation (3ie). 2008. Principles for Impact Evaluation. 3ie New Delhi.
Khandker S, B. Koolwal G, Samad H. 2009. Handbook on Impact Evaluation. Washington DC: The World Bank.
Kilian B, Pratt L, Jones C, Villalobos A. 2004. Can the private sector be competitive and contribute to development through sustainable agricultural business? A case study of coffee in Latin America. Int. Food Agribus. Manag. Rev. 7(1030-2016–82554):21–45.
Lyngbaek AE, Muschler RG. 2001. Productivity and profitability of multistrata organic versus conventional coffee farms in Costa Rica. Agrofor. Syst. 53(2):205–213.
Maclellan N. 2016. Regional Trade Agreements in the Pacific Islands: Fair Trade for Farmers? Di dalam: Fair Trade, Corporate Accountability and Beyond. Routledge. hlm. 313–332.
Mancini L, Sala S. 2018. Social impact assessment in the mining sector: Review and comparison of indicators frameworks. Resour. Policy. 57:98–111.
McGill E, Er V, Penney T, Egan M, White M, Meier P, Whitehead M, Lock K, de Cuevas RA, Smith R. 2021. Evaluation of public health interventions from a complex systems perspective: a research methods review. Soc. Sci. Med. 272:113697.
Meemken E-M, Barrett CB, Michelson HC, Qaim M, Reardon T, Sellare J. 2021. Sustainability standards in global agrifood supply chains. Nat. Food. 2(10):758–765.
Philpott SM, Bichier P, Rice R, Greenberg R. 2007. Fieldâ€testing ecological and economic benefits of coffee certification programs. Conserv. Biol. 21(4):975–985.
Ranjan Jena P, Grote U. 2017. Fairtrade certification and livelihood impacts on smallâ€scale coffee producers in a tribal community of India. Appl. Econ. Perspect. Policy. 39(1):87–110.
Raynolds LT. 2004. The Globalization of Organic Agro-Food Networks. World Dev. 32(5):725–743.
Raynolds LT, Murray D, Leigh Taylor P. 2004. Fair trade coffee: building producer capacity via global networks. J. Int. Dev. J. Dev. Stud. Assoc. 16(8):1109–1121.
Riedl E. 2009. Fair trade: Evaluating opportunities for smallholder coffee growers in Papua New Guinea. Pacific Econ. Bull. 24(2):122–140.
Van Rijsbergen B, Elbers W, Ruben R, Njuguna SN. 2016. The ambivalent impact of coffee certification on farmers’ welfare: a matched panel approach for cooperatives in Central Kenya. World Dev. 77:277–292.
Ruben R, Fort R. 2012. The Impact of Fair Trade Certification for Coffee Farmers in Peru. World Dev. 40(3):570–582.doi:10.1016/j.worlddev.2011.07.030.
Ruben R, Zuniga G. 2011. How standards compete: comparative impact of coffee certification schemes in Northern Nicaragua. Supply Chain Manag. An Int. J. 16(2):98–109.doi:10.1108/13598541111115356.
https://www.elgaronline.com/abstract/edcoll/9781782549550/9781782549550.00020.xmlSchrefler%20L.%202016.%20Social%20impact%20assessment.%20Handb.%20Regul.%20Impact%20Assess.:108–126.doi:10.4337/9781782549567.00020.%20%5bdiunduh%202003%20Mei%2022%5d.%20Tersedia%20pada:%20http:/www.gsvc-sea.org/?page=social_impact_assessment
Schweizer ML, Braun BI, Milstone AM. 2016. Research methods in healthcare epidemiology and antimicrobial stewardship—quasi-experimental designs. Infect. Control Hosp. Epidemiol. 37(10):1135–1140.
the Organization for Economic Co-operation and Development. 2022. Outline Of Principles Of Impact Evaluation. [diunduh 2022 Jun 15]. Tersedia pada: https://www.oecd.org/dac/evaluation/dcdndep/37671602.pdf
Traldi R. 2021. Progress and pitfalls: A systematic review of the evidence for agricultural sustainability standards. Ecol. Indic. 125:107490.
US Environmental Protection Agency. 2022. Program Evaluation Glossary. [diunduh 2022 Jun 25]. Tersedia pada: https://sor.epa.gov/sor_internet/registry/termreg/searchandretrieve/glossariesandkeywordlists/search.do?details=&glossaryName=Program Evaluation Glossary
Valkila J. 2009. Fair Trade organic coffee production in Nicaragua—Sustainable development or a poverty trap? Ecol. Econ. 68(12):3018–3025.
White H. 2006. Impact evaluation: the experience of the Independent Evaluation Group of the World Bank. Munich, Germany: Independent Evaluation Group, the World Bank.
Williams CC, Kedir AM. 2020. Evaluating the impact of registration on future firm performance in the Middle East and North Africa region: evidence from the World Bank Enterprise Survey. Int. J. Entrep. Small Bus. 41(4):473–489.
Williams MJ. 2020. External validity and policy adaptation: From impact evaluation to policy design. World Bank Res. Obs. 35(2):158–191.
Bacon CM, Ernesto Mendez V, Gómez MEF, Stuart D, Flores SRD. 2008. Are sustainable coffee certifications enough to secure farmer livelihoods? The millenium development goals and Nicaragua’s Fair Trade cooperatives. Globalizations. 5(2):259–274.
Blackman A, Naranjo MA. 2012. Does eco-certification have environmental benefits? Organic coffee in Costa Rica. Ecol. Econ. 83:58–66.
Blackman A, Rivera J. 2011. Producer-Level Benefits of Sustainability Certification. Conserv. Biol. 25(6):1176–1185.doi:10.1111/j.1523-1739.2011.01774.x.
Bolwig S, Gibbon P, Jones S. 2009. The economics of smallholder organic contract farming in tropical Africa. World Dev. 37(6):1094–1104.
Calo M, Wise TA. 2005. Revaluing peasant coffee production: Organic and fair trade markets in Mexico. Glob. Dev. Environ. Institute, Tufts Univ.
Crato N, Paruolo P. 2018. Data-driven policy impact evaluation: How access to microdata is transforming policy design. Springer Nature.
Dietz T. 2021. Social impact assessment as a tool for rangeland management. Di dalam: Developing Strategies for Rangeland Management. CRC Press. hlm. 1613–1634.
Dietz T, Estrella Chong A, Grabs J, Kilian B. 2020. How effective is multiple certification in improving the economic conditions of smallholder farmers? Evidence from an impact evaluation in Colombia’s Coffee Belt. J. Dev. Stud. 56(6):1141–1160.
Ferraro PJ. 2009. Counterfactual thinking and impact evaluation in environmental policy. New Dir. Eval. 2009(122):75–84.doi:10.1002/ev.297.
Gertler PJ, Martinez S, Premand P, Rawlings LB, Vermeersch CMJ. 2016. Impact evaluation in practice. World Bank Publications.
Glasbergen P. 2018. Smallholders do not eat certificates. Ecol. Econ. 147:243–252.
Grabs J. 2020. Assessing the institutionalization of private sustainability governance in a changing coffee sector. Regul. Gov. 14(2):362–387.
International Initiative for Impact Evaluation (3ie). 2008. Principles for Impact Evaluation. 3ie New Delhi.
Khandker S, B. Koolwal G, Samad H. 2009. Handbook on Impact Evaluation. Washington DC: The World Bank.
Kilian B, Pratt L, Jones C, Villalobos A. 2004. Can the private sector be competitive and contribute to development through sustainable agricultural business? A case study of coffee in Latin America. Int. Food Agribus. Manag. Rev. 7(1030-2016–82554):21–45.
Lyngbaek AE, Muschler RG. 2001. Productivity and profitability of multistrata organic versus conventional coffee farms in Costa Rica. Agrofor. Syst. 53(2):205–213.
Maclellan N. 2016. Regional Trade Agreements in the Pacific Islands: Fair Trade for Farmers? Di dalam: Fair Trade, Corporate Accountability and Beyond. Routledge. hlm. 313–332.
Mancini L, Sala S. 2018. Social impact assessment in the mining sector: Review and comparison of indicators frameworks. Resour. Policy. 57:98–111.
McGill E, Er V, Penney T, Egan M, White M, Meier P, Whitehead M, Lock K, de Cuevas RA, Smith R. 2021. Evaluation of public health interventions from a complex systems perspective: a research methods review. Soc. Sci. Med. 272:113697.
Meemken E-M, Barrett CB, Michelson HC, Qaim M, Reardon T, Sellare J. 2021. Sustainability standards in global agrifood supply chains. Nat. Food. 2(10):758–765.
Philpott SM, Bichier P, Rice R, Greenberg R. 2007. Fieldâ€testing ecological and economic benefits of coffee certification programs. Conserv. Biol. 21(4):975–985.
Ranjan Jena P, Grote U. 2017. Fairtrade certification and livelihood impacts on smallâ€scale coffee producers in a tribal community of India. Appl. Econ. Perspect. Policy. 39(1):87–110.
Raynolds LT. 2004. The Globalization of Organic Agro-Food Networks. World Dev. 32(5):725–743.
Raynolds LT, Murray D, Leigh Taylor P. 2004. Fair trade coffee: building producer capacity via global networks. J. Int. Dev. J. Dev. Stud. Assoc. 16(8):1109–1121.
Riedl E. 2009. Fair trade: Evaluating opportunities for smallholder coffee growers in Papua New Guinea. Pacific Econ. Bull. 24(2):122–140.
Van Rijsbergen B, Elbers W, Ruben R, Njuguna SN. 2016. The ambivalent impact of coffee certification on farmers’ welfare: a matched panel approach for cooperatives in Central Kenya. World Dev. 77:277–292.
Ruben R, Fort R. 2012. The Impact of Fair Trade Certification for Coffee Farmers in Peru. World Dev. 40(3):570–582.doi:10.1016/j.worlddev.2011.07.030.
Ruben R, Zuniga G. 2011. How standards compete: comparative impact of coffee certification schemes in Northern Nicaragua. Supply Chain Manag. An Int. J. 16(2):98–109.doi:10.1108/13598541111115356.
https://www.elgaronline.com/abstract/edcoll/9781782549550/9781782549550.00020.xmlSchrefler%20L.%202016.%20Social%20impact%20assessment.%20Handb.%20Regul.%20Impact%20Assess.:108–126.doi:10.4337/9781782549567.00020.%20%5bdiunduh%202003%20Mei%2022%5d.%20Tersedia%20pada:%20http:/www.gsvc-sea.org/?page=social_impact_assessment
Schweizer ML, Braun BI, Milstone AM. 2016. Research methods in healthcare epidemiology and antimicrobial stewardship—quasi-experimental designs. Infect. Control Hosp. Epidemiol. 37(10):1135–1140.
the Organization for Economic Co-operation and Development. 2022. Outline Of Principles Of Impact Evaluation. [diunduh 2022 Jun 15]. Tersedia pada: https://www.oecd.org/dac/evaluation/dcdndep/37671602.pdf
Traldi R. 2021. Progress and pitfalls: A systematic review of the evidence for agricultural sustainability standards. Ecol. Indic. 125:107490.
US Environmental Protection Agency. 2022. Program Evaluation Glossary. [diunduh 2022 Jun 25]. Tersedia pada: https://sor.epa.gov/sor_internet/registry/termreg/searchandretrieve/glossariesandkeywordlists/search.do?details=&glossaryName=Program Evaluation Glossary
Valkila J. 2009. Fair Trade organic coffee production in Nicaragua—Sustainable development or a poverty trap? Ecol. Econ. 68(12):3018–3025.
White H. 2006. Impact evaluation: the experience of the Independent Evaluation Group of the World Bank. Munich, Germany: Independent Evaluation Group, the World Bank.
Williams CC, Kedir AM. 2020. Evaluating the impact of registration on future firm performance in the Middle East and North Africa region: evidence from the World Bank Enterprise Survey. Int. J. Entrep. Small Bus. 41(4):473–489.
Williams MJ. 2020. External validity and policy adaptation: From impact evaluation to policy design. World Bank Res. Obs. 35(2):158–191.
Published
2022-07-31
How to Cite
Ibnu, M. (2022). Evaluating the Impact of Intervention Programs in Agricultural Sector. AGRIMOR, 7(3), 102-113. https://doi.org/https://doi.org/10.32938/ag.v7i3.1775
Section
Original research article
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
The journal is committed to free-open access that does not charge readers or their institutions for access. Readers are entitled to read, download, copy, distribute, print, search, or link to the full texts of articles, as long as not for commercial purposes. The license type is CC BY-NC 4.0.
