Operational guidelines for assessing the impact of
agricultural research on livelihoods

Good practices from CIMMYT


Coordinated by: Roberto La Rovere, John Dixon

Impacts Targeting and Assessment (ITA) Unit
CIMMYT, Int., El Batan, Mexico

 

Why, and for whom are these guidelines?
See 1, 2

The context for these guidelines for IA. See 1, 2
Institutionalizing IA at your center
IA capacity, elements of training

 

 

 
Background. See 1, 2, 3
IA quality standards. See 1, 2
Logic models
What is IA: definition, types
See 1, 2, 3
Priority Seeting and Targeting
and the link to IA
IA for evaluating impacts on poverty / livelihoods
How do impacts materialize
- the impact pathways
 

How to conduct an IA: steps of the IA framework
Clarify the IA!
- What is IA? Key aspects. (1, 2, 3)
- What to assess?
- Why do I need an IA?
- Is IA good for me?
- Who can do an IA with me? (1, 2)
- How much do IAs cost? (1, 2)
Who to involve, when, how? (1, 2)
 
How to measure impact: the indicators
- Where and how to get the data?
How to plan an IA: Tools (1, 2, 3, 4) teams (1, 2) resources (1), skills?
How can I build an impact story?
How and to whom do I communicate the impact?

... a step back: using the IA to reflect and learn. See 1, 2

 

Preface

This manual responds to the need of CIMMYT scientists and field partners for guidance on impact assessment (IA) . It has been developed through a two-year process involving colleagues at CIMMYT and various stakeholders of CIMMYT work.

The guidelines naturally draw on many sources of information on IA, and condense and enhance what is known about IA for hands-on users: researchers and managers of crop improvement projects and their NARS partners interested in IA, and social scientists who are not expert in IA.

This manual will initially be published on-line so that it can be easily upgraded and linked to other sources. Users will be able to provide feedback, upgrade and enrich the content, and add details to IA case studies.

The aim of this document is to help ensure quality in IA, institutionalize good IA practices, provide a resource list of approaches, tools, and suggestions, and give examples of how CIMMYT does good IA with partners in diverse places and conditions.

The manual often refers to complementary documents, such as the “Strategic Guidelines” that are forthcoming from the Standing Panel on IA (SPIA) of the CGIAR, and draws from many sources.

Please inform us of any instances where we have neglected to cite original sources, so we can revise the manual accordingly.

Correct citation: La Rovere, R. and Dixon, J. 2007. Operational guidelines for assessing impact of agricultural research on livelihoods. Good practices from CIMMYT. Working Document, Version 2007.1.0. CIMMYT, Mexico, D.F.

 

Click here to download entire manual in PDF (693kb)

Contents (in html version)
1. Introduction
  The need for guidelines for assessing impacts on livelihoods
  Applications of the guidelines
    Definition of impact and impact assessment
  Overview of key operational concepts for Impact Assessment
    Livelihoods
    Poverty
    Other key concepts in impact assessment
  Links between impact assessment, priority setting and targeting
  External demand for IA
    Investor demands
    Partner demands
  Internal demand from CIMMYT
  Capacity for IA
  IA approaches to date: strengths and weaknesses
  Impact assessment quality standards
2. Good practices in conducting impact assessment
  2.1 Good practices in designing an impact assessment
    (I) Clarify the purpose, context, scope, and limitations of the IA
      What is an IA?
      What types of impact need to be assessed?
      Why do I need an IA? How can we use the outputs of an IA?
      Is IA good for my work? What if the IA shows little or no impact?
      Who do I need to develop an IA?
      Who does IA in and outside the CGIAR?
      How do we get rapid and cost effective IA?
      What are the risks in doing an IA?
      How can the outputs of an IA study be (made more) credible?
    (II) Focus on the key questions and hypotheses
    (III) Towards implementation: ensuring partners’ involvement, and planning for learning and communicating the results
    Responsibilities and roles
  2.2 Good practices in implementing an impact assessment
    (IV) Select/develop the analytical instruments
    Livelihood IA indicators
    Good practice for choosing an assessment method
      A: Qualitative methods
      B: Quantitative methods
      C: Integrating Quantitative and Qualitative Methods
    (V) Describe the impact pathway of the program/project
    (VI) Acquire and manage the data and information
    (VII) Analyze and validate impacts, and interpret the findings
    (VIII) Report, disseminate, communicate externally and internally
    (IX) Evaluate the assessment, reflect and learn internally
3. Incorporating IA into projects and institutions
  Training in IA
  Writing IA into projects, and developing a budget
  Institutionalizing impact assessment
   
Annexes  
Annex 2 Logic models
Annex 3 The extra need to learn and change
Annex 4 Modified standards of the Africa Evaluation Association
Annex 5 Selected IA resources
Annex 6 Selected impact Studies of CIMMYT
Annex 7 Key web resources on IA
   
Figures  
Figure 1 Flowchart of contents and links for easy interactive navigation
Figure 2 Sustainable Livelihoods Framework
Figure 3 Links between impact assessment, priority setting and targeting
Figure 4 The IA framework: Key steps in designing an IA
Figure 5 Rigor, costs, and timeframes of methods for IA studies
Figure 6 Window of DREAM showing input, output, and scenario fields
Figure 7 An institutional Impact Pathway
Figure 8 Impact Pathway of CIMMYT ‘Maize and Wheat cropping systems project
Figure 9 Window of the template for project proposals with the Impact Assessment box
Figure 10 Charting complex impact pathways: wheat stem rust
   
Boxes  
Box 1 Application of livelihood concepts to a typical farmer
Box 2 Assessing livelihood changes and impacts of CIMMYT projects in Oaxaca, Mexico
Box 3 Roles in a typical IA study
Box 4 Key elements of the Terms of Reference for an IA study
Box 5 Nutritional and health indicators: Disability-Adjusted Life Years (DALYs)
Box 6 Typical livelihood indicators
Box 7 Integrating indicators to assess impact of natural resource management research
Box 8 Classification of methods for IA
Box 9 An example of a participatory IA process at CIMMYT
Box 10 Cost-Benefit Analysis
Box 11 Integrating poverty classes and economic surplus analysis: Bt maize in Kenya
Box 12 Ethical and practical issues in field research
Box 13 CIMMYT IA courses: examples of IA training good practice
Box 14 Resource requirements for a ‘local’ IA study
Box 15 Institutionalizing IA: A model of workshops and events from CIMMYT’s experience
Box 16 Key contents of the CIMMYT IA inception workshop, May 2005
Box 17 The results-based monitoring model of GTZ


Acknowledgements

We are grateful to Olaf Erenstein, Hugo de Groote, Erika Meng, and Jonathan Hellin for reviews, comments and contributions to the document; Janin Trinidad for literature searches and management; participants in the 2005 IA workshops and in the “IA platform process”; Mike Listman for editing and other input on design and layout; María Delgadillo for web design and layout; CIMMYT management for recognition of and support to the improvement of IA culture in the Center; Henning Baur for contributing as a consultant to parts of the study; and Isabel van Bemmelen for a review of selected sections and elaboration of the case studies from the Oaxaca IA study in Mexico.

 

Acronym Website
IA and M&E Impact Assessment and Monitoring and Evaluation of impacts  
CGIAR IMPACT Consultative Group on International Agricultural Research http://impact.cgiar.org/
CGIAR Consultative Group for International Agricultural Research www.cgiar.org
CIAT International Center for Tropical Agriculture www.ciat.cgiar.org
CIMMYT International Maize and Wheat Improvement Center www.cimmyt.org
DAC Development Assistance Committee (OECD) www.oecd.org/dac
DFID Department for International Development www.dfid.gov.uk
DREAM Dynamic Research EvaluAtion for Management www.ifpri.org/dream.htm
EIARD European Info System on Agric. Research for Development www.eiard.org;http://www.infosysplus.org/
FAO Food and Agriculture Organization www.fao.org
GTZ German society for technical cooperation www.gtz.de
ICARDA International Center for Agricultural Research in Dry Areas www.icarda.cgiar.org
IDRC International Development Research Centre (evaluation) www.idrc.org/evaluation
IDS Institute of Development Studies www.ids-ps.org
IFAD International Fund for Agricultural Development www.ifad.org
IFPRI International Food Policy Research Institute www.ifpri.org
ILAC Institutional Learning and Change www.cgiar-ilac.org
IMPACT (model, IFPRI) www.ifpri.org/themes/impact.htm
IRRI International Rice Research Institute www.irri.org
ISNAR International Service for National Agricultural Research (NL) www.isnar.cgiar.org
MDGs Millennium Development Goals www.un.org/millenniumgoals
ODI Overseas Development Institute (UK) www.odi.org.uk
OECD Organization for Economic Cooperation and Development www.oecd.org
PSIA Poverty and Social Impact Analysis spa.synisys.com/resources/draftPSIAUserGuide.pdf
RWC Rice-Wheat Consortium for the Indo-Gangetic Plains www.rwc.org
SC Science Council (of the CGIAR) www.sciencecouncil.cgiar.org
SPIA Standing Panel on Impact Assessment www.sciencecouncil.cgiar.org/activities/spia/
WB The World Bank www.worldbank.org

Top

Go to Introduction