Operational guidelines for assessing the impact of agricultural
research on livelihoods
Good practices from CIMMYT
La Rovere and Dixon, CIMMYT, 2007
Back to Contents
Impact assessment (IA) in agricultural research is the effort to measure its social, economic, environmental, and other benefits. IA is important because stakeholders expect research organizations such as CIMMYT to account for their use of resources, as well as learning from and adjusting to new challenges. These guidelines present major considerations to be addressed in designing and implementing IA. They are intended for partners in national agricultural research systems, universities, non-government organizations, or others who may have limited background in IA or economics and who are charged with conducting IA for their projects and programs. It may also be of interest and direct use to colleagues in other CGIAR centers.
The need for guidelines for assessing impacts on livelihoods
Many methods, tools, and standards are available for doing IA, yet there are two essential requirements:
- IA must be integral part of the organization’s core business and knowledge management.
- Formulating the right questions, designing the study, communicating throughout the assessment, and taking action on recommendations are as important as the actual results of an IA.
Neglect of these requirements can seriously jeopardize the value of IA, resulting in studies that simply comply with pre-established rules and targets, playing it safe, or adopt a defensive stance or displace goals (Perrin 2002).
Applications of the guidelines
The guidelines essentially follow a livelihoods approach to arrive at a comprehensive, poverty-explicit IA. The document will assist in:
- Understanding what is meant by IA and why IA studies are needed and important.
- Designing IA studies that respond to external and internal demands.
- Increasing awareness of available approaches.
- Identifying good practices for quality IA and making informed choices on data and methods.
- Teaching the key elements of well designed IA studies or projects with IA elements.
Structure of the guidelines
This document contains step-by-step guidelines for IA and presents procedures, methods, and options to help users develop appropriate IA for projects or studies. The document provides:
- A selection of IA good practices, tools, and methods with examples, in the form of Boxes, often with a brief description of their main strengths and weaknesses.
- Suggestions on minimum datasets from which to select indicators of impact on livelihoods.
- Key approaches for assessing impacts of agricultural technologies on farmers’ livelihoods.
- A checklist for designing an IA project or identifying projects that would benefit from an IA component.
- References from the international (Annex
5) and CIMMYT (Annex 6) literature;
and web links to resources and organizations (Annex
7) from CIMMYT’s experience.
Figure 1. Flowchart of contents and links for easy interactive navigation.
(Click on numbers / active links to be redirected to topic)
|
|
|
|
Definition of impact and impact assessment
IA involves observing, measuring, and describing how the condition being assessed (e.g., poverty) has been influenced by intentional human action. It should compare achievements with planned targets, or how things were before and after the intervention, and include a critical review of the assumed chain of causal influence. A classic understanding of impact is that of direct effects on income from increased adoption and use of technologies, as measured by numbers of farmers or area planted with an improved technology, yield increases, productivity growth, and the economic effects of adopting new technologies. Yet there is increasing recognition of the need to go beyond these forms of understanding of impact and include other and more comprehensive measures.
In this sense, having an impact means having an effect on farmers’ livelihoods and well-being, contributing to policy debates, influencing processes and outputs, creating change, and providing benefits to users. The effect is the intended or unintended change due directly or indirectly to an intervention. “Process impacts” are currently not fully defined but are important: for instance, process impacts may refer to changes in institutional, developmental, and policy level impacts that directly or indirectly, and in the longer run, lead to improved livelihoods.
Impact Assessment (IA) is defined as “a process of systematic and objective identification of the short and long-term effects–positive and negative, direct or indirect intended or unintended, primary and secondary–on households, institutions and the environment caused by on-going or completed development activities such as a program or project.” An IA helps researchers in development to better understand the extent to which activities affect the poor, which objectives are fulfilled, and the magnitude of their effects on people’s welfare.
An IA evaluates the effects of the different stages of an innovation system or intervention, from:
Research Inputs > Research Outputs > Outcomes > Final Impacts
The IA should provide information and results that are credible and useful, enabling lessons learned to be used for decision making by all stakeholders.
Impacts are the broader, longer-term, economic, social, or environmental effects resulting from research or development interventions. Evaluation is a systematic and objective process of judging, appraising, or assessing the worth, value, or quality of interventions in terms of their relevance, efficiency, effectiveness, and sustainability, as well as impacts. Linkages exist between the two terms and practices, and several elements and results of an IA can be used for the purposes of evaluation, but there is a clear distinction between the two.
Impact monitoring and evaluation (M&E) is a systematic, ongoing process of data collection on given indicators, to ascertain the long-term, widespread, intended/unintended consequences of an intervention and to monitor progress towards wider livelihood improvement goals. M&E seeks to provide stakeholders with an indication of the extent to which objectives are being or have been achieved. Monitoring and evaluation are complementary, but distinct processes.
Overview of key operational concepts for impact assessment
Livelihoods have been defined at CIMMYT as the “stocks and flows of assets and the ways these contribute to farmers’ well-being” (based on definition by staff, see section: Institutionalizing impact assessment). A livelihoods approach means considering the impact of technologies or of projects on farmers’ livelihoods. This shifts the focus from maize or wheat crops alone, to approaches that link them to the stocks and flows of household assets and activities. These guidelines define livelihoods as "the capabilities, assets, and activities required for a means of living. Livelihoods are sustainable when they can cope with and recover from stresses and shocks and maintain or enhance the capabilities or assets, while not undermining the natural resource base."
The livelihoods approach focuses on people’s lives, rather than resources or project outputs.
- A livelihoods approach to IA means that:
- Poor people become the focus.
- A wide range of beneficiaries will be involved.
- Impacts and the links between impacts need to be understood at the local, national, and policy levels. This means looking beyond households to impacts on organizations (capability, culture) and on society at large (values, attitudes).
Impact assessment that takes a livelihood approach (Figure 2) measures changes in the factors that affect livelihoods: capital assets, institutional structures or processes, the resilience or vulnerability of households, and livelihood strategies and outcomes.
A livelihoods approach (see Sustainable Livelihoods Guidance Sheets) can be used as a checklist of important issues to be considered systematically in doing an IA, to design indicators, and to understand how indicators link to each other. Adato and Meinzen-Dick (2007) describes applications to implement an IA and draws the attention to the core influences and processes and emphasizes the multiple interactions between the factors that in practice affect livelihoods.
Livelihood assets can be classified into five groups.
Natural capital -
natural resources from which resources and services for livelihoods are
derived (e.g., vegetation, land, water).
Social capital - social organisations
that facilitate or constrain cooperative enterprises, inter-household
relationships, formal/informal networks.
Human capital - education, knowledge,
health that enable people to solve their own problems and to pursue different
livelihood strategies.
Physical capital - infrastructure, equipment,
property to support livelihoods (affordable transport; shelter; water
supply, sanitation; energy).
Financial capital - financial resources
that people use to achieve their livelihood objectives (e.g., access to
credit, loans, savings and remittances).
The interaction of livelihood assets with policies, institutions, and processes and with livelihood strategies (combinations of farming and non-farming activities, for example, migration, off-farm work, abandoning farming for urban employment, farming diversification or, intensification) influence people’s livelihoods.
People in rural areas have complex livelihood strategies. Box 1 shows the application of livelihood concept to a typical farmer in a marginal maize- and wheat-growing area.
Box 1. Application of livelihood concepts to a typical farmer.
(CIMMYT work contributes directly or indirectly to the areas shown in bold)
| Livelihood capitals |
The farmers’ context before research or development | The farmers’ context after research or development |
| Natural | Limited land which may be degraded. No irrigation and poor rainfall. Does not use improved technologies. |
Limited land which may be less degraded. No irrigation and poor
rainfall. Uses improved germplasm and conservation agriculture options. |
| Social | Large family. Member of farmers’ groups. Not participant in
selection trials. Belongs to political association and church |
Large family. Member of farmers’ groups. Participant in selection trials. Belongs to political association and church. |
| Human | Limited education. HIV/AIDS in the family. Not member in participatory variety selection activities. Poorly nourished children. Labour focused on food crop production. | Limited education but more aware of varietal and crop management options. HIV/AIDS in the family. Member in participatory variety selection activities. Better nourished children. Labor released for cash crops or off-farm income. |
| Physical | Modest house. No post-harvest storage. Poor sewage system. | Modest house. Post-harvest storage (metal silos). Poor sewage system |
| Financial | Savings in good years only. No credit access. Receives remittances. | Savings in some bad years also. No credit access. Receives remittances |
To implement a livelihood approach, centers like CIMMYT are adopting a broader view of productivity that includes improvements to the livelihood capitals and specifies the circumstances in which better productivity will improve livelihoods in maize and wheat systems.
The causal relationships between adoption, productivity, and livelihood improvements depends on the nature of the farming and livelihood systems. To define causal relationships as integral parts of IA approaches, Centers should partner with specialists from a variety of fields and endow their staff and partners with a wide set of skills1 to conduct IA projects. This also entails the need to recognize that attributing impacts becomes more difficult, although several analytical tools are available for the purpose (see for instance Alston and Pardey 2001).
1 Other examples are contained in Adato and Meinzen-Dick (2007). One characteristic of the Oaxaca case
used in these guidelines is that it explicitly defines and uses the counterfactual, according to the good
practices described in this document
As agriculture is only part of rural livelihoods, IA of agricultural technologies needs an integrated, interdisciplinary approach combining conventional quantitative economic tools with systems modeling and qualitative tools. This means integrating household surveys, social analyses tools, gender, institutional, stakeholder, and markets analysis and measuring unintended as well as intended impacts (whether positive or negative).
Implicit in the livelihoods approach is the need for quantitative and qualitative baseline data that include indicators beyond those relating to maize and wheat. Each IA will need to be tailored to specific circumstances. The need for broad sets of impact indicators means that stakeholders need to agree on indicators at the design stage. Changes in measurable indicators (e.g., cash, yield) must be assessed in terms of how they contribute to livelihoods directly (e.g., to income, food) or indirectly (on assets, options, ability to cope with shocks). Changes in how people live may therefore become just as important as the more obvious changes in what people achieve. Livelihood approaches to impact assessment seek to answer questions such as:
- What are people’s livelihood priorities and which of these is the project aiming to meet?
- In what ways will the project affect the livelihoods of target groups?
- How do livelihood strategies affect how different groups get involved in a project?
- How can the project enhance the livelihoods of target groups?
Livelihood approaches to impact assessment also assess the impact of technologies that:
- Maximize average production and stabilize yields.
- Are successful with high inputs under good conditions and low inputs in poor conditions.
- Reduce workloads, freeing up household labor for other uses.
They also seek to answer questions about the context in which technologies work:
- Does the technology change people’s ability to cope with temporary changes and shocks, and capitalize on and adapt to positive trends and permanent changes?
- How does the technology relate to long-term trends, and does it compensate for or amplify their effects?
The livelihoods approach considers different levels, factors, and driving forces, and captures a broad picture of impacts in rural areas. This means that IA of agricultural technologies through a livelihoods lens draws upon conventional quantitative economic methods, tools for modeling systems and pathways, and qualitative tools.
To illustrate the application of livelihood approaches in IA we have drawn on the following:
- Livelihood guidance sheets (Sustainable Livelihoods Guidance Sheets).
- Methodologies for conducting a livelihoods baseline (LFP 2004).
- Methodologies for livelihood IA (WWF in East Africa), ODI.
- Tools for sustainable livelihoods M&E, IDS, 2001.
- A manual by Bellon (2001).
- Meinzen-Dick, Adato, Haddad, Hazell (2004).
Even when agricultural research generates large gains in yield (Evenson and Gollin 2003), poor farmers may not benefit (Kerr and Kolavalli 1999). Poverty is not only about low incomes but includes food insecurity, social inferiority, exclusion, lack of assets, and vulnerability.
To assess the impact of agricultural research on poverty means using tools, such as poverty mapping and ex-ante assessment, to identify where impact can be achieved. These take into account diverse factors, technologies, and externalities, and measure the impact of research products on poor consumers, as well as on food security and policies that affect poverty. The impact of research and development (R&D) interventions on poverty can be measured in:
- Monetary terms, where poverty is viewed as a shortfall in income below a given level. The impact is the effect of the R&D on income. If the tools are appropriate, monetary metrics can take into account the heterogeneity of the effects of a R&D intervention across target groups.
- Terms of capabilities, where poverty is viewed as a failure to achieve minimal or basic capabilities. The impact is the effect of the R&D on capabilities. A capability approach sees development as the expansion of human capabilities (for instance improving the capacity of farmers to cope with risk), rather than maximizing income.
- Terms of social exclusion, meaning total or partial exclusion from participation in society. The impact is the extent to which R&D fosters inclusion. The social exclusion concept deals with the processes of marginalization and deprivation, usually of specific groups rather than specific individuals.
- Terms of self-perception. In the self-perception approach people themselves, rather than outsiders, decide on the impact of R&D.
Other key concepts in impact assessment
Other key concepts for IA are listed below and described in more detail in Annex 1.
Adoption is the process by which innovations are accepted and used by people. Adoption is influenced by factors such as perceptions, the policy environment, socioeconomic characteristics, and the technology.
Attribution is the process by which a causal link is ascribed between observed (or expected) changes and interventions. It serves to assess those who at different levels and stages were involved in a project, program or in the development and diffusion of a technology, and their roles.
A counterfactual is what would have happened without the intervention. See Step:
(II) Focus on the key questions and hypotheses, for details.
The impact pathway is the chain of events and outcomes that link outputs to goals.
Outputs are products of development interventions and result in changes that achieve outcomes.
Outcomes are the likely or actual short-term and medium-term effects of intervention outputs.
Links between impact assessment, priority setting and targeting
IA is increasingly recognized as a set of related activities:
- Ex-ante IA to forecast impacts.
- Monitoring during projects.
- End-of-project evaluations.
These link with priority setting and targeting ( Figure 3). Starting points are often a baseline study and ex-ante forecasts of future events that coincide with or precede interventions. Targeting and describing the pathways that lead to intended impacts normally precede a project or take place during early stages. Monitoring during the project tracks progress on project indicators. At the end of projects, or some time after, ex-post IA studies take place, ideally as a comparison with baseline circumstances. Lessons learned can be fed into priority setting for follow-up phases. This framework is related to that which appears at www.impact.cgiar.org.
Figure 3. Links between impact assessment, priority setting and targeting.
There are two major trends in the external demand for IA. First, demand for externally conducted IAs is growing. Second, the demand for IA comes increasingly from larger national agricultural research systems and research centers in developing countries—Africa, India, China, Brazil. CGIAR centers and their partners now account for only a limited fraction of international agricultural research for development.
Strong demands for evidence of the impacts resulting from work by international agricultural research Centers comes among donors (Raitzer and Winkel 2005). The impacts of agricultural research on mission-level development goals, chiefly poverty alleviation and the distribution of benefits, constitute an increasing focus. Often managers and scientists in the CGIAR want concise summaries and briefs from an IA, which they also use to inform higher-level decision makers or the public.
Whereas social scientists increasingly demand ex-ante studies, breeders and other partners tend to require ex-post studies. The external pressure to ensure credibility means as well that the “learning” aim of impact studies is of growing importance. Finally, the demand for IA studies commissioned and conducted by experts external to a Center or project is increasing.
Donors require evidence of the impacts of agricultural research (Raitzer and Winkel 2005), particularly how it reduces poverty and how benefits are distributed. Decisions on priorities and funding are mostly driven by ex-post rather than ex-ante IA studies.
Few investors, however, have reported a direct instrumental use of IA information to decide on funding. IA is said to influence decision making more indirectly, through an improved understanding of overall research and science issues. While most CGIAR members appreciate economic metrics, such as internal rates of return, others are concerned that economic metrics do not always inform adequately about the distribution and social implications of research benefits. According to EIARD (2003), good IA studies need to enhance the developmental impacts of research investments for poor people. Information about returns on investment is important, but analyses should go beyond easily-measured impacts, seeking to capture complex, non-linear innovation processes and effects on livelihoods. Because of the difficulty in attributing impact to specific research outputs, searching for plausibility rather than proof of impact can help to produce useful information and insight at reasonable cost.
National agricultural research systems, non-government organizations, and advanced research institutes have their own expectations and uses for IA, including the following:
- Guidance for rationalizing research budgeting.
- Improving the likelihood of sustained financial support.
- Comparisons across research projects and identifying areas for improvement.
Many well-known impact assessment from CIMMYT have focused on adoption rates and rates of return to investments in crop improvement. The vision document “Seeds of Innovation” (CIMMYT 2004) emphasizes people-centered, livelihoods- and poverty-oriented, systems-based approaches to research. The CIMMYT Business Plan for 2006-10 states that IA must assess a broader range of impacts than in the past, including vulnerability, poverty, and the distribution of benefits. Direct and indirect impacts arising from linkages within farming systems and between agriculture and the non-farm economy should also be recognized. Finally, current strategies propose embracing diverse stakeholders, each with different expectations for IA. In this context, IA helps CIMMYT staff and partners to conceptualize and communicate project and program results internally or externally.
To enhance its IA capacity, CIMMYT analyzedkey strengths and weaknesses of center staff and partners in this area. The strengths of CIMMYT staff were in traditional IA—adoption studies, financial analyses, estimating the number of varietal releases, estimating the areas planted to new varieties, and biophysical analyses. CIMMYT staff were relatively weaker in assessing impacts on livelihoods, assessing impacts on policy, equity and poverty, and in training on IA. However, CIMMYT could count on a team of social scientists familiar with innovative approaches and able to appreciate farmer realities, both skills that complement livelihood approaches. CIMMYT partners generally lacked IA skills and experience, except in biophysical analyses, although capacity varied. The concluding section of this document “
Training in IA summarizes the key elements of training
required for a livelihoods approach to IA.
IA approaches to date:
strengths and weaknesses
The CGIAR has a long history of IA that has produced a wealth of information and understanding, for example on adoption of new varieties and returns on investment in germplasm improvement. Yet, according to Matlon (2003), weaknesses also exist: “A primary objective driving many studies was to demonstrate impact, to show donors that their investments in center research were well spent, and thereby to mobilize additional resources. Departing, often unconsciously, from the classic scientific method of hypothesis testing to move towards a demonstration mode, methodological problems became increasingly apparent: selection of successful cases for IA studies, inconsistent use of counterfactuals, overestimating benefit attribution to center activities, and restricting the dissemination of less favorable studies biased results and undermined their credibility and value. Donors and an increasing number of critics, also within the CGIAR, began to challenge the accuracy and representativeness of the exceptionally high published rates of return. As a result, both the resource mobilization and accountability goals of IA studies were often not achieved.”
Many difficulties stem from inadequate conceptualization of the innovation process itself and from the challenges of attributing impact. Innovation is a complex process in which technology is only one factor (Kuby 1999). Because innovation is the result of social interaction, development impact is never the result of the activities of a single factor such as agricultural research. Research can work towards development goals but it cannot guarantee that the goals will be reached.
For effective IA of research it is necessary to recognize that innovation is a social process. IA research needs to abandon the idea of scientific proof and aim for plausible arguments and claims as to the causes of impacts (EIARD 2003, Alston and Pardey 2001).
Impact assessment quality standards
Quality IAs meet accepted social science and international standards for:
- Utility, Feasibility, Accuracy, Propriety, Transparency.
These standards are modified from the African Evaluation Society (Annex 4). Since implementing all the standards may be impractical, the principles of usability and feasibility are recognized as being the most critical. Standards address the ethics of IA (see also Box 12).
| Back to Contents |  |
Next Part |
