Operational guidelines for assessing the impact of agricultural research on livelihoods
Good practices from CIMMYT
La Rovere and Dixon, CIMMYT, 2007

Back to Contents

2. Good practices in conducting impact assessment

The key steps in designing an IA are shown in Figure 4. They address the issues of clarifying the purpose of the IA, planning to involve stakeholders, communicating the results, identifying the conceptual framework that guides the IA, and drawing up timeframes and budgets. The steps, developed at CIMMYT, build upon the framework proposed by Patton (1995) and the EIARD (2003):

• Clarify the IA: Clarify the background, context, key hypothesis, demand, purpose, intended uses and users, and (involve) key stakeholders.
• Focus on the key IA issues: What is the innovation that needs to be assessed, its scope, timing? What is the logic model? What are counterfactual and attribution questions?
• Plan the IA: Identify the key disciplinary expertise needed, set up the best possible team for the assessment, plan to learn from and use the IA results.
• Select from a variety of methods; focus on the key data and indicators for the IA.
• Assess the roles that different agents and factors have played in achieving impact, the pathways by which impact was / was not achieved, and the expected magnitude of impacts.
• Acquire the agreed key data and information from primary and secondary sources.
• Assess and analyze impacts, interpret the findings, and develop recommendations.
• Report to facilitate understanding; disseminate and communicate the findings.
• Evaluate the assessment; reflect and learn internally.

Figure 4. The IA framework: Key steps in designing an IA.

Although Figure 4 shows a linear presentation of the steps in IA, in reality these are rarely sequential but are iterative and interrelated. The framework is constructed from the assessor’s point of view, but can also be used by other stakeholders starting at other entry points. It is crucial to negotiate and communicate with users during the IA process, since users need to be involved in key decisions to get their acceptance and buy-in to the results.

Examples from CIMMYT’s recent experience of designing IA studies for research projects are used to illustrate the framework in practice. The examples mostly discuss the impact of activities in which downstream livelihood impacts were intended. The study on the livelihood changes and impacts of previous maize diversity projects in Oaxaca, Mexico (Box 2) is used more frequently. However, it has not been published yet and will be dealt with in more depth in next updates of these guidelines. Other CIMMYT IA case studies will be added online as soon as completed, as practical illustrations of the elements, steps, and practices discussed in these guidelines².

Box 2. Assessing livelihood changes and impacts of CIMMYT projects in Oaxaca, Mexico.

The next section (2.1) focuses on good practices for designing an IA (steps I to III), while the following section (2.2) focuses on implementation (steps IV to IX).

Top

2.1 Good practices in designing an impact assessment

(I) Clarify the purpose, context, scope, and limitations of the IA

Impact assessments take time and resources, so there need to be good reasons for doing them. Formulating the questions, study design, communication, and actions on recommendations are as important as the substantive results of an IA. Neglect of these can seriously jeopardize the value of an IA. Problems can also arise when an IA is conducted in a rigid, unimaginative, and bureaucratic way. The main questions that need to be answered to justify whether or not an IA needs to be done are given below:

What is an IA?

An IA is an “evaluation to determine consequences of an intervention.” Social science and economic tools can be used to systematically quantify and measure values and indicators, and capture perceptions. IA includes:

• Ex-ante studies, done before an intervention is initiated or an outcome is generated to ensure appropriate targeting of research, resource allocation and priority setting;
• Monitoring and evaluation to monitor progress and impact of research activities; and
• Ex-post assessment to measure the outcomes of interventions and research.

What types of impact need to be assessed?

The main types of impact that need to be assessed are: quantitative (measurable), qualitative (observable), direct (e.g., yield increase), and indirect (e.g., less need to work off-farm). Adato and Meinzen-Dick (2007) provide a comprehensive classification and a list of practical examples.

Other types of impact are (examples in parentheses):

Top

Why do I need an IA? How can we use the outputs of an IA?

Demand for IAs is growing because:

• Resources are scarce and must be targeted and spent effectively.
• Organizations such as CGIAR centers need to show that they can—and do—alleviate poverty.
• Proof is required that public investments (from tax-payers’ money!) to research and development organizations actually pay off in the field.

Other reasons for doing an IA are to feed information back into programs, encourage internal learning, and better target and implement ongoing and future research. Yet traditionally, and most often, IA is done to ensure accountability: to give stakeholders evidence that investments in research are effective and relevant, and that continued investments are justified. IAs also:

• Provide estimates of the scale of outcomes for different demographic groups, regions and over time. These help target research and make it more effective.
• Measure the effects of an activity and distinguish these from the influence of other factors.
• Compare the effectiveness of alternative interventions.
• Clarify whether the costs of an activity are justified. This helps to inform decisions on whether to expand, modify, or eliminate projects or programs, and how to improve future activities.

The effort and resources invested in IA are particularly justified when the research is innovative, replicable, of practical use, and with defined applications, uses, and users.

Is IA good for my work? What if the IA shows little or no impact?

For all the above reasons, an IA can be extremely useful, even if it shows little, no, or negative impact, provided that the IA has captured the reasons and factors limiting impact. This is because one of the main and increasingly important purposes of IA is that of learning. IAs are effective and practical provided that a center has institutionalized IA as a tool for learning and for project or program improvement—meaning the results of IA are used in a context conducive to learning. Essentially, the results of an IA should provide answers to central development questions, for example, whether a project or institution is making a difference and to what extent, and should demonstrate impact on the ground. The systematic analysis and rigor achieved by using the results of IAs can give managers and policy-makers added confidence in decision-making and often lead to more and more effective funding.

Who do I need to develop an IA?

Donors, the main stakeholders, and the relevant policy-makers need to be involved in an IA from the beginning, to foster their buy-in to the results and the legitimacy of the design and recommendations. The IA team needs implementers with strong skills in the design of social science research, management, analysis and reporting, as well as a balance of quantitative and qualitative research skills. The actual mix of expertise and staff needed to conduct an IA will ultimately depend on the type of IA required (see also below).

Who does IA in and outside the CGIAR?

A typical CGIAR IA team is led by a social scientist. National partners and international experts play an important role. A list of roles in IA is given; and Annex 7 gives a list of websites of organizations both within and outside the CGIAR that are involved in IA.

How do we get rapid and cost-effective IA?

The cost and speed at which IA can be done varies, depending on the type of project, its scope, its purpose, the resources (financial, human, data, time) available, and the location. Figure 5 indicates some of the costs and timing of IA, based on the actual costs and duration of previous CIMMYT projects.

What are the risks in doing an IA?

IAs may be expensive and time-consuming. Unless they are written into projects from the start, they may not be easily funded. There are, however, quick, cheap approaches to IA. Assessments that require more time, are not designed for rapid use by stakeholders, or are more academic risk being of less use, when decision-makers need information quickly. An IA may be of little credibility or scientific value, if appropriate counterfactuals are not identified.

Obviously, some IA studies may show limited or no impact, or may be perceived by users as negative relative to their initial expectations. This may happen especially if the lessons from the IA are not used positively for learning. Strategies to avoid negative perceptions of an IA are:

• Building IAs (and any evaluative analysis) into project proposals from the very beginning.
• Fostering a culture of continuous improvement in the institution that makes it safe for people to make mistakes and even to fail. This is only realistic if the mistakes and failures happen at the early stages of the work, before significant time or money are invested.
• Promoting self-assessment and peer review. Often people are more critical than outsiders of their own work.
• Be clear on what people will be held accountable for and discourage them from playing it safe. One way to do this is to hold people accountable for their behavior—it should be responsible and professional—rather than for specific impacts that cannot be guaranteed. If, for instance, a new variety is not accepted by farmers, the scientists cannot be held responsible, whereas they can be held responsible for taking action on the causes of the rejection.

How can the outputs of an IA study be (made more) credible?

IA is more likely to be credible when:

• The recommendations in these guidelines and in other mainstream good practice literature on IA are followed.
• The IA conforms to appropriate standards.
• Proper indicators, data, methods are used.
• The right IA questions are asked from the beginning.
• The right people are involved at the right time.

Given the complexity and cost of doing an IA, the costs and benefits must be assessed realistically at the outset, and appropriate alternatives considered (e.g., M&E instead of an ex-post³ assessment). Alternatives should be seen as complementary rather than as substitutes for IA. The objectives of the IA need to be determined for the benefit of both the assessors and those assessed, and a common ground for assessment developed.

The first step is to describe the background. This means describing the political, social, cultural, and ecological aspects of the project or program in detail:

• History and current status of what is to be assessed.
• Names and types of organizations involved.
• Goals, scope, and size of the project or program.
• Sources of existing information (e.g., previous reports, performance monitoring).
• Who—people or institutions—requested the IA, and the reasons why it was requested.
• How and for what the information will be used.
• The intended audience for the findings and recommendations.

Clarifying the purpose of an IA means answering questions such as:

• What exactly is to be assessed? What is being analyzed?
  • (Impact of what?)
• What are the welfare (distributional), social, ecological impacts being assessed?
  • The distribution of costs and benefits among groups (e.g., rich and poor, men and women) is an important consideration when judging developmental impact.
  • Whose welfare is being analyzed? What is the impact being analyzed?
    • (Impact on who?)
• How, and by whom impacts are channeled?
  • (What are the expected impact pathways?)
• Who commissioned the IA?
  • Who has/should have a stake in it and should/may (want to) influence it?
• What are the risks of an unexpected outcome?
• Who should be involved in developing the IA?
  • (What expertise is needed? Who should author the IA?)
• Who will use the results?
  • Users may include for example, project staff, beneficiaries, policy makers, donor.
• Which conceptual framework or perspective will be used and will guide the IA?
• How much time and money are available and needed for the assessment?
• What is the impact of (and on) governments, NGOs, the private sector and others?
  • How do institutions affect the outcomes of the project?
    • Does the innovation affect the external forces (organizations, institutions, policies, markets, and social norms) that influence local livelihoods?
    • Does the innovation change the policies or behavior of others towards local residents, people’s access to institutions, and their influence over them
  • How do stakeholders benefit or lose from the project?
  • How do stakeholders affect the nature and scale of impacts on local people?
  • How does the policy, institutional, and political environment influence the project and its impacts and the sustainability of project impacts?

Other questions address the strategic, spatial, and temporal dimensions of the IA:

• What are the system boundaries? What should/will be included and what should/will be excluded from the assessment?
• When are the impacts expected to materialize?
• At which level(s) should the IA be conducted? This questions calls for differentiating between the geographic and intervention levels, as discussed further below.

The answers to these questions should help both the assessors and the users of an IA study to identify the key factors affecting the impact, the distribution of impacts between stakeholders, and the wider development impacts of a project or program.

Top

(II) Focus on the key questions and hypotheses

This section presents the key considerations and trade-offs in the choice and design of an IA. The key aspects are:

• Identifying realistic counterfactuals.
• Accounting for lag times.
• The timing of the assessment.
• Defining the spatial dimensions.
• Attributing effects and impacts in the context of often complex, multi-player partnerships.
• Defining the logic model (Annex 2) to conceptualize the IA.

The counterfactual. Every ex-post IA starts with this, since the impact is the difference between the observed events and the counterfactual. If counterfactuals are not realistic, the results of an IA will have little credibility. Constructing a realistic counterfactual of projects or programs allows before/after and with/without scenarios to be generated (Baker, 2000) and attributes causal pathways to specific influences of a particular element relative to other drivers. Because of the complexity and typical dynamics of the agricultural context, “before” scenarios cannot always be assumed to be accurate counterfactuals to “after” scenarios. The before/after scenarios are not sufficient as counterfactuals; with/without scenarios are also needed. With/without counterfactuals are normally made of participants (in innovations or programs) versus non-participants, or of adopters (beneficiaries, for instance of a new variety) versus non-adopters (non-beneficiaries).

Building counterfactuals in agriculture is complex because of the dynamics, externalities, policy influences, conflicts, and social, ecological and technological changes, which are the product of the interaction of different innovations. It is not easy to isolate the role of single innovations, as these are the result of collaborative efforts of scientists and institutions. It is thus challenging to determine what the course of events would have been if single contributions were removed. Counterfactuals must take into account the dynamic nature of innovations and capture valid technological alternatives for farmers, including innovations that would be produced by other institutions in the absence of the assessed research. In the case of international-public-good research outputs, a true control sample for comparison with a treatment group can hardly be isolated since public good information is freely available. As a result, experimental controls, as described later in the methods, are rarely possible and quasi-experimental controls must be used. Adato and Meinzen-Dick (2007) give a definition of the ideal counterfactual that comprises a quasi-experimental design (B: Quantitative methods) with randomly chosen adopters and non-adopters, supported by baseline and panel data collected over time.

Attribution means ascribing a causal link between observed (or expected) changes and specific interventions. It serves to assess who—institutions, stakeholders, researchers, or farmers involved at different levels—had a role in the development and diffusion of an innovation and, therefore, an impact. At the project or program level, establishing a counterfactual relative to a specific program is equivalent to attributing the causal pathway of specific actions to a particular institution, relative to other drivers of change. Attribution refers to what is credited for observed changes or results achieved. It represents the extent to which observed effects can be attributed to a specific intervention or to the performance of one or more partners taking account of other interventions, anticipated or unanticipated confounding factors, or external shocks.

Attribution can be difficult because of:

• Spatial differences: for example, local interventions, or the wider spillovers.
• Stakeholder diversity: for example, researchers, farmers, institutions, investors.
• Temporal differences: for example when given players entered (or exited) the process that ultimately led to given impacts.
• Different outputs: technologies, capacity building, knowledge, empowerment; this gets more complex when natural resource management technologies are included.
• The lack of a counterfactual or a wrongly-defined counterfactual.

Yet attribution is needed because of:

• Different interests and pressures, and the need to anticipate stakeholders’ claims.
• Bias (for example, bias towards wining projects), neglecting costs, or overestimating benefits (see section IA approaches to date: strengths and weaknesses), or neglecting certain stakeholders, partners or previous projects or investments.

It is not always feasible or desirable to attribute results to the actions of partners in collaborative research efforts, since often the actions of one partner alone would have not produced adoptable outputs without the contributions of others. Attempts to attribute credit may offend the partners involved. In such cases a viable solution is to consider and attribute collaborative efforts jointly.

Identifying the application of agricultural and related research outputs may often be complex, especially in the case of research programs that do not directly produce finished tools or improved physical inputs. Good examples are the intermediate genetic research outputs of CIMMYT that are used by others but do not directly impact on livelihoods; or documents, recommendations, and policies that draw on agricultural research results but do not produce direct impacts. The impact of these can only be attributed by gathering evidence (through interviews and case studies) on the contribution they made from those involved.

Lag times and timing are other critical issues to be considered in doing an IA. It is important to consider lag times, because research is typically a cumulative and evolutionary process in which new findings are partially a product of past findings. Problems arise in attributing impacts from previous projects, the sunk costs of previous investments, the direct costs (evaluations, travel, field work, building data systems, analysis, overheads), and the opportunity costs (scientists’ time, participatory research or ex-ante studies performed at the beginning of the process). Each new finding or technology that leads to successful innovations takes time to be applied broadly. It is thus important to be careful in the temporal attribution of research efforts, as current achievements may stem from previous research.

Research investments are often regarded as sunk costs, so internal rates of return are calculated for the marginal investment of new research and can vary significantly, depending on assumed lag times. This may be reasonable, if the counterfactual assumption of no alternate provision of the output is valid. Lag times also present challenges for the timing of ex-post IA, as it may be several years before research products are widely adopted and produce benefits (for example, the impact of conservation agriculture on soil health in farmers’ fields may not be evident in less than 5 years). IA studies often attempt to project benefits into the future, but time lags may complicate quantification. Ideally, ex-post IA should take place at the program or institutional level every 5-10 years.

The dimension or level of the assessment depends on the geographic or institutional mandate of the study, and can be interlinked with and differentiated between geographic and intervention levels. Recognizing the presence of different levels and factoring this into the design and analysis of impacts is critical to capture the effects and interpret the explanatory factors leading to impact.

The geographic levels comprise the following:

• International (e.g., global impacts of drought-tolerant maize research).
• National (e.g., impacts of maize breeding in Mexico).
• Regional (e.g., impacts of various CIMMYT and partner projects in Oaxaca State, Mexico).
• Community (e.g., impacts of a CIMMYT project in Huitzo village, within the Oaxaca project).
• Household (type) (e.g., impacts of maize varieties on poor households in Oaxaca).
• Field (e.g., impacts of improved maize varieties on clay soils).

The intervention levels comprise the following:

• Global or regional (e.g., global or Africa-wide impacts of maize improvement).
• System-wide (e.g., impacts of wheat breeding by the CGIAR).
• Institution (e.g., impacts on the internal dynamics of implementing organizations, their policies, service delivery mechanisms, management practices, and links among these).
• Program (e.g., impacts of the wheat improvement program in Turkey), in general of a development program involving multiple activities that cut across sectors, themes and/or geographic areas, grouped to attain specific development objectives.
• Project (e.g., impacts of the Nepal hill maize project on Nepalese maize farmers), in general of a development intervention designed to achieve specific objectives with given resources and implementation schedule, often within the framework of a program.
• Study (e.g., impact of a study on maize diversity in Oaxaca in terms of learning and targeting).

Top

(III) Towards implementation: Ensuring partners’ involvement, and planning for learning and communicating the results

This section gives good practices for planning IA to:

• Ensure stakeholder involvement and buy-in.
• Learn from and use the outputs.
• Enhance the credibility, use, and dissemination of the results.

An IA needs relevant, action-oriented findings and, to encourage action and reflection, the involvement of clients and users from the beginning. Stakeholders involved in or affected by an IA (including the beneficiaries) should be identified and included, so that their needs can be addressed and they can use the findings. An IA should thus include key users and anticipate and gain the cooperation of interest groups, to avoid any attempts to influence the findings. The main stakeholder groups should be identified, and grouped into those with common interests: direct participants (e.g., owners, workers, customers), affected non-participants (e.g., local residents), or those who may want to influence the project. Stakeholder groups may be sub-divided further depending on factors such as scale and benefits.

Once the groups of stakeholders have been identified, consultation and negotiation are needed to get agreement on indicators, how to measure impact, baseline data, and the standards to be applied throughout the IA process. The IA should be planned, conducted, and reported in a way that encourages follow up by stakeholders and increases the chances of the findings being used. An IA should be presented as an asset and opportunity for those being assessed, since it requires their time and resources, and their input should, therefore, not be taken for granted. Staff may be concerned about participating in the assessment of a project or program that evaluates their own work. It is thus important to examine honestly and openly what has/hasn’t worked, to include both successes and failures and identify positive lessons.

Obligations should be formalized in writing, so that participants have a common understanding of the IA and of options for renegotiating the agreement. Informal and implicit expectations by all parties should be considered. Conflicts of interest, if any, should be dealt with openly, so as not to compromise the reliability and credibility of the process and results.

The CIMMYT experience of institutionalizing IA flagged the need to establish control mechanisms to ensure that IA and M&E achieve and maintain high quality. This can be done by establishing an IA focal point for quality control. The focal point can be supported by scientists representing each program who perform peer evaluation, give guidance, harmonize, synthesize and support the IA process.

Responsibilities and roles

A multidisciplinary IA team is crucial for the success and credibility of an IA. Depending on the objectives, team members may contribute at different stages and participate to different degrees. If the IA involves collecting field data, the staff should be a mix of people from the area or region and external specialists. The responsibilities of the IA team (Baker, 2000) are to:

• Develop objectives, the timetable, logistics, budget, team composition and roles.
• Design and organize the IA system.
• Collaborate with partners and hosts.
• Train staff and other individuals involved.
• Organize collection of primary data collection and gather secondary data.
• Coordinate data analysis.
• Present and feed back information.

The key roles (see also Adato and Meinzen-Dick, 2007) are:

• Leader or manager
  • Establishes the IA design and methods, data needs, indicators (with the stakeholders), identifies the IA team, and drafts the Terms of Reference (ToR).
• Policy or other assessment experts
  • For example, economist, anthropologist.
• Sampling expert
  • Guides the choice of who, where, and how many participants and non-participants in an innovation should be sampled.
• Survey expert
  • Designs data collection instruments and codebooks; pilot tests the survey.
• Data processors
  • Map household, crop, plot, and other data. May be analysts based in the institution or unit that commissioned the IA, and often include a GIS technician.
• Field work supervisor or manager
  • Directs field operations, may collect some data but mainly gathers it from enumerators, harmonizes data types, checks for consistency and quality of data.
• Field enumerators
  • Collect the data in the field, often enter it, and report it to the supervisor.

   

  Box 3. Roles in a typical IA study.

The terms of reference of impact assessment studies should summarize the following:

• Purpose and scope.
• Needs for and types of training.
• Methods and data to be used.
• Standards against which performance will be assessed.
• Resources and time allocated.
• Reporting requirements and outputs.
• Deadlines and deliverables.
• Overall cost of the IA.

Baker (2000, pp. 169 – 187, pp. 188-197) provides good examples of standard terms of reference for an IA study. An example based on the terms of reference for the Oaxaca project is presented in Box 4.

Box 4. Key elements of the terms of reference for an IA study.

The primary functions of M&E and IA have been to provide accountability to donors and assess the achievement of projects or programs, but they also build capacity for ongoing learning beyond the life of the project, and produce information that can be used for planning, making policies, or resource allocation. To establish a learning process that uses the outcomes of IA, the capacity of individuals must be strengthened and a culture of reflection, learning, and communicating knowledge must be institutionalized. These issues need to be considered in the early planning stages of an IA. Their application within the context of CIMMYT is described in more detail in the section on Institutionalizing impact assessment.

Top

Back to Contents

Previous part

Next Part