Operational guidelines for assessing the impact of agricultural research on livelihoods
Good practices from CIMMYT
La Rovere and Dixon, CIMMYT, 2007

Back to Contents

2.2 Good practices in implementing an impact assessment

(IV) Select/develop the analytical instruments

The choice of the approach and methods for an IA of agricultural research projects or programs can be challenging. One difficulty is that the causal chain from research to improved well being for the intended beneficiaries of projects is often long and complex, with significant lags between research operations and impact on the ground. Moreover, research is in most cases not the only influencing force; there are many other causal and confounding factors, such as changes in prices, policies, various externalities and shocks, the institutional environment, etc.

This section presents analytical methods and tools that can be used to capture the complexity of the anticipated impacts. The objective is to give an overview of proven methods, with short descriptions of their main characteristics and how they can be used.

Livelihood IA indicators

Livelihood IA indicators are designed to measure the changes in household access to assets, institutional structures and relationships, or changes in livelihood strategies. Livelihood IA indicators should be Simple, Measurable, Achievable, Realistic, Time-bound (SMART) and:

- Outcome focused (indicators that assess outcomes of the intervention, such as changes in yields, reduced input use and cost, soil fertility improvements, etc.). Outcome-focused indicators address changes in peoples’ livelihoods by recognizing that outcomes are diverse and go beyond simple quantitative changes in the variables measured.
- Process based (indicators that show whether innovations are being used and are being used as intended).
- Negotiable and open-ended (indicators that show negative and positive trends, and recognize the context).

Livelihood IA indicators should capture the dynamics of livelihoods by looking at vulnerability and how vulnerability changes over time in relation to the context, rather than just the livelihood status, due to a technological innovation or intervention by a project or program. The list that follows gives commonly-used indicators.

Livelihood capital

- Human, social:

• Household demographics (e.g., labor assets, family composition, ethnicity).
• Social organisation (e.g., inter-household relations, participation in community).
• Knowledge, levels of literacy/illiteracy, school drop out rates.
• Sanitation and hygiene awareness, health status.
• Nutritional indicators (e.g., linked to consumption of nutritionally-enhanced crops).
• Number of meals consumed in the lean season; number of months food-insecure.
• Ability to borrow money from other households for consumption.

- Physical and natural:

• Natural resources from which inputs, resources, and services for livelihoods are derived (e.g., natural vegetation, land, water).
• Physical productive assets (e.g., soil types, equipment, animals, labor), and the changes in their levels under mild, medium, and severe stress or risk conditions.
• Farm land distribution, production, crop area, productivity (change), yields by crop and variety; for example under mild, medium, or severe drought conditions.
• Basic infrastructure, equipment, and property to support livelihoods.
• Affordable transport, road access (especially in rainy season), adequate shelter; access to water, to energy, sanitation.
• Access to common property resources.
• Level of natural vulnerability of the area of residence (e.g., flood prone).

- Economic and financial resources people use to achieve livelihood objectives:

• Farm-level seed demand.
• Prices; for example, of seeds, grains, feeds.
• Household financial assets, asset stability and change, expenditures.
• Crop consumption, sales, and marketable surplus.
• Investment by farmers in farm inputs.
• Access to credit, use and amounts (includes formal and informal loans).
• Grain and livestock products.
• Institutional investment in research and development.
• Inventory of germplasm by source.
• Household savings (including ownership of livestock).
• Remittances from household members working outside the area.
• Subsidies (assess their presence to see if technologies would be viable in their absence).

Indicators of crop productivity, production, and adoption capture impacts of germplasm diffusion:

• Preferred crop traits, varieties planted, crop management and input use (fertilizers, chemicals), yields, seed systems, management practices.
• Crop post-harvest and marketing: grain consumption and sale, amount of marketable surplus, amounts stored by the household for later sale.

The IA indicators should also allow capture of the following factors:

• Seasonality (e.g., duration of labor-scarce agricultural periods).
• Shocks (e.g., frequency of drought years, price collapses).
• Stresses (e.g., chronic lack of water for irrigation).
• Trends in assets and resource availability and use (e.g., increasing fertilizer use).
• Policies (e.g., subsidized inputs, prices, credit), institutions (e.g., providing microfinance for agricultural investments), availability of services (e.g., mobile phone networks).
• Livelihoods status (e.g., nutrition, health, social).

These indicators are linked to household livelihood strategies (e.g., intensification of use of inputs, migration, diversification, exiting from agriculture, specialization, intensification). Changes in these indicators contribute to livelihood outcomes. These are location-specific and can vary across households. It is important to assess how the technology or project contributes to improved outcomes. Outcome indicators include:

• Better well-being (health, education, etc.), more (cash) income, less vulnerability, more food security), improved asset base (land, labor, livestock), better food security, more physical security, lower farming or climate risk, personal or community empowerment, natural resources preservation, more job opportunities, etc.
• Economic effects (income, returns to investment, etc.), by source, including from improved maize and wheat.
• Improved market access (roads, markets, access to types of information and extension).
• Environmental effects (sustainability, natural resources use).
• Policy changes that affect livelihoods and determine type and degree of impact.
• Capacity building (e.g., of farmers, of NARS), public awareness, empowerment.

Top

The indicators need to be established in the planning stage of an IA, preferably with the participation of the key stakeholders and users. Indicators should ideally also be geo-referenced.

Box 5. Nutritional and health indicators: Disability-Adjusted Life Years (DALYs).

Baker (2000, pp. 178-180), the Sustainable Livelihoods Guidance Sheets and materials posted on www.livelihoods.org, and the web-based literature on the Millennium Development Goals list additional livelihood indicators in different contexts, as well as examples and case studies.

Box 6. Typical livelihood indicators.

Adoption. Adoption is a dynamic process determined by factors such as farmers’ perceptions of the advantages and disadvantages of technologies, efforts made by extension services to disseminate the technologies, the policy environment, the characteristics of farmers, the characteristics of the farming systems, and of the technologies themselves (see Adato and Meinzen-Dick, 2007, for a list of factors drawn from a variety of case studies). Adoption studies aim to derive an overall understanding of the farming systems in which innovations and technologies are diffused by identifying the technical, socioeconomic, and policy constraints. The objectives of adoption studies are to improve the adoption of the technology and its diffusion among farmers, and provide information that for impact studies. Assessment of adoption should distinguish between early and complete adoption. Adoption can be quantified by considering:

• The effect of diffusion of germplasm, technology, or management option on productivity.
• Farm-level productivity gains.
• Sustainability of gains.
• Area planted (primary and secondary estimation methods).
• Adoption intensity.
• Substitution effects.
• Asset requirements.
• Cultural and culinary preferences.
• Inter- and intra-household effects.
• Identification of adoption constraints.
• The economic surplus generated.

Costs for technology development are mainly incurred through research and extension. The ratio between benefits and costs decreases, as the duration of research and extension increases and as the benefits derived from the technology decrease. Innovations that are quickly adopted are more profitable than those that are adopted more slowly, because the benefits arrive more quickly and the ceiling of adoption is reached earlier, all the rest being equal. The higher the level of adoption achieved at a given time, the higher the benefits. The likely extent of future adoption of research results strongly influences the efficiency of research. Research activities are beneficial if their results are transferred to farmers—the faster the adoption and the more farmers who adopt, the greater the benefits.

The speed and ceiling of adoption for each technology/innovation are a function of the relationship between the characteristics of the new and the traditional technologies. However the decision to adopt does not easily fit into conventional econometric models (Adato and Meinzen-Dick 2007), hence the need to complement adoption studies with a livelihoods assessment. The assessment of adoption is comprehensively dealt with in the following sources:

• CIMMYT manuals on good practices for economic assessment of technology adoption, of the adoption of technology, and of productivity (CIMMYT 1993, 1998).
• Parts of the Bellon (2001) manual on participatory research on the rapid assessment of adoption, through participatory and qualitative approaches for varietal selection.

  Box 7. Integrating indicators to assess impacts of natural resource management research.

Top

Good practice for choosing an assessment method

There is no one best method for IA; the method chosen depends on, for example, the availability of data, the economic environment and the type of results required. Methods to evaluate the impact of crop breeding research are relatively well established, but there is no consensus yet on how to measure the impact of other research, such as natural resource management or farming-system projects.

Conventional methods include, for example, econometrics, use of production functions to determine, test and compare the influence of alternative drivers, economic surplus, and use of Net Present Values or Rates of Return or Benefit/Cost ratios of research investment. Mathematical models are appropriate for some tasks. However, to assess impacts on poverty, livelihood approaches have become more widespread. This section focuses on the specifics of livelihood approaches.

Because projects, data, cost, time constraints, and country circumstances vary, IA studies require a combination of appropriate methods (Adato and Meinzen-Dick 2007). Quantitative experimental design is often a good option and matched comparisons a second-best alternative. But these methods are not mutually exclusive. Estimating a counterfactual can be done by

• Using random assignments to create a control group (experimental design).
• Appropriately using other methods to create comparisons (quasi-experimental).

The best approach combines with/without and before/after counterfactuals and baseline data. Baseline data are crucial to reconstruct why certain events took place and to control for them. When more rapid assessments are required, social and poverty assessments are appropriate. When more complete assessments are required, household surveys, econometrics or modeling are needed. Incorporating cost-benefit or cost-effectiveness analysis (Box 8) is recommended, to compare alternative interventions, especially where funds and other resources are limited.

Box 8. Classification of methods for IA (adapted from the World Bank).

For a more comprehensive treatment of IA methods and guidance on which method to use, refer to Baker (2000) or Masters (1998), for surplus analysis approaches. The World Bank website hosts a comprehensive list of methods for doing Poverty and Social Impact Analysis (PSIA), describing the key elements that characterize the different tools required for methodological decisions.

Top

A: Qualitative methods.

Qualitative techniques are used to determine impact without depending on the counterfactual to make a causal inference. The focus is on processes, behaviors, and conditions as perceived by individuals or groups; for example, how a community perceives a project and how they are affected by it. Open-ended methods are used during design, data collection, and analysis. Qualitative data can also be quantified. Approaches used in qualitative IAs include rapid rural IA or participatory IAs in which the stakeholders—involved at all stages—determine the objectives of the study, select the key indicators, and participate in data collection and analysis.

Qualitative assessments are flexible, can be tailored to the needs of the IA, can be carried out using rapid techniques, and can enhance the findings of IA by providing a better understanding of stakeholders’ perceptions on the factors that may affect the impact. Various types of cause-effect diagrams can be used to capture farmer and stakeholder perceptions; for instance, on livelihood threats, opportunities, priorities, and preferences (www.livelihoods.org/info/tools/Diagrams.html).

Drawbacks in qualitative assessments are the subjectivity involved in data collection, the lack of a comparison group (without which it is impossible to determine causality), lack of statistical robustness given the small sample sizes, all of which make it difficult to generalize the results. The reliability of qualitative data is dependent on the skills, sensitivity, and training of the assessors because data collected may be misinterpreted.

Participatory methods are approaches in which representatives of stakeholder groups and beneficiaries work together to design, carry out and interpret an IA. This actively involves those with a stake in a project or program in decision-making and, by involving the key players, can generate a sense of ownership in the results.

Participatory methods can be used to learn about local conditions, perspectives, priorities, to design more responsive and sustainable interventions, identify and sort out problems during implementation, identify changes resulting from the project, identify who benefited and who did not benefit, identify the project’s strengths and weaknesses, and empower the those involved. Participatory methods can be effective in identifying intangible outcomes and unforeseen impacts, and in harnessing the opinions of those who are less involved by providing opportunities for discussion. They can also strengthen the capacity of individuals and organizations to participate in the development process. Information from specific groups can be compared with the opinions of key informants and information from secondary sources by triangulating findings. Participatory methods are however often regarded as less objective—less quantitative and thus supposedly less rigorous—and were often not part of conventional economic practice and economist-led assessments. It can be time-consuming to involve stakeholders in a meaningful way and the process may be hijacked by some stakeholders for their own interests. Resources on participatory research at CIMMYT and elsewhere are: Hellin et al., 2006; and 2008; Bellon et al., 2001; Lilja and Dixon, 2008, Lilja and Bellon, 2006; Lilja et al., 2006.

Types of qualitative methods

Key informant interview—a series of open-ended questions posed to individuals known for their knowledge and experience in the matter of interest. Interviews are qualitative, in-depth, and semi-structured. They rely on interview checklists of topics or questions.

Focus group discussion—a facilitated discussion in small groups of carefully selected participants from similar backgrounds and a common interest in the topic discussed. The facilitator uses a checklist of topics for discussion, and note-takers record comments.

Community group discussion—questions and facilitated discussion in a meeting open to all community members. The interviewer follows a checklist of questions.

Direct observation—recording of observations of facts seen and heard at a program site.

Stakeholder analysis is a prerequisite for understanding poverty and social impact and is the starting point of most participatory work. It is used to understand the relationships, influence and interests of those involved in given activities and determine who should participate in a project or in its components. It identifies the interest and influence of those who should be involved in an IA.

Beneficiary assessment is a systematic consultation with project beneficiaries to identify and design development initiatives, constraints to participation, and to provide feedback. It comprises participatory assessment and monitoring that incorporates a process of direct consultation of those affected by and influencing an intervention or policy. It is primarily qualitative, though with relatively lower emphasis on the use of visual techniques and of community-level research.

Participatory poverty assessment (PPA) approaches include the poor directly in discussions and debates on policies and priorities. They mainly use qualitative, visual, participatory rural appraisal. Data collection techniques are similar to those in beneficiary assessments, though with a greater focus on consultation with the poor, and on a broader set of policy issues.

Participatory rural appraisal (PRA) focuses on sharing learning with local people. It enables researchers and local people to assess interventions collaboratively, often using visual techniques so that illiterate people can participate. Group discussions between scientists and farmers include different members of the household. Formal surveys of households in the baseline may use participatory, rapid, or visual techniques to evaluate new technologies (Bellon, 2001), or include specific questions on the indicators identified earlier during the project to reassess the new technology.

Scenario analysis is a tool to help decision-makers and stakeholders think through how a given intervention may perform in different situations (scenarios). Each scenario focuses on a discontinuity (e.g., price changes), takes into account significant but predictable factors (e.g., demographic trends) and explores how successful the intervention or policy would be in this new scenario. It pre-tests changes under a variety of circumstances. The qualitative scenario exercises can be the basis of quantitative scenarios using modeling tools.

Box 9. An example of a participatory IA process at CIMMYT. (draws on Bellon 2001)

Top

B: Quantitative methods

Quantitative methods for IAs include, for example randomization, quasi-experimental designs, statistical control, and modeling (Baker, 2000 for details, and Scott, 1985 for sample sizes and the trade-offs between sample size, analytical rigor, and resources).

Experimental Designs / Randomization is a method of creating treatment and control groups statistically equivalent to one another. Treatment and control groups should be sufficiently large to establish statistical inferences with minimal attrition. Randomly generated control groups are the counterfactual. Subjects are randomly assigned to treatment or control groups. The impact is the means of samples of treatment groups minus the means of samples of control groups. Randomized methods of IA involving collection of data on project and control groups at different times are the most rigorous. Questionnaires or other instruments are applied to both groups before and after a project. In practice it is rarely possible to use randomized designs because of the cost, time, and ethical or other constraints. Thus, most methods of IA are less rigorous and less expensive. The most frequent problems with randomization designs are that:

1. They may be unethical because eligible members of the population are denied benefits or services for the purposes of the study (e.g., denial of medical treatment).
2. It can be politically difficult to provide an intervention to one group and not another.
3. The scope of the program may mean that there are no non-treatment groups.
4. The identifying characteristics of certain individuals in control groups may change during the experiment and may invalidate or contaminate the results (e.g., people move in and out).
5. It may be difficult to ensure that assignment is truly random and is not being modified.

Quasi-experimental (non-random) methods that compare project and control populations before and after interventions are an alternative to randomization. A non-equivalent control group is selected to match the characteristics of the project population as closely as possible. Comparison groups can be used to determine, test, and compare the influence of different drivers of change. Treatment and comparison groups are selected non-randomly after an intervention. Statistics are used to discriminate among groups, and matching techniques to build comparison groups with similar characteristics to treatment groups. Quasi-experimental methods draw on existing data, are quick, cheap, and can be done after a program has been implemented. Their disadvantage is that the results may be less reliable because the methods are less robust in statistical terms, they can be complex, and there can be selection bias⁴.

- Matching methods or constructed controls are a second-best to randomization. They pick an ideal control group to match the treatment group from a larger survey.

- Propensity score matching matches control groups to treatment groups on the basis of observed characteristics or by a propensity (to participate) score; the closer this score, the better the match. A good control group is from the same economic environment and is asked the same questions by similar interviewers as the treatment group. This technique is valuable when lots of time and baseline data are available, since it over-samples beneficiaries and then matches them.

- Double difference compares a treatment and control group (first difference) before and after a program (second difference). This can be an effective approach if the interaction between the adopter/beneficiary group and the non-adopter/non-beneficiary control group is small, and the groups are under reasonably similar conditions. This compares relative changes in metrics over time between two groups to establish how trends are influenced by interventions.

- Reflexive comparisons compare data from baseline data of participants before the intervention and data from a follow-up survey after the intervention. The baseline provides the control group; the impact is measured by the change in outcome indicators before and after the intervention.

Ex-post comparisons of project and non-equivalent control groups use data collected from beneficiaries and a non-equivalent control group after a project has ended. Multivariate analysis is used to statistically control for differences in attributes of the two groups.

IAs can range from large-scale sample surveys that compare project populations and control groups before, after, and possibly at several points during the intervention, to small-scale rapid assessment and participatory appraisals where estimates of impact are obtained combining group interviews, key informants, case studies and secondary data.

Formal household surveys are a method of collecting standard data from a sample of people or households in particular target groups for the quantitative approaches outlined above. The findings of such surveys can be up-scaled to the wider target group or population and quantified estimates made on size and distribution of impacts. Formal household surveys can provide:

• Baseline data to assess the performance of a project or program
• Key input to a formal IA of a program or project.

They can be designed to compare:

• Different groups at a given point in time
• Changes over time in the same group
• Actual conditions with the targets established in a program or project design.

Some types of information are difficult to obtain from formal surveys. Also, formal surveys often:

• Deliver results only after lengthy periods
• Are expensive and time-consuming, and
• Require sound technical and analytical skills to design, analyze and process the data.

Good practice suggests that questionnaires for IA surveys should be kept short and should focus on the main questions. So that answers are reliable and consistent across locations, enumerators and field data collectors should be instructed in the actual and intended meaning of the questions at the outset (see also models of Training Courses for IA on livelihoods Training in IA). Surveys should be adapted to local realities and cultural sensitivities. It is also advisable to complement household interviews with objective measurement, for example of yields, areas, to increase the accuracy, reduce subjectivity, and triangulate the findings.

Box 10. Cost-Benefit Analysis.

Top

C: Integrating quantitative and qualitative methods

Combining quantitative and qualitative methods both quantifies impacts of projects and explains given outcomes. Adato and Meizen-Dick (2007) outline the advantages and disadvantages of this combination and give examples from case studies. While quantitative data from samples that are statistically representative provide better assessments of causality by means of econometrics, qualitative methods are better for studying selected issues or events, provide critical insights into beneficiaries’ perspectives and illuminate quantitative analyses.

Additional benefits from integrating quantitative and qualitative methods include:

• Consistency checks can be built in by triangulations that independently estimate key variables (e.g., income, opinions about projects, reasons, or specific impacts).
• Different perspectives can be obtained (e.g., in terms of gender differences).
• Analysis on different levels. Surveys can give estimates of individual, household, or community level welfare; qualitative tools are more effective for analyzing social processes (e.g., conflicts) or institutions (e.g., effectiveness of services).
• More options for interpreting findings. Surveys often lead to inconsistencies that cannot be explained by the data. Qualitative methods can be used to check on outliers (responses that diverge from a general pattern) and for rapid field check of such cases.

Rapid ex-post IA is a low-cost approach that combines group interviews, case studies, key informants, and review of secondary data to gather the views of beneficiaries and other key stakeholders. This approach is useful when there is a need to respond rapidly to decision makers requests for information. Rapid ex-post IA can be used to provide a qualitative understanding of socioeconomic changes and social situations, people’s values, motivations, and reactions, and can provide the context and help interpret quantitative data. The findings, however, often relate to specific communities or localities and are thus difficult to generalize. This means that quantitative economists or evaluators see the recommendations as less valid, reliable and credible than those from formal surveys. Rapid approaches require skills such as interviewing, facilitation, field observation, note-taking, and basic statistics.

Figure 5 shows how the cheaper, quicker methods sacrifice methodological rigor. Participatory methods are not always cheaper than quantitative ones as the costs of staff, and training, as well the costs of surveys and data analysis, can be significant.

Figure 5. Rigor, costs, and timeframes of methods for IA studies

The choice of methods involves tradeoffs, mainly in terms of time, skills and resources (Figure 5 is based on international costings and need to be adjusted to local costs). This issue is also discussed in the last section of these guidelines on writing IA into projects. In general:

• Large-scale IA surveys require moderate amounts of time and skills, and significant resources. The results are widely publishable in the academic literature but their policy relevance and usefulness is limited.
• Small-scale IA surveys require moderate amounts of time, skills, and resources. Though the findings are more difficult to publish in academic outlets than the findings of large-scale surveys they are often covered in popular science publications and the media, so their policy relevance, usability and usefulness are relatively valuable.
• Informal participatory and rapid studies need less time and resources, but substantial skills and training; their policy relevance, usability and usefulness are also significant.
• Study and compilation of secondary data to derive IA conclusions are useful when time, resources and skills are limited and the results are needed urgently, but their policy use is also limited.

Methods of analysis

Econometrics applies mathematical and statistical methods to analyze data in the field of economics. In IA econometrics is used to analyze defined relationships between variables in survey data. Production and cost functions determine, test, and compare the influence of alternative drivers and estimate change in productivity due to research investment. Econometrics require good quality time series or panel data that capture variability (see Maredia et al. 2000, Bellon et al. 2007).

Economic surplus models are used to evaluate the adoption, spillover and economic impact of agricultural research. Various methods estimate economic indicators (Net Present Value, Internal Rate of Return to Investments, Benefit/Cost ratio, changes in consumer/producer surplus) deriving from changes in technology.

The economic surplus approach is based on a partial equilibrium model⁵. Initially developed for ex-ante IA, such models are now more often used for ex-post analysis. Models require data on inputs and outputs, budgets with and without the new or improved technology (or intervention in general), prices, yield (increasing or stabilized) and input (reduced) change due to the technology, rates of adoption, adoption lags, costs and discount rates. For example, DREAM (Dynamic Research EvaluAtion for Management) software runs economic surplus analyses that simulate market, technology adoption, research spillover and trade policy scenarios. The models and framework use information gathered through farm household surveys to determine adoption by households (and non-adoption or dis-adoption). The models can be integrated with GIS mapping techniques to identify and map the specific areas that could benefit from particular activities. DREAM (Alston et al., 1998) can be downloaded from the IFPRI website.

Figure 6 shows a window of the DREAM model with key input and output parameter fields. Results are often sensitive to input and output parameters and, in these cases, it is important to obtain good parameter estimates. DREAM requires robust estimates of supply elasticity.

Figure 6. Window of DREAM showing input, output, and scenario fields.

Box 11. Integrating poverty classes and economic surplus analysis: Bt maize in Kenya.

Top

IMPACT models

The IMPACT model series developed at IFPRI are computable general equilibrium (CGE⁶) models that analyze baseline and alternative scenarios for global food demand, supply, trade, income and population. IMPACT covers more and more countries, regions and commodities, and all cereals. IMPACT is a representation of a competitive world agricultural market for crops and livestock. In IMPACT, country and regional agricultural sub-models are linked through trade. The model uses a system of supply and demand elasticities incorporated into equations, to approximate production and demand functions. Productivity growth is estimated by its component sources, including crop management research, conventional plant breeding, biotechnology, and transgenics. Other sources of growth considered include private sector agricultural research and development, agricultural extension and education, markets, infrastructure and irrigation. IMPACT models factors that have the potential to impact future developments in the world food situation, including growth in populations and incomes, rates of growth in crop and livestock yield and production, feed ratios for livestock, agricultural research, irrigation and other investments, price policies for commodities, and elasticities of supply and demand. For any specific factor, the model generates projections for crops (area, yield, production, demand for food, feed, prices, and trade) and livestock (numbers, yield, production, demand, prices, and trade). The model includes tropical or semitropical fruits, temperate fruits, vegetables, fish commodities, distributional impacts on three income groups, and nutritional information⁷ . Parameter estimates are drawn from econometric analysis, past trends, expert judgment, and literature syntheses.

⁷ Endogenous variables determined by the by country–region model are: Commodity prices and quantity; Trade quantities (imports, exports); Cropped area by commodity; Commodities consumed; Calories per capita; Agricultural incomes, Percent children malnourished. Exogenous variables are: Population by year; Non-agricultural income by year; Total land area by country–region; Non-price (productivity) supply growth including contributions from: schooling, extension, public- and private-sector agricultural research.

Social accounting matrixes (SAMs) are related to national income accounting. They provide a conceptual basis for examining both economic growth and distributional issues within a single analytical framework. SAMs are used to organize information from the interaction of production, income, consumption and capital accumulation in a matrix.

Top

(V) Describe the impact pathway of the program/project

Describing the pathways that interventions will take to have an impact shows how proposed research will contribute to the innovation process. If the impact pathway is appropriately formulated, claims of impact become stronger and the potential for learning is greater. At a workshop in December 2006, the impact pathways of research projects at CIMMYT were described to assess processes by which impact is (or isn’t) achieved, the magnitude of impacts, the roles that different agents play in achieving the planned impact, what is expected to lead to intended impacts, and to make sure impact pathways were explicit in planning documents.

References on impact pathways are: Patton (1995) on utilization-focused evaluation; Virtanen and Uusikyla (2004) on links between cause and effect; Baker (2000) on theory-based evaluation, the World Bank M&E guidelines; GTZ on results-based monitoring, Douthwaite et al. (2003) on evaluation by impact pathways Smutylo and Carden (IDRC) on outcome mapping, and Adato and Meinzen-Dick (2007) on impact pathways and livelihoods.

(VI) Acquire and manage the data and information

The data for an IA should respond to the needs and interests of clients and stakeholders, focus on what is actually needed, and be amenable to updating, especially if the IA is long term or of an M&E type. Good practice for acquiring and managing data and information means:

• Accurately describing and documenting the sources of information so it can be verified.
• Checking data that is likely to be biased—by using a variety of methods and sources, and triangulating data with qualitative information.
• Choosing, developing, and implementing the information gathering process to ensure that the information obtained is sufficiently reliable for the intended uses.
• Identifying the type of data required to address the evaluation issues or questions, outlining the main indicators, and giving reasons for the chosen approach.

The data collection methods for IA depend on why and how the data will be used, the level of analysis, local conditions, and the data requirements and availability. Every method has advantages and disadvantages and well-defined applications (see Baker, 2000). The main methods of data collection are:

• Case studies
• Focus groups
• Interviews
• Observations
• Questionnaires
• Review of secondary data

It is always desirable to gather multiple lines of evidence. This can be done in stages (see Baker, 2000). In the first phase explorative and qualitative instruments are useful for refining the focus of the study and deciding which instruments to use (Baker, 2000). Multiple lines of evidence make arguments and conclusions more credible. This is important as IAs are typically done in a context of uncertainty. Triangulation, tapping into different sources of information, has similar aims.

Typical steps in data collection are:

1. Sample design and selection
2. Development of the data collection instrument
3. Selecting staff and training fieldwork personnel
4. Pilot testing
5. Data collection
6. Data management and access

The World Bank (2002, 2004) and Baker (2000) provide recommendations and checklists of good practices for data collection. CIMMYT (Carrion et al., 2007) developed a manual for real time data collection, management, and sharing, based on the integration of socioeconomic surveys into Personal Digital Assistant devices. The manual is being used to guide socioeconomic household data collection for IA in various projects. The ethics of data collection for IA are also important, and surveys should be carried out with particular care and tact.

Box 12 shows the CIMMYT guidelines for training partner enumerators, for both household surveys and participatory community surveys. These should be used in conjunction with the propriety aspects in IA standards (see Annex 4).

Box 12. Ethical and practical issues in field research

Baselines: The first step in M&E (see also Figure 3), and in ex-post IA, is to establish a baseline. Baseline indicators should be sufficient to describe the context in which knowledge and technologies are being applied. A livelihoods baseline is more comprehensive than conventional baselines that measure poverty in terms of income, productivity or household facilities. Livelihoods baselines establish

• The current livelihoods context of local people, their livelihoods strategies, and priorities
• The effect of policies and institutions on the lives of the rural poor (e.g., food security, vulnerability)
• The current livelihoods status of people, households, and the circumstances that make them vulnerable
• Problems, strengths, weaknesses, opportunities, and risks related to their livelihoods.

As with all baselines, livelihoods baselines should be used to identify different typologies of farmers, for example those who benefit versus those who don’t benefit. Typologies are useful for defining the counterfactual (with and without the project), in selecting farmers, and designing trials, as well as to measure and show impacts.

At the program level, a livelihoods baseline can help in analyzing key areas that were overlooked, determine whether activities should be redesigned, examine whether critical issues identified by the baseline should be addressed, and develop M&E systems informed by baseline indicators. A livelihood baseline examines resources in broader ways than conventional baselines by linking resources with their use and with poor people's access to them, looks at broader livelihood relationships, is designed with the participation of stakeholders, uses quantitative and qualitative techniques in data collection and analysis, and examines the impacts beyond the project or program's outputs.

Once data has been collected, it should be appropriately managed, archived, and made accessible at appropriate levels. Institutional Property Rights should be respected at all times. The documentation consists of:

• Metadata: basically data about data. See also Baker 2000.
• Information to interpret the data and conduct the analysis, contained in a document that describes the focus and objective of the IA, the methodology, the instruments, the sample information to be collected, contacts of team members, description of field work, etc.

Top

(VII) Analyze and validate impacts, and interpret the findings

To effectively answer IA questions both quantitative and qualitative IA data should be systematically analyzed. Relying on one type of data is likely to miss key facts and reduce the validity of the assessment. The perspectives, procedures, and rationale used to interpret the findings should be described. Conclusions should be clearly explained and justified.

Good practices for strengthening the quality and rigor of the analysis, interpreting the findings in collaborative ways, and increasing the plausibility of IA conclusions are:

• Testing alternative explanations and discussing competing impact hypotheses
• Addressing attribution of observed productivity trends to different causal factors
• Giving explicit consideration to unintended effects
• Seeking additional evidence and triangulating results
• Reviewing and adjusting the model impact pathway
• Drawing conclusions and developing recommendations.

The main sources of information on the topics mentioned in this section include:

• Baur et al., 2001, on the plausibility of IA studies and conclusions
• Miles and Huberman (1994), with discussion on standards for the quality of conclusions
• Baker (2000), Marsland et al. (2001), Adato and Meinzen-Dick (2007) on combining quantitative and qualitative methods.

Adato and Meinzen-Dick (2007) observe from the case studies they present that impacts are mixed and variable. In many cases research had a positive impact, but only on some dimensions of livelihoods, and not on others. Or, at times there was little evidence in terms of conventional impact measures such as income and consumption. Impact literature in the last few decades has mostly reported successes and high rates of return; the Adato and Meinzen-Dick studies provide examples of a more realistic assessment of the outcomes of research investments.

(VIII) Report, disseminate, communicate externally and internally

To convey and strengthen the findings of an IA, promote uptake and use of the results, and package and present the information for different audiences, it is important to plan and budget appropriately and plan for reporting, dissemination, and communication from the very beginning,. There are various ways in which IA results can be communicated effectively. Reports should clearly document the purpose of the IA so that the findings can be understood in context. Timeliness of reporting and dissemination is crucial, especially if the IA will be used to make strategic decisions. The findings and limitations of the IA should be made accessible to those affected by the IA, as well as to others who may be interested. Feedback on interim findings should be incorporated prior to producing final reports.

IA reports should generally be concise, since they are mainly aimed at executives who have little time to read lengthy documents. Supplementary information should be put in annexes or references. Impact reports should include information on projects, funding, and completion dates.

Presentation of the IA findings should be modest and realistic, recognize the limitations of the methods and the uncertainties in the results.

Means of communicating the IA results include workshops, publications, videos, meetings with policy makers and stakeholders. It is good practice to explicitly recognize the contribution of partners and all those involved.

(IX) Evaluate the assessment, reflect and learn internally

Upon completion the IA should be formatively and summatively evaluated against the principles presented in this document so that stakeholders can assess its strengths and weaknesses. Much of the relevance, effectiveness and efficiency of an IA derives from what is learned from it, from open discussion of the findings, reflection, dialogue, and action. Further guidelines on using IA for learning and reflection purposes are presented in section: Institutionalizing impact assessment. As Adato and Meinzen-Dick (2007) conclude, “research organizations should ask how technology development and dissemination could have been done differently, asking less often “how much poverty we reduced” and more often ‘what did we miss and could have done better.”

Top

Back to Contents

Previous Part

Next Part