CIMMYT. Operational guidelines for assessing the impact of agricultural research livelihoods. Good practices from CIMMYT. Annex 1

Annex 1. Glossary of key terms and concepts

The concepts and definitions used in these guidelines, as understood by CIMMYT, are presented below. The definitions integrate understanding from CIMMYT’s IA learning process with the conventional definitions of organizations such as OECD and the World Bank (operational concepts that are generally limited in covering livelihoods, poverty and a balance between accountability and learning) and more recent definitions that refer to a livelihoods framework.

Impact monitoring and evaluation (M&E) is a systematic, ongoing process of data collection on given indicators, to ascertain the long-term, widespread, intended/unintended consequences of an intervention and to monitor progress towards wider livelihood improvement goals. M&E is aimed at providing assessors and stakeholders with indications of the extent to which an ongoing intervention is achieving its objectives. Monitoring and Evaluation are complementary but distinct:

Monitoring focuses on tracking inputs, outputs, outcomes and impacts of interventions
Evaluation assesses the efficiency and impact of interventions (after implementation).

Together, M&E allow results to be tracked over time, corrections to be made during implementation, success to be assessed, and ownership of achievements and accountability to be promoted. M&E is a tool for steering projects, and for checking that they are on track and that progress is being made towards the intended impact. When M&E takes a livelihoods approach, this is not ultimately concerned with attaching values to livelihood outcomes, but with understanding whether livelihoods are moving in the right direction. Hence, it is more relevant to focus on determining the trends and direction of change rather than to attaching values to change and differentiating impacts between groups. Participatory M&E, because it considers the perspectives and insights of all stakeholders (beneficiaries as well as project implementers) provides feedback on ongoing program effectiveness. Stakeholders identify issues, analyze findings, recommend, and take responsibility for action. The participatory process effectively ensures the ownership and commitment of stakeholders for any corrective action.

Ex-post IA measures the actual outcomes of an intervention at a given point after completion of a project, to determine whether or not intended impacts on individuals, households, communities and institutions were achieved as expected, whether those effects are attributable to the project, and whether or not there were other outcomes. Ex-post IA aims to identify the factors in success or failure, assess the sustainability of the impacts, and draw conclusions to inform subsequent interventions. Ex-post IA is a summative form of IA, concerned with the effectiveness and value of a project; it is conducted after project completion for the benefit of the implementing institutions and for external audiences, to determine the extent to which intended outcomes were achieved. Other types of IA are formative, that is, they are aimed at providing information during the planning, preparation or implementation phases of projects, in order to improve performance. Ex-post IAs, however, are increasingly seen as informative studies for feeding back information into strategic and funding decisions to help institutions learn and improve project management. A comprehensive IA package includes, for example, ex-ante, program reviews, ex-post, performance M&E, and process evaluations. In research studies, process evaluations often take the form of early-acceptance and adoption studies that provide feedback on the research process as it proceeds. Other key IA terminology, used throughout these guidelines, is presented below:

Accountability is the demonstration that research complies with agreed rules and standards, and reports performance results fairly and accurately vis-à-vis plans. This includes the obligation of those involved in a project to act according to defined responsibilities, roles and expectations, in terms of a wise use of resources; for assessors, to provide accurate, fair and credible monitoring reports and performance assessments; and for public sector managers and policy-makers, to ensure accountability to taxpayers and citizens.

Adoption: is a dynamic process by which innovations are accepted and used by people. The process is influenced by a variety of factors:

Farmers’ perception of the relative advantages and disadvantages of technologies.
The efforts made by extension services to disseminate these technologies.
The policy environment.
The socioeconomic characteristics of farmers.
The characteristics of the farming system under consideration.
The characteristics of the technology.

The degree of adoption is measured by the proportion of land under the new technology (rate), whereas the intensity of adoption is measured by multiplying adoption rate by degree of adoption.

Appraisal: is an overall assessment of the potential relevance, feasibility and sustainability of an intervention prior to a decision on funding or implementation, used by decision makers to decide whether an activity represents an appropriate use of resources. Appraisal is related to ex-ante IA.

Attribution is the process by which a causal link is ascribed between observed (or expected) changes and specific interventions. It serves to assess those who, at different levels, were involved in a project or program or in the development and diffusion, and impact, of a technology.

Baseline is an analysis describing the situation prior to an intervention, that uses benchmark reference points against which progress can be assessed or comparisons made. An appropriate baseline is often an advantage because it allows before/after counterfactuals.

Beneficiaries are individuals, groups, or organizations, whether explicitly targeted or not by the project or intervention, that are expected to benefit from a development intervention.

Counterfactual: is the situation, forecasted scenario or course of events which might prevail for individuals, organizations, or groups were there no development intervention, or that would have occurred without the intervention. The design of realistic counterfactuals is a key to successful IA.

Effect: The intended or unintended change due directly or indirectly to an intervention.

Effectiveness is the extent to which given objectives of the intervention were achieved, or are expected to be achieved, taking into account their relative importance, whereas efficiency is how economically the given resources and inputs (funds, skills, time, etc.) are converted into results.

Empowerment: is the expansion of the assets and capabilities of people that enable them to participate in, negotiate with, influence, control and hold accountable institutions that affect their lives. It is the expansion of freedom of choice and action, a process which places or transfers decision-making responsibility and resources into the hands of those who are intended to benefit.

Ex-ante IA can be defined as an IA process carried out before interventions or policy change take place, typically by simulating events and forecasting the impacts of these changes. It is used to set the right direction, identify the existing situation, and opportunities for impact generation before an intervention is initiated or outcomes are generated, to ensure proper targeting of research, and for priority setting. A related term to ex-ante IA is “appraisal.”

Feedback is the transmission of findings generated through the evaluation process to parties for whom it is relevant and useful in order to facilitate learning. This may involve the collection and dissemination of findings, conclusions, recommendations and lessons from experience.

Impact pathway: a conceptualization of a project, program or organization’s envisioned pathway towards achieving impact. It is a visual description of the causal chain of events and outcomes that link outputs to goals. It includes network maps, which show the evolving relationships necessary to achieve the goal (implementing organizations, stakeholders, beneficiaries, etc.), and shows the project rationale and logic. It can be used in both ex-ante and ex-post IA contexts.

Indicators are quantitative or qualitative factors or variables that provide simple and reliable means to measure achievement, reflect changes connected to an intervention, or help assess the performance of a development actor. Indicators are increasingly important in summarizing the progress and direction taken by development-related activities.

Livelihoods have been defined at CIMMYT as the “stocks and flows of assets and the ways how these contribute to farmers’ well-being.” Thus the impact of technologies on farmers’ livelihoods must be considered in the broad context in which people live and operate. This implies a shift in thinking from maize or wheat as central objects of research, towards an approach that more comprehensively links crops to the stocks and flows of household assets and activities.

A meta-assessment is used for IAs designed to aggregate findings from a series of IAs or to denote the assessment of an IA to judge its quality or assess the performance of the assessors.

Performance: is the degree to which a research or development intervention or partner operates according to specific criteria/standards/guidelines or in accordance with stated goals or plans.

A performance indicator is a variable that allows verification of changes due to the intervention or relative to what was planned.

Performance measurement is a means for assessing the performance of interventions against stated goals;

performance monitoring is a process of continuous data collection and analysis to assess how well a project or program is implemented.

Outputs are products, goods and services resulting from development interventions. Outputs may also include changes resulting from interventions which are relevant to achieving outcomes.

Outcomes are the likely or achieved short-term and medium-term effects of intervention outputs.

Poverty is the failure to meet basic needs, in terms of food, health, housing, security, education, and vulnerability to climatic and economic shocks. Conventional definitions are based on having less than a given minimum disposable income, based on national averages from consumption baskets or $/day. Exclusion is another aspect of poverty linked to limited access to institutions (e.g., seed systems, credit organizations, markets, social networks).

Poverty map: is a geographic profile showing the spatial distribution of poverty within a country, used to identify where policies or interventions had or can have greatest impact on poverty. They are used to estimate geographically disaggregated welfare and inequality levels, and changes in small areas, therefore allowing geographic heterogeneity to be taken into account in IA studies.

Quality assurance: encompasses any activity concerned with assessing and improving the merit or the worth of a development intervention or its compliance with given standards. Examples of quality assurance activities include appraisal, reviews during implementation, and evaluations.

Social capital: the social capital of a society comprises the institutions, relationships, attitudes and values that govern the interactions among people.

Social capital assessment is an approach that integrates quantitative and qualitative tools that allow investigating institutions and networks and enabling collective action. It is normally implemented together with other tools.

Social IA is a framework to identify the range of social impacts and responses to an intervention or policy by institutions or people. It is used to assess how costs and benefits of interventions are distributed among stakeholders and over time. It is based on stakeholder analysis, and is useful to disaggregate data on livelihood assets and capabilities into meaningful categories. It uses a range of qualitative data collection tools (focus groups, semi-structured key informant interviews, stakeholder workshops, etc.) to determine impact, stakeholder preferences, priorities and constraints on implementation.

SPIA: the CGIAR’s Science Council comprises a Standing Panel on IA (SPIA) that contributes to the overall performance monitoring of the CGIAR centers by developing methodological inputs and by evaluating center performance in the domain of IA. SPIA provides CGIAR members and the public with information on system level impacts and a website with impact studies and methodological documents (www.sciencecouncil.cgiar.org/activities/spia/). SPIA is developing “Strategic Guidelines,” complementary to the present “Operational guidelines for IA on livelihoods.”

Stakeholders are agencies, organizations, groups or individuals having direct or indirect interests in a development intervention or its evaluation. They include those affected by a given intervention.

Vulnerability: denotes a condition characterized by risk and reduced ability to cope with shocks or negative impacts. It may be due to socioeconomic conditions, gender, age, disability, ethnicity or other criteria that influence people's ability to access resources and opportunities. Vulnerability is always contextual, and it must be assessed in the context of a specific situation and timeframe.

Top

Back to Contents

Previous Part

Next Annex