Operational guidelines for assessing the impact of agricultural
research on livelihoods
Good practices from CIMMYT
La Rovere and Dixon, CIMMYT, 2007
Back to Contents
3. Incorporating IA into projects and institutions
This section outlines the key elements of training in IA, and describes how to incorporate and budget for IA in projects. The last section describes CIMMYT’s experience in institutionalizing IA.
Box 13 gives three examples of training courses given by CIMMYT aimed at building overall capacity in IA as well as specific IA capacity within special projects. The three courses involved:
- Mexican partners and field work consultants as part of the planning phase of the Oaxaca IA project in CIMMYT, El Batan, Mexico, November 2005.
- Ethiopian and Ugandan partners and consultants monitoring and assessing impacts of SG2000 interventions, July 2006 in Kampala, Uganda.
- Turkish national program partners assessing impacts of the Winter Wheat Program (CIMMYT–ICARDA) in Turkey, November 2006, Ankara, Turkey.
Box 13 CIMMYT IA courses: examples of IA training good practice
A. Training workshop for assessing impacts of CIMMYT maize projects in the Central Valleys of Oaxaca’, CIMMYT, El Batán, November 2005 The purpose of the workshop was to familiarize staff and consultants with livelihoods and related IA concepts, and to show them how to develop and test a survey, and use automated data collection tools
B. Training workshop on IA of innovations promoted by National Research Systems and SG 2000 in Ethiopia and Uganda; Kampala, Uganda, July 2006
C. Joint CIMMYT – ICARDA training on IA, Ankara, Turkey, November 2006
|
Writing IA into projects, and developing a budget
Securing funding to incorporate IA in projects can be a challenge. In recent years, the average project budget has decreased and IA is often one of the elements cut during project planning or negotiations with donors. So, it is crucial to match IA objectives with available resources, and acceptable levels of rigor, transparency, and reliability. We touch in here what an IA plan needs:
- Terms of Reference, budgets, workplans, formation of teams, reporting, and internal and external communication of IA results, and so on.
The first consideration is whether an IA is required or necessary for a project or program. Not all projects merit a full IA, and some projects require only partial, rapid or light assessments. To clarify these, IA team members and managers have to link with others in the planning phases:
- At the Institutional Level, center management should consider the need for IA at the project proposal stage. If management identifies the need for IA, the next step is to discuss what IA is appropriate with the Center’s IA unit and ensure that the project budget adequately covers the IA.
- At the Program Level, directors should determine whether the project budget includes proper funds and, if not, should seek more funds from local, regional or global sources as appropriate.
- At the Scientist Level, an IA should be viewed as a component to maximize the benefits of a project, add value, enhance impact, and disseminate and communicate the results.
It is important to determine early on who will fund the IA. If it draws on the Center’s core budget it may be done inefficiently and with resentment. Donors are rarely willing to approve resources for IA in projects that they fund, since they usually conduct their own IA for projects.
Figure 9. Window of the template for project proposals with the Impact Assessment box.
The recommended procedures for building IA into projects are:
- Define an impact pathway and define in the proposals how impact will be demonstrated.
- Design and complete the IA plan before formally starting an IA. This will provide the IA unit or team with a framework that sets out the purpose of the IA, how it will be organized, the resources required. The plan covers the questions outlined in Step I: Clarify the IA.
- Tailor the definition of IA to the project. In many cases IA may be already embedded in the proposal and the existing evidence may be sufficient.
- Define the IA at the beginning of the project. Focus on the type of impact that the project seeks. Depending on the objectives of the project, IA can be a useful tool to ensure the effectiveness of the project.
- Carry out IAs at the institutional level regularly so that IA becomes routine practice.
- Do ex-ante IA and rapid scoping studies for main projects with core funds.
Criteria that need to be considered when deciding on the design of an IA design include:
- Type of project and purpose: for example, accountability, attribution, learning, monitoring.
- Financial resources: adequate funds/scarce funds, IA costs.
- Human resources: various (skilled)/few (unskilled) staff.
- Objectives: M&E versus IA, or external independent evaluation versus internal self-assessment.
- Scope, boundary, and time: urgent and rapid/comprehensive or more complex study without time limits, time required to complete.
- Balance between internal and external input: the former provides more insights and better data, while the input of external assessors will enhance credibility.
- Context: location, data, skills, capacity,
time, attitude, partners. This ranges between:
- Favored location, lots of data, positive attitude of participants.
- Marginal location, absence of data, defensive participants.
The main costs of Livelihoods M&E and IA studies and projects include:
- Design of methods and indicators.
- Real costs for each item, overheads, indirect costs, staff time (national, international).
- Cost for field data collection, local/international travel, field expenses, equipment.
- Cooperation with NARS and salaries, allowances and per diem for local collaborators.
- Communication, review, publication and dissemination costs (including modern media).
- Contingencies, normally 5% of total net costs.
- Collection and analysis of data. Indicative cost
for conducting socioeconomic field surveys for IA, including all costs
except international staff time, is often between:
- $25-30 to $170-200 per household, depending, for example, location
- RRA, PRA cost: $100-250 per community or focal group
The average duration of local IA studies is from 4 months to 2 years, but may be as much as 5 years for integrated and more complex global studies or M&E projects. IA studies with substantial field and field survey work are much more expensive.
- Impact specialist and GIS specialist: 4 total months - Support technical staff: 7 total months - Field consultants: 8 total months - Field expenditures and travel: US$12,000 - Publication and dissemination: US$5,000 - Institutional overheads and costs: US$7,000 (drawn from Oaxaca Study, with international scientist input) |
A rule-of-thumb estimate, based on metadata from CIMMYT socioeconomic studies, suggests that 3-5% of budgets (core funds and special project resources dedicated to IA) are spent each year on IA. As CIMMYT recognizes that IA supports the Center’s mission, it may be appropriate to allocate a similar percentage of budgets to IA. Obviously, the actual amount will depend on local costs, conditions, the types of IA, the locations at which the IA study is conducted, and the mandate and type of activities of the Center. The costs of conducting IA studies are often underestimated even though they represent only a small fraction of research costs. The costs of not obtaining impact data, improper targeting, and of missed opportunities to learning from IA, can in reality be much higher than the costs of conducting an IA.
Institutionalizing impact assessment
This section (see La Rovere et al., forthcoming) describes how CIMMYT institutionalized IA by means of a process that started in 2005, as an example for other CGIAR centers and NARS.
CIMMYT has a long tradition of IA. Until recently, IA focused on rates of adoption of improved germplasm and rates of returns on investments. Less attention was directed at measuring impact in terms of poverty reduction or livelihood security. Challenges facing the institutionalization of IA in CIMMYT involved its inherent complexity, the development of credible methods to measure broader impacts, mechanisms to ensure adequate engagement of staff and partners in learning through IA, and packaging the results to better meet the needs of users. CIMMYT’s Impacts Targeting and Assessment Unit (ITAU) developed and promoted wide-ranging IA to ensure that IA contributes to staff and institutional learning and improves future work. The process included:
- Collectively assessing the understanding of impact among CIMMYT’s scientists.
- Developing a people-centered framework for IA, with a focus on systems and livelihoods.
- Strengthening individual skills and capacity for high quality IA research.
In May 2005 the ITAU led a CIMMYT-wide workshop to develop and launch a learning and operational platform for IA. The workshop involved biophysical and social scientists from CIMMYT’s regional and global programs. The multi-stakeholder process embraced diverse stakeholders with different expectations of IA. The platform was built on existing staff competencies, and CIMMYT’s economic and IA experiences, and reflected the principles in the new CIMMYT strategy. The workshop also aimed to develop a practical framework for CIMMYT IA incorporating people-centeredness as well as a focus on systems, livelihoods, and poverty reduction, and to strengthen individual skills and the capacity for high-performance team research.
Box 15 and Box 16 show the main elements of the process.
Box 15. Institutionalizing IA: A model of workshops and events from CIMMYT’s experience.
| Event | Participants | Purpose |
| Inception IA workshop | Social and biophysical scientists, corporate communications, managers | Initiate the IA process and platform |
| Special sessions on IA for breeders | Wheat breeders Maize breeders |
Familiarize breeders with the key elements of IA |
| Workshop on priority setting and targeting | Scientists from different disciplines, and management staff | Apply IA practice to enhance targeting |
| Mid-term IA follow up workshop | Social scientists from the ITA program, plus key participants from SPIA, IFPRI, and IPGRI | Reflect, review progress, and share learning on IA |
| IA sessions on poverty and livelihoods during a science forum | All CIMMYT scientists | Familiarize CIMMYT with IA and present concrete applications |
| Impact pathways workshop | Social and biophysical scientists, corporate communications | Write impact pathways for main MTP Projects |
At the workshop participants shared their views, expertise and expectations.
Reaching consensus was not as important as learning from and understanding
each other. The approach fostered buy-in from participants. The workshop
included plenary and small-group sessions. The former were sometimes made
of pre-identified members to ensure geographic or disciplinary diversity
or homogeneity, or at other times participants were allowed to join groups
based on their interest.
Box 16. Key contents of the CIMMYT IA inception workshop, May 2005.
Understanding experiences with IA Eco-regional groups discussed: What is our capacity for IA? How do we define impact? What are our strengths and best practices in IA? What are challenges/weaknesses in IA? Increasing understanding of livelihoods, poverty and systems. Discussion in thematic groups focused on questions for participants such as: Are we working together effectively? Have we gained enough clarity on the concepts of livelihoods, poverty and systems? Have we gained clarity on the implications for IA? Impact assessment framework. An IA framework – described in this document - was proposed and adapted. Exercises were held on IA cases including impact of stem rust (Figure 10). Panel presentations on methods, approaches and best practices were conducted on CGIAR approaches and guidelines on IA, an application of the Sustainable Livelihoods Framework in Mexico, and on aspects of IA deriving from ILAC’s approach for Learning and Change. Implementing an IA platform, framework, and guidelines: to define best practices and an action plan to implement IA, and build IA into projects and operational modalities for global and regional programs. |
An IA framework (based on Patton, 1995) was proposed, discussed extensively in the workshop, and finally adopted for use by practitioners (resulting in Figure 4). This framework covers the actions and aspects that IA leaders or project managers need to consider before and during an IA.
Using this framework, participants discussed a series of practical case studies, describing the:
- Intended users and uses of the information generated,
- Stakeholders that need to be involved,
- Topics to be assessed,
- Scope and boundaries of the study,
- Critical questions to be asked,
- Disciplinary expertise required and ways to mobilize it, and
- Ways to use the results of the IA.
The case studies included one on the potential impacts of the spread of a new strain of wheat stem rust, the UG99 (Figure 10, Hodson et al., 2005). The impact pathway shows that direct impacts on yield, grain quality, and prices could result in increased grain imports, migration to cities, changes in cropping patterns, and reduced exports, which could ultimately affect both producers’ and consumers’, food security and livelihoods. The impact on national economies could also affect global markets, if wheat stem rust affected large exporters or several export countries simultaneously.
Figure 10. Charting complex impact pathways: wheat stem rust.
The IA workshop helped define the roles and modus operandi in IA for the
participants and overcame disciplinary and knowledge barriers. It also
started a process of continuous learning about how to conduct IA and integrate
work among programs. As a result, the IA capacity of scientists from regional
and cross-disciplinary programs was strengthened. One of the key benefits
was the opportunity to reflect on IA experiences; on individual, programs,
and regional capacities for IA; and on how impact is being understood
and defined.
Activities since then (Box 15) include a follow-up workshop in Rome in October 2005 to foster and assess progress on the action plan agreed during the inception workshop. This workshop brought together social scientists from CIMMYT headquarters and regional programs, other CGIAR centers (Bioversity, IFPRI), and representatives from SPIA. Participants reviewed case studies and methods that integrated traditional economic and livelihood methods, and discussed the skills required to implement them effectively. An overview of ongoing CIMMYT IA activities in the areas of breeding maintenance research, ex-ante studies, monitoring and evaluation of technology use, adoption, and ex-post impact studies was presented. As a result, more IA studies were generated that reflect the new thrusts and that integrate economic, qualitative and livelihood IA approaches.
On the recommendation of the CGIAR, a final workshop was held in late 2006 to map the impact pathways of projects in CIMMYT Medium-Term Plan.
Project-based training courses in IA started in 20068. These were supplemented by discussions, and sharing of data and documents among participants, and IA focal points were identified.
8 As a modality for tracking the impact of a public good and as an indicator of use of this document, since 2006, earlier versions of this document had already been used for training purposes in 5 projects in 12 different countries, including Mexico, Turkey, and several African countries. Over 60 staff and partners were trained directly and more than 60 indirectly.
Lessons learned included the realization of the benefits of building on past achievements and successes, of moving to broader livelihood, systems, and poverty IA approaches, and of integrating more closely with other disciplines than in the past, while still maintaining a strong emphasis on economics. Workshop participants felt a continued need for guidance on:
- Defining relevant datasets and the key variables to be included in an IA,.
- How to prioritize studies that need an IA.
- Budgeting for an IA when developing proposals.
The need to reduce the use of IA jargon and demystify IA terms was flagged to foster a broader understanding and use of IA at CIMMYT. This manual responds to this demand for guidelines, within the context of the new strategy, as well to the demand raised to strengthen capacity for IA.
Most people gained a better understanding and appreciation of IA principles and practice. The process of assessing the learning and change that occurred through these activities is on-going.
Much learning was implicit and passive rather than explicit. Behavioral change, chiefly for project managers, was often reflected in greater attention to IA in projects and a better common understanding of IA in collegial discussions. A good example is that, in developing the Medium-Term and Business Plans staff wrote impact statements based on impact workshop outputs.
Concrete changes were the institutionalization of templates for project proposals that have budget lines for IA (Figure 9). Since then, IA and innovation systems received increased consideration by CIMMYT management. There is also a growing acceptance of the technical and institutional complexity of IA, and that it has a key role in making strategic decisions and setting priorities, as testified by the increase in demand for ex-ante IA studies. Another outcome of the process is that it did raise the profile of IA within CIMMYT and initiated an institution-wide learning process. Biophysical and social scientists are now more aware of, interested in and curious about what IA means for them. IA is being developed increasingly around issues (rather than commodities) and, while still maintaining robust economic analyses, is moving from the previous focus on crops and adoption to the broader issues of livelihoods, different impacts, and attribution.
CIMMYT’s experiences (see also Annex 3) in enriching its IA practices are relevant and potentially directly useful to NARS, other partners, and other centers.
| Back to Contents |  |
Previous part | |
Next Part |
