back
Epilogue
Impact Assessment and Evaluation in Agricultural Research for Development
Task Force on Impact Assessment and Evaluation, European Initiative for Agricultural Research for Development (EIARD) [1]
Abstract
The European Initiative for Agricultural Research for Development (EIARD) is concerned with maximizing the contribution of agricultural research to the improvement of the livelihoods of poor people in developing countries. A clear view on means of measuring and enhancing the effectiveness of agricultural research for development [2] is, therefore, of considerable importance to its work. For EIARD, impact assessments and evaluations have four main objectives: (a) to enhance the developmental impact of agricultural research investments for poor people; (b) to provide information on the returns to investments in agricultural research for development; (c) to derive strategic and programmatic lessons for future investments in agricultural research for development; and (d) to provide information for use in public awareness work. These objectives suggest three sets of issues. First, if the main objective of impact assessment and evaluation is to increase the developmental impact of research, it is important to recognize and address the complex social, economic and political dimensions of pro-poor innovation. Thus, impact assessments and evaluations should not be limited to directly measurable impacts; they should seek to capture the complexity and non-linear nature of agricultural innovation and sustainable development. Impact assessments and evaluations should also be integrated as far as possible into research programmes, to facilitate internal learning processes and changes that enhance the probability of impact. Second, the difficulty of attributing impact to specific research investments requires special attention. In agricultural research organizations, exhaustive analysis of specific innovation processes is rarely possible because it is extremely difficult, costly, and time consuming. In most cases, a more feasible goal is to establish plausible links between research investments and development impacts. Third, for impact assessments and evaluations to be useful for decision making, policy design, learning, and public awareness, stakeholder interests and intentions need to be assessed, and appropriate information needs to be prepared for different audiences. It is also important to integrate evaluation lessons with other forms of learning.
Key words
Agricultural Research, EIARD, Impact Assessment, Attribution, Plausibility
Introduction
The proposal to establish EIARD was submitted by the European Commission in 1995, and recognised by the European Council of Ministers and the European Parliament in 1997. EIARD is implemented by a European Coordination Group (ECG) consisting of representatives of the European Commission, the fifteen Member States of the European Union, plus Norway and Switzerland. EIARD's purpose is to enhance the appropriateness and effectiveness of European policies and investments in agricultural research for development at national, regional and international levels, both in Europe and in developing countries.
A Working Group ensures the continuing activities of EIARD, and operates as a "Steering Committee". It meets about four times a year, and is fully accountable to the ECG, which meets formally once a year. The Working Group is assisted by ad hoc Task Forces, with experts provided by members. Task forces prepare draft positions on specific issues, for endorsement by the Working Group and eventually the ECG. An Executive Secretary, provided by one of the members of EIARD on a rotational basis, and hosted by the European Commission, is responsible for the day-to-day management of EIARD.
A Task Force on Impact Assessment and Evaluation was established in 2000 to develop a statement that could help to make impact assessment and evaluation studies more useful both for investors in agricultural research and for the scientific community. It was felt within the European donor community that most impact assessments and evaluation studies have little influence on policy making and the thinking of the professional communities concerned.
The present paper, a result of the Task Force's work, is addressed to people who prepare, conduct and read impact assessments and evaluations. Its intention is to provide a fresh view on the link between agricultural research and sustainable development, thereby helping to address the difficulty of attributing impacts to specific research efforts. It is hoped that this paper will inspire readers to design and conduct impact assessments and evaluations that contribute more to the livelihoods of the poor and to learning, and to do so more quickly than studies have in the past. Readers of impact assessments and evaluations may also find ideas and concepts that will enable them to judge the comprehensiveness and quality of impact studies and evaluations, and to make best use of them.
Impact assessment and evaluation from poor people's point of view
The sustainability of poor people's livelihoods is often very fragile, and the impact of development initiatives needs to be assessed in this context. Do development initiatives allow poor people to cope with and recover from stresses and shocks and maintain or enhance their capabilities and assets while preserving the natural resource base? To contribute to the improvement of sustainable livelihoods, it is necessary to address the following types of needs, felt by the poor:
· The need for more secure access to and better management of natural resources
· The need for a more supportive and cohesive social environment
· The need for more secure access to financial resources
· The need for improved access to high quality education, training, knowledge, technologies and information
· The need for better nutrition and health
· The need for better access to facilitating infrastructure
· The need for a more supportive policy and institutional environment
In this context, it is necessary to go beyond evaluation of research performance in terms of the products and services delivered, and assess the developmental outcomes for poor people in terms of poverty, food security, and natural resources use and conservation. Agricultural innovation and rural development are social, political and institutional processes as much as technological ones. Technology is just one among many other sources of innovation, and innovation is only one among many factors that influences sustainable livelihoods.
Evaluations of agricultural research for development should be designed and carried out within a holistic livelihoods framework. To raise poor people out of poverty in a sustainable way, all their needs (physical, human, economic, natural and social) must be addressed. Measuring research products and rates of return are important, but ultimately impact assessments and evaluations should examine how research products and services are being used and how their use affects people's lives, their societies and environments.
Impact assessment and evaluation from an investor's point of view
Inter alia EIARD members (as investors in research) need information on the overall return on their investment, lessons learnt in program implementation, and information that can be used for public awareness work. Each of these types of information serves different purposes and requires different approaches and methodologies.
Return on investment
Public investors account to their governments by providing evidence that agricultural research for development contributes to the overall goals of development policy - poverty alleviation, food security, conservation and sustainable use of natural resources. The desired developmental outcomes of agricultural research for development can usually be measured only long after research projects have been completed or terminated. For practical purposes, therefore, it is necessary to assess intermediate processes, products and outcomes that are requisites for impact and provide evidence of the likelihood of impact. Such "intermediate" or "process" indicators may include:
· Stakeholder perceptions of the programmes and their constituent projects
· Community-based opinions on the quality and relevance of projects, programmes, and outputs
· Changes in behaviour at the level of individuals, groups and organizations
Stakeholders in partner countries are the ultimate owners of the developmental outcomes of research investments, and they will almost certainly outlive most research programmes. Truly innovative research and development projects are risky and have uncertain outcomes; some of them are likely to fail. Therefore, insisting that each research project should accomplish its original goals, or predefined performance targets, may discourage innovative research and those who wish to explore the unknown. When evaluating agricultural research investments, criteria similar to those employed by venture capitalists and portfolio managers in assessing the value of their investments may be appropriate. Returns to portfolios of projects may be a more appropriate performance criterion than returns to individual projects, as a few big successes will compensate for many failures. Only an evaluation of the benefits and costs of the whole portfolio will provide useful information for investment decisions and policy making.
Learning
Impact assessments and evaluations have a major potential role to play in helping to identify lessons learned and their implications for future programming. Evaluation results can best be used in a context of discussion and debate. The open exchange of views can open up new possibilities for learning and organizational change. Greater understanding and new comprehension can emerge from this kind of collective deliberation.
The willingness to learn from failures as well as successes is a key component of effective agricultural research for development. Impact assessments and evaluations must recognize that a "failure" may actually represent "work in progress". This implies a shift of focus from ex-post evaluations, where impact is often assessed long after the end of the project, towards more on-going, participatory assessments which allow participants to adjust the programme's direction while it is still active.
Information for public awareness
Impact assessments and evaluations can provide valuable information for public awareness activities. To be of use they must provide understandable accounts of impacts and how and why they were achieved. This cannot be accomplished by simple input-output evaluations that obscure the key links in the chain from research towards impact on the ultimate development goals. Convincing arguments are needed that address public concerns and issues. Impact assessments and evaluations should ensure that truly innovative projects and programmes are recognized and rewarded.
EIARD's approach to impact assessment and evaluation
Plausibility of impact
Improvements in poverty alleviation, food security and the state of natural resources result from dynamic, interactive, non-linear, and generally uncertain processes of innovation, and lead to incremental changes over time. It is seldom possible to identify clear, sound cause-and-effect relationships between a given research project or its outputs and changes observed on the ground. It should be possible, however, to show plausible associations between research outputs and aggregate impact, and to rule out rival hypotheses.
The most crucial questions for impact assessment and evaluation are: "Was it worthwhile to carry out the research work?" and "What did we learn from it?" But linking research outputs to broad development results (attribution) is difficult and generally not feasible at a justifiable cost. The two main problems are the attribution of development results to specific research outputs, and the estimation of a realistic counterfactual or scenario of what would have happened without the research. A reasonable compromise is to trace out impact pathways and to establish plausible links between the research investment and the observed development impact(s). Determining plausibility along impact pathways is more feasible than proving impact, and it provides more meaningful information for learning, program steering, and accountability.
Elements of good practice in impact assessment and evaluation
Impact assessments and evaluations need to address and incorporate the following points:
· Identification of the agricultural research investment, and a description of its context
· The model or concept of innovation
· The objectives, scope and limitations of the evaluation
· The logic model underlying the project or programme
· The statement and testing of an impact hypothesis
· A discussion of other factors that could have affected the observed changes
· A critical review and comment
The rationale for each of these is considered below.
Identification of the object of the evaluation and its context. To construct plausible links between an agricultural research investment and impact targets, it is fundamental to identify and clearly describe them both. The severity of the attribution problem depends on the type of research activity and outputs and the role of different actors in the innovation process. Where the research output is a "hard" technology such as a new crop variety, it may diffuse more or less spontaneously among the farm population. In such a case, the researchers could be said to be the main innovators. This may have been essentially the case of the Green Revolution. Today, however, such a case is rare and researchers are only one among many sources of innovation. More typically, researchers produce "soft" results, such as information or advice, which other actors use as inputs in innovation processes. The researchers contribute to the innovation process but others contribute even more. In such cases significant challenges are encountered in attempting to attribute broad development impacts to specific actors, particularly to researchers who are more involved at early stages of the innovation process. Listing and describing the activities and processes through which research produced its outputs and early outcomes is the first step in constructing a chain of plausible impact. It is also important to note that research often generates new knowledge and ideas for future research, as well as political and administrative benefits, agricultural sector benefits, and broader economic benefits.
Model or concept of innovation. Any model that assumes a single and uninterrupted causative line between research and development is likely to be unrealistic. Research usually impacts on the livelihoods of poor people through highly complex, dynamic and interactive processes involving many different factors and actors in addition to agricultural research. Politics, cultural traits, social conditions, economic interests and the requirements of the surrounding technology are all being amalgamated into some development change. No single model will be applicable to describe the path from basic research to highly-aggregated development impacts in a systematic way. The path from research to development is long and winding and it is very difficult to attribute development impact to research outputs unambiguously or unchallenged by rival models of causation. By making the nature of the model or concept of innovation explicit and superimposing the research strategy onto it, helps the users of the studies to understand the internal logic of the impact assessment and to check the completeness of the inquiry.
Objectives, scope and limitations of the evaluation. It is important to clarify the objectives, scope and limitations of the impact assessment or evaluation. In our view, the primary value of impact assessments and evaluations is to enhance the developmental impact of investments in agricultural research for development. Other motivations include generating lessons from experiences; providing information for project management, priority setting and planning; and justifying research investments by providing information on the potential or actual developmental impacts. The objectives and scope of an impact assessment or evaluation should indicate the level of aggregation at which results are examined. Impact assessments can take on many different forms and, depending on the objective, their scope may be narrower or wider. For example, one evaluation may focus on a programme's outputs and their direct use, while another may seek to document impacts all the way up to highly aggregated effects on the environment or poverty. Whatever the level at which impacts are assessed, it is important that the limitations of the evaluation be noted, in particular in attempting to bridge the "attribution gap" between results that can in fact be documented and plausible impacts further down the impact pathway. This is important for establishing the credibility of the evaluation.
Logic model underlying the project or programme. A logic model (for example, a logical framework) aims to make explicit the rationale behind an agricultural research programme and the assumptions connecting inputs, outputs, outcomes and ultimate impacts. A logic model should state the reasons why the research programme was launched - what its designers and implementers hoped to achieve with the investment and how the research activities were meant to contribute to reaching development objectives. A plan for impact assessment and evaluation should be prepared before the project commences and it should be an integral part of project implementation. Nevertheless, development policies and the context of rural innovation change rapidly over time, so that the object of evaluation is, in fact, a moving target. Impact assessments can help to address the dynamics of development policy and knowledge generation by making transparent the evolution of a programme's logic.
Statement and testing of an impact hypothesis. The plausibility of IAE results can be enhanced if they have been explicitly developed in relation to an impact hypothesis - a statement about the impact that is expected to be found. Once constructed, the impact hypothesis drives the assessment. Where the project or programme has a log frame, the impact hypothesis can often be derived from that.
Discussion of other factors that could have affected the observed changes. Arguing plausibly and believably for the linkage between a research outputs and observed development impacts requires that other significant factors of influence be addressed and their potential effects weighed. Not all other factors can be examined, but the main ones (other than the research program) that could have affected the observed impacts should be identified and discussed.
Critical review and comment. Plausibility is generally built upon informed opinion - but opinions can differ and people can reasonably disagree. The findings of impact assessments and evaluations can be strengthened by outlining how other informed people agree or disagree with the plausibility of the cause-effect relationships. Especially important here are the views of partners and beneficiaries who are directly affected by the project or programme being evaluated. Even the best of programmes have negative effects on some stakeholders, and there are bound to be some people who feel they have been disadvantaged in some way. Plausibility and credibility are strengthened when dissenting points of view are presented and discussed. How do stakeholders' views differ? How and why do some disagree with the findings of the evaluation? Why do the authors believe their findings are nonetheless plausible?
Conclusion
There is a need to improve the practice of impact assessment and evaluation feedback in agricultural research for development. Scientific rigor is necessary but it is not sufficient to meet the demands for developmental information and insight that stakeholders expect from impact assessments and evaluations. The primary motivation for undertaking impact assessments and evaluations should be to enhance the probability that investments in agricultural research will improve the livelihoods of poor people. Other important uses of information from impact assessments and evaluations include informing donors on the returns on their investments, deriving strategic and programmatic lessons for future investments and providing material for public awareness campaigns.
As the number of actors involved in decision making concerning agricultural research increases, the need for negotiation and discussion also increases. Searching for plausibility rather than proof of impact can help to produce useful information and insight at reasonable cost. Application of the key criteria of plausibility does justice to the complexity of research-based innovation, encourages well-grounded arguments, and directs impact assessment and evaluation towards a more reasoned debate.