Harmonizing Evaluation Standards. The ECG was chaired by the director general of OED in 2005–2006 and, in October 2005, the group met for the first time at ADB’s headquarters. The ECG has developed good-practice standards for various types of evaluations and undertakes benchmarking studies to assess how each member applies them. Standards have been completed for public and private sector project lending and program lending, and are being prepared for the evaluation of country strategies and TA. Two benchmarking studies have been completed for private sector project lending, and one is under way for public sector project lending. ADB took the lead in developing the standards for evaluating policy-based lending and is leading ongoing work to develop standards for country assistance program evaluations.
The ECG is examining the feasibility of developing a robust peer review of evaluation functions in the multilateral development banks that will address issues such as (i) the independence of the evaluation office and the role of Management and Board of Directors, (ii) the selection of evaluation topics, (iii) adherence to good-practice standards, (iv) the quality of evaluation reports, (v) use of lessons and recommendations, and (vi) staffing and budgets. The group has appointed ADB to a task force to develop the peer review methodology.
Box 12: OED’s Work Programa
In 2007–2009, OED will (i) undertake fewer evaluations of individual operations but validate ratings in all project or program completion reports; (ii) increase the number of broad evaluations, such as corporate and policy evaluations, country and sector assistance evaluations, thematic evaluations, and impact evaluations; (iii) improve the evaluative content of the Annual Evaluation Review and Annual Report on Loan and Technical Assistance Portfolio Performance; (iv) prepare or revise evaluation guidelines; (v) sequence evaluations so that those done early in the 3-year rolling work program feed those planned for the later years; and (vi) promote knowledge management, including dissemination of findings and recommendations in accessible and digestible ways.
a Available: www.adb.org/evaluation/2007-work-program.pdf
Evaluating Country/Sector Assistance Programs. Because country and sector assistance program evaluations are having an impact on the formulation of the subsequent country partnership strategies, these evaluations will receive priority in allocating OED staff resources. The DEC strengthened feedback between evaluation findings and formulation of country strategies by requiring country assistance program evaluations for major countries to be prepared and discussed by the DEC before a new country partnership strategy is completed. A new appendix has been added to the document template for country partnership strategies to indicate how the strategy addresses the country assistance program evaluation and the DEC’s recommendations. A new product, the country assistance program evaluation update, will be introduced because (i) completion reports for country partnership strategies are being prepared, which will provide a better basis for preparation of country assistance program evaluations; (ii) some country portfolios are relatively small, and do not merit the depth of analysis undertaken in past country assistance program evaluations; and (iii) OED will undertake second country assistance program evaluations for an increasing number of countries.
Jointly Evaluating Country/Sector Assistance Programs. The Evaluation Network of the Development Assistance Committee of the Organisation for Economic Co-operation and Development has identified the evaluation of total official development assistance flows to a country as an important topic that has not been addressed by evaluators. Besides the ECG, and as part of the international harmonization agenda, there is increasing interest in undertaking joint evaluations for greater consensus and usefulness of results. The Joint Evaluation of Global Environment Facility Projects of 2005–2006 was OED’s first involvement in such an evaluation. In 2007, work will begin on a joint country assistance program evaluation in Bangladesh in partnership with the World Bank, the Department for International Development of the United Kingdom, and the Japan Bank for International Cooperation. This evaluation is expected to be finished in 2009.
Validating Country Partnership Strategy Completion Reports. ADB is beginning to produce country partnership strategy completion reports. OED will pilot a new evaluation product in 2007 in the Maldives, the country strategy completion report validation, primarily for countries with small portfolios for which production of a full country assistance program evaluation would not be an efficient use of resources.
Validating Project/Program Completion Reports. OED will change the way that individual projects/programs are selected for evaluation. The strategic objective is for the quality of completion reports to be sufficient to rely on this self-evaluation. Efforts to improve the quality of completion reports appear to have removed the upward bias of ratings at completion. The changes include the following: (i) moving away from the 25% random selection of projects and programs for which evaluation reports are prepared to a smaller, purposeful sample of 10 per year; (ii) OED would validate, based on a desk review, all ratings (with the validation assessment to be attached to the completion report) rather than commenting on draft completion reports—much of this work will be outsourced using funds that were previously spent on staff consultants for preparation of evaluation reports; and (iii) reporting success based on the combined completion and evaluation ratings as has been done in the more recent issues of Annual Evaluation Reviews. The projects and programs for evaluation will not be randomly selected. Selection triggers will include dispute over a rating, by OED or external stakeholders, and special interest.
Evaluating Impact. OED agrees with the general conclusions of the debate in the international evaluation community (IEC) about impact evaluations: (i) more rigorous impact evaluations are desirable; (ii) the methodology will be determined by issues related to data availability, time, and resources; and (iii) impact evaluations will be undertaken selectively, largely in the social sectors. OED is undertaking its first rigorous impact evaluation on microcredit in the Philippines as part of an evaluation study on the effectiveness of ADB’s microcredit operations. This will be completed in 2007. One impact evaluation is programmed per year.
Developing Evaluation Capacity. Evaluation capacity development is part of OED’s mandate. As of January 2007, OED had formulated 15 TA operations to this intent for a total amount of $4.35 million in Bangladesh, the People’s Republic of China, Nepal, Papua New Guinea, Philippines, Sri Lanka, and Thailand. Three phases can be distinguished in this assistance:
• Phase 1: TA focused on building a postevaluation capacity within a central agency and providing the means for disseminating postevaluation findings for decision making.
• Phase 2: TA aimed at establishing ADB’s project performance management system in central and sector agencies.
• Phase 3: TA aimed at building more generic results in monitoring and evaluation capability.
In the future, OED expects to work with evaluation units in DMCs to provide on-the-job evaluation experience and knowledge transfer, building on lessons learned from the evaluation of the 15 TA operations, not all of which were successful.
Promoting Portfolio Performance. OED began to provide real-time feedback on portfolio performance in 2001. The 2005 Annual Report on Loan and Technical Assistance Portfolio Performance highlighted serious and fundamental corporate issues. At the DEC’s recommendation, ADB’s Management prepared an action plan to address these issues.
Evaluating Business Processes. In connection with ADB’s reorganization of 2002, a working group on business process change was appointed in 2001 to review country strategies and programs and subregional cooperation strategies and programs; public sector loans (including project preparatory TA operations), private sector loans, and nonlending products and services (including advisory TA operations); and portfolio management. In addition to its reporting on portfolio performance, OED has included business process-related evaluation studies in its work program. Forthcoming evaluation studies will examine, for instance, the effectiveness