Part A

INTRODUCTION

1

Overview

1.1 Introduction

This Study was commissioned by the OECD/DAC Expert Group on Aid Evaluation at its Paris meeting in October 1996. The agreed purpose was:

...to undertake a synthesis study of non-governmental organisation (NGO) evaluations, supplemented where easily accessible with related data and information, in order firstly to provide an analysis and assessment of the impact, including efficiency and effectiveness, of NGO development interventions, and secondly, to provide an analysis and assessment of evaluation methods and approaches used.

NGOs are involved in a range of different activities.¹ One subset of activities involves emergency and relief activities. An early decision was that the synthesis should focus on NGO development interventions and not on NGO emergency and humanitarian activities.² Likewise, within the þdevelopmentþ umbrella, northern NGOs are involved in development education, advocacy and lobbying work within their own countries, and in networking internationally. The research team also took the decision neither to trawl the literature to assess impact, nor to synthesise studies examining the methods of assessing the impact of this cluster of development interventions and initiatives. Thus, the main focus of this Study has been development interventions implemented within developing countries. In that context, and as discussed further in Chapter 3 below, because of the nature of such interventions and the availability of literature, the major focus of the Study is on assessing the impact and analysing the methods of assessing the impact of discrete projects. Additionally, some (though far less) attention is also focused on institutional and capacity building projects and longer term development processes, not least because it has only been in the relatively recent past that NGO development work has expanded into these increasingly important areas and types of activity.

1.2 Methods and approaches used in this Study

How should the Studyþs purpose be achieved, and what methods and approaches should be used? Initially, it might be thought that this would be relatively straightforward: use a comprehensive database in order to gather all evaluation studies, or, if the numbers of studies are so large as to be unmanageable, obtain and make use of a representative sample of such studies. A late 1996 search of the OECD/DAC database of evaluation abstracts for the years 1986 to 1997 recorded a total of 74 entries using the category "NGO" and a total of 337 items using the keyword "NGO" from a total listing of 6,341 entries. Discussions with officials from almost all donor agencies during the course of this Study confirmed the initial view of the researchers that this database was incomplete and partial, and thus that it formed a wholly inadequate data set upon which to make a reliable synthesis of the impact of NGO development interventions. Indeed, an early conclusion of this Study, corroborated repeatedly throughout the research period, is that an international database of NGO impact evaluations simply does not exist. What is more, the case study work confirmed our initial hypothesis that there is not even a reliable and comprehensive database of all NGO evaluation studies at the country level in any of the 13 donor/country case studies.

The method of gathering evaluation reports was initially to ask the different members of the OECD/DAC Evaluation Group to gather together and send evaluation reports focusing on the impact of NGO development interventions, and then synthesise them in order to summarise what they were saying about impact and methods of evaluation used. It was the view of the researchers that for all its merits, this approach to data gathering would probably be deficient both in relation to impact data and in relation to evaluation methods. Their experience and knowledge of NGO development activities suggested that relying on donor evaluation departments to forward reports to the researchers would:

1. run the risk of omitting evaluations and related studies undertaken by official aid agencies but not commissioned by evaluation departments;
2. be highly likely to omit evaluations undertaken and/or commissioned by northern NGOs; and
3. would almost certainly omit evaluations undertaken and/or commissioned by southern NGOs and community-based organisations.

Additionally, it was the view of the researchers that such an approach would be unlikely to provide a rounded picture of methods of evaluation used to assess the impact of NGO development initiatives. This latter concern was rooted in the criticisms which have been voiced by NGOs of methods used to evaluate official aid interventions and, relatedly, because, in undertaking and commissioning their own evaluations, NGOs were unlikely to use methods of which they have been critical. Again, and as discussed in Parts B and C, these concerns were strongly reinforced in the evidence gathered in the country case study evidence.

As a result, it was decided that it would be necessary to try to supplement the data and information obtained from donor evaluation departments with data and information from NGOs within donor countries, and from NGOs and community-based organisations (CBOs) within developing countries. These data and information would focus both on evaluations of the impact of NGO development interventions and on evaluation methods and approaches. In short, the revised approach to be used aimed to gather data and information on impact and methods from three clusters of sources: from official donor agencies, from northern NGOs and from southern NGOs and community-based organisations.

The initial method of obtaining information from donors has already been described: it involved making contact with all members of the OECD/DAC Expert Group on Evaluation requesting them to send all relevant evaluation and related reports to the researchers. The initial request for studies and reports was supplemented in two ways: first by follow-up letters, and secondly by telephone/fax and face-to-face discussions as and when researchers went to different countries.

The attempt to fill gaps in knowledge about impact and evaluation methods by contacting northern and southern NGOs faced the major constraint of time and resources. The time set aside for the whole Study was only five months, of which the first six weeks involved reading the initial (donor-sent) evaluation reports, working out methods of gathering additional data and writing the Study's Inception Report, and the last three to four weeks were spent agreeing the Report's conclusions and finalising this document, including the Appendices. This left scarcely two months for additional data gathering. It was decided that the main additional data gathering process would involve the following:

• making postal and telecommunication contact with NGOs and NGO network and umbrella organisations explaining the purpose of the study and asking them to send what they considered were important evaluation studies and reports and manuals on evaluation methods;
• undertaking case studies in a selection of donor countries; and
• undertaking case studies in a (smaller) selection of southern countries.

In selecting countries for the case studies, the researchers were faced with a choice of undertaking a comparatively large number of case studies, but devoting only a very few days to each, or undertaking far fewer studies, but undertaking a more in-depth study. As explained in the Inception Report, it was decided to undertake a relatively large number of case studies. In all, 13 donor/country case studies were carried out, more within donor countries (eight) than in developing countries (five). The case study countries are listed in Box 1.1.

The fourfold purpose of the donor-based country case studies was:

1. To ensure that the donor-based evaluation studies sent to the researchers consisted of a complete set of recent donor-commissioned impact evaluations and, where necessary, to collect important additional studies.
2. To gather data on development impact from evaluations undertaken or commissioned by northern NGOs, focusing in particular on any synthesis, thematic or sectoral studies which might have been carried out.
3. To obtain information from NGOs on current attitudes, methods and approaches to the evaluation of development interventions.
4. To obtain data and information on linkages with southern NGOs in relation to evaluations undertaken or commissioned and interaction vis-…-vis methods and approaches.

The fourfold purpose of the southern-based country case studies was:

1. To gather data on impact from evaluations undertaken or commissioned by southern NGOs, focusing in particular on any synthesis, thematic or sectoral studies which might have been carried out.
2. To obtain information from southern NGOs on current attitudes, methods and approaches to the evaluation of development interventions and, in that context, to assess the extent to which methods are influenced by northern or other southern NGOs.
3. To obtain information on the extent to which southern NGO evaluations are commissioned by northern NGOs vis-…-vis being home-grown southern-based initiatives.
4. To obtain data and information on self-evaluation activities of community-based organisations and the extent to which knowledge about impact and methods are shared with southern NGOs, northern NGOs and donors.

The extent to which the Study was successful in meeting these objectives is discussed in Parts B and C.

1.3 Outputs and timing

Using the data gathered, the researchers carried out a number of tasks. The first was the production of the Study's Inception Report which was completed by the end of December 1996 and circulated to the members of the OECD/DAC Evaluation Group. It was discussed at a meeting of the Group in Copenhagen in February 1997. A second task was to undertake a synthesis of the main donor- commissioned evaluations of the impact of NGO development interventions. A third was to undertake and write reports for the 13 donor/country case studies. A fourth was to gather together some of the main/larger thematic and sectoral reports to analyse what data and information these provided on impact and evaluation methods. A fifth task was to use the data and information from the reports and country case studies to provide an analysis and assessment of evaluation methods and approaches used. A sixth task was to bring together all these different components and attempt both to provide some overarching reflective conclusions and to draw lessons from the data analysed. The final task was to draw together the threads in order to compile this Report and its appendices.

The timing of the work was as follows:

1.4 The structure of this Report

Following this introduction, the rest of the Report is divided into three parts. Part B, Searching for Impact summarises what is known about the impact of NGO development interventions. Chapter 2, Data and Data Quality discusses the sources and quality of the data used. Chapter 3, Donor-Based Impact Studies, summarises the impact results from the 10 main donor-commissioned studies, highlighting the areas of agreement and disagreement between these studies, and ending with a summary of the factors these studies consider critical in accounting for successes and failures. With the conclusions of these donor-commissioned studies as a backdrop, Chapter 4, The Country and Donor Case Studies draws out the main impact conclusions from the 13 donor/country case studies, highlighting where these confirm or challenge the results and conclusions of the donor-commissioned studies. Finally, Chapter 5, Thematic and Sectoral Studies of Impact summarises some of the main conclusions on impact drawn from some key thematic and sectoral studies.

Part C, Searching for Methods switches from impact to a discussion of methods of evaluating NGO development interventions. Chapter 6, Introduction provides an overview of this part and draws a range of conclusions concerning the degree of consensus among donors and between donors and NGOs on methods to be used, the gaps in methods, and possible future directions. Chapter 7, Methods and Approaches in Donor-Commissioned Studies, looks more closely at methods used and discusses the ways in which, and the extent to which, donors and NGOs are approaching evaluation differently, and the gaps between rhetoric and reality. Chapter 8, Methods and Approaches beyond the Donor- Commissioned Studies briefly summarises the data on methods provided by the case studies, drawing in places on the wider literature.

Finally, Part D, Lessons Learnt and Recommendations, draws a number of overarching conclusions, points to continuing gaps in knowledge about impact and evaluation methods, highlights a number of lessons drawn directly from the Study and outlines a number of ideas for follow-up.

This main Report ends with Annex A, References Cited. This is split into two parts: the first lists the main donor-commissioned studies used in the synthesis chapters (3 and 7); the second lists only additional texts cited directly in the main Report.

The Report contains 14 separate appendices. Appendices 1-8 consist of the eight donor-based studies commissioned for this Study and Appendices 9-12 are the five southern-based country studies commissioned for the Study. While each appendix contains its own annex and listing of studies cited, Appendix 14 brings all these references together, supplementing them with other texts used in the overall Study. It needs to be acknowledged, however, that some texts, especially some confidential evaluation studies and reports, were forwarded to the team on condition they would not be cited directly. In these cases, the works provided have not been referenced.

[contents] [next]

¹ While the term NGO is used throughout this Report, the authors acknowledges that there has been considerable debate about the desirability of using the term 'non-governmental organisation' at all, both because of the negative connotations of the term and because it clearly fails to capture the range of different organisations outside government, private sector organisations and official donor agencies involved in the development process. The report also follows the United States convention of using the term 'private voluntary organisation' (PVO) to describe United States' non-profit organisations.

[text]

² This decision was influenced in part by the knowledge that at least one major parallel initiative was under way, supported by the former British Overseas Development Administration (ODA), to undertake a synthesis of NGO activities in emergencies. See Borton and Macrea, (1997). Following the British elections in May, the ODA ceased to exist and the Department for International Development (DFID) was created distinct from the Foreign and Commonwealth Office.

[text]