Part C SEARCHING FOR METHODS 6 INTRODUCTION
Part C moves away from impact issues to focus on methods and approaches of assessing impact. It is divided into two chapters. Chapter 7 summarises the methodological approaches used in the donor-commissioned studies. Chapter 8 looks at methodological issues beyond this narrow cluster of studies. Most of this chapter summarises the information provided in the country studies; however it also makes reference to some additional literature, highlighting not merely interest in the issue of methods and methodologies but also the fact that this interest extends well beyond the confines of what NGOs themselves are doing both individually and in groups. There would appear to be six clusters of conclusions emerging from this discussion. First, and as just noted, there is considerable interest across donors, NGOs and the wider research community in evaluation and evaluation methods. Some ten years ago outside the United States (where there has been a far longer tradition of interest in, and practice of evaluation of PVO development initiatives), most interest in NGO evaluation was probably focused within the official donor community rather than within and among NGOs. This has changed: today, there is widespread and growing interest in evaluation and evaluation methods among many NGOs and within the linked and wider research community. Secondly, however, there is far from unanimity about how to evaluate NGO development interventions. In part, this can be traced directly back to two of the main conclusions arising from the discussion of impact: that the impact of discrete projects is usually profoundly influenced by the wider context, and that development is an immensely complex process. Thus uncertainty about impact and the relevance of its differing causes ripples through into uncertainties about methods of assessing it. In part, too, as discussed most fully in Chapter 8, uncertainty about how to evaluate arises because NGOs are involved in a variety of different types of development intervention, many of which are ill-suited to more orthodox/traditional approaches. Additionally, however, differing views about methods arise because of differing views about the "why" of evaluation. If the purpose of evaluation is to provide an ex-post assessment of achievements to date, then the process of evaluating is likely to be different from evaluation whose purpose is to deepen understanding of what has happened in order to enhance future performance. Together, these different factors and influences provide at least some of the explanations for the fact that there is both so much activity focused in trying to develop and use indicators (qualitative and quantitative) with which to judge impact, and so little evidence of a growing consensus on the appropriateness and use of more "holistic" indicators. Thirdly, and relatedly, it would appear that there is probably as much, if not more, experimentation within the NGO and linked research community focusing on new and different evaluation methods than there is within the official donor community. But this does not imply that it is only donors who need to learn from and listen to NGOs, and not vice versa. In particular, it is apparent that a growing number of NGOs have seen merit in, if not, in some cases, the necessity of addressing some of the mainstream issues which donor evaluations have long considered to be essential tools of evaluation: the focus on achievement against objectives, efficiency, effectiveness and sustainability. But to the extent that these lessons/insights are being absorbed and, to an increasing degree, accepted by NGOs – though there remains some vigorous criticism of the use/misuse of such approaches – interest and debate have moved on to the less tangible and quantitative aspects of evaluation, around which debate, discussion and research are now increasingly focused. The fourth conclusion is that both the debate and discussion of evaluation methods and experimentation tend to be focused predominantly on larger and more medium-sized NGOs. The evidence suggests that far more smaller NGOs, and even CBOs, are involved in evaluation, especially different forms of self-evaluation, far more than one would suspect from reading documents housed in donor organisations, or even within northern NGOs. Yet, they tend not to be involved in, or able to take part in and make use of many of the approaches being debated and discussed. In part, this is because of ignorance, rooted in failures or gaps in communication, but, in part, it lies in an inappropriateness of methods for the smaller organisations. Smaller NGOs often do not want to become involved in more complex and sophisticated methods because the time, money and human resources required would often change the nature and size of the NGOs involved, perhaps eroding the very attributes in which their (potential) successes are rooted, not least their smallness and the flexibility this provides. Fifthly, it would appear that in spite of much talk about partnerships – between donors and NGOs, between northern and southern NGOs and between southern NGOs and CBOs – the closer one reaches down to the immediate beneficiaries at the grass-roots the more vocal is the complaint that the flurry of activity in relation to evaluation and evaluation methods continues predominantly to be a top-down externally-driven exercise. Large gaps remain both in relation to sharing written reports and discussing conclusions with beneficiaries. Such gaps reinforce the view that those beyond view evaluation more as an audit to ensure that funds are well spent than a process dominated by the desire to learn in order to enhance future impact. This is not only a complaint which northern NGOs make about donor-commissioned evaluations; it would appear to be a complaint which southern NGOs make about northern NGO evaluations and which community-based organisations make about southern intermediary NGOs. Sixthly, and finally, the evidence gathered confirms that one of the reasons why impact data on NGO development interventions is often so poor lies in inadequate to non-existent monitoring of project performance and the absence of any base-line data against which to judge performance. This leads to the important policy conclusion that in these cases it is insufficient to focus solely on evaluation methods and techniques if one is trying either to improve impact or to learn more about impact; it is necessary to focus on the wider issues and gaps in planning appraisal and monitoring. |