There is significant variation when it comes to who Norwegian NGOs are using as evaluators, but little evidence to suggest that evaluations are mostly carried out by people who have had a stake in the project themselves. There is little fear of external consultants as such, but also a tendency to select consultants from a relatively small group of consultants external to, but familiar to the NGOs. A typical team will consist of a representative from the project/organisation, a local consultant and an international consultant.

CARITAS never or seldom uses international consultants, but may participate with their own staff and national or regional expertise. Redd Barna and Norwegian Church Aid use independent Norwegian or international experts, and in addition their own staff and national expertise. Strømme Memorial Foundation depends to a large extent on its regional representatives, but are also using national and international consultants. Red Cross draws on a small group of consultants who are familiar with the principles and programmes of Red Cross work. The organisations will always involve project staff and national consultants, but will not necessarily participate with staff from their own headquarters. Internal evaluations or self-assessments are carried out by the local organisations, but are not always categorised as evaluations since they may not always result in written products.

The NGOs appear pragmatic when it comes to the internal/external dimension. All emphasise the importance of increasing the partner's involvement in and ownership of the evaluation process in order to enhance the potential for learning. On the other hand 'in-house blindness' and conflicting interests among project staff and/or partners are acknowledged as a constraint where external people can provide a fresh and independent look - given that the consultants are knowledgeable about the country and the technical aspects of the projects and have the necessary interpersonal skills. The ability of the team to carry out a 'good' evaluation seems to be more important than where the team members come from.

3.7.1 Building local evaluation capacity

The changing relations described between NORAD and Norwegian NGOs could be repeated in the new relations between Norwegian NGOs and their partners, where the latter increasingly prepare and integrate evaluation plans as part of their own programmes.

In a partner concept mutual control and learning would be required, and southern NGOs would have to define their own evaluation needs, prepare their evaluation strategies and negotiate such plans with their funding organisations. The partners of Norwegian NGOs are said to be in the process of building their evaluation capacity, but there are so far few examples of southern NGOs being in control of the evaluation function or of Norwegian NGOs supporting such a move.

External national and regional consultants are used in most evaluations, but there are problems finding qualified personnel, with often ad hoc procedures of selecting consultants (e.g. personal networks), weak analytic and writing skills among local consultants, weak efforts by Norwegian NGOs to systematically build institutional evaluation capacity in partner organisations and in other relevant southern resource centres.

3.8 Gathering dust or creating change?

It is difficult to assess the relative importance of evaluations for introducing new ideas or generating change and renewal in NGOs. The internal use of evaluations seems to be most important, for instance in initiating discussion both between the Norwegian NGO and their counterparts, and within the organisations themselves. But we have little evidence to help us understand and assess the dynamics between the various partners in the planning, implementation and follow up of evaluations.

NGOs which have both guidelines and procedures for how to prepare, implement and follow up evaluations, and a board to whom findings are reported, seem to benefit most from evaluations. Where there are no such organisational mechanisms, benefits depend more upon individual initiatives and have less impact on institutional memory.

All evaluation reports are legally the property of the organisations and/or NORAD, but with few exceptions the organisations are willing to share the reports publicly. A common problem is, however, that in practice reports are not easily available. The best and sometimes only source of information about such reports is the desk officer, who may be able to retrieve one of the few available copies in his/her own file. It is exceptional to find evaluation reports registered in a database or easily available in a library or documentation centre, either for sale or free distribution. Most of the larger NGOs have their own information desks, but evaluation reports are rarely used for broader dissemination of findings or as inputs to development education and fund-raising.

4

SUMMARY

• The report shows that the number of NGO evaluation studies started to increase from the mid- 1980s, and Norwegian NGOs are currently involved in up to 500 evaluations each year. Significant changes have taken place, but there is considerable scope for improvements and change in policy and practices. There are also large variations between the organisations.
  
• The quality of the reports vary considerably and are in general more process than impact oriented. There is a lack of information that would allow impact to be measured.
  
• NORAD played an important role in introducing evaluations to the NGO sector, but the NGOs adapted rapidly to the new requirements, acknowledged the relevance and usefulness of evaluations and have now taken the lead in setting evaluation priorities.
  
• Evaluations are still more the concern of Norwegian NGOs than their southern counterparts, but a change of roles is underway.
  
• Several NGOs are searching for a new evaluation concept which is less external and donor-driven, with a narrow project focus, limited in scope and time, focus on control, accountability and numerical achievements due to new objectives oriented planning systems and broad framework agreements with NORAD.
  
• NGOs are searching for new approaches to evaluating institutional development, sustainability, country/regional policies and programmes and some have also allowed partner evaluations.
  
• Nearly all NGO evaluations are currently initiated by Norwegian NGOs with a few broad studies commissioned by the Ministry of Foreign Affairs. All NGOs emphasised the learning aspects of evaluations, but accepted also the control and accountability dimensions.
  
• The internal capacity for carrying out evaluations is low with no separate staff or monitoring and evaluation units in any of the organisations.
  
• Few organisations have specific evaluation policies and guidelines, and they are all using a small part (one to three per cent) of their budget for evaluations.
  
• Norwegian NGOs are increasingly expected to measure and document results and impact, and some of the larger NGOs have tried to meet these expectations. There is, however, considerable confusion about direction and how to measure results.
  
• NGOs appear pragmatic when it comes to who would be the best evaluators. Emphasis is given to partner involvement and local knowledge, but the ability of the team to carry out high quality evaluations seem to be more important than who the team members are or where they come from.
  
• Evaluation reports are in principle available to the public, but for practical reasons not easily available in practice.

[contents]

BIBLIOGRAPHY 1

Selected NGO evaluation reports³⁰

Bangladesh

Lily F.B. et al. (1991) An endeavour for better life: Final evaluation on first phase January 1989 to 1991, Nilphamari: Local Initiative for Emancipation (LIFE) Bangladesh.

Russell N. et al. (1991) Assessment of the Bangladesh Red Crescent Society's disaster preparedness, cyclone preparedness and community development programmes, Ruthin, North Wales: CDR Resource Group.

Breidlid A. & Raustøl A.K. (n.d.) Evaluation Report. The Bangladesh Northern Evangelican Lutheran Church (BNELC) - Education programme,, Oslo.

Elgsæther 0., Lindøe P. & P. Baidya (1987) Inntektsbringende tiltak for å øke selvfinansieringen av helseprosjekt på Sylhetfeltet.

Greiner, T. (1992) Review of SIDA-financed Vitamin A Deficiency Control Programs in Bangladesh, Uppsala: International Child Health Unit, Uppsala University, Worldview International Foundation (WIF), Nutritional Blindness Prevention Programme / Pastor Strommes Minnestiftelse.

Haugen, P. C. (1990) Sluttrapport fra 1985-90, Lalpur Mulberry Programme, Bangladesh, The Development Fund.

Hoel V., Mathias M. & Rahman A. (1988) A review of the project implemented by the Society for Care and Education of the Mentally Retarded, Bangladesh (SCEMPB) with support of the Norwegian Association for Mentally Retarded (NFPU) and NORAD, Dhaka, NORAD.

Mia A., Momen M.A. & Islam M.H. (1986) An interim evaluation of the WIF pilot project on Nutritional Blindness Prevention, Dhaka: Institute of Social Welfare and Research, University of Dhaka.

Maal B. (n.d.) Evaluering av RDHCF (The Rural Development and Health Center Foundation).

Steen S. H. & Fernando G. (1990) Process report. SUNB Health Project: Planning for Nationalisation. Report from a consultancy visit, Oslo: Diakonhjemmets Intemasjonale Senter.

Steen S. H., Sannesmoen J. & Malakar M. (1989) Project Review. Santal Mission Norwegian Board - Health Project Bangladesh.

Vold 0. (1990) Evaluation Report Amnura and Adompur clinics , Bangladesh Lutheran Mission, Norwegian.

Wilding J. & Russell N. (1991) Bangladesh disaster preparedness assessment for the League of Red Coss and Red Crescent Societies, Geneva, Peasmarsh, East Sussex CDR Resource Group.

Chowdury M.H. (red) (1989) Worldview International Foundation Nutritional Blindness Prevention Program Bangladesh. Final report (2nd phase), Dhaka: Institute of Nutrition and Food Science, University of Dhaka.

Eritrea - Ethiopia

Aricultural Rehabilitation Programme:Evaluation report 1986,1988, Kharthoum: Eritrean Relief Association, 1989.

Agricultural Rehabilitation Programme, Norsk Folkehjelp, Kharthoum: Eritrean Relief Association, 1987þ1990. SSE Programme.

Agricultural Rehabilitation Programme:Evaluation Report, 1986þ1988, Kharthoum: Eritrean Relief Association, 1989.

Agricultural Rehabilitation Programme in Eritrea, Framtiden i våre hender. The Development Fund. Kharthoum: Eritrean Relief Association, 1985þ1989.

Araya, Z., Sahel-Senhit-Barka agricultural rehabilitation programme (SASEB-ARP) with special reference to Rora-Nacfa, Wada Lebca / Sheib Gedged, Western Zara / Erota and Algede /Cufa Areas: a proposal, Oslo, Norsk Folkehjelp, 1987.

Araya Z. (1989) Midtveisevaluering av Sahel-Senhit-Barka, Semharjordbruksrehabiliteringsprogram: konsulentrapportforelagt Norsk Folkehjelp, Stockholm: Orgut Cons. English version: A mid term evaluation of the Sahel Senhit-Barka-Semhar.

Nordstokke K. et al. (1990) Bale Water Project, Ethiopia. Project Evaluation for Norwegian Church Aid (NCA).

Berg T. (1987a) Ethiopian Evangelical Church Mekane Yesu-s / South Ethiopia Synode - Community Development Programme - Review of the Agricultural Research Activities. Norwegian Church Aid.

Berg T. (1987b) Norwegian Church Aid / Ethiopia - Dello Project: agricultural development planning: Report from a project visit, NORAGRIC.

Berg T. (1987c) Review of the agricultural research activities, NORAGRIC, Norwegian Church Aid.

Berg T. & Skjerve E. (1992) The Seed Bank and Bull Servicing programmes of Tigray. Reports from a review mission, The Development Fund.

Brodersen C. (1986) Eritrea, 1986, Rapport fra en studiereise for Landbrukets Eritreaaksjon, SAIH.

Community Development Programme (Ethiopia), Oslo, Norwegian Church Aid. Six reports.

Deelstra J. (1988) Bama/Eritrean Relief Association Project on reafforestation and soil and water conservation in Asmat, Eritrea: a feasibility study, NORAGRIC.

Norwegian Church Aid (1987) Evaluation report - Rora Habab Comprehensive Programme.

Grova M.E. (1989) Experience with implementing integrated community development projects with an environmental bias in Ethiopia, Addis Ababa: Redd Barna.

Hauge T. & Kvalvaag L. (1990) Mega Development Center: Agriculture Extension Programme 1985þ90. From research field to unit farins,Community Development Programme (Borana, Ethiopia), SSE- programme. Norwegian Church Aid.

Haugen T. (1987) Tree planting in Ethiopia: experiences from the Konso area. Oslo: Landbruksforlaget, Care Norge.

Helland J. (1989) An anthropologist looks at the Wogda project. A report to Redd Barna - Ethiopia. Bergen: Christian Michelsens Institutt, DERAP, 36. [Wogda Project, Conservation and Rural Development Project, Tegulet and Bulga, Ethiopia]

Lexow J. (1988) Community development and sociocultural factors: Redd Barna - Ethiopia, Oslo: Development Consulting (DECO), [Bolosso Rural Development Programme, Conservation and Rural Development Project, Tegulet and Bulga].

Lexow J. (1989) Report from socio-economic consultant to Redd Barna, Ethiopia, Oslo, Development Consulting (DECO).

Mitchell J. & Slim H. (1989) A review of EECMY-SES' Community Development Programme in Sidamo and Gamo Gofa regions of Ethiopia, Peasmarsh, Sussex: Rural Evaluations.

Mitchison N. A., Tsenga E. & Bloom B.R. (n.d.) Report to the Norwegian Agency for International Development (NORAD) in appraisal of the Armauer Hansen Research Institute, Ethiopia, Redd Barna.

Olafsson H. & Hammer T. (1989) Mid-term Review Dello Development Project 1987-1991. Government of EthiopialNorwegian Church Aid.

Rahmato D. (1990) A resource flow systems analysis of rural Bolosso (Wollaita): a report for Redd Barna-Ethiopia, Addis Ababa: Addis Ababa University. Institute of Development Research.

Berg T. et al. (1986a) Report from 'Baseline study of Redd Barna Ethiopia's conservation and rural development project in Gosh Bado, Aregesh Area, Shewa Region Ethiopia', Oslo.

Berg T. et al. (1986b) Report from Baseline study for phase II of Redd Barna Ethiopia's integrated rural development programme in Bolosso - Welayfa - Ethiopia, Oslo.

Mogedal S. et al. (1985) R eport on NOR,4D supported health services through Norwegian Lutheran Mission in Ethiopia, Oslo: Norwegian Lutheran Mission.

Faaland J. et al. (1990) Review of Norwegian Church Aid's assistance to Ethiopia, Bergen: Chr. Michelsen Institute.

Ruden F. (1986) Bale Water Supply Development Programme, Norconsult, Norwegian Church Aid.

Sivertsen T. (1986) Med veterinterutstyr til Eritrea. Rapport fra en reise til Eritrea for Veterinxr Eritreaaksjon, aprillmai 1986.

Suleiman M. (1987), Ethiopian Evangelical Church Mekane Yesus / South Ethiopia Synode - Community Development Program - 1985-87 Agricultural Afforestation Evaluation team reportment Committee, Norwegian Church Aid.

Tekle T.G. & Smith P.D. (1989) Soil and water conservation in Tigray, Ethiopia, Oxfam, The Development Fund.

Kirkens Nodhjelp (1987) Evaluation team report, Ethiopia.

Vinsrygg S. (1989a) Analysis of women involvement in a Community Development Programme in SidamoGamo Gofa, Ethiopia, Oslo: Norwegian Church Aid.

Vinsrygg S. (1989b) Analysis of women involvement in Evangelical Church Mekane Yesus, South Ethiopian Synode, Community Development Programme, Ethiopia, Norwegian Church Aid, Oslo.

Yetola A. & Yetola A. (1987) Evangelical Church Mekane Yesus, South Ethiopian Synode, Community Development Programme Evaluation and Planning Mission of Water Development Committee, Norconsult, Norwegian Church Aid.

Nicaragua

Bye V., Baumeister E. & Quinteros E. (1990) Prosjektevaluering. Utviklingsprosj ektet Det Kgl. Selskap for Norges Vel med UNAG - ECODEPA. Innkjopslag Tienda , Skjetten: Det Kgl. Selskap for Norges Vel.

TECNOPLAN (1989) Evaluación del Perfil Pecuario Pueblo Nuevo. Esteli Midinra Region I, Managua: Tecnologia y Planificacion.

Hersoug B. (1990) 'Innviklet utvikling - en analyse av et norsk fiskeriprosjekt i Nicaragua' In Internasjonal Politikk, Vol. 48 (3).

Holt-Gim6nez E. et al. (1991) Integrated farms systems project, PNþ51. Mid-term evaluation, CARE Norway.

Lie M. (1985) Rapport I. Prosjekt gjennomgang av det norske helseprosjektet i Nicaragua. Rapport 11

Norheim R.H. (1989) Kvinner i samvirke: integrering av bygdekvinner i kooperativene i region IV, Nicaragua, Skjetten: Det Kongelige Selskap for Norges Vel.

Sandved E. (1989) Administracion y organization de la pesca artesanal en la region Atlantico sur: un estudio sobre la. administracion publica en relacion a la implementacion de un plan de desarollo de la pesca artesanal: un infome a NORAD, Managua.

Stålsett S. & Thorburn M. (1991) Rapportfra evaluering av CIEETS, Nicaragua, Norwegian Church Aid.

Torres 0., Morales N. & Montenegro W. (n.d.) Evaluación del pryecto 'Desarollo Comunal Integral'. In- fonne final de consultoria, Managua.

Zimbabwe

Andersen, T. (1988) Lærerprosjektet i Zimbabwe. 1983-1988. Prosjektgjennomgåelse. 1989þ1994, 116 s. (Fellesrådet for det sorlige Afrika.)

Chinemana, F. (1986) Evaluation of the Redd Barna - Zimbabwe training programme, with special reference to Nyarungu training centre, Redd Barna, Zimbabwe.

Chinemana, F. & Jareg P. (1989) Supplementary feeding in Zimbabwe. A case study, Redd Barna - Zimbabwe.

Goerdt, A. et al. (1987) Evaluation of the Zimbabwe Red Cross Socieiy's comniunity-based rehabilitation programme in Gutu district, Zimbabwe, Harare.

Zimbabwe Christian Council (1989) Evaluation of Zimbabwe Christian Council's work, Norwegian Church Aid.

Leesland, A., De Ridder C. & Manda S. (1990) 1990 1NORIWI111: Zimbabwe - Training programme for women trade unionist. Evaluation report, Landsorganisasjonen i Norge.

Moyana, R. (1988) Evaluation of the Norwegian Council of Southern Afica - Zimbabwe Teacher Programme, University of Zimbabwe, Fellesrådet for det sørlige Afrika.

Steen, 0. I. (1989) Norwegian non-governmental involvement in Zimbabwe. Country report within the project: 'non-governmental development actions - aims and means', Oslo: Norsk Utenrikspolitisk Institutt, NUPI rapport, vol. 138.

Other NGO reports

Berg, T. (1988) The Abakaliki development project, Nigeria: an ex-post evaluation for Norwegian Church Aid. NORAGRIC, Centre for Partnership in Development.

Drifts- og stillingsstotte: utredning av prosjekter som har hatt drifts- eller stillingsstotte i mer enn 10 år / Norsk MisjonsrAds Bistandsnemd. - Oslo: Norsk Misjonsråds Bistandsnemd, 1987.

Mukhopadhyay, M. et al. (1985) Evaluation of Redd Barna development activities, Oslo: Ministry of Development Cooperation (DUH), Evaluation Report, 4.85. 80 s.

Myklebust, P. et al. (1989) Evaluation of the Employment Promotion Programme (Epp-Programme) in Africa and Asia and the Preceding Employment Campaign of the ICFTU 1978-1988.

Evaluation of the Red Cross Action Team project in Southern Africa. Proposal of an integrated disaster preparedness programme, Geneva: Nils Gussing Development Consulting Service, 1988.

Thorsrud, E. et al.(1985) Evaluation of trade union training: financed under the LO-NORAD framework agreement 1980-1985, Oslo: Ministry of Development Cooperation (DUH), Evaluation report, IA.85.

Archetti, et al. (1983) Evaluation of two NORAD-supported development projects in Latin America under the responsibility of Norwegian mission organizations, Oslo: NORAD, Evaluation Report, No. 1/1983.

Mogedal, et al. (1989) Fra ideologi til praksis. En evaluering av Norsk Folkehjelps internasjonale arbeid, Oslo: Diakonhjemmets Internasjonale Senter.

Grogaard, J.B. & Gleinsvik A. (1984) A study of the organisations and countries supported through the agreement between NORAD and LO-Norway,.Oslo: NORAD.

Itotoh, J.0., Obanya P.A.I. (1986) Evaluation of WCOTPIFIPESOINUFOIAATO Programmes on 'Content of Secondary Education in Africa' 1984-1986.

Agapitus, et al. (1991) Learning by production: a realistic method of education? Evaluation of the Loudima Model, Elverum: Namibiaforeningen.

Manger, L.0. (1986) S ocial anthropological report, Development in Sinkat, the Red Sea Hills: Norges Rode Kors.

Mide, B. (1988) From policy to practice. Women Support Unit as an implementation for increasing women's participation in Equatoria Programme's development activities. Lillehammer: Oppland Distriktshogskole.

G. Haaland, et al. (1981) Norwegian Church Aid: Sudan programme,Oslo: Norwegian agency for international development (NORAD), Evaluation Report, 2.81.

Norwegian Red Cross international AIDS programmes, Geneva: Nils Gussing Development Consulting Service, 1991

Skjonsberg, E. (1986) Private organisasjoner og kvinnerettet bistand, Oslo: Women in Development Consulting (WIDCO).

Steen, 0. I. (1988) Mål og midler i norske private organisasjoners utviklingshjelp: En undersokelse av 11 organisasjoners prosjektvirksomhet i den tredje verden. Oslo: Norsk Utenrikspolitisk Institutt, NUPI rapport, nr. 119. 114 s.

Study on administrative strategy for rural development, volume 1-3 Juba, Sudan: Norwegian Church Aid, 1987.

Tvedt, T. (1987) Norwegian Church Aid Sudan Program and aid addiction in Southern Sudan, Juba: Norwegian Church Aid.

Fuglesang, A. & Chandler D. (1989) What's inside? Learnig with children in the streets, Colombo: Redd Barna Sri Lanka, 1989. 132 s.

Wicknertz, L., Woldsnes M. & de Vylder S. (1987) Stotte til Intemational Cooperative Alliance's Development Programme 1985-88.

Tvedt, T. et al.(n.d.) De private organisasjonene som kanal for norsk bistand. En analyse av utviklingstrekk i forholdet mellom den norske staten og den tredje sektor.

These are the main reports from the Norwegian NGO evaluation. Other sub-reports are:

Dalseng, T. Statlige utbetalinger til private organisasjoner.

Hodnebø, K. Norske niyndigheters ritningslinjer for private organisasjoners bruk av norske statlige midler.

Morvik, K. Kunnskapsutvikling og løring i fire norske frivillige organisasjoner.

Steen, 0. I. Privat bistand og humanitart hjelpearbeid pd 1980-tallet. En studie av offentlige stotteformer i utvalgte giverland.

Tvedt, T. The 'civil society', the state and the NGO sector in the Southern Sudan.

Wiik, A. Et økonomisk perspektiv på seks private organisasioners rolle som agent for filantropisk arbeid.

Barkved, E. Bibliografi over evalueringsstudier av norsk privat utviklingssamarbeid og generell litteratur om private organisasjonar med vekt pd einskilde utvalde land.

Barkved E., Manger L, Steen O.I. & Tvedt T. De private organisasjonene som kanal bistand. En tilleggsstudie.

Indgjerd, T. (1987) Integrert utviklingsprogram i privat og offentlig regi: en studie av Redd Barna og Departementetfor Utviklingshjelp, NORAD (hovudoppgave), Bergen: Universitetet i Bergen. Institutt for administrasjon og organisasjonsvitenskap

Planning guide for savings and credit schemes: non-governmental project activities, Departementet for Utviklingshjelp, Oslo: Ministry of Development Cooperation, 1989.

Aasland, T. & von Hanno, U (1984) Iandsinformasjon til organisasjons-Norge: en rapport om tolv norske organisasjoners informasjonsvirksomhet om u-land og u-hjelp finansiert av offentlige midler over NORADs budsjett, Oslo: DUH, Evaluation report, 5.84.

[contents] [next]

³⁰ Most of the reports are taken from Barkved 1995.

[text]