External Evaluator

I. INTRODUCTION

On Friday, 28 September 2018 at 17.02.44 WITA (Central Indonesia Time), a tectonic earthquake of 7.4 on the Richter scale struck 4 districts / cities in Central Sulawesi Province. This triggered a tsunami on the coast of West Donggala and Palu Bay. It also triggers liquefaction in 4 areas: Balaroa and Potobo in Palu and Jono Oge and Langaleso / Sibalaya in Sigi District. Meanwhile, Bowa Village, which is in Banawa District, Donggala Regency, sank into the sea with a depth of 23 meters.

Based on data published by the Kogasgabpad Post for the Central Sulawesi Earthquake and Tsunami Disaster Management on February 28, 2019, there were 4,402 victims who died with the following details: The death toll was 2,685 which were recorded in Palu: 2,132, Sigi: 289, Donggala: 249 and Parimo: 15. Meanwhile, 1,106 unidentified victims were buried in the graves of Poboya and Pantoloan. There were 4,438 injured victims with the following details: Palu: 1,549, Sigi: 1,112, Donggala: 1,750 and Parimo: 27. There are still 1,309 missing people and 122 points are still being used as temporary shelters with the following details: Palu: 80,034, Sigi Regency: 84,888, Donggala Regency: 41,019 and Parigi Moutong Regency: 553. There were 65,733 houses destroyed and several other buildings as well such as 327 houses of worship, 265 schools, 3 hospitals, 7 bridges, Airport: Towers and terminals, 168 road crack points at the Anjungan Talise, 78 office building units and 362 shops.

In response to the disaster, several organizations took the initiative to conduct joint assessments (JNA). The aim is to determine the impact of the earthquake and tsunami during the first weeks which includes primary data collection and community needs in the shelter. Thus, NGOs working to support affected families will be able to decide strategically based on the JNA findings. The sectors that are the target of JNA include: shelter, WASH, health, food security, protection, education and access to information and markets, and by looking at the vulnerability of people living in shelter centers.

Caritas Manado, one of the members of Caritas Indonesia National Network which has Palu as its service area coverage, has been actively helping survivors since the first day after the disaster by opening a soup kitchen at the St. Maria, Palu, to provide food for 500 survivors every day. Support from Caritas Manado reached 8,664 families, or the equivalent of 38,110 people from 47 villages, 17 sub-districts in Palu, Sigi and Donggala districts. During the transition to recovery, with the full support of Caritas Indonesia and Facilitating Partners, Caritas PSE Keuskupan Manado implemented the EA27 / 2018 project funded by the Caritas Internationalist network with the activation of the Emergency Appeal Protocol. The main activities of this project are NFI, MPCA / CTP, Livelihoods, DRR, Psychosocial and Protection. The number of beneficiaries supported from this project are through the distribution of NFI supports covered 2,000 households, MPCA / CTP supported 3,648 households, DRR supported 12 villages and psychosocial supported in 12 villages. This project officially closed in December 2019.

With CI’s approval, Caritas Indonesia and Caritas PSE Manado Diocese developed the EA04 / 2020 project. This project was developed, apart from being based on the remaining budget for the EA27/2018 Project, which is still quite a lot, but also based on real conditions in the area, where there are still so many survivors who have not received proper housing. This project is a 12-month-project (1 January 2020 – 31 December 2020), with an extension period of January – February 2021. The number of beneficiaries is 329 households, of which 250 households receive shelter and toilet assistance, while 79 households receive livelihood support assistance.

In general, project implementation has provided several benefits to beneficiaries in the form of; safety, comfort, and guaranteed privacy for those who receive shelter and toilet assistance. Meanwhile, families who receive livelihood support benefit in the form of restoration of livelihoods damaged by the earthquake. However, to ensure that the assessment of project benefits is not biased, it is necessary to carry out an external evaluation by an independent person or institution.

II. OBJECTIVES OF THE EVALUATION

 The objectives of this external evaluation are:

• Assessing the Relevance, Effectiveness, Efficiency, Participation, Impact and sustainability of the project implementation for 14 months carried out by Diocese Caritas PSE of Manado.
• Get lessons learned and recommendations for planning, implementing, evaluation and monitoring of future projects.

III. MAIN POINTS OF EVALUATION

3.1  Relevance

The key questions:

• To what extent did the interventions for building houses, toilets and livelihood support address the problems faced by the survivors of the earthquake, tsunami, and liquefaction in Palu-Donggala-Sigi?
• To what extent can farmer group activities restore and / or improve their livelihoods that are lost and or damaged by the earthquake, tsunami, and liquefaction?
• To what extent did the constructed houses and toilets provide a sense of security and comfort to the beneficiaries?
• To what extent is the flexibility of project implementation able to respond to changing contexts that occur in the communities served?

3.2  Impact/Outcome

 The key questions:

• To what extent has project implementation resulted in positive and negative changes for all people served?
• To what extent does the project implementation benefit the assisted communities? As an example; Economic benefits, behavior change, knowledge, and skills benefits.
• Economic Benefits: To what extent can farmer group members assisted through livelihood support activities increase group or family income? How to explain? Very good for describing according to the experience of each group member; economic conditions before and after being the direct beneficiaries of the project.
• Behavior Change: To what extent do housekeeping and toilet assistance lead to a change in attitude or behavior? What is the evidence of change? It is better if given real examples.
• Benefit of knowledge and skills: To what extent were members of the farmer groups assisted in this project able to apply the knowledge and skills acquired? Are they able to do it themselves? How to prove this?
• To what extent are the field facilitators from Caritas-PSE Mando who have been trained able to facilitate livelihood training at the farmer group level?
• To what extent are the benefits of the project enjoyed equally by men and women? How can that be proven?
• Were there any unplanned impacts that occurred during project implementation? If so, how could that be?
• To what extent has working with network partners helped achieve project objectives?
• How large is the area (hamlet or village) that will enjoy the impact of the project activity? What impact did they enjoy from this project?
• What important lessons did Caritas Indonesia and Caritas-PSE Manado gain during project implementation? Which important lessons need to be documented specifically to serve as future references for implementing projects with the same intervention? Has there been a process in terms of documenting it?

3.3  Effectiveness

It includes all efforts or actions to achieve goals or objectives.

The key questions:

1. Have the mentoring activities been carried out realistic enough?
2. Was the implementation of all activities carried out easily?
3. Have all the implementation of activities been planned for a measurable and beneficial to the assisted community?
4. To what extent have the objectives been achieved? What information is available regarding the indicators set at the start of the project design that will help answer this question? What other information is available regarding the achievement of goals?
5. What are the main factors that influence the achievement or not achieving of the project objectives?
6. Are the objectives set at the time of project design realistic?

 3.4  Efficiency

It is the accuracy of the method and the ability to do a job well. In other words, minimal resources to achieve optimal results.

The key questions:

1. What evidence shows that the project was carried out efficiently under certain circumstances? Is the project carried out in an economical and cost-conscious manner?
2. What are the parameters on which the assessment will be based (eg cost per project output: cost per training course or trainee, cost per beneficiary…)? Are there benchmarks for these parameters available from other projects or agencies?
3. How well is the organization’s performance (Caritas Indonesia and Caritas PSE Manado)? Possible areas to consider include, for example, management systems, administration, and communication structures.
4. How well can the organization monitor and reflect on all project activities? How do you assess the PME (Planning, Monitoring and Evaluation) system, the effectiveness of performance at the diocesan (Caritas-PSE Manado) and national (Caritas Indonesia) level? How useful are the monitoring formats and meetings held?
5. What is the relationship between the observed impacts and the resources used? Are the financial resources being used as planned or are they experiencing weaknesses and or excesses? Are there activities that have had excess or lack of funds? If so, why do you think it happened that way?
6. Is the project implementation running on time?
7. Does the capacity of the diocesan staff (Caritas-PSE Manado) match the needs?
8. What has been done to achieve optimal efficiency?

3.5  Participation

The main stakeholders in project implementation are all people served. They are the goal of the project. Therefore, their participation in project implementation is a must.

The key questions:

1. To what extent are all the people being served actively involved at all stages of project implementation? For example: in all phases of housing and toilet construction and livelihood support activities.
2. To what extent do those who are most vulnerable actively participate in decision making?
3. To what extent did women and men participate equally in this project?

3.6  Sustainability

The ability of all people served to continue and manage project benefits, after project implementation ends.

The key questions:

1. To what extent are project benefits likely to continue after project implementation ends?
2. What are the main factors affecting the achievement or failure of the project sustainability?
3. Does everyone served remain committed to maintaining the house and toilets and using them? In the judgment of everyone served, will the house and toilet last or not last long? What factors make homes and toilets last or not last long?
4. Do farmer groups remain committed to continuing the activities they have planned? What factors make them continue to carry out activities or not continue the planned activities? Is there a possibility that farmer group members can access village funds to support their activities?

IV.  EVALUATION TARGETS

The target group of this external evaluation is 329 people KK served by the EA 04/2020 Project spread across several villages in Sigi and Donggala Regencies. Rogo, Sidera and Oloboju villages in Sigi Regency, while Saloya, Batusuya, Enu, Tibo, Kaliburu and Bulupountu Jaya Villages in Donggala Regency. Of the 329 households served, 250 of them received housing and toilet assistance and 79 households received livelihood support assistance.

V.  EVALUATION STAGES

5.1  Preparation

The stages of preparation for this external evaluation include preparatory meeting between evaluators and Caritas Indonesia (program team), review of documents related to planning, implementation, and evaluation of the EA04/2020 project.

5.2  Field

Activities in the field include Socialization of evaluation objectives and targets to all staff and former Caritas-PSE Manado staff who will be involved, collecting related data and information; relevance, effectiveness, efficiency, impact, participation, and sustainability of Project EA04 / 2020.

5.3  Debriefing Session

Evaluators will conduct debriefing as soon as they dig up data and information in the field. This provision is carried out in two levels, among others; The first level will be conducted at the Caritas-PSE Manado level, with participants consisting of the Director of Caritas PSE Manado and his staff (or former staff) and key figures as representatives of the people being served. The aim is for the evaluators to get feedback from Caritas PSE Manado and representatives from the people served on the findings and recommendations from the field. It is important to give this feedback, before the evaluator starts writing the report. The second level will be carried out at the Caritas Indonesia Office after all field work is completed.

5.4  Final Report

The final report includes an analysis of the data and information collected related to the Relevance, Effectiveness, Efficiency, Participation, Impact and Sustainability of the EA04/2020 Project.

VI.  EVALUATION METHOD

1. Reviewing documents related to the planning, implementation, and evaluation of the EA04/2020 Project.
2. Collect stories and qualitative data relating to the EA04 / 2020 Project Relevance, Effectiveness, Efficiency, Participation, Impact, and Sustainability in assisted communities through interviews with key figures, FGDs and observations.
3. Analyze data and information to be able to provide an assessment of the Relevance, Effectiveness, Efficiency, Participation, Impact and Sustainability of the EA04/2020 Project.

VII. EVALUATOR RESPONSIBILITY

In general, the responsibilities of the evaluator are as follows:

1. Reviewing documents related to the planning, implementation and evaluation of the EA04/2020 Project.
2. Develop appropriate methods and tools to collect data and information related to EA04 / 2020 Project Relevance, Effectiveness, Efficiency, Participation, Impact and Sustainability.
3. Coordinating all activities and providing updates on the progress of the evaluation process to Caritas Indonesia.
4. Discussing the draft final evaluation report with Caritas Indonesia.
5. Submit the final evaluation report in Indonesian and English to Caritas Indonesia on time.

VIII.  EVALUATION SCHEDULE AND EVALUATOR CAPACITY REQUIRED

8.1  Implementation Schedule

This external evaluation activity will be conducted between 1 – 31 March 2021. The final report will be submitted on 5 April 2021.

8.2  Evaluator

To get optimal results, this project evaluation requires the following personnel:

1. Having experience, knowledge and skills to evaluate empowerment programs and programs during the disaster emergency response period, especially in shelter construction and economic recovery programs for affected communities.
2. At a minimum, know and understand about issues and community strengthening programs, community management programs, community resilience, and disaster risk reduction.
3. Know and understand the specifics of Caritas’ work and services

IX.  EVALUATION PRODUCT

The result that is expected to be prepared by the evaluator is a final report written in Indonesian and English which contains detailed findings and conclusions from the evaluation, accompanied by realistic recommendations that are useful for future learning for Diocese Caritas and Caritas Indonesia. The report should include both quantitative and qualitative information including the following structure.

How to Apply

REGISTER/LOGIN to read more, it’s FREE

The post External Evaluator appeared first on Devjobsindo ORG.