Terms of Reference for Evaluation At Save the Children

Introduction

This document provides the Terms of Reference for the external evaluation of the Dutch Ministry of Foreign Affairs funded Nigeria Joint Response running from January 2019 to December 2021.

The Dutch Relief Alliance

The Nigeria Joint Response is implemented by the Dutch Relief Alliance (DRA). In response to the challenges of the humanitarian system and the growing gap between humanitarian needs and humanitarian funding, the Dutch Minister for International Trade and Development Cooperation in 2015 set up a Dutch Relief Alliance to increase effectiveness of Dutch humanitarian aid. The DRA is an alliance of 14 Dutch NGOs which respond to chronic crises as well as acute crises, for which they receive funding from Netherlands MFA. The main objectives of the Dutch Relief Alliance are to deliver effective, efficient, relevant and timely humanitarian aid through collaboration to more beneficiaries in a better way. DRA partners implement Joint Responses (JRs) to address protracted crises enabling capacity building, localisation and investment in community resilience. The four strategic pillars of the DRA include collaboration, localization, accountability and innovation.

The Nigeria Joint Response

The Nigeria Joint Response is implemented by 5 DRA partner organisations: Save the Children (lead agency), Tearfund, ICCO, Terre des Hommes and ZOA who work together with 7 local partner organizations: JDF, JDPH, EYN, GEPADC, CRUDAN, SAHEI and YIPDI to provide lifesaving assistance people affected by the crisis in Northeast Nigeria. The period under review runs from January 2019 to December 2021 during which the partners implemented three annual programs, each phase a continuation of the previous one. The budgets for each consecutive year were EUR 5.923.341, EUR 6.521.700 and EUR 5.160.952. We have reached 147,388 and 200,798 beneficiaries in 2019 and 2020 respectively and plan to reach 81,572 beneficiaries in 2021.

Project background

Context

The Lake Chad crisis is one of the most severe humanitarian crisis in the world that involves Lake Chad Basin countries i.e. Chad, Cameroon, Niger and Nigeria and 17.4[1] million people have been directly affected by this crisis to date. Out of 17.4 million affected people, 8.7 million affected people are from northeastern states BAY (Borno, Adamawa and Yobe) of Nigeria who have been displaced and are under a protracted conflict crisis for more than a decade due to an active militancy that has severely damaged lives and livelihoods. In 2021, more than 5 million people are estimated to face severe food insecurity, an increase of 19% and 34% compared to 2020 and 2019 respectively. Conflict, climate change and COVID-19 are the main drivers of the increasing humanitarian needs.

The NJR programme is implemented in Borno State focusing on the Local Governate Areas (LGAs) with the highest needs and where DRA partners can work together to generate collaborative impact. In 2019 these LGAs were Mafa, Konduga, Dikwa, Monguno and Gzowa, in 2020 Mafa, Dikwa, Monguno and Gwoza and in 2021 Mafa, Biu, Monguno and Gwoza. NJR is providing a multi-sectoral integrated response with the aim of holistically meeting the needs of the affected population. In 2019 and 2020 the response focussed on FSL, WASH, Nutrition, Protection, Health and Multi-Purpose Cash and in 2021 the response focussed on FSL, WASH, Protection, Health and Multi-Purpose Cash.

The target group consists of Internally Displaced Persons (IDP), returnees and host communities. Among these, vulnerable groups have been identified especially unaccompanied and separated children, orphans, child-headed households, households with children under five (especially with SAM cases), female-headed households, Pregnant and Lactating mothers, elderly and physically challenged.

The NJR programme is designed as an adaptive and flexible grant structure which encourages a high level of cooperation across partners with the aim of delivering quality programming through continuous learning, adaptive implementation and information sharing. Moreover, the high level of collaboration including joint activity planning, sharing of resources and integrated programming is leading to increased efficiency and effectiveness of the actions of all agencies in the joint response.

DRA and MFA both value collaboration, gender mainstreaming and disability inclusion, localization, accountability to affected population, protection mainstreaming, and innovation. Therefore, these cross-cutting themes are being addressed in the programming.

Scope of evaluation

The NJR partners have a contractual obligation towards the donor, the Dutch Ministry of Foreign Affairs, to ensure the realization of an evaluation. The evaluation will cover the period 2019-2021 during which the NJR partners delivered three annual humanitarian programs in North-East Nigeria. In the technical proposal the consultant should deliberate on how primary data will be collected for earlier project periods; 2019 and 2020.

Objective 1 Results achieved: The evaluation report will assess the overall performance of the NJR against selected OECD DAC criteria and the Core Humanitarian Standards, ensuring accountability towards the Dutch Government, the Dutch public and the beneficiaries of the program. It should be noted that the consolidated logframe with project results will not be ready in time for this evaluation. The consultant is expected to work with individual partners to gain insight into the achievement of targets.

Objective 2 Learning: Evaluate to what extent the adaptive management system and learning strategy of the project was effective.

Objective 3 Accountability to affected populations: Evaluate to what extent relevant information on the program and partner organisation has been shared with the communities; to what extent communities were able to participate in all phases of the project cycle (from design up to evaluation processes); to what extent communities were aware and have made use of feedback mechanisms and to what extent NJR partners were able to close the feedback loop.

Objective 4 Collaboration: Evaluate to what extent the collaboration between NJR partners contributed to reaching beneficiaries and targets in a more effective, efficient, relevant and timely manner, as set in the log frame and in the narrative proposal. In addition, evaluate to what extent the collaboration between NJR partners has led to increased positive impact on the lives of the beneficiaries.

Objective 5 Localization: Evaluate to what extent the NJR programme has contributed to increased capacity and ownership of local actors including local NGOs, CSOs, community structures, local authorities etc. To what extent have local partners gained the capacity to continue the delivery of humanitarian aid beyond the NJR programme.

Locations:

The suggested locations to be evaluated are Biu, Mafa and Monguno LGA in Borno State. Mafa and Monguno LGAs were evaluated during a Real Time Review (RTR) conducted in May 2019. Going back to these two locations will allow us to determine to what extent the recommendations from the RTR have been taken into account in programming going forward. Biu is a new location of implementation where NJR partners started implemented in January 2021. This will allow for a comparison in results between two locations where NJR has implemented for a longer period of time and the results of collaboration in a new location. Naturally, considering the volatile situation in North-East Nigeria, the accessibility and security of these locations will need to be verified prior to the evaluation.

Key Questions

We have listed a range of possible questions below. The consultant in consultation with the NJR partners is expected to propose a selection of the most relevant questions as part of the inception report. The questions should cover all five objectives.

Objective 1 Results achieved

Relevance/Appropriateness **

• How relevant have the objectives and activities of the NJR programme been in addressing humanitarian needs in Nigeria?
• To what extent was NJR able to adapt and provide appropriate response to context changes and evolving needs and capacities, and the priorities of the people, taking into account the specific needs of the most vulnerable groups, including women, children and disabled persons?
• To what extent and how was protection mainstreamed across the program activities?
• To what extent and how was gender sensitive programming implemented?
• To what extent and how was disability inclusion mainstreamed across the program activities? **

Effectiveness

• To what extent have the planned results (outputs, outcomes) been achieved in all the phases of NJR?
• What factors were crucial for the achievement or failure to achieve the project objectives?
• What value do beneficiaries, stakeholders and communities attach to the results achieved?
• How did the project coordinate with and/or complement other similar actions in the field – geographical and thematic?
• How did partnership with local NGOs inform DRA partners implementation process?

Efficiency

• Was the process of achieving results efficient? To what extent has the NJR programme been implemented in a timely and cost-effective manner?
• Was there good value for money for the activities undertaken?
• To what extent has the joint response model of collaboration led to cost-effectiveness?

Impact

• How has NJR impacted the lives of crisis affected population (IDPs, returnees and host communities) in sustaining their lives through integrated life-saving humanitarian assistance?
• What has changed as a direct result of the NJR intervention? Are there unintended positive or negative consequences of the NJR programming on target population and/or on national and international NJR partners?
• Which groups have been affected (or unaffected) by the changes?

Sustainability

• To what extent does the intervention reflect and consider factors which have a major influence on sustainability, i.e. economic, ecological, social and cultural aspects?
• To what extent have the outcomes and results of the NJR programme been sustained after the first phases of the Nigeria joint response and to what extent will they be sustained after the completion of the project?
• What are the contributing factors and constraints that require attention in order to improve prospects of sustainability of the project outcomes?
• How have localization efforts contributed to sustainability of project activities?

Objective 2 Learning

• To what extent has the NJR collaborative way of working facilitated peer-learning between NJR partners?
• What kind of learning activities have been most effective according to NJR partners?
• Did partners make any changes to their programming as a result of these learnings, leading to better quality program?
• Is there any substantial anecdotal evidence on how activities to increase learning have affected the delivery of humanitarian aid by NJR partners?
• How did we learn from previous phases of the joint response? What have we improved compared to phase 1 – 4 and how did we improve between 2019 and 2021? To what extent did we implement the recommendations of the NJR3 final evaluation, the 2019 Rapid Real Time Review, the 2020 Impact Assessment, the 2020 cash pilot evaluation and the 2020 accountability pilot evaluation?
• What have been the reasons for (not) implementing the recommendations? What has been the result of the decision to implement or not implement?

Objective 3 Collaboration

• Is there any substantial (anecdotal) evidence of how collaboration between NJR partners, specifically related to complementarity, has led to positive impact on the lives of beneficiaries?
• Are there any concrete examples of successful models of collaboration of NJR partners on geographic level affecting the reach and impact on beneficiaries? What are barriers and/or enablers to this?
• To what extent, and how, did NJR coordinate activities with external parties, i.e. national Government, UN OCHA, non-NJR INGO partners?
• How have the comparative advantages[2] of different DRA partners contributed to the effectiveness of the joint response (either in individual programming or in collaboration), both at location-level and at a broader level?

Objective 4 Accountability to the affected population

• How and to what extent have communities (IDPs, host communities and returnees) been involved in the development of selection criteria to identify the most vulnerable?
• To what extent did NJR partners involve beneficiaries in project design (e.g. development of indicators), implementation (e.g. community committees) and MEAL (e.g. monitoring progress)?
• How and to what extent did NJR partners provide opportunities for community members to provide feedback? What (joint) feedback mechanisms are in place and how inclusive are these feedback channels? Are there any concrete example of adjustments in the program accordingly to beneficiaries’ feedback and preferences?
• How did the accountability pilot influence feedback and complaints reporting by beneficiaries?

Objective 5 Localization

• To what extent have capacity strengthening activities fulfilled the needs and priorities of the participants (especially local partners); what influence has this had on local partners’ way of implementing? Provide concrete examples of how it contributed to their work and improved program quality benefitting the beneficiaries.
• To what extent have NJR localization efforts contributed to the capacity of local partners to fund, design, and deliver humanitarian actions in northeast Nigeria?
• To what extent did local partners feel empowered / felt ownership in the design and implementation of the NJR programme?
• What has been the impact of the NJR localization efforts on the localization practices of the DRA INGO partners?
• What has been the impact of the NJR localization efforts on the localization agenda of the humanitarian community in Nigeria?

Evaluation Methodology

The evaluation will be carried out in a transparent and participatory manner by involving relevant stakeholders (UN, Cluster representatives, NJR partners and affected people). The COVID-19 pandemic is a key consideration at the time of finalizing the methodology, however a mixed-method approach is anticipated including but not limited to the following methods:

• Desk study and review of all relevant program documentation.

• Key Informant Interviews with key stakeholders (Project teams, senior officials of national and international partners, Cluster leads, Government and UN agencies)

• Questionnaire for NJR partner staff

• Focus Group discussions with target population

• Household surveys with target population.

The evaluation should be inclusive taking into account gender, age, disability, and other vulnerability considerations, sensitive of social norms and practices, and be considered of ethical data collection. In addition, the evaluation methodology should consider consultations with child and youth beneficiaries and highlight key approaches for undertaking it:

• Safety and ethics considerations for engaging children in evaluation
• Data collection methods which are age and gender appropriate

Roles and responsibilities

NJR coordination team

• Starts the evaluation process;
• Prepares and publishes the ToR for the selection of the consultant;
• Participates in the consultant selection process;
• Is the contact person for the consultant;
• Leads and/or coordinates the evaluation

NJR partners

• Make necessary logistical and security arrangements to receive data collection team. This includes organizing and providing a security briefing upon arrival. The JR lead organization however makes final decision on security-related matters;
• Informs key staff, crisis-affected people and other relevant stakeholders of the upcoming visit;
• Makes key documentation and background information available to the consultant;
• Makes sure that key staff, crisis-affected people and other relevant stakeholders are available for participating in interviews and focus group discussions

Consultant

• Prepares the inception report;
• Prepares the data collection tools;
• Conducts data collection; transcription and data cleaning
• Executes the data analysis;
• Writes the evaluation report.

Timeline

• Deadline for applications – 15 October
• Interviews with pre-selected candidates – 26-28 October
• Selection of evaluation team By 2 November
• Inception report – By 12 November
• Data collection – 24 November – 17 December
• First draft evaluation report – 21 January 2022
• Validation workshop – Last week January 2022
• Final evaluation report – 15 February 2022

The timeline detailed above is an indicative timeline. The consultant can amend this timeline in the inception report keeping in mind that project staff might not be available after December 2021 and the bulk of data collection will need to happen before the end of the year. The end date for the consultancy is flexible.

Expected Outcomes and deliverables

Process

The assignment shall start the latest by first week of November 2021. A first meeting shall be held before the assignment starts in order to review the ToR and agree on tentative work plan.

An inception report shall be submitted to NJR Consortium Coordinator detailing the work plan, sampling frame and data collection tools, no later than one weeks after holding the first meeting. The consultant will finalize the report, incorporating feedback and suggestions coming from the NJR Coordinator and the NJR partners.

The consultant will be responsible for data collection including hiring enumerators where required. Data collection and fieldwork by the consultant shall take place in November – December 2021, possibly extending into January 2022 where needed and the methodology will be mutually agreed upon by NJR coordination and consultants, keeping the constraints generated by the security situation and the COVID-19 pandemic in mind. The methodology shall take into account that staff might leave after December 2021.

A briefing will be conducted for partners in Maiduguri and/or Abuja by the consultant before the start of the data collection.

A draft evaluation report will be ready by mid-January 2022. One feedback and verification workshop will be conducted where the main findings are presented to the NJR partners who will be provided with the opportunity to provide verbal and written feedback.

The final report should be delivered no later than 15 February 2022. The content of the report will be finalized through mutual discussion between NJR coordination and consultants.

Deliverables:

Inception report: The Inception Report will highlight the methodology and the guiding principles of the evaluation. The inception report will include: Objectives and key questions, methodology, data collection methods, sampling approach, timeline and logistics, tools to be used for data collection. The report should be no longer than 20 pages.

Data collection and data analysis tools: The evaluator(s) will develop the tools for data collection and data analysis in line with the structure of the tools in the inception report.

Evaluation Final Report (between 30 and 40 pages, annexes excluded, in Microsoft Word format) including tabs and graphs representing the data:

• Table of Contents
• List of Acronyms
• List of Tables
• Executive Summary
• Background
• Scope of Evaluation
• Methodology (including sampling)
• Main Findings
• Main Learnings
• Conclusions and Recommendations
• Annexes
• o Project logframe
• o Evaluation ToR
• o Objectives and key questions
• o Methodology
• o Study schedule
• o List of people involved
• o Bibliography of consulted secondary sources
• o Finalised data collection tools

PROFESSIONAL SKILLS AND QUALIFICATIONS

Interested experts/consultancies are required to provide CVs detailing the experience with similar type of assignments completed in the past.

Technical expertise **

The experts/consultancies should demonstrate the following areas of technical expertise in his/her Expression of Interest:

• Experience in design, planning and implementation of mixed-method evaluation exercises.

• Experience in quantitative data collection and analysis, use of sound statistical methods to identify causal relationships and address threats to internal validity.

• Experience in qualitative data collection and analysis of complex qualitative information, drawing findings from multiple sources and handling potential contradictions between data sets.

• Relevant subject knowledge and prior experience of working on multi-year programming in the emergency and humanitarian sector to ensure that design and research methods are as relevant and meaningful as possible as given in the work scope of this assignment.

• Proven knowledge and experience with using humanitarian sector frameworks for Quality and Accountability (e.g. CHS; SPHERE) in evaluation assignments;

• Statistical analysis: a range of statistical modelling and analysis of impact data; highly proficient user of: SPSS, STATA or equivalent; and qualitative data analysis techniques.

• Language Proficiency: Proficiency in English and the ability to produce good quality written documents in English is a mandatory requirement of this assignment.

Desired expertise and experience

Members of the evaluation team should hold at least a bachelor’s degree in relevant field of study.

The evaluation team should have prior experience of developing research designs that involve remote data collection and management.

Regional experience: it is particularly desirable that the evaluation team has experience working in the West-African context and/or Northeast Nigeria in particular.

[1] https://reliefweb.int/sites/reliefweb.int/files/resources/Lake%20Chad%20Snapshot.pdf

[2] Comparative advantage refers to the ability of individual actors to contribute to humanitarian effectiveness where they can clearly and consistently add value. This will increase impact, predictability and quality

How to apply

APPLICATION REQUIREMENTS

Save the Children is inviting both individuals as well as teams of consultants to apply for this assignment. Interested candidates who meet the key qualifications and have relevant experience in designing and delivering similar type of assignments must submit their technical and financial proposals to the following email address: [email protected] no later than 15 October 2021.

Shortlisted candidates will be invited for an interview after which the final application will be selected. Interviews are expected to take place between 26-28 October.

In case of questions about the assignment or the application process please contact: Janna de Jong at [email protected]

Expression of interest

The Expression of Interest (EoI) should include:

1) A technical proposal that responds to the asks of this ToR. The technical proposal should contain a clear outline of the research methodology, data collection and analysis techniques, a tentative work plan with clearly defined milestones to achieve within the given timeline of the assignment and a financial proposal.

2) Consultants’ CV;

3) At least one sample of similar previous work;

4) two references, to be attached as annex to the technical proposal.

Please note that incomplete EoI will not be assessed.

Financial Proposal

The financial bid must be prepared and submitted with clearly defined breakdown of consultants’ daily fee and other associated costs. Please read carefully section Payment Schedule of the ToR while developing and finalizing financial proposal. Interested applicants should consider that approximately €30,000 inclusive of VAT is available to conduct this assignment. Please be aware that, along with the quality of the technical proposal and of the sample of previous work submitted, the amount of the financial proposal will also be a criterion for assessing the EoI received.

Disqualification

Save the Children Netherlands reserves the right to revoke the call or stop the process of hiring services without giving any prior reasons to the applicants. Incomplete application, applications submitted after deadline or application without financial proposal or vice versa will not be considered in the selection process.

Payment Schedule

The consultant will be paid in three instalments with following terms and conditions;

1. 30% advance payment will be released upon acceptance of inception report and tools by assignment focal point.
2. 40% payment will be released upon submission of the draft assessment report
3. 30% payment will be released upon successful completion of assignment in the field and acceptance of deliverables by assignment lead.

Consultant will be paid fixed fee as per signed contract after deduction of all taxes applied by Netherlands Law. Save the Children will cover field travel and accommodation if required in Maiduguri and Abuja during assignment days. All other costs including international travel, life and health insurance, visa and other costs associated for traveling to Nigeria as well as costs for hiring enumerators for data collection, will be borne by the consultant and should be included in the financial proposal.

Child Safeguarding and PSEA

Safeguarding considerations: Ensuring the whole evaluation process adheres to best practice for research with children including the implementation of child safeguarding policy and procedures to ensure safety of participants.

Conflict of interest

The incumbent is required to demonstrate the necessary independence and declare any conflict of interest and potential biases, including bias towards any of the stakeholders, target groups, type of approach etc.

code of conduct

Save the Children’s work is based on deeply held values and principles of child safeguarding, and it is essential that our commitment to children’s rights and humanitarian principles is supported and demonstrated by all members of staff and other people working for and with Save the Children. Save the Children’s Code of Conduct sets out the standards which all staff members must adhere to and the consultant is bound to sign and abide to the Save the Children’s Code of Conduct.

mandatory security training

The consultant shall be expected to go through mandatory on-line security training and submit the certificate of completion to SC before the commencement of the task. Failure to deliver this will lead to an automatic disqualification. SC will advise on the security plan on appointment.