Consultancy Program Evaluation

1.      Background

International Medical Corps (Pakistan Program) is planning to conduct programmatic evaluation of its recently completed projects (July 1, 2018 – June 30, 2020) as part of mandatory annual evaluation exercise. The final evaluation report will be submitted to Govt. as per MoU with Ministry of Interior.

Projects Background and Context

The purpose of these TORs is to hire a consultant to evaluate projects implemented since July 1, 2018 – June 30, 2020 by IMC in KP province.  The first project “Prevention and Response to Gender-Based Violence (GBV) in Afghan Refugee Villages” and Expanding MHPSS and Strengthening GBV Prevention and Response for Afghan Refugees in KP Province is funded by PRM to implement in 9 refugee villages of Districts Haripur, Mansehra and Lower Dir of Khyber Pakhtunkhwa (KP) Province in Pakistan. Its key objectives are to improve protection and sustained access to quality services for prevention and response to GBV and MHPSS services for Afghan refugees of different refugee villages in Pakistan.

The programmatic evaluation will be done for the two years of the project recently completed (July 2018 to June 2019) and the ongoing 3 years project’s first year, July 2019 to June 2020. Total direct beneficiaries of the project were 52,450 Afghan Refugees living in 7 RVs. The evaluation will be focused on to review that program level outputs and outcomes were met as agreed upon with relevant donor.

 Key expected results/objectives of the three years program were:

1. Strengthen the community mechanisms for prevention and response to Gender-based Violence (GBV) among Afghan refugees in Panian I & II, Padhana; Ichrian, Khaki, Barari, Chakdara, Toor, Taimer RVs (year 3 July 2018-June 2019)

2. Ensure continued support through quality care provision around the response to GBV survivors and strengthen community-based mechanisms for GBV prevention. (July 2019 – June 2020)

3. Provision of integrated mental health and psychosocial support services at the primary health care level and in community-based settings (July 2019-June 2020) through:

·         Building the capacity of IMC MHPSS staff in IMC mental health case management approach and on the IASC guidelines on MHPSS in emergency settings to be able to provide case management services to clients in the community facing mental health conditions.

·         strengthening of MHPSS referral pathways for provision of various services required to mental health clients.

·         Providing mental health awareness-raising sessions regularly by community mobilizersat targeted RVs in Haripur, Mansehra, and Lower Dir districts, The key mental health conditions discussed during these sessions were: Attention deficit hyperactivity disorder, Conduct disorder, Dementia, Depression, Enuresis, Epilepsy, Grief, Postpartum depression, Psychosocial stressors, PTSD, Schizophrenia, Torture, and other significant MH complaints.

The second project namely, Saving Lives by implementing Minimum Initial Service Package (MISP) for Reproductive Health (RH) and GBV in KP and merged districts from July 2019 to June 2020 funded by UNFPA in target areas/health facilities at Basic Health Unit Phandu and Urmar Payan in Peshawar district, Tehsil Headquarters hospital Bara, Civil Hospital Jamrud and Rural Health Center Ali Masjid in Landi Kotal Tehsils of Khyber district. Total direct beneficiaries of the project are 25,000 (Afghan Refugees: 13,170, Non-refugees: 11,830).

The overall outcome of the project is - by end of project period, women and girls from Afghan refugees, Temporarily Displaced Persons (TDPs) and host populations in merged district and KP have increased access to services for sexual and reproductive health, as well as awareness around prevention of gender-based violence.  

The project outputs are:

1. Increased availability of life-saving Sexual and Reproductive Health (SRH) including family planning and GBV services,.

2. Strengthened capacity of public healthcare service providers in the proposed areas to deliver life-saving SRH services, and counselling to GBV survivors.

3. Increased public awareness on the available SRH and GBV services in proposed areas of merged district and KP.

Below here are draft evaluation design and Terms of Reference (TOR) for this assignment (to finalize evaluation parameters, methodologies of data collection and analysis & reporting requirements).

2.      Objectives of program evaluation

The purpose of this summative evaluation is to learn from the implementation of the project in achieving the defined goals and document the lessons for future programming and implementation in the same thematic areas. Moreover, one of the key purposes of the evaluation will be to know about whether the approaches used to implement the projects were successful in terms of relevance, effectiveness, efficiency of planning and implementation, impact and potential for sustainability and replication. The evaluation will review International Medical Corps’ GBV and MHPSS project covering the period July 1, 2018 to June 30, 2020 and MISP/GBV project July 2019 to June 2020 by identifying gaps, documenting success and making programmatic recommendations.

3.      Methodology and Tools

The Program Evaluation lead will develop a work plan that includes the frequencies and timelines for data collection, cleaning and analysis. The plan with methodology and tools will be finalized in consent with International Medical Corps technical team at HQ. The suggested methods for data collection are:

a)      Secondary data, desk review (various reports and documents will be reviewed as per need), for example

i.            Project proposal including log frames

ii.            Procurement plan (timelines of key procurement items)

iii.            M&E plan and other project management tools

iv.            Baseline Knowledge, Attitudes and Practices (KAP)  Survey of project

v.            Quarterly and final project reports to donor

vi.            Trip reports or monitoring field reports

vii.            Client Satisfaction Survey files, etc

viii.            Training reports and assessment report conducted by a consultant for UNFPA Project.

ix.            MHPSS need assessment report, training evaluation reports

b)      Primary data collection from the field through:

i.            Focus Group Discussions with projects beneficiaries

Focus Group Discussions (FGDs) will be conducted by following SOPs keeping in view the COVID-19 situation with project’s primary beneficiaries (community elders, male and female gender support group members, health committee members, service users with experience of mental health conditions, carers / family members of MHPSS beneficiaries etc. but no survivors of GBV) in each refugee village and host communities specifically for MISP/GBV project. Separate FGDs will be conducted for MHPSS, GBV and MISP project beneficiaries to evaluate the immediate impact of the relevant sector. For each of the focus group discussion qualitative tools will be developed by the consultant to collect information in a clear and focused manner. These FGD tools will be reviewed and approved by IMC’s global technical advisors per sector, prior to being used.  These focus group discussions will be conducted with 7 – 8 participants in a group, with separate groups for men and women, girls and boys (not younger than 15 years old) a criteria-based approach will be applied to qualify participants of the focused group discussion. For each of the FGDs, participation will be voluntary, and participants will be allowed to opt out of the FGD at any point. There will be two teams, which will both consist of two members who will conduct/lead the focus group discussion; one team will perform the role of moderator and other one will perform the role of note taker and observer. There will be one team consisting of a woman and a man one of men to facilitate gender sensitive discussions. Focus group discussions will be noted down on the specialized templates in local language. Each of the focus group discussions will be concluded in a time span of    one hour plus. Prior to the start of the focus group discussion participants will be fully briefed on the purpose of the FGD and expectation and utilization of the FGD findings and also participants’ privacy

ii.            Key Informant Interviews with key stakeholders

Key Informant Interviews (KIIs) will be conducted by following SOPs keeping in view the COVID-19 situation with the projects’ different stakeholders (including Refugee Village Administrators (RVAs), Health Facility In charge/Psychiatrist/relevant staff of IMC and partner organizations, Community Development Unit (CDU) representatives, religious leaders, health cluster, Department of Health MoH,  GBV sub-cluster partners, etc.) to get their inputs against pre-defined questions. The consultant will develop tools i.e. KIIs and FGDs for conducting the evaluation and these tools will be reviewed and approved by the TUs prior to starting work in the field. The selection of interviewees will be as gender balanced and equal number of men and women will be interviewed or will be contacted to capture their views during FGDs and KIIs. Apart from Refugee Village Administrators, which have only male staff, all other stakeholders i.e. CDU, religious/local leaders, health cluster and GBV sub cluster have enough women representations as possible. The interviewer will be selected according to sex of the interviewees. Each of the KIIs will be conducted on the explicit approval of the relevant stakeholder on a pre-agreed schedule – for each KIIs a maximum of 40-60 minutes will be required. The KIIs will be noted down on a pre-defined template in Pashto or English language by the interviewer. Each interview will be conducted by a team member, with the explicit approval of interviewee. The conversation will be recorded for the purpose of transcription later. The participants will be allowed to drop any question(s) they wish not to respond to and they will also be allowed to discontinue the interview at any point.

4.      Data Validation and Quality Control

Each of the FGD and KIIs will be supported through signed list of participants or with thumb impression as applicable. If permissible, team will make photographs of the proceedings as an evidence of the activities they had carried out in the field. The evaluation lead will independently verify with the participants on the random basis through their phone numbers and also through meeting with them in person. The evaluation lead will do a detailed debrief with the project evaluation team for the FGDs and IDIs they will conduct at the start to provide them feedback on the areas of improvement. For the FGDs a moderation guide will be provided to the team to help them build the conversation and guide them in that. 

5.      Data Transcription

All the FGDs and KIIs will be transcribed into the digital format of Microsoft Word from the handwritten notes and recordings as available. The transcription will include participants inputs and observations with the details of the proceeding and different modes and gestures those groups exhibited as a whole. These transcriptions will be created in the narrative form to help feed into the analysis.

6.      Data Analysis

During the analysis process, the team will How ensure the disaggregation of data by gender, age, and other considerations such as geographic location..

a)      Thematic Analysis

Thematic analysis approach will be used to analyse the data. Based upon tools different themes will be identified and a matrix will be created in Microsoft excel.  Against each of the different theme’s convergence and divergence trends and unique inputs will be noted down and populated into the framework.

b)      Triangulation

Data will be analyzed separately for each method of collection (FGD, KII, ..), but results and  conclusions should then be compared to each other to validate the findings.

For the purpose of report project’s finding will be drawn through the triangulation of project’s data sets and reports findings against the FGDs and findings against the KIIs IDIs. The findings will be organized against the key questions proposed against the evaluation criteria. This process of triangulation, is essential to controlling and reducing bias in data collection and analysis.

7.      Ethical Considerations Interactions with the respondents will only be made after their formal consent is obtained via a consent  form  and the purpose of the data collection and use of the collected information will be explained to them accordingly. The evaluation team will follow the culturally appropriate dress code. The evaluation team will strictly avoid engaging in any conversation of a political or religious nature, and will not use any jargons, words or gestures those can be offensive on religious, ethnic, gender, age or any other ground. The picture of community meetings will be taken with the formal consent of the FGD members. The evaluation team will strictly avoid any sexual advances, offers or favours to beneficiaries, neither will they promise any favours to survey participants. The team will not guide/or instigate respondents to provide specific answers, nor will they misinterpret the survey participants’ inputs or distort these. Safety and security of the survey participants is of paramount importance and evaluator will not act in any manner putting security & safety of the survey participant at risk. Interaction with the respondents will be guided by INTERNATIONAL MEDICAL CORPS’ Policy and protocols. 8.      Limitations

a)      The evaluation is a validation of the reported results through qualitative methods thus these may not present a statistically valid picture of findings.

b)      As only qualitative methods are applied any differencing based upon the findings may lead to serious errors and shortcomings.

c)       Field team will do utmost in their capacity to extract information as close to reality as possible however the possibility of some bias cannot be ruled out of qualitative information collected.

9.      Management Plan

Profile of the Evaluation Team

This assignment will be completed within the given time period through a close monitoring of the plan by MEAL Manager,  Gender  Coordinator, MHPSS Coordinator and Medical Coordinator who will be part of team in hiring process for ensuring technical skills of consultant. They  will also closely monitor assignment with MEAL Manager and global Technical Unit. The consultant will provide team composition such as the role of team lead, male and female enumerators. They will also share a schedule including field teams’ orientation, data collection, data entry, data analysis and report writing.  The key deadlines, identification of field team and tasks will be systematically tacked to ensure these are completed in the timely manner with required standards of quality. FGDs in the field will be closely monitored to ensure all the data collection is being properly captured, transcribed and reported. The FGDs moderators will be hired with very right set of capacities to ensure FGDs are properly facilitated, observed and documented. The assignment in the field will need robust logistical resources to ensure field work is completed without any issues. The moderators will be trained on the purpose of the work, interview questions and GBV basic concepts, guiding principles of working with GBV, MHPSS and SRH and what should they do if anyone discloses violence in the FGDs/KIIs and how to make safe survivor centered referals, etc.

10.  Geographic Areas

The study will be conducted in the province of Khyber Pakhtunkhwa covering the following three five districts (7 out of total 9 Refugee Villages/camps):

Districts

 Locations (Refugees Villages)

Haripur

Panian I

Panian II

Padhana

Mansehra

Khaki

Lower Dir

Chakdara

Taimer

Toor

Peshawar

BHU Phandu including RV Mera Kachori

BHU Urmar Payan including RV Baghbanan

Khyber

Bara

Jamrud

Landi Kotal

Note: Two Refugee Villages - Ichrian and Berari in Mansehra district - have been closed by Governmen of Pakistan and all the families are either being relocated to Khaki RV or have opted for repatriation.

11.  Deliverables

 The required outputs of this assignment will be as follows: 

·         An assessment plan (maximum five pages) outlining the approach/methodology and execution program/timetable. This plan shall be submitted for review and approval by International Medical Corps' Technical Units/Program Director/Gender Coordinator/MEAL Manager/MHPSS Coordinator/Medical Coordinator well before commencement of the work.

·         Data collection tools will be developed after acceptance of the methodology for review and approval by International Medical Corps Technical Units/Program Director/Gender Coordinator/MEAL Manager/MHPSS Coordinator/Medical Coordinator.

·         Draft evaluation report will be prepared within 15 days after completion of the field work and shall be accompanied with the raw data as collected by the data collection tools.

·         Presentation of the key findings with HQ TU colleagues and SMT via skype

·         Will share the draft report for review. The process for review can take 3 to 4 revisions depending on the work presented.

12.  The final evaluation report, which shall be submitted within 15 days after the inputs provided by technical units i.e. Gender Coordinator/MEAL Manager/MHPSS Coordinator/Medical Coordinator, and global Technical Unit sector specialists.

13.  Report

Outline of the Report

The evaluation report will contain the different elements mentioned below. All parts will be clearly distinguished from each other and of sufficient quality.

·         Cover page

·         Table of contents

·         List of acronyms

·         An executive summary that can be used as a document in its own right. It will include the major findings of the programmatic review and summaries conclusions and recommendations.

·         The objectives of the programmatic review.

·         The main evaluation questions and derived sub-questions.

·         A justification of the methods and techniques used (including relevant underlying values and assumptions, theories) with a justification of the sample selections made (of persons interviewed, camps visited, or activity sites visited).

·         A presentation of the findings and the analysis thereof (including unexpected, relevant findings). All research questions and analysis should be addressed, paying attention to gender and age sensitivities and variations. It also needs to reflect any geographical differences between RVs.

·         Conclusions, which will analyze the various research questions. Conclusions will have to be derived from findings and analysis thereof, disaggregated by sex, age and geography.

·         The report should also include the analysis of challenges/limitations that may have affected the collection of data, as well as the quality of data collected.

·         Recommendations should be practical and if necessary, should be divided up for various actors or stakeholders as well as sectoral and include guidelines of how they can be implemented addressing gender, age and geographical variances and needs.

·         Report annexes including FGDs participant’s attendance sheets, consent forms, questionnaires, etc.

The reporting style will be clear and accessible. References to sources used, such as interviews, literature review, reports must be provided.

Confidentiality of information: All documents and data collected will be treated as confidential and used solely to facilitate analysis. Interviewees will not be quoted in the reports without their permission.

14.      Assignment Duration

The entire period of this consultancy will be 60 working days that will include preparation, development of the survey methodology, tools, testing and reviewing of the tools, actual field work, and data analysis, dissemination of the key findings to stakeholders and reviewing of the final report. 

15.      Schedule of Key Deliverables

Task /Outputs /Deliverable

Expected due date

Assessment plan

Methodology

Tools

Inception report

Report draft

Three to four  drafts

Final report

16. Selection Process and Criteria:

The potential firm/consultant will submit detailed technical and financial proposal/s. The proposal/s will devise technical strategy and all costs.  The proposal/s should include:

• Cover letter with Expression of interest and addressing the stated requirements
• Detailed CVs of expert(s)
• Description of proposed methodology to clearly address the criteria and content of the Terms of Reference and may detail what additional improvements / best practices the firms can introduce while collecting and computing the data.
• Samples of previous work including evaluation tools used and reports produced.
• Experience and expertise in health specifically in SRH and  in gender sensitive work, MHPSS and GBV is preferred.
• Three references with contact details (name, email and phone number  from previous clients, where similar work was performed.

The proposals will be evaluated on technical and financial basis. The technical score will be awarded upon maximum of 80% score, whereas, the financial will be awarded maximum of 20% score. The client will award the contract to the firm/individual with the highest accumulative score in technical, financial and the presentation of the competition process. The review team will review proposals based on the following general criteria:

Marks

1. Methodology                                                                                                                    30
2. Demonstrated relevant experience i.e. GBV, MHPSS and health                  20
3. , Composition and JDs of key personnel and enumeration staff                    25
4. References                                                                                                                         5
5. Cost                                                                                                                                       20

Total:                                                                                                                                            100

All documents related to this tender shall be in English and all costs shall be expressed in Pak Rupees. Proposals may be submitted electronically or by hard copy. If proposals are submitted electronically, all pieces to the proposal must be labelled clearly. Acceptable formats include Microsoft Word, Microsoft Excel, and PDF.

17. Logistical Support

External evaluator will have the trained enumerators in each district under evaluation. All the enumerators will be local residents and well aware of the local culture and norms. There will be no need for accommodation of enumerators in their respective areas. The accommodation of external evaluator in the districts and travel of enumerators and external evaluator will be the external evaluator’s responsibility. International Medical Corps’ Pakistan Program will give a lump sum amount to the external evaluator in accordance with the related clauses of the consultancy agreement. 

18. Qualifications and experience required:

Qualifications: Well-recognized and reputable research consultancy firms/company or individual consultants having documented/proven experience in conducting evaluations in GBV Prevention and Response, MHPSS, SRH and gender sensitive work in Pakistan are eligible to submit their proposals. Preference will be given to those who have  prior work experience with Afghan Refugees in KPK.

General Conditions:

• No contract may commence unless the contract is signed by both parties.
• The team members of the consultancy firms/consultant are not entitled to payment of overtime. All remuneration must be within the contract agreement.
• Accommodation and meals will be covered by the consulting firm/company/consultant. Therefore, the price component must contain an overall quotation including Food and Stay. this. In addition, the detailed budget is required to show the actual breakup.
• The consulting firm should provide its team with relevant IT equipment and other office supplies related directly to the task.
• International Medical Corps will facilitate the provision of addresses, organizing community meetings movement of the teams in field and geographic areas. The travel to and from and within the field needs to be reflected in financial proposal.
• Final payment to the consulting firm will be dependent on the completion of deliverables (to be specified in the contract) as well as handover notes and relevant data.

19. Validity of Proposal

Proposals will be valid for 30 days only . International Medical Corps reserves the right to reject any or all proposals received in response to this RFP and is in no way bound to accept any proposal. The client additionally reserves the right to negotiate the substance of the finalists’ proposals, as well as the option of accepting partial components of a proposal if appropriate.

20. Human Subject Protection Issues

The consultant must be familiar with and observe the “WHO Ethical and safety recommendations for researching, documenting and monitoring sexual violence in emergencies”, and follow the confidentiality, privacy and informed consent policies of International Medical Corps program/projects such as GBV, MHPSS and SRH according to national and international protocols.

21. Intellectual Property Rights:

All outputs under the assignment shall be the sole property of International Medical Corps. The Consulting Firm/consultant shall not exercise any rights on any of the outputs and cannot share them with any outsiders without the prior consent of International Medical Corps.

22. Deadline for submission of Technical & Financial proposal:

All outputs under the assignment shall be the sole property of International Medical Corps. The Consulting Firm/consultant shall not exercise any rights on any of the outputs and cannot share them with any outsiders without the prior consent of International Medical Corps.