International Medical Corps (Pakistan Program) is planning to conduct programmatic evaluation of its recently completed project (July 1, 2017 – June 30, 2018) as part of mandatory annual evaluation exercise. Final evaluation report will be submitted to Govt. as per MoU with Ministry of Interior.
Project Background and Context
The project “Prevention and Response to Gender-Based Violence (GBV) in Afghan Refugee Villages” is funded by PRM to implement in 9 refugee villages of Districts Haripur, Mansehra and Lower Dir of Khyber Pakhtunkhwa (KPK) Province in Pakistan. Its key objectives are to improve protection and sustained access to quality services for prevention and response to GBV for Afghan refugees of different refugee villages in Pakistan.
Overall the project is for three years started from July-2016 to June 2019. Now we are in its third year implementation period July 2018 to June 2019. The programmatic evaluation will be done of the second year of the project recently completed (July 2017 to June 2018). Total beneficiaries of the project were 22,000 Afghan Refugees living in 9 RVs. The evaluation will be focused on to review that program level outputs and outcomes were met as agreed upon with relevant donor.
Key expected results/objectives of the three years program were:
1). Monitor and mentor the community-based GBV prevention and response mechanism in the former sites of Mera Kachowri, Baghbanan, Koga, Barakai I and II RVs (year 1, July 16-June 2017)
2). Establish new GBV prevention and response services among Afghan refugees in Panian I & II, Padhana; Ichrian, Khaki, Barari, Chakdara, Toor, Taimer RVs (year 1, July 16-June 2017)
3). Introduce the community-based mechanisms for prevention and response to GBV among Afghan refugees in the new sites of Panian I & II, Padhana; Ichrian, Khaki, Barari, Chakdara, Toor, Taimer RVs (year 2, July 2017 to June 2018)
4). Strengthen the community mechanisms for prevention and response to Gender-based Violence (GBV) among Afghan refugees in Panian I & II, Padhana; Ichrian, Khaki, Barari, Chakdara, Toor, Taimer RVs (year 3 July 2018-June 2019).
Below here are draft evaluation design and Terms of Reference (TOR) for this assignment (to finalize evaluation parameters, methodologies of data collection and analysis & reporting requirements).
2. Objectives of program evaluation
The purpose of this summative evaluation is to learn from the implementation of project in achieving the defined goals and document the lessons for future programming and implementation in the same thematic areas. Also one of the key purpose of the evaluation exercise will be to know about whether the approaches used to implement the projects were successful in terms of relevance, effectiveness, efficiency of planning and implementation, impact and potential for sustainability and replication. The evaluation will review International Medical Corps’ GBV prevention and response project covering the period July 1, 2017 to June 30, 2018 by identifying gaps and making programmatic recommendations.
3. Methodology and Tools
The Program Evaluation lead will develop a work plan that includes the frequencies and timelines for data collection, cleaning and analysis. The plan with methodology and tools will be finalized in consent with International Medical Corps technical team at HQ. The suggested methods for data collection are:
a) Secondary data, desk review (various reports and documents will be reviewed as per need), for example:
i. Project proposal including log frames
ii. Procurement plan (timelines of key procurement items)
iii. M&E plan and other project management tools
iv. Baseline KAP Survey of project
v. Quarterly and final project reports to donor
vi. Trip reports or monitoring field reports
vii. Client Satisfaction Survey files, etc.
b) Primary data collection from the field through:
i. Focus Group Discussions with projects beneficiaries
Focal Group Discussions (FGDs) will be conducted with project’s primary beneficiaries (community elders, male and female gender support group members, health committee members, etc. but no survivors of GBV) in each refugee village. For each of the focus group discussions qualitative tools will be developed by the consultant to collect information in a clear and focused manner. These focus group discussions will be conducted with 7 – 12 participants a group, with separate groups for men and women, a criteria based approach will be applied to qualify participants of the focused group discussion. For each of the FGD, participation will be voluntary and they’ll be allowed to quit FGD any moment they will opt for. There will be two teams. One team consist of two members who will conduct the focus group discussion; one will perform the role of moderator and other one will perform the role of note taker and observer. There will be one team consisting of women and one of men to facilitate gender sensitive discussions. Focus group discussion will be noted-down on the specialized templates in Pashtu language. Each of the focus group discussion will be concluded in a time span of 30 – 60 minutes. Prior to the start of the focus group discussion participants will be fully briefed on the purpose of the FGD and expectation and utilization of the FGD findings and also participants’ privacy.
ii. In-depth Interviews with key stakeholders
In-depth Interviews (IDIs) will be conducted with the projects’ different stakeholders (including Refugee Village Administrators (RVAs), Community Development Unit (CDU) representatives, religious leaders, GBV sub-cluster partners, etc.) to get their inputs against pre-defined questions. The selection of interviewees will be as gender balanced and equal number of men and women will be interviewed or will be contacted to capture their views during FGDs and IDIs. Apart from Refugee Village Administrators which have only male staff, all other stakeholders i.e. CDU, religious/local leaders, and GBV sub cluster have enough women representations as possible. The interviewer will be selected according to sex of the interviewees. Each of the IDI will be conducted on the explicit approval of the relevant stakeholder on a pre-agreed schedule – for each IDI a maximum of 40-60 minutes will be required. The IDI will be noted down on a pre-defined template in Pashto or English language by the interviewer. Each interview will be conducted by a team member, with the explicit approval of interviewee the conversation will be recorded with for the purpose of transcription later. The participants will be allowed to drop any question(s) and also they will be allowed to discontinue interview at any point.
4. Data Validation and Quality Control
Each of the FGD and IDI will be supported through signed list of participants or with thumb impression as applicable. If permissible, team will make photographs of the proceedings to support the activities they had carried out in the field. The evaluation lead will independently verify with the participants on the random basis through their phone numbers and also through meeting with them in person. The evaluation lead will do a detailed debrief with the project evaluation team for the FGDs and IDIs they will conduct at the start to provide them feedback on the areas of improvement. For the FGDs a moderation guide will be provided to the team to help them build the conversation and guide them in that.
5. Data Transcription
All the FGDs and IDIs will be transcribed into the digital format of Microsoft Word from the handwritten notes and recording as available. The transcription will include participants inputs and observation with the details of the proceeding and different modes and gestures those groups exhibited as a whole. These transcription will be created in the narrative form to help feed into the analysis.
6. Data Analysis
a) Thematic Analysis
Thematic analysis approach will be used to analyze the data. Based upon tools different themes will be identified and a matrix will be created in Microsoft excel. Against each of theme convergence and divergence trends and unique inputs will be noted down and populated into the framework.
For the purpose of report project’s finding will be drawn through the triangulation of project’s data sets and reports, findings against the FGDs and findings against the IDIs. The findings will be organized against the key questions proposed against the evaluation criteria.
7. Ethical Consideration
All interaction with the respondents will be only made after their consent and they will be explained purpose of the data collection, and also use of the collected information. A consent script will be used for interaction with the respondents will be made after consent. The evaluation team will strictly avoid indulging into any conversation of political or religious nature, and will not use any jargons, words or gestures those can be offensive on religious, ethnic, gender, age or any other ground. The evaluation team will strictly avoid any sexual advances, offers or favors to beneficiaries, neither will promise any favors to survey participants. S/he will not guide/or instigate respondents to provide specific answers, also will not misinterpret survey participant inputs or distort these. Safety and security of the survey participants is of the paramount importance and evaluator will not act in any manner putting security & safety of the survey participant at risk. Interaction with the respondents will be guided by INTERNATIONAL MEDICAL CORPS Policy and protocols. Field team will be trained on Bureau of Population, Refugees, and Migration (PRM) assessment on guiding principles, referrals and how to care for survivors.
a) The evaluation is a validation of the reported results through qualitative methods thus these may not present a statistically valid picture of findings.
b) As only qualitative methods are applied any differencing based upon the findings may lead to serious errors and shortcomings.
c) Field team will do utmost in their capacity to extract information as close to reality as possible however possibility of some bias cannot be ruled out of qualitative information collected.
9. Management Plan
Profile of the Evaluation Team
This assignment will be completed in the given period through a close monitoring of the plan by both Evaluation Lead MEAL Manager and GBV Coordinator who will be part of team in hiring process for ensuring technical skills of consultant. She will also closely monitor assignment with MEAL Manager and technical unit. They key deadlines, identification of field team and tasks will be systematically tacked to ensure these are completed in the timely manner with required standards of quality. FGDs in the field will be closely monitored to ensure all the data collection will be properly captured, transcribed and reported. The FGDs moderators will be hired with very right set of capacities to ensure FGDs are properly facilitated, observed and documented. The assignment in the field will need robust logistical resources to ensure field work is completed without any issues. The moderators will be trained on the purpose of the work, interview questions and GBV basic concepts, guiding principles of working with GBV, and what should they do if anyone comes forward in the FGDs/IDIs saying they have experienced GBV, etc.
10. Geographic Areas
The study will be conducted in the province of Khyber Pakhtunkhwa covering the following three districts (nine Refugee Villages/camps):
Districts Locations (Refugees Villages)
Haripur Panian I
Lower Dir Chakdara
The required outputs of this assignment will be as follows:
• An assessment plan (maximum five pages) outlining the approach/methodology and execution program/timetable. This plan shall be submitted for review and approval by International Medical Corps Technical Units/GBV Coordinator/MEAL Manager well before commencement of the work.
• Data collection tools will be developed after acceptance of the methodology for review and approval by International Medical Corps technical units/GBV Coordinator/MEAL Manager
• Draft evaluation report will be prepared within 15 days after completion of the field work and shall be accompanied with the raw data as collected by the data collection tools.
• Presentation of the key findings should be included in the draft report.
12. The final evaluation report which shall be submitted within 15 days after the inputs provided by technical units/GBV Coordinator/Manager.
Outline of the Report
The evaluation report will contain the different elements mentioned below. All parts will be clearly distinguished from each other and of sufficient quality.
• Cover page
• Table of contents
• An executive summary that can be used as a document in its own right. It will include the major findings of the programmatic review and summaries conclusions and recommendations.
• The objectives of the programmatic review.
• The main evaluation questions and derived sub-questions.
• A justification of the methods and techniques used (including relevant underlying values and assumptions, theories) with a justification of the selections made (of persons interviewed, camps visited or activity sites visited).
• A presentation of the findings and the analysis thereof (including unexpected, relevant findings). All research questions should be addressed, paying attention to gender issues.
• Conclusions, which will analyze the various research questions. Conclusions will have to be derived from findings and analysis thereof.
• The report should also include the analysis of challenges that may have affected the collection of data, as well as the quality of data collected.
• Recommendations should be practical and if necessary be divided up for various actors or stakeholders as well as sectoral and include guidelines of how they can be implemented.
• Report annexes Including FGDs participant’s attendance sheets, consent forms, questionnaires, etc.
The reporting style will be clear and accessible. References to sources used, such as interviews, literature, reports, must be given.
Confidentiality of information: All documents and data collected will be treated as confidential and used solely to facilitate analysis. Interviewees will not be quoted in the reports without their permission.
14. Assignment Duration
The entire period of this consultancy will be 60 working days that will include preparation, development of the survey methodology, tools, testing and reviewing of the tools, actual field work, and data analysis, dissemination of the key findings to stakeholders and reviewing of the final report.
Selection Process and Criteria:
The potential firm/consultant will submit detailed technical and financial proposal/s. The proposal/s will devise technical strategy and all costs. The proposal/s should include:
• Cover letter addressing the requirements stated
• Detailed CVs of expert(s)
• Description of proposed methodology to clearly address the criteria and content of the Terms of Reference and may detail what additional improvements / best practices the firms can introduce while collecting and computing the data
• Samples of previous work including tools used and reports produced
• Three references (names, email and phone contacts) from previous clients where similar work has been conducted
The proposals will be evaluated on technical and financial basis. The technical score will be awarded upon maximum of 80% score, whereas, the financial will be awarded maximum of 20% score. The client will award the contract to the firm/individual with highest accumulative score in technical, financial and the presentation of the competition process. The review team will review proposals based on the following general criteria:
1. Methodology 30
2. Demonstrated relevant experience 20
3. Proposed methodology, key personnel and enumeration staff 25
4. References 5
5. Cost 20
All documents related to this tender shall be in English and all costs shall be expressed in Rupees. Proposals may be submitted electronically or by hard copy. If proposals are submitted electronically, all pieces to the proposal must be labelled clearly. Acceptable formats include Microsoft Word, Microsoft Excel, and PDF.
External evaluator will have the trained enumerators in each district. All the enumerators will be local residents and well aware of local culture and norms. There will be no need of accommodation of enumerators in their respective areas. The accommodation of external evaluator in the districts and travel of enumerators and external evaluator will be responsibility of external evaluator. International Medical Corps Pakistan Program will give lump sum amount to external evaluator as per related clauses of consultancy agreement between external evaluator and International Medical Corps.
18.-Qualifications and experience required:
A well-recognized and reputable research, consultancy firms/company or individual consultants having documented/proven experience in conducting evaluations in GBV Prevention and Response in Pakistan are eligible to submit their proposals. Preference will be given to those who have worked with Afghan Refugees in KPK.
• No contract may commence unless the contract is signed by both parties.
• The team members of the consultancy firms are not entitled to payment of overtime. All remuneration must be within the contract agreement.
• Accommodation and meals will be covered by the consulting firm/company. Therefore, the price component must contain an overall quotation. However, the detailed budget required to show the actual breakup.
• The consulting firm should provide its team with relevant IT equipment’s and other office supplies related directly to the task.
• International Medical Corps will provide facilitation such as addresses, meetings with community, etc., for the movement of the teams in field and geographic areas. The travel to and from and within the field needs to be reflected in financial proposal.
• Final payment to the consulting firm will be dependent on the completion of deliverables (to be specified in the contract) as well as handover notes and relevant data.
19. Validity of Proposal
Proposals will be valid for 30 days. International Medical Corps reserves the right to reject any or all proposals received in response to this RFP, and is in no way bound to accept any proposal. The client additionally reserves the right to negotiate the substance of the finalists’ proposals, as well as the option of accepting partial components of a proposal if appropriate.
20. Human Subject Protection Issues
The consultant must be familiar with and observe the “WHO Ethical and safety recommendations for researching, documenting and monitoring sexual violence in emergencies”, and follow the confidentiality, privacy and informed consent policies of International Medical Corps program/projects such as GBV according to national and international protocols.
21. Intellectual Property Rights:
All outputs under the assignment shall be the sole property of International Medical Corps. The Consulting Firm shall not exercise any rights on all the outputs and cannot share the same with any outsiders without the prior consent of International Medical Corps.