Background
The Balochistan CDWA evaluation was conducted as part of the National Evaluation Capacity Development (NCED) initiative, which began in 2015, when Phase II of CDWA began. The initiative focuses on developing and implementing effective monitoring frameworks, establishing SGD- focussed indicators and collecting and analysing accurate data for those indicators. An effective monitoring framework will contribute significantly to the ultimate outcome of developing evidence- based policies.
Since 2015, UNICEF has been working with the Planning and Development Division (P&DD) Balochistan on M&E capacity development. This evaluation contributed to that capacity- development in several ways, as described in this report, including the concept and content of a theory of change as an integral element of an effective M&E framework.
Access to safe drinking water is both a basic need and a basic human right. Yet, water on our planet has become both scarce and impure while the demand increases continuously. Access to safe drinking water is also a critical issue in Pakistan. Despite remarkable improvements in the proportion of the population using an improved water source and an improved sanitation facility, 27.2 million Pakistanis do not have access to safe drinking water. Diarrhoea is the second leading cause of death in children under five years old. In Balochistan, between 2011 and 2015, the use of many water sources decreased, and the water supplied through drinking water sources is not always safe due to contamination from various sources. Many of the existing water sources in 2011-12 were either unsafe at the consumer end or were non-functional.
There have been efforts by both the federal government and the provincial government to improve the availability of clean water in Balochistan. In 2007, the Federal Government decided to install 567 water purification plants in Balochistan at the UC level. In the initial stages, the effort was led by the Environmental Protection Agency, the district government and contractors who were selected through a bidding process to construct and staff the plants. Later, responsibility for the implementation of the initiative, called the Clean Drinking Water for All (CDWA) project, shifted to the Public Health Engineering Department (PHED). The project officially began in 2008 with a target of 575 (567 + 8 additional plants) total plants (including both Ultra-Filtration (UF) and Reverse- Osmosis (RO) plants). The plants were modular, and four different classifications of plants were used, depending on the characteristics of the water source. Phase 1 of the CDWA closed in 2011, by which time only 409 UF plants and 13 RO plants had been installed, and 45 of those UF plants remained non-functional. Phase 1 faced many challenges, including lack of electricity, high operation and maintenance costs, political interference, and a devolution of power in 2010 which shifted responsibility for the CDWA from the federal government to the provincial government. In 2015, the design phase for Phase II of CDWA began, with implementation beginning in 2017. This included 85 solarized plants, 24 RO plants and 61 UF plants.
In 2011, a third-party verification exercise was carried out by Balochistan Engineering Services Agency (BESA), which identified significant issues, including that 41% of the installed plants were non-functional. Since 2015, UNICEF has been working with the Planning and Development Division Evaluation of CDQ Balochistan Component P&DD/UNICEF Joint Evaluation
(P&DD) Balochistan on M&E (Monitoring & Evaluation) capacity development under its National Evaluation Capacity Development (NECD) initiative. In 2015, the Government of Balochistan (GoB) asked UNICEF to support the province in developing an evaluation policy. Thus, a joint evaluation was suggested where the P&DD was supposed to select a project from the social sector (ADP) for such an evaluation. UNICEF agreed to provide technical support for the evaluation and agreed to support a joint-country led evaluation for an ADP project within the WASH sector.
The overall purpose of conducting an independent and objective evaluation was to gauge the effectiveness of the CDWA project implemented by PHED in 15 districts of Balochistan, over the period 2010-11 upto and including May 2018. Additionally, it was to inform programming decisions for improving water supply to households while demonstrating accountability to the stakeholders, drawing lessons learnt, and forming recommendations to inform continuity and scale-up.
OBJECTIVES
The evaluation aims to:
- measure the program in terms of how successful it was in addressing problems related to water;
- whether there were more effective ways of addressing the problem for different cost and
- to build the capacity of the GoB to conduct evaluations by helping to develop a policy for the evaluation of development projects in Balochistan. The evaluation is focused only on long-term outcomes and has therefore not employed any impact evaluation methodology using experimental methods.
A three-day training workshop was conducted from 29 April to 1 May 2018. The workshop was held at the Gardenia Hotel Quetta and was attended by both field supervisors and officials of the Government of Balochistan. The training focused on a variety of topics relating to evaluations, tool development, sample framework, practical data collection and processing of data gathered; the training was delivered through multiple approaches, including discussions and exercises.
The Evaluators have prepared data collection tools that meet applicable national and international best practices, including UNEG/UNICEF guidelines on participatory approaches, respondent- friendly methods of data collection, human rights, equity and gender in the design, data collection and data processing stages and in the formulation of recommendations. The latter ensures ownership of the actions to be taken in the post-evaluation period. Human rights, equity and gender aspects were included in the evaluation design.
The Evaluators included quality assurance approaches and methods for both qualitative and quantitative data collection, including random re-testing, spot checks, verification visits, cross- checking, independent reporting by different team members and picture verification amongst others. The quality assurance function was embedded into all training, supervision and review processes. The qualitative and quantitative data was validated prior to analysis and was triangulated to achieve greater clarity. All final reports were professionally developed and edited.
Evaluation
Challenges and limitations (geographic, security, cooperation, documentation, fieldwork and others) that would likely be faced by the evaluation team during the fieldwork were identified ahead of time and appropriate management and mitigation measures were put into place.
Findings
The findings described in this report are based on 56 KIIs , 30 FGDs, 300 HHs surveyed and an inspection of 532 WFPs conducted in 15 districts of Balochistan, in the period 4th to 8th May 2018.
Both of the hypotheses framed by the evaluation (see Section 2.6.1) were proved false. The subsections below summarize the key qualitative (QL) and quantitative (QT) findings of the evaluation against the OECD-DAC evaluation criteria. Please refer to Table 06 for an overview of the findings against each of the key evaluation questions.
Conclusions
The CDWA programme Balochistan component remains relevant to the province. However, to create a meaningful impact, the programme must i) have WFPs at the ward level and ii) account for population density. What was inherited was weakly designed to begin with, and weak programme management threatens what has been achieved. The lack of timely interventions in operations & maintenance is one significant factor, stemming mostly from improper attention by the P&DD and PHED in 2010, a fact that remains visible in the ADP allocations to date. The technology selected remains appropriate, and because of its modular nature it can be upgraded or downgraded to meet changing input water quality; implying that an active monitoring system is required. Expensive equipment and consumables will be/are wasted due to inefficient district- level management.