5. Ratings.

OUTCOME

5.1 Relevance. The initial overall design was, and remains, substantially relevant, despite the dilution of the targeting during appraisal and execution, in the context of Indonesia's priorities and the Bank's country and sectoral strategies. Current high-risk target populations would focus somewhat more on IDU than CSW. The Bank's 2003 Country Assistance Strategy (CAS) (17) establishes, as a key pillar of strategy, making service delivery responsive to the needs of the poor, with particular reference to health services. The HSPMP concept fits well into this. Similarly, the Project objectives are consistent with the 2004 East Asia and Pacific regional AIDS strategy. (18)

5.2 The first objective, of strengthening institutions needed for the prevention and management of HIV/AIDS and STDs, is highly relevant in Indonesia today. With dramatic decentralization of public services since the financial and other crises of the late 1990s, provincial-level services have assumed increasing importance. Strengthening epidemiological and behavioral surveillance continues to be important in Indonesia. In particular, while the HSPMP did not address this problem, dealing with issues in the financing of surveillance is a high public priority, as Indonesia's health system is increasingly decentralized. Similarly, enhancing the capabilities of Indonesia's public health laboratories continues to be important to the country's priorities and the Bank's global and sectoral strategies. Finally, the effective establishment and operation of policies and procedures for improving the flow of public funds to NGOs for health services, and of making NGOs an active partner in the provision of services to hard-to-reach populations continues to be highly relevant today, particularly under a country strategy that emphasizes making service delivery responsive to the needs of the poor.

5.3 The second objective, of piloting interventions to reduce transmission of STDs and HIV, continues to be substantially relevant in Indonesia. The key problem concerns moving beyond disparate, discrete donor interventions, often associated with high quality technical support, into large national programs. This is an issue of both intervention design and institutional support.

5.4 Efficacy. Since most of the Bank loan was cancelled and only a fifth of the resources were actually disbursed, efficacy--the extent to which the Project objectives were achieved, or expected to be achieved--of the HSPMP is rated negligible, overall.

5.5 The efficacy of the project with respect to the first objective, of strengthening of AIDS institutions, is rated modest. As indicated above, the most significant achievements were in the areas of public health laboratories (see Annex D) and establishment of mechanisms for the flow of government funds to NGOs for public health action. The latter occurred near the end of the Project's limited period of execution, so they were only put partly into practice. Their apparent lack of use since then, and the rapid intervening change in the organization of the Indonesian health system, risk requiring future review and revision of the policies and procedures established during execution of the HSPMP.

5.6 The efficacy of the project with respect to the second objective, piloting of interventions, is rated negligible, despite the involvement of a considerable number of NGOs. The weaknesses in M&E made it impossible to build on project experience and ensure the learning that is an essential element of a successful pilot operation.

5.7 Efficiency. Overall efficiency is rated negligible. Utilizing the iwgAIDS projection model that assumed a rapid escalation of the epidemic in Indonesia, with parameters similar to an African epidemic, the SAR estimated rates of return on Project investments, ranging from 13 to 41 percent. No economic analysis was prepared for the Implementation Completion Report. Available evidence on Project execution did not justify or permit estimating economic rates of return for this report. While the AIDS literature suggests that the interventions planned to be piloted under the HSPMP are normally cost-effective, particularly in environments of high STD prevalence, no estimates are available of the cost-effectiveness in Indonesia of the specific interventions piloted under the HSPMP. With respect to the first objective, on institutional development, efficiency is rated modest on the grounds that the limited achievements on public health laboratories and NGO engagement likely improved the cost-effectiveness of the AIDS program. With respect to the second objective, despite the sound theoretical underpinnings of the planned pilot interventions, their efficiency as executed is rated negligible, since little was implemented and there was substantial dilution of the emphasis on HRG in implementation.

5.8 Overall, despite the substantial relevance of its design, the HSPMP outcome is rated unsatisfactory, due to its negligible efficacy and efficiency. This flows largely from the failure of the authorities to execute nearly four-fifths of the HSPNP in the wake of management problems and the Indonesian financial crisis. The assessment of relevance, efficacy, and efficiency by project objective is summarized in Table 1.

INSTITUTIONAL DEVELOPMENT IMPACT

5.9 Overall, institutional development impact is rated negligible. The HSPMP had a modest impact on the enabling environment by facilitating and requiring the adoption of policies and procedures for use of public funds for provision by NGOs of health services deemed to be in the public interest. Other impacts on the enabling environment were negligible. Aside from its favorable impact on health laboratories, the Project had a negligible impact on building AIDS institutions within the MOH and provinces. The ICR reports that the number of NGOs working on HIV/AIDS increased dramatically during project implementation, despite the rigidity of MOF contracting procedures. However, only the NGOs already active prior to the Project could maintain their activities after the HSPMP ended.

SUSTAINABILITY

5.10 Sustainability is rated unlikely. First, despite indications to the contrary from core agency personnel during preparation and appraisal, there was very little ownership of the Project among the many agencies and personnel concerned. Key champions in Bappenas changed jobs early in execution. Second, only a small part of the Project was actually executed, and only 20 percent of the loan was disbursed. Third, there were few measurable outcomes associated with those HSPMP activities that were executed. Only the public health laboratories component may have some modest sustainability. If future use is made of the procedures for government financing of NGO services, the arrangements made in connection with the Project could be sustainable, but no GOI, Bank or other donor program or project is known currently to be using them. Fourth, the Project was conceived as a pilot, but had essentially no effective monitoring or evaluation. In these circumstances, sustainability could not be expected.

BANK PERFORMANCE

5.11 Bank performance was unsatisfactory on both quality at entry and supervision. The concept of early intervention with high-risk groups and support to public goods was, and remains, fully consistent with Bank corporate and country strategy. Nonetheless, the pressure to advance rapidly led the Bank's preparation support and project appraisal to focus on specific technical dimensions of the proposed project and to pay insufficient attention to breadth and depth of ownership, institutional analysis and NGOs. The absence of sound sector work prior to identification and preparation, the complexity of the Project, and the limited period planned for its execution magnified these problems. These weaknesses meant that project approval was premature, and contributed to subsequent, managerial problems during project execution that were never resolved.

5.12 As observed in the ICR, the Bank's supervisory support tended to focus on current issues rather than the underlying fundamental problems of ambitious expectations and disagreements on target populations. While the staff paid attention to the Project's development objectives and demonstrated concern for M&E, the solutions proposed did not lead to meaningful action by the MOH or the core agencies of government. The tone of Bank communications was occasionally inappropriate and led to tensions with local stakeholders. The ICR points out that when supervision responsibilities were transferred from Washington to the World Bank Jakarta office, supervision missions became rare. ' Closeness of the task manager to day-to-day Project activities did not strengthen supervision in this case, but appeared to make the conduct of regular supervision missions seem unimportant.

BORROWER PERFORMANCE

5.13 Overall, borrower performance was unsatisfactory. During project preparation, leadership was taken by the central planning agency Bappenas. This is not surprising, in light of the then prevailing patterns of government action in Indonesia. However, the MOH implementing agencies, such as the Center for Health Education, and provincial level staff had a relatively passive role. Preparatory activities concentrated on technical aspects of the project, such as health laboratories and their requirements, and gave insufficient attention to definition of the respective roles and responsibilities of the many parties to be engaged in Project execution. Readiness for implementation was weak at the time of Loan signature. During implementation, macro factors--delays in receiving government counterpart funds under investment budget allocations, delays in ensuring adoption of an MOF circular to facilitate NGO participation, and the Indonesian financial crisis--had a strongly negative impact on the Project. Similarly, implementing agency performance factors were also negative, including conflicts between MOH units, unresolved role ambiguities between the PMU and the line departments of the MOH, reluctance to make use of technical assistance, and low levels of commitment to the Project. Monitoring was all but non-existent. While detailed indicators were set out in the SAR, target values were not fixed and baseline data were only collected late in Project execution (Annex D). There was no evaluation of the Project. While a high quality external review of Indonesia's HIV and AIDS programs was carried out near the end of the Project, discussions with stakeholders suggest that it had little impact. There were modest levels of compliance with requirements of the Loan Agreement, as indicated by the problems of delays in release of counterpart funds, PMU role confusion, and procurement performance.

Table 1. Summary of Project Outcome Ratings by Objective

                                        Relevance    Efficiency

Objective 1: Develop Institutions to      High         Modest
reduce STD and HIV Transmission

Objective 2: Pilot Interventions to    Substantial   Negligible
reduce STD and HIV Transmission

Overall Project                        Substantial   Negligible

                                        Efficacy       Outcome

Objective 1: Develop Institutions to     Modest       Moderately
reduce STD and HIV Transmission                     unsatisfactory

Objective 2: Pilot Interventions to    Negligible   Unsatisfactory
reduce STD and HIV Transmission

Overall Project                        Negligible   Unsatisfactory