ICT 4 RBF: when technological innovation meets health care financing innovation. A PBF CoP meeting. 28 April – 1st May 2014 Burundi

19 Feb

Do you think that new technologies can boost efficiency and good governance in RBF systems? You have ideas or experiences to share? Join us in a technical meeting of the Performance-Based-Financing Community of Practice organized from 28th April to 1st May 2014 in Bujumbura, Burundi.

You can access the detailed program here.

Towards a more efficient health sector in Africa

For many years now, multiple countries in Africa have engaged in large-scale health financing reform for improving health outcomes. Result Based Financing is frequently at the heart of these reforms with countries adopting both supply side (such as PBF) and demand side RBF mechanisms (such as conditional cash transfers and vouchers).

Both supply side and demand side systems rely on effective data systems that need to be supported by adequate technology platforms.

In supply side RBF systems, information technology (IT) is used for administrative management of RBF/PBF such as decentralized result declaration, RBF subsidies calculations, managing verification processes or production of bank transfer orders. Information technology is also used for strategic purchasing, budget follow up, and transparency and benchmarking. Currently, countries use a set of web based open source instruments such as OpenRBF or DHIS2 or sometimes stand alone technical solutions. For example, you can visualize RBF data  from Zambia, Benin, or Burundi.

In demand side systems, IT is used for conditional cash transfers (CCT) and voucher management. The features of the CCT and voucher system include: patient follow up, provider accreditation, voucher and fraud tracking, budget follow up and reseller management. Currently, projects and countries use a broad set of IT instruments.

Why a technical meeting on ICT and Result based financing?

New technologies are already used in existing Result Based Financing systems. Cloud computing has improved data flows, reduced transaction costs and facilitated financial transfers. Mobile phones are facilitating demand side financing instruments. But future RBF systems will even more rely on new technologies.

New projects are going in several directions. For example, countries are starting to explore how to better share verified RBF data with the general public, partners and authorities. This requires the development of specific web dashboards and data visualization instruments that Cameroon, Benin and Burundi are currently piloting. Data could also be shared in an active way by email or text messages. Vouchers scheme are migrating to eVouchers, and the voucher coupons are becoming text message codes. Some projects are trying to reinvent community participation: patients will be allowed to send their feedback about providers by mobile phone and the comment will be displayed on the public OpenRBF provider webpage. Other projects aims to use smartphones to collect patient feedback and verification data, or tablets for quality assessment of providers.

The potential of new technologies is obvious. It is the right time to work together on how these technologies can be harnessed effectively to boost health outcomes.

Share your experience, learn from your peers

The objective of this meeting is that CoP members share new technological developments, implementation experiences, identify shared issues and problems, draw best practices. Like any other CoP events, we expect fruitful interactions, interesting encounters and multiple new ideas.

During the meeting, we will also test the idea to create a working group within the RBF CoP that will take the lead on new technologies for RBF.

If you want to know more details, please check the detailed program.

If you are interested in the meeting, please register here or fill the registration form at http://goo.gl/lXvYEH

Nicolas de Borman, Alfred Antoine Uzabakaliho, Cheikna Toure, Randy Wilson and Olivier Basenya

Proving impact: options in terms of research designs

3 Jul

In this blog post, Ellen van de Poel (Erasmus University Rotterdam) summarizes discussions held in Bergen  between academics and PBF implementers about the desirability and feasibility of various research designs to identify causal impact of PBF. She covers discussions held within the dedicated working group (facilitated by Atle Fretheim from the Norwegian Knowledge Center for the Health Services, Oslo) and in plenary sessions.

Being a novice to the field of Performance Based Financing (PBF), this interesting two day workshop brought quite some new insights to me, but also raised many questions. Does paying for performance crowd out intrinsic motivation? Are the poor benefiting most from PBF, or do these schemes mostly cater for the better off?  If PBF increases health care utilization, does it do so in a cost-effective way, compared to the traditional input based financing? Many interesting questions, ideas for research, … but, being a quantitative researcher, the most burning question on my mind is whether indeed PBF has a substantial impact on access to quality care in low and middle income countries. Prove of the impact of PBF is also urgently called upon by a recent Cochrane review, but robust impact evaluations seem hindered by the complexity of the intervention. This blog summarises our discussions in Bergen.

Getting the research question right

Let’s first make sure we got the research question right. “Does PBF work?”  is too simple.

For whom does it work? Which populations are we interested in? Patients using the health facilities, all households in catchment areas, poor or better off individuals? Defining the right target population is important for data collection. Exit interviews reveal valuable information about patient experiences but are limited to users of care, household surveys provide a more general picture, but might be hard to link to facility information; average effects may mask important inequities.

What do we mean by PBF?  Various terminologies like PBF, P4P, RBF and contracting are often used interchangeably while such programs can vary to a great extent.  In Bergen, practitioners stressed several key features. Let’s agree here that with ‘PBF’ we mean programs including incentivized payments, additional financial resources, reinforced supervision and increased managerial autonomy.

Works compared to what? Are we interested in evaluating PBF against the counterfactual of traditional input based financing, holding resources constant or at least equal – the incentive effect? Or do we want to measure average effects of the complete PBF program – resource and incentive effect?  This distinction obviously has important consequences for defining an appropriate counterfactual and knowing what we capture.

Works to achieve what?  What are the important hypotheses, and main outcomes of interest?  To avoid ending up testing for a significant effect on each (of the many!) incentivized outcomes, and finding some effects by chance only, researchers should be more transparent about the main hypotheses that they want to test before conducting the study.

Research designs

Once we agree on the research question, let’s see how it can be answered in a scientifically rigorous and robust way. Let’s go through the standard impact evaluation toolbox:

We agreed that the simplest thing to do is to compare districts/facilities with PBF to those without PBF. Easy enough, but the difference between both can be driven by many things other than the PBF intervention, so drawing causal conclusions is really not possible here.

We could compare outcomes from within the same district/facility from a period before PBF to a period after. While this approach is useful for monitoring purposes, and therefore typically embedded in the project cycle of PBF, it leaves room for too many factors other than PBF (e.g. other nationwide programs) to drive the change to really claim causality.

To be able to net out the effect of such nationwide changes that might bias the before-after comparison of PBF, we can compare the trend in outcomes in PBF facilities/districts to the trend in other, non-PBF, facilities/districts and only attribute the difference between both trends to the PBF intervention. In Bergen, some presented studies used the so-called difference-in-differences design. Basically, the non-PBF units (controls) serve as a counterfactual for what would have happened to the PBF’ed ones (treated) should there have been no PBF.  This design can produce very robust impact estimates, but the validity of the counterfactual is crucial.  Many of the controlled before-after studies that were presented in Bergen showed baseline differences between treated and controls that were much larger than the PBF effects identified. Furthermore, many of these effects seemed to derive from a negative trend among the controls, and not so much from huge increases among the treated. This does raise questions about comparability. When a country moves forward with PBF in some areas it is possible that attention/resources are being reduced in the ‘control’ areas, hereby biasing the counterfactual. In such instances, contextual and more qualitative information is crucial to build up credibility in the estimate.

Ideally, the allocation of districts/facilities to control or intervention is done randomly to make sure that there are no systematic differences between them – the so called Randomized Controlled Trial (RCT).

While the argument often made that it is unethical to randomize subjects in social policy experiments seems not justified – in fact there’s nothing more ethical than tossing a coin if you don’t have the resources to cater to everyone in one go – RCTs can be challenging in practice.

Randomization seems politically difficult.  How to explain to some districts/facilities that they will have to wait, how to convince local health officials/providers that the assignment was done in a fair way? Transparency of the randomization process  is crucial and can go a long way in mitigating these difficulties.

Another challenge relates to the design of the RCT. Randomization really only works if there is a sufficient number of units to sample from. Randomly drawing 200 from a total of 500 facilities obviously works better than drawing two provinces from a total of 5. A lower level of randomization not only ensures a higher degree of comparability between treated and controls but also increases statistical power, i.e. it is easier to pick up small effects that are statistically significant.

So ideally researchers would want to randomize facilities into a PBF program.  But is PBF really a facility level intervention, as it not only changes the way in which providers are rewarded and monitored but also the ways in which districts supervise and operate? While it may be difficult – but not impossible? – to evaluate the entire PBF program through randomization at the facility level,  an RCT design can be very feasible to identify effects of program components. For example, a design in which randomly chosen treated facilities obtain the full PBF program, while controls (in the same area) only receive the additional supervision, managerial autonomy and financial resources could very well identify the effect of paying for performance only. Such a design is currently under test in Cameroon, but some practitioners have been really harsh at such designs, which may be a nightmare for PBF implementers and district managers.

A more acceptable design is to randomly vary the fee structure across facilities within the district. This could be a way to identify the optimal reward scheme (but will not answer the question whether PBF works). There are some nice possible sophistications, such as randomly adding bonuses for equity-targets to existing PBF programs in some facilities, which can allow to identify their effects on the distributional impact of PBF, etc. This is the research design Burkina Faso is heading for.

Another method to evaluate the effects of PBF is interrupted time series analysis – two studies presented in Bergen used this technique. Basically the idea is to look for kinks in high frequency (administrative) data that coincide with the starting date of the PBF program. Ideally this is compared to data from a control group in which no such kink is evident. These studies can be quite convincing, but are subject to some constraints. First, one needs high quality and frequently collected data. Second, to convincingly attribute the kink in any trend to the PBF program, it is important to establish that this trend has been relatively stable prior to the intervention.  Third, as an important component of any PBF program is to improve the reporting and verification of the health care system, we need to be careful interpreting the kink in the trend in reported outcomes as this may be very well due to facilities having improved their registration of services provided, rather than having increased the volume.  So while looking at time series can be useful and powerful to evaluate modifications to PBF programs that do not touch on the monitoring system, they may be less suitable to evaluate entire PBF programs. Yet, in some settings, it may prove to be the main design available. It could also be an interesting track in countries where data reliability has already been improved. This could be the case for instance in Burundi, where the forthcoming impact evaluation will look at the addition of new indicators within a nationwide PBF system.

Finally, it may be worthwhile for researchers to consider using secondary data for evaluating PBF programs. Supplementing household survey data that is collected for many countries on a regularly basis (like the Demographic and Health Surveys, or Households Living Standards Measurement Surveys) with information on the (sequential) PBF rollout can reveal robust results if appropriate statistical techniques are used to correct for the non-randomized rollout.

Conclusion

A lot of research questions related to PBF, a lot of methods to use, so what are we waiting for? Well from the Bergen workshop I realized ‘we’ are not waiting at all. Research teams from LSHTM and Heidelberg University talked about interesting impact evaluations currently being done in Tanzania and Malawi, and also the World Bank is currently funding a large number of impact evaluations. It would be useful if such projects will not only contribute to the evidence base on the effects of PBF, but also to the evidence base on the practicalities of setting up prospective studies (see also here). It seems like many of these efforts are unnecessarily duplicated because this information is not easily available to researchers setting up new impact evaluations.  These new evaluations are indeed needed to establish whether paying for performance is a cost-effective way of increasing access to good health care, but also to find out how the incentives should be designed to get the biggest bang for the buck.

Financement Basé sur la Performance : structurons mieux le débat

9 Jun

Dans ce blog post, Bruno Meessen revient sur les critiques qu’il a pu entendre sur le FBP. Il  identifie 7 principales causes de désaccord et propose pour chacune, des pistes pour améliorer le débat.

A bientôt 15 ans d’engagement à promouvoir ce qui est devenu le FBP, j’ai bien sûr pu entendre les critiques les plus diverses à son égard. Ces critiques sont parfois prudentes et formulées sous forme de questions ou d’hypothèses, mais on sent parfois que le différend est plus profond. L’inconvénient alors est que l’auteur de la critique mélangera souvent différents types d’arguments. Si une telle stratégie peut être efficace comme technique rhétorique face à un auditoire spécifique, elle n’aide pas au débat.

Pour permettre un bon dialogue, je me suis rendu compte qu’il était important de décanter la nature de ces critiques en essayant d’identifier les causes sous-jacentes du désaccord. J’ai identifié sept motifs principaux ; à chaque fois on peut identifier une piste de solution. Continue reading

Researching PBF: time to open the black box

2 Jun

In this blog post, Gaute Torsvik (University of Bergen) introduces a forthcoming scientific workshop, which will take place in Norway on June 13 and 14. He argues that researchers studying PBF schemes should go beyond measuring impact only.

Performance-based incentive schemes and reforms are receiving growing attention from governments, aid agencies, donors and researchers. As for Africa alone, 29 countries are today at least piloting a PBF scheme; three of them (Rwanda, Burundi and Sierra Leone) have adopted performance-based financing as a core component of the funding of their whole health system. A major impact evaluation program led by the World Bank assesses the rapid expansion of performance-based financing of health care.

 Why opening the black box is important Continue reading

Formation au CESAG: 4 – 9 mars 2013

5 Feb

Après le succès de la première édition, AEDES et le CESAG organisent une deuxième édition de la formation en financement basé sur la performance. La formation se tiendra du 4 au 9 mars 2013.

Les formateurs de cette cession seront :

- Dr Olivier Basenya. Coordinateur adjoint de la cellule de Financement Basé sur la Performance au Burundi. Le Dr Basenya a accompagné les différentes réformes du financement de las santé au cours de ces dernières années.

- Dr Serge Mayaka. Dr Mayaka est médecin de santé publique, doctorant à l’UCL  et assistant à l’école de santé publique de Kinshasa. Dr Mayaka a accompagné la mise en place du FBP en RDC, au Benin et au Tchad.

- Maud Juquois. Maud Juquois est économiste de la santé au bureau régional de la Banque Mondiale à Dakar. Elle travaille sur la réforme du financement de la santé en Afrique de l’Ouest, et sur le FRB au Benin.

- Morris Kouamé. Morris est diplômé du CESAG et spécialiste en financement de la santé. Il travaille actuellement sur les questions des réformes visant à améliorer la couverture maladie dans la région.

Cliquez ici pour avoir accès à la plaquette de la formation, au  programme, et au formulaire d’inscription.

PBF: the pitfall of relying on a single sponsor

29 Aug

Bruno Meessen reflects on the lessons from a recent regional workshop organized by the World Bank on results-based financing for Anglophone African countries. In a previous blog post, he identified several positive developments. In this second contribution, he shares a point of concern.

The workshop in Livingstone, Zambia was a great opportunity to get a broader view on the development of PBF in Anglophone Africa. This nicely complemented the information I already had on the current situation in Francophone Africa. Although a rigorous mapping of PBF schemes in Africa still needs to be done (in fact, the PBF CoP offered to perform such a mapping as a wiki project already more than 2 years ago, so if you are a possible sponsor and willing to help, do not hesitate to contact us!), I realized that a worrying pattern is emerging: PBF is increasingly turning into a story of a single sponsor per country.

In a few countries (e.g. Rwanda and more recently Benin), PBF champions have been successful in building a broad and strong commitment among several funders of the health system. Yet, this is a minority. In many countries, PBF today is mainly a (strong) bilateral relationship between the government and one of its partners (often the World Bank, sometimes USAID or another bilateral aid agency).  Other donors are just watching things closely for now, it seems.

PBF sponsored by a single donor means at least two missed opportunities.

First, it creates a misinterpretation on what PBF really is about: from the outside, PBF is seen as a single project of that particular agency, whereas inside the agency, it is seen as an innovative approach to study. I repeat what I already wrote elsewhere: PBF ‘s ambition is to be an entry point to address several structural weaknesses constraining health systems in low-income countries, and in sub-Saharan Africa in particular. Such a reformist view has many implications. One of them is that one responsibility of PBF champions is to look for possible synergies with other efforts to strengthen the health system, as happened for example with selective free health care in Burundi or the Health Sector Services Fund in Kenya.

There is a second missed opportunity: the possibility to use the PBF approach as a virtual basket of funds. Indeed, PBF offers the possibility to any donor to obtain results in its areas of interest (geographical or programmatic ones) in a manner consistent with a national strategy. The main requirement is to accept a shift from an input-based approach to an output-based one. It is a concrete way to realize principles promoted by Sector Wide Approaches or even the Paris Declaration. Conversely, this indicates that PBF champions have to engage and collaborate with agencies involved in basket funding arrangements.

It is not yet clear what prevents so many donors from committing to the strategy in new ‘PBF countries’. Skepticism? Inadequacy of their aid instruments? Disinterest in structural changes in the operation of the health system? A lack of funds? A conflict of interest? A wait-and-see attitude? I suspect that this outcome is also partly the result of mistakes made by ‘PBF champions’.

In 2004-2005, PBF was still in its infancy as a policy; several of us spent time and energy to defend and explain the strategy inside our organizations (ministries, aid agencies, research institutes…) or to the outside world. Today, thanks to the financial support of Norway and the UK and via the World Bank, PBF is becoming mainstream all over Africa. Yet, if its full potential is to be achieved, more work needs to be done. Are we not making the mistake to ‘lazily’ rest on our laurels, i.e. this acquired funding? In fact, PBF needs more donors to join and more fundamentally, still has to access public funding in many countries – integration in public finance will indeed be the real litmus test for the full realization of any PBF pilot project.

As a community, we have to bring the ongoing pilot projects to success, but we should not diminish our efforts to advocate for the strategy. Let us not make the mistake to believe that conclusive evidence from the impact evaluations will naturally bring key actors to commit to PBF. The world is simply not that rational. Seizing the transformative power of PBF is a common responsibility for all of us: the World Bank, the implementing PBF agencies, but also individual experts.

The fact that participants to the Zambia workshop formulated the request to receive assistance for these advocacy efforts is a clear indication that this is a widely shared view.

 

The challenges of implementing results-based financing health in Anglophone Africa

2 Jul

Bruno Meessen reports on a recent regional workshop organized by the World Bank on results-based financing for Anglophone African countries. In this first blog post, he identifies some reassuring facts and lessons. In a second blog post, he will come back on matters of concern.

Around 80 experts from 9 countries (Cameroon, Gambia, Kenya, Lesotho, Liberia, Sierra Leone, Tanzania, Zambia and  Zimbabwe) joined their forces in Livingstone (yes, next to Victoria Falls) for a 4-day workshop organized by the African Region of the World Bank and the World Bank Institute. The general theme of the event was “Challenges of implementing results-based financing health in Anglophone Africa.” As you may remember, a similar workshop  had been organized for Francophone African countries in December in Limbe (a town which, by the way, used to be called “Victoria” between 1858–1982. It  looks like Queen Victoria was popular among explorers and missionaries in the 19th century!).

The program covered most of the major issues faced by any country willing to adopt an RBF/PBF strategy: institutional arrangements, budgetary forecasts, operational challenges, ICT solutions for data management, the importance of pilot projects, institutional and financial sustainability, PBF & Universal Health Coverage , monitoring & evaluation, contractual options, procurement and financial procedures, etc.

Most participants were familiar with PBF and its key principles. The faculty team was also impressive. National experiences were the central topic of the workshop. Some countries like Zambia, Sierra Leone, Liberia and Zimbabwe are well advanced in their PBF policies. Others like Cameroon, Kenya or Tanzania are at the pilot stage. Gambia and Lesotho have yet to start their first pilot.

Lessons in terms of design

The workshop was rich in lessons. Here are a few that I would like to highlight.

First, one could observe a convergence in terms of design. This is particularly true for projects supported by the World Bank. Some schemes will probably need some corrections. In Zimbabwe for instance, the health staff is not allowed to earn some personal income from their performance (PBF revenue collected by the health facility has to be allocated to purchase of inputs). The current approach in Liberia is more a performance-based contracting experience than what we are used to call a PBF scheme in sub-Saharan Africa. While ‘contracting out’ could be necessary in a post-conflict setting, one should hope that donor countries will realize that contracting out districts to their national contractors has limits, especially in terms of maximizing resources for frontline services. The most efficient approach to transfer technical assistance to countries is a topic which deserves more attention. At another level, I was particularly interested in the case of Kenya: the pilot scheme there is building on the ‘Health Sector Services Fund’, an existing scheme which transfers public resources directly to the bank account of health facilities. This should allow us to see the added value of linking the payment to results (if any).

Lessons in terms of implementation

Many discussions in Livingstone were consistent with discussions we had elsewhere. For instance, the challenge of geography and distance to health facilities came back. This is obviously not a challenge specific to PBF. However, it is important to keep in mind that PBF was conceived and fine-tuned in one of the most dense regions of Africa (the Great Lakes Region). Obviously, distance between the population and the health facilities are much greater challenges in some other parts of Africa. This was already much-discussed fact in the 2012 joint Communities of Practice workshop of Bujumbura with delegations from vast countries like Chad or Niger. It is probable that we have not yet factored in this challenge in the PBF strategy. (1)

This is a not scoop, but as mentioned by many participants, context matters a lot. PBF has sometimes been promoted with too simple messages (e.g. the guidance that the right budget for a PBF scheme is 3$/year/inhabitant). In a country like Nigeria, where a midwife may get a salary of $500, the amount required to motivate the personnel to work more would have to be substantial. Similar problems may be faced by South Africa’s neighboring countries: the pays have to be attractive to dissuade staff to migrate. We need a technique (like discrete choice analysis? ) to better identify what amount is required for a PBF to succeed in motivating a health staff. (2)

In general, throughout the workshop, one could feel a very good understanding among participants of challenges faced by PBF at country level. The partnership with the World Bank seems also strong. This is reassuring. For instance, there is large consensus on both sides that PBF requires to be piloted before to be scaled up at national level (by ‘pilot project’, we mean : a small experiment in a couple of districts to adapt the strategy to the local context). I remember that it was not the case 2-3 years ago: some countries, but also some agencies and experts, believed that one had to seize the window of opportunity and go to scale from day 1.

Points for action

During the workshop, we also identified things which should be addressed at regional or even global level. A first one is certainly the need to be more systematic in the transfer of tools and methods for the monitoring of PBF schemes. A lot of emphasis has been put on impact evaluation, but probably not enough on more classical monitoring and evaluation.  Another one is to move away from regional workshops gathering… mainly us: technocrats from the aid sector and the government. PBF possibly has major transformative value for African societies. The strength to play forcefully requires that we engage much more with civil society, including the media and non-governmental organizations.

Endnote:

(1)    Yet, one lesson I learned from Cambodia is that once the economy takes off, road building and growing urbanization can be more effective than covering the whole country with health facilities.

(2)    Yet, these methods will have to factor in that PBF modifies not only the income but also the effort to exert and to some extent, the decision rights held by the staff.

 

Follow

Get every new post delivered to your Inbox.

Join 27 other followers