Immpact Toolkit:
A Guide and Tools for Maternal Mortality Programme Assessment

Version 2

Preface

The need to evaluate the effects of interventions on health is still often questioned. Today, as in the past, this is partly because of a belief that only benefit (health gain) can ensue from such initiatives, and partly because of a concern with the costs - time, finance, personnel, and skills - incurred in conducting outcome evaluations. The need for evidence-based resource allocation, on the other hand, is almost universally accepted. It makes common sense for scarce resources to be allocated to interventions where they are likely to make a difference. However, if these interventions are not evaluated for their effects, future resource investment will necessarily be opinion-based rather than evidence-based. Moreover, the shift towards results-based financing by donor agencies is now creating a further demand for evaluation of health outcomes. The challenge is to make such evaluations desired and realistic where capacity is limited but the need great.

This volume speaks directly to one such need - evaluation of composite health interventions for safe motherhood in contexts challenged by weak health systems, scarce resources and lack of relevant information. It is built upon the experience of Immpact - a global research initiative to identify cost-effective strategies to reduce maternal mortality in developing countries. The focus on strategies and programmes is intentional: there is no 'magic bullet' but rather multiple inputs and processes are needed to bring about an effect on maternal mortality. Assessment of such composite interventions currently represents a frontier for evaluation methodology and best practice has yet to be established. Immpact adopted a pluralist perspective to evaluation, utilizing qualitative and quantitative techniques, and acknowledging different stakeholder views to judging a strategy. As such, we have drawn upon best available methods and experiences from different disciplines. Where fit-for-purpose tools were lacking, Immpact developed and tested new ones. At the same time, we sought to build capacity in collaborating countries to use best methods and the evidence generated. The three interrelated outputs of Immpact in its first phase (2002-2006) have thus been enhanced methods and tools, new evidence on the effects strategies, and stronger evaluation capacity.

We have learned much else along the way, but six lessons stand out:

• Firstly, we have seen how stakeholder engagement is crucial from the outset, particularly in agreeing on the focus of evaluation. In our three initial collaborating countries - Burkina Faso, Ghana and Indonesia - we were extremely fortunate to have high level support from government, providing 'champions' for the evaluation who remained engaged through out the four years of work.
• Secondly, we have learned that refining the evaluation question is a lengthy but crucial process if stakeholders' expectations are to be met and if focused measurement of inputs, processes and outcomes is to take place.
• Thirdly, Immpact has been exposed to the difficulty of defining the content and delivery of strategies when they become implemented programmes in real world contexts where many other influences are at work.
• Fourthly, we have grappled with the enormous breadth and depth of primary and secondary data to be captured - from populations, health facilities, reports, and key informants. These data reflect the hard work of the Immpact research teams across the seven collaborating institutions, and highlight the range of expertise needed to conduct composite evaluations.
• Fifthly, quantifying the inevitable uncertainty of attributing health effects to specific intervention programmes has presented us not only with analytical challenges but also a communication challenge. We have sought to provide evidence helpful for decision-makers, whilst not disguising uncertainties of attribution.
• Sixthly and finally we have learned that such evaluations are like a story - involving key characters, specific settings, intense activities, a beginning and an end. This toolkit is part of Immpact's story.

But the story is not quite finished. This is the second edition of the Immpact Toolkit, with the story told up to the point of having completed evaluations in three countries. Here we give details on the process of defining evaluation questions and agreeing on evaluation designs and data capture approaches and instruments. We will continue to refine the story as we apply the toolkit in other evaluations and as we use the findings of these evaluations to tell us more about the pros and cons of the methodology adopted. The uptake of these findings and how they contribute to policy and programmes are the final 'proof of the pudding', providing ultimate assurance of the toolkit's worth.

The very use of the term 'Toolkit' as a collection of tools with guidance on their selection and use emphasizes the importance of evaluations being tailor-made for specific contexts. In composite evaluations, tools need to be selected and used not independently, but as complementary devices which together construct a story. From our experience, one can thus offer broad guidance - as we do here - on designs and tools, but local adaptation, flexibility and interpretation will always be needed.

Our ultimate hope in undertaking Immpact and in preparing this toolkit is that we have convinced stakeholders that composite evaluations, encompassing context, process and outcome measurement, is a worthwhile and legitimate challenge in the strive for safe motherhood.

Wendy Graham
Immpact Principal Investigator
Aberdeen, September 2007

Toolkit content overview

This toolkit brings together a collection of research tools on safe motherhood developed by the Immpact research programme. Immpact is a global research initiative with the ultimate goal of reducing maternal mortality in developing countries. The toolkit is available on CD, or as a web-based resource (www.immpact-international.org) and as a paper-based resource, downloadable in .pdf format.

The selection of research tools has been developed based on Immpact's experience in designing various evaluations. Not all of Immpact's research tools are included in this second edition of the toolkit. Other tools will be added and improvements made to the existing tools in future editions, as greater experience is gained on the use and potential of the different tools. The tools contained in this volume include techniques, methods, instruments or approaches used during the process of conducting evaluations of safe motherhood programmes.

This Toolkit Guide is not meant to be read from cover to cover but is designed to be used as a reference resource for grappling with evaluations of complex programmes. Although users can pick out individual tools for particular purposes, evaluations should be seen as a process of discovery, with optimal insights to be gained from the appropriate use of a variety of tools developed from a variety of perspectives. A list of Acronyms used in all Tools, and of all References is included for ease of referral.

The Immpact Toolkit consists of five modules described here, and presented as a summary table below.

We would be pleased to receive feedback from users on this second edition of the toolkit. Immpact also provides formal training and capacity building on the use of Immpact tools. Please send your queries and comments to info@immpact-international.org.

Table of Contents