Evaluation of the National Preterm Birth Prevention Collaborative
Evaluation Methods
The Every Week Counts Collaborative will be utilising a mixed methods approach to evaluation, utilising both qualitative and quantitative data to provide greater depth of insight into the complex realities of participating in the project. The information obtained from qualitative inquiry will be combined with the standardised and generalisable quantitative data to strengthen learning related to the evaluation questions.
It should also be noted that both formative and summative evaluation approaches are used within Collaborative methodology to ensure maximum impact of the approach within the participating sites. Formative evaluation is a process of gathering and analysing feedback during the development or implementation of a project to identify areas of strength, weakness, and improvement so that adjustments may be made to improve the effectiveness of the work. This formative evaluation has occurred through a series of Milestone Reviews held periodically over the course of the project. Summative evaluation occurs at the end of a project to measure the project’s outcomes and summarise learning. The summative evaluation of the Every Week Counts Collaborative will be conducted following the project end in June 2024.
Collaborative Data Reporting
Improvement is about testing, adapting, implementing, and spreading change that results in better outcomes. Data serves as feedback to help us learn about what is working and whether we are achieving the intended results. The Every Week Counts Collaborative measurement strategy includes a family of measures comprised of outcome, process and balancing measures. The process measures provide teams with early feedback on the changes they are testing to inform their learning about what works. While outcome data supports understanding about whether the changes implemented ultimately contribute to the achievement of the desired results of the project, in this case a reduction in early term (birth from 37.0 – 38.6 weeks’ gestation), or preterm birth (Births from 20.0 – 36.6 weeks’ gestation). While balancing measures ensure the program of work is not having any unintended consequences.
This data is reported by participating teams into a quality improvement platform, Life QI, that supports service to share learning and data and facilitate collaboration amongst participating services. Participating teams enter process data weekly and outcome data on a monthly basis to provide as near to real time feedback if the changes teams are introducing are resulting in improvement.
Interviews and Focus Groups with Participating Teams
The Australian Preterm Birth Prevention Alliance has engaged researcher from the University of Melbourne to independently evaluate the activities of the National Collaborative to understand the impact of the program and key factors that were supportive of success. Interviews and focus groups are particularly suited to providing in-depth understanding of participant perspectives of implementation, outcomes and factors influencing both of these.
Some of the key themes to be explored using this approach include:
- understanding participating team experiences in implementing evidence-based changes
- Determining the enablers and barriers to implementing changes
- Identifying the drivers that accelerated improvement
- Exploring the internal and external factors that may have impacted on implementation
The information obtained from qualitative inquiry will be combined with the standardised and generalisable quantitative data to strengthen learning related to the evaluation questions. The evaluation results will be used to assess the effectiveness of the program during 2022-24 and will be available to guide future collaborative work in Australia and around the world.
Understanding the impact that consumer engagement played in the Collaborative
Consumer engagement plays an important role in the Every Week Counts Collaborative and in the improvement work undertaken by many of the participating teams. Consumer engagement is a part of every level of the project. To best understand the experience of consumers in the Collaborative and in reducing preterm and early term birth, interviews and focus groups will be held with consumers from the Consumer Advisory Committee as well as consumers actively participating in maternity service improvement activities at participating hospitals.
Evaluation Timelines
Data collection for the National Collaborative will continue until the end of June 2024. Following this data will be analysed and prepared for publication in international journals. An end-of-project report is also being prepared outlining details of the quality improvement approach, change ideas introduced and results from both quantitative and qualitative evaluation methods. It is anticipated that the release of the Every Week Counts Collaborative Public Report will be delayed until such time as the peer reviewed publications have been published. This is to prevent early release of data affecting the ability for articles to be accepted into quality journals.
