The proposed evaluation of Safewards in Victoria for 2016-2019 follows the completion of the Victorian Safewards Trial, including the one-year follow up phase. The evaluation of the Trial was conducted by researchers at the Centre for Psychiatric Nursing, the University of Melbourne.
Safewards was associated with reduced seclusion events overall and improvement was especially evident in adult and youth wards. Also, Safewards had a highly favourable impact in terms of staff and consumer perceptions of increased safety and more positive inpatient environments.
Independent evaluation of the Victorian Safewards Trial was built on a program logic map and conducted across project phases of training, trial and sustainability. The evaluation provided project partners with clear findings regarding acceptability, effectiveness and impact of Safewards. The program-logic defined the objectives of the trial project. A mixed method design and multiple data sources were used. Data included: Safewards fidelity measures, based on observation by evaluators; several staff and consumer surveys; organisation-level diaries of project activity; and state wide data regarding seclusion events in all inpatient settings.
For detailed evaluation of Safewards in Victoria see the outcomes article: Outcomes of the Victorian Safewards trial.
The Trial evaluation answered six major evaluation questions:
- Was Safewards effective in reducing containment (seclusion events) of consumers in the trial wards?
- How did Safewards impact on safety and conflict in participating wards?
- How acceptable was Safewards to consumers in the wards participating in the Victorian Safewards trial?
- How was Safewards implementation enabled and impeded?
- How acceptable and applicable was Safewards, according to staff participating in the Victorian Safewards trial?
- Did the participating wards achieve fidelity with the Safewards interventions, beyond the trial phase?
In terms of key objectives:
Effectiveness: Safewards was associated with a 36% reduction in seclusion rates, from baseline to follow-up.
Applicability: The Safewards model made sense to staff, most interventions were keenly taken up by staff (n=103), and consumers enthusiastically engaged with several interventions (n=72).
Acceptability: Safewards was highly acceptable to staff (n=103) and consumers (n= 72).
Sustainability: Fidelity was achieved to a very good standard in the trial period (n=18 X 3) and to an excellent standard by the end of the sustainability period (n=14 X 5).
The researchers concluded that further implementation should be supported by a refocused evaluation effort. The focus should be on four things:
- enhancing outcome evaluation for Safewards by developing feasible data collection for the primary outcome of conflict events
- measuring effects in wards where Safewards is continuing and where Safewards is newly implemented, via outcome measures and consumer feedback
- enabling ongoing monitoring of use of Safewards interventions across all wards, and via fidelity measurement
- reporting effectiveness of staff training via brief pre-and post- surveys and quizzes.
Victorian Safewards evaluation for 2016-2019
Considering the overarching program logic and the findings from the Safewards Victorian Trial, further evaluation should be directed to monitoring, publically reporting and understanding performance and variations both in Safewards implementation and in conflict and containment outcomes, between wards and service types, over time. Such an evaluation will inform ongoing actions to achieve sustained implementation and optimal outcomes.
The Evaluation will focus on the following:
Access the list of locations from the map below in accessible format
- Training outcomes - Given the development of Victorian Safewards resources and the proposed approach to DHHS staff consulting into services for train-the-trainer programs, the project includes evaluation of effective learning and will include Post survey evaluation of Safewards training.
- Fidelity - The project includes evaluation of the extent of success of implementing Safewards across all mental health units and sustained use across Trial wards. Fidelity evaluation will include the completion of fidelity checklists.
- Outcomes – There is an opportunity to develop and pilot a real-time measure of conflict on units, using an abbreviated version of the current conflict measure (PCC items) on a mobile device. Evaluators will work with participating services, through stages of design, pilot and trial and roll out of an app. The team will continue to analyse and report containment events/restrictive practices, as key Safewards outcomes of high value to consumers and staff.
- Impact - Consumers’ experiences of Safewards will also be directly canvassed, across new and continuing sites. The emerging consumer/peer workforce in participating services will be key to enabling consumer surveys.