What question does this study focus on?
Our study examined the impact of a checklist intervention on social workers’ forecasting abilities (how well they could predict the likelihood of different events and outcomes) and confirmation bias (the extent to which respondents sought information to confirm, rather challenge, their current view). You can see a copy of the checklist used in this study here.
How did we study it?
We enrolled 87 social workers (and one student social worker) from England in a randomised controlled trial (RCT), which involved completing an online survey. Each respondent read two initial case studies and answered four questions for each one about the likelihood of different events and outcomes. For example, after reading a case study involving domestic violence, to estimate the likelihood of another police call-out within 6 weeks, or that there would be a child protection conference. Respondents were then randomly allocated to a control group or intervention group. Respondents in the control group were asked to complete two additional case studies before accessing the checklist intervention, and completing a task designed to measure confirmation bias. Respondents in the intervention group were asked to complete the checklist intervention first, before completing the two additional case studies, and the task to measure confirmation bias.
What did we find?
There was a post-intervention improvement in forecasting accuracy for respondents in the intervention group. However, the difference between the intervention and control groups was small and not statistically significant. Thus, the checklist intervention did not have a significant impact on forecasting accuracy.
The checklist intervention also appeared to make no meaningful difference in relation to confirmation bias with similar scores between the two groups. None of the personal or professional characteristics that we measured were associated with forecasting accuracy or confirmation bias (including things like gender, age, and length of experience as a qualified social worker).
What are the implications?
Checklist interventions are typically designed to be used within real-life decision-making settings, for example in health care settings. Because of the Covid-19 pandemic, we had to design a checklist that could be used online, which probably made it less effective and limits our ability to generalise from these findings. We do not know what might happen if social workers were asked to use checklists in real-life, or whether the checklist might be improved by including different items within it.
When considered alongside results of a previous study, these findings suggest that interventions to improve forecasting accuracy (or mitigate confirmation bias) in social work probably need to be more in-depth than the relatively brief interventions we have tested here and previously.