Social workers’ AI tools risk errors in care records

New research suggests that use of artificial intelligence transcription tools for social work risks false details and ‘gibberish’ being entered into people’s care records. In some cases, erroneous warnings of suicidal ideation have been reported in official records.

Research conducted by the Ada Lovelace Institute has found that AI transcription tools are currently in use by 85 local authorities for social care, where they are used to record the interactions with, and decisions about, people in care. 98% of all GPs also have access to AI transcription through the Accurx digital healthcare platform.

The study found that there were benefits to using artificial intelligence to help with administrative tasks. However, they introduce risks of hallucinations and bias, which will be used to make decisions about care and are entered into that person’s official records.

The report found that: “Unpredictability in the outputs of AI transcription tools can lead to harmful inaccuracies being incorporated into official documents. Research has shown that AI transcription tools can generate text that promotes harmful perceptions and stereotypes of people in a recorded conversation, which often have no relation to the original recording.”

The study also found worrying examples of AI ‘hallucinations’ being entered into people’s records. One social worked spoke of an occasion where “the tool had incorrectly ‘indicated that there was suicidal ideation’, but ‘at no point did the client actually, you know, talk about suicidal ideation or planning, or anything’.” Without oversight, this error could have had a significant impact on this person’s care.

Other social workers reported inaccuracies in the AI transcriptions, such as “foul language or other irrelevant words and phrases.” Another interviewee detailed how their transcriptions often included ‘gibberish’. The AI tool might refer to “fishfingers or flies or trees” when in fact a child was talking about their parents fighting.

Andrew Reece, of the British Association of Social Workers said about the report: “The risk here is that people aren’t checking what’s been written for them… The time you spend writing helps you make sense of what you have heard. If the computer is doing that for you, you’re missing out on the important parts of reflective practice.”

The study notes that “government is making a bet on AI’s potential to transform the public sector.” In a speech on 13th January this year, Prime Minister Keir Starmer lauded AI’s ability to “halve the time social workers spend on paperwork” and set out a commitment to unlock the “vast potential” of AI across the UK.

The report concludes that “the use of AI technologies to create efficiency savings in the delivery of public services requires continuous monitoring and evaluation, alongside context-specific guidelines and safeguards, to protect the workers that use these tools and the people who draw on public services.”