London Mayor Sadiq Khan highlighted the potential dangers of unregulated artificial intelligence (AI) after a deepfake audio clip, imitating his voice making inflammatory remarks, nearly incited serious disorder before Armistice Day.

The deepfake, which spread rapidly online, included falsified statements disparaging Remembrance weekend and prioritising pro-Palestinian marches planned for the same day, falsely attributing these views to Khan.

It said, “What’s important and paramount is the one-million-man Palestinian march takes place on Saturday.”

The artificial clip also appeared to show Khan asserting control over the Metropolitan Police saying, “I control the Met Police, they will do as the Mayor of London tells them… the British public need to get a grip.”

Khan criticised existing laws for their inadequacy in addressing AI-generated fakes, as the individual responsible for the clip faced no legal consequences.

The spread of this deepfake not only led to a spike in hateful comments against Mayor Khan online but also risked sparking significant unrest.

The Mayor told the BBC, “What was being said was a red rag to a bull for the far right and others.”

Counter protests took place on the same day, including people connected to far-right groups, which led to clashes with police, and arrests for offences including inciting racial hatred.

“We almost had serious disorder,” he said. “People should be able to criticise me. But I think what you shouldn’t do is use AI to manufacture lies. And I think the person that made this needs to realise the consequences on that Saturday.”

Khan highlights the responsibilities of social media platforms in managing the spread of such content, noting that immediate action from these companies could prevent potential harm.

This incident underscores the broader implications of deepfake technology and unregulated AI in influencing public opinion, inciting disorder, and undermining trust in public figures.

It calls attention to the urgent need for comprehensive legislation and robust mechanisms to identify, control, and mitigate the spreading of AI-generated fake content.