ChatGPT used to create violent and sexualised images

Researchers have used OpenAI’s artificial intelligence platform ChatGPT to create sexualised images and scenes of graphic violence. This adds the platform to a list of other AI tools, such as xAI’s Grok, which have been criticised for being able to create explicit images.

British artificial intelligence security firm Mindgard were able to get ChatGPT to create graphic images with a single prompt. The researchers slightly altered a widely-shared prompt that is used to create humorous results, from which the chatbot created gory and sexualised images of ‘its own volition’.

Even without being given detailed instructions on what to create, ChatGPT was able to create content that was “very gruesome, sometimes sexualised, sometimes both together” according to Mindgard’s founder Peter Garraghan. “This is a perfectly innocent-looking instruction to an AI, but the consequence is it generates very, very bad imagery and content,” he said.

Jim Nightingale, Mindgard's AI safety and security researcher, said he was left “shaken, and in tears” by the kinds of images the AI platform could be made to generate. While the images depicted people who were AI-generated, it has been shown that ChatGPT, like other AI tools, can be tricked into creating images using the faces of real people.

After being contacted by the BBC, OpenAI said that it had multiple layers of protection that should prevent users from creating content that breaches the company’s terms and conditions. However, it said that it had taken steps to stop ChatGPT from responding with these graphic images, saying: “After investigating this trend, we've introduced additional safeguards against this type of prompt”. Nevertheless, after adding further tweaks to the prompt, the researchers were still able to create disturbing content.

In his report, Nightingale said: “I'm struck that while what I saw was generated, an artificial image, it has ties to real images, and the real world,” raising concerns that the images produced reflect the data that was used to train the AI model.

Dr Rumman Chowdhury, Chief Executive of Humane Intelligence, an organisation working to make AI systems fairer and more responsible, told the BBC: “Models do not understand intent. They do not understand context. They do not understand propriety or right or wrong,” which means AI companies face a “mountainous” task in developing protections.