Sexual deepfake apps on Apple’s App Store, research finds

Updated on Aug. 20

Nearly 50 apps on Apple’s App Store were found to deliver sexually explicit face- and body-swapping once installed, according to research published by Copyleaks, an AI-detection company.

Apple, responding to a request for comment from Unbowed, said only three examples were provided by Copyleaks in its article and all had been removed on July 29, two weeks before Copyleaks released its report. It had no record of Copyleaks reporting any of the apps to Apple.

Apple said App Review Guidelines specifically prohibit overtly sexual or pornographic content, including ‘nudification’ apps and that the App Review team works round the clock to keep the App Store a safe and trusted place.

Copyleaks’ research describes listings that disclose no sexual capability. The advertising that brings users to those listings runs on TikTok. And a number of the apps block screenshots and screen recording, which makes it harder for researchers or journalists to document what the apps actually generate, it says.

Copyleaks says its researchers reviewed the apps over a six-month period.

TikTok did not immediately respond to a request for comment.

What the research describes

Every app reviewed promoted benign features in its App Store listing, Copyleaks said, and none disclosed that it could generate sexualised imagery. Users encountered the explicit features only after downloading. The company said several user reviews came from people who had downloaded an app on the strength of its description and found the sexual content afterwards, including one reviewer who identified themselves as under 18.

Almost all the apps operated on subscription or virtual-credit models, with sample templates viewable free and generation requiring payment. Some, Copyleaks said, may not work at all: several appeared to rely on pre-recorded or wholly synthetic material rather than processing uploaded photographs, and reviews reported apps that stopped functioning after payment.

The finding with the widest implications concerns discovery. Virtually all the apps were found through advertisements on TikTok, Copyleaks said, and the advertisements were sexual in nature. In several cases, the company said, they explicitly promoted non-consensual use.

“There is virtually no barrier to entry for these services anymore,” Copyleaks chief executive Alon Yamin said in the research.

Apple said that when its App Review team identified malicious and fraudulent apps, they were analyzed in order to strengthen detection models. It noted that customers can visit https://reportaproblem.apple.com to report offensive, illegal, or abusive content, as well as scams or fraudulent activity on the App Store.

The end of the chain

The Independent also reported that it had found apps on Apple’s App Store offering to “strip” uploaded photographs.

That app listed dozens of generated scenarios into which a user could place a photograph of a real person, the paper reported, including one labelled “bedroom rape”, each illustrated with graphic video.

The app was removed after the publication flagged it to Apple, it said. Apple declined to comment directly but pointed to guidelines prohibiting overtly sexual or pornographic content, and said a number of the apps flagged to it had added violating features after being reviewed — which, if correct, describes a review process that inspects an app once and not again.

Janaya Walker, interim director of the End Violence Against Women Coalition, told the Independent that such material allowed users to “role play violent and dehumanising acts against women” and had a “desensitising effect”, and that images created this way were “directly used to intimidate, threaten and harass women”. She called for legally binding safety-by-design requirements on platforms.

Regulation shortfall

The Online Safety Act reaches TikTok. By Ofcom’s account, it does not reach the App Store.

Asked by the Independent about generative AI pornography sites and apps, an Ofcom spokesperson said the regulator was “actively assessing” their compliance with the Act and would “not hesitate” to act on failings. The spokesperson then added: “We have no powers to regulate app stores at present, but have been tasked with assessing the role they play in children encountering harmful content. We will report our findings later this year.”