Malaysia and Indonesia have become the first countries to block Grok, an artificial intelligence chatbot created by Elon Musk's company XAI (xAI), over concerns that the program is being misused to generate images featuring explicit and non-consensual sex.
There is increasing scrutiny of generative artificial tools that can produce realistic images, sound, and text, and growing concern that existing measures are failing to prevent their misuse.
The chatbot Grok, accessed through Musk's X platform, has been criticized for its ability to generate manipulative images, including depictions of women in bikinis or in sexually explicit poses, as well as images with children.
"The government considers sexual non-consensual deepfakes (images in which the identity of real people is manipulated) to be a serious violation of human rights, dignity and security of citizens in the digital space," said Indonesian Minister of Communications and Digital Affairs Meutja Hafid.
Control over Grok is also strengthening in the European Union, India, France and the United Kingdom, which announced today that it is taking measures to criminalize "nudification apps".
A British regulator has launched an investigation into whether Grok broke the law by allowing users to share sexualized images of children.
Last week, Grok restricted image generation and editing for paying users after a global backlash over sexualized deepfakes of people. Critics say that hasn't fully addressed the problem.
A request for comment from The Associated Press to xAI resulted in an automated response from the company's media support email address that read: "Media lies from a bygone era." The same message was received from another email address of the company, when asked for comment on the global response to deepfakes.
Regulators in two Southeast Asian countries have said that existing controls are not preventing the creation and dissemination of generative pornographic content, especially that involving women and minors.
The Indonesian government temporarily blocked access to Grok on Saturday, and Malaysia on Sunday.
Initial findings showed that Grok lacked effective safeguards to prevent users from creating and distributing pornographic content based on real photos of Indonesians, said Aleksandar Sabar, director general of digital space oversight. He explained that such practices risk violating privacy and rights in photography when those photos are manipulated or shared without consent.
The Malaysian Communications and Multimedia Commission has noted "repeated misuse" of the tool to generate indecent, sexually explicit and manipulated images without consent, including content involving women and minors.
The regulator announced that requests for stronger protection measures were issued to the companies Iks and IksAI this month.
"The restriction has been imposed as a preventive and proportionate measure while legal and regulatory processes are ongoing," they said, adding that access will remain blocked until effective safeguards are in place.
Launched in 2023, Grok is free to use on X. The image generator feature, Grok Imagine, was added last year and includes a so-called "spicy" mode that can generate adult content.
In recent days, almost all the important political and entertainment figures in Serbia have received fake "bikini" photos. Namely, it was enough for one of the users to "ask" Grok to turn a regular photo into an explicit one and the program would do it very quickly.
Bonus video:
