The Peril of 'Realness': AI's New Front in Online Harassment
In an era increasingly shaped by digital innovation, a darker facet of artificial intelligence has emerged, redefining online harassment. Experts now identify a clear trend: the weaponization of AI to generate sexualized imagery and propaganda, primarily targeting Muslim women in India.
The disturbing reality was brought home for Ayoub, a 24-year-old freelance model from India-administered Kashmir. Last year, she discovered a video circulating on Instagram, a fabricated narrative of her life in New Delhi. The clip, complete with AI-generated voiceover, scrolling captions, and news-style headlines, stitched together photographs from her student days at Jamia Millia Islamia University. These images, depicting everyday campus moments from group projects to selfies, were manipulated to falsely claim she was a Muslim woman “selling her body” to Hindu men, even misidentifying her own brother as her “pimp.” Ayoub's chilling assessment: “It looked so real that if someone, even my parents, saw the video, they would think it was real.”
This isn't an isolated incident. Researchers at the Washington, DC-based Center for the Study of Organized Hate (CSOH) have documented a clear pattern. Their study, which analysed 1,326 publicly available AI-generated images and videos collected from 297 public accounts on X, Facebook, and Instagram between May 2023 and May 2025, revealed a grim statistic: sexualized depictions of Muslim women generated the highest engagement, accounting for over 6.7 million interactions across platforms. Zenith Khan, a co-author of the study and digital research analyst at CSOH, succinctly articulated the shift: “Generative AI has made the transformation of sexual fantasy into imagery possible at speed and at no cost,” adding that “Image generators and deepfakes allow individuals to convert hostile narratives into highly realistic visual material with minimal technical expertise.”
The implications of this technological weaponization are profound. The Mumbai-based RATI Foundation’s online safety helpline, Meri Trustline, confirms a growing volume of such cases. While media attention often gravitates towards celebrities, their 2024 report highlights that ordinary women are increasingly targeted. The fabricated images, despite their artificial origin, possess a capacity for real harm, inflicting shame and leading to retraumatization for victims who often decline to speak publicly.
This surge in AI-powered abuse unfolds against a paradoxical backdrop: India's active participation in global AI governance discussions. The nation hosted a high-level AI Impact Summit earlier this year in New Delhi, focusing on innovation and regulatory frameworks. The stark contrast between these forward-looking policy dialogues and the burgeoning domestic misuse of AI signals a critical challenge. It underscores the urgent need for regulatory frameworks that can keep pace with rapid technological advancements, ensuring that discussions on innovation are robustly paired with concrete measures against misuse and harm.
Ultimately, the weaponization of AI against vulnerable groups marks a significant escalation in digital warfare. It democratizes the tools of abuse, allowing hostile narratives to be rendered with chilling realism and disseminated with unprecedented speed and reach. The profound social and psychological damage inflicted by these fabricated realities demands immediate and comprehensive responses, moving beyond mere innovation rhetoric to actionable governance.