Meta announced changes to its Instagram parental supervision tools that will notify parents if their teenage children discuss suicide or self-harm with Meta AI. The company said the alerts will be accompanied by expert resources designed to help parents navigate sensitive conversations with their children.
According to Meta, the system relies on a dedicated AI model tasked with identifying chats in which teens make explicit references to hurting themselves. The company emphasized that every conversation flagged by the AI will undergo a manual review process prior to any alert being issued to a supervising parent.
Meta said it consulted with parents and mental health experts to set parameters for which AI-detected exchanges should generate a notification. The company stated it will adopt a cautious posture when a young person's intent is unclear, acknowledging that this conservative approach could lead to some alerts when there is ultimately no real concern.
How the feature is being rolled out
The alert capability is currently available to parents who use Instagram parental supervision in the United States, United Kingdom, Australia and Canada. Meta plans to make the feature available to supervising parents worldwide by the end of the year.
In addition to notifications and resources, Meta is developing functionality that would contact emergency services if a Meta AI conversation suggests someone may be at imminent risk of suicide. The company reported it made more than 19,000 referrals to emergency services last year to request wellness checks for people it identified as potentially at risk.
Clinical input and safety controls
Meta said it worked with over 75 clinicians specializing in teen mental health to refine how Meta AI handles prompts related to suicide and self-harm. These clinicians reviewed AI responses to hundreds of prompts and provided feedback on what was appropriate and where improvements were needed.
The company also introduced a stricter Limited Content setting for Meta AI chats. When parents opt their teens into this mode on Instagram, Meta AI will refuse to respond to a wider range of prompts, according to the company.
Meta described the combined approach of automated detection, human review, clinician input and parental controls as its strategy to reduce risk and provide supportive resources, while acknowledging the potential for cautious thresholds to produce alerts in ambiguous situations.