Meta announced Thursday that it will now notify parents if their teen talks about suicide or self-harm with the company’s Meta AI chatbot. Meta says it is also working on the ability to contact emergency services if someone’s conversations suggest they may be at risk of self-harm. These changes come as Meta and other tech
Meta announced Thursday that it will now notify parents if their teen talks about suicide or self-harm with the company’s Meta AI chatbot. Meta says it is also working on the ability to contact emergency services if someone’s conversations suggest they may be at risk of self-harm.
These changes come as Meta and other tech companies face scrutiny from regulators and parents over how AI chatbots respond to users in crisis, particularly teens, a liability issue that is increasingly shaping the way AI companies design and market their products.
Meta says it has created a dedicated AI system to identify conversations in which a teen makes a clear reference to harming themselves.
“We understand how distressing receiving these alerts can be for a parent,” Meta wrote in a blog post. “That’s why, as we continue to improve our detection, all chats flagged by our AI will be manually reviewed before sending an alert. If a teen’s intent is ambiguous, we’ll err on the side of caution and alert parents. While that means we can sometimes notify parents when there’s no real cause for concern, we believe this is the right starting point and will continue to monitor to help make sure we’re in the right place.”
These alerts are now available to parents using Instagram Parental Monitoring in the US, UK, Australia and Canada, and will roll out globally by the end of the year, Meta says.
This update builds on alerts Meta already sends parents when their teen repeatedly searches for suicide or self-harm terms on Instagram. It also relies on a feature that allows parents to see the topics their teens discussed with Meta AI over the past week.
Meta also announced that its “Limited Content” setting, which allows parents to place their teens on a more restrictive experience on Instagram, now applies to Meta AI as well. Meta AI is already trained to avoid sexual or romantic conversations or alcohol-related discussions with teens, and the Limited Content setting expands those safeguards by having the chatbot reject a broader range of prompts. Meta did not specify what those additional prompts include, but TechCrunch asked the company for more information.
Additionally, Meta says it will contact emergency services if someone’s conversation with Meta AI, whether an adult or teenager, suggests someone is at risk of suicide. It’s worth noting that Meta already takes this step when someone posts something on Facebook or Instagram that suggests they’re at risk, so it extends that same practice to conversations with its chatbot.
When you buy through links in our articles, we may earn a small commission. This does not affect our editorial independence.
Keep following us for the latest insights.
















