Google is rolling out its Gemini AI chatbot to kids under 13. It’s a risky move

Google has announced it will roll out its Gemini artificial intelligence (AI) chatbot to children under the age of 13.

While the launch starts within the next week in the United States and Canada, it will launch in Australia later this year. The chatbot will only be available to people via Google’s Family Link accounts.

But this development comes with major risks. It also highlights how, even if children are banned from social media, parents will still have to play a game of whack-a-mole with new technologies as they try to keep their children safe.

A good way to address this would be to urgently implement a digital duty of care for big tech companies such as Google.

How will the Gemini AI chatbot work?

Google’s Family Link accounts allow parents to control access to content and apps, such as YouTube.

To create a child’s account, parents provide personal details, including the child’s name and date of birth. This may raise privacy concerns for parents concerned about data breaches, but Google says children’s data when using the system will not be used to train the AI system.

Chatbot access will be “on” by default, so parents need to actively turn the feature off to restrict access. Young children will be able to prompt the chatbot for text responses, or to create images, which are generated by the system.

Google acknowledges the system may “make mistakes”. So assessment of the quality and trustworthiness of content is needed. Chatbots can make up information (known as “hallucinating”), so if children use the chatbot for homework help, they need to check facts with reliable sources.

What kinds of information will the system provide?

Google and other search engines retrieve original materials for people to review. A student can read news articles, magazines and other sources when writing up an assignment.

Generative AI tools are not the same as search engines. AI tools look for patterns in source material and create new text responses (or images) based on the query – or “prompt” – a person provides. A child could ask the system to “draw a cat” and the system will scan for patterns in the data of what a cat looks like (such as whiskers, pointy ears, and a long tail) and generate an image that includes those cat-like details.

Understanding the differences between materials retrieved in a Google search and content generated by an AI tool will be challenging for young children. Studies show even adults can be deceived by AI tools. And even highly skilled professionals – such as lawyers – have reportedly been fooled into using fake content generated by ChatGPT and other chatbots.

Will the content generated be age-appropriate?

Google says the system will include “built-in safeguards designed to prevent the generation of inappropriate or unsafe content”.

However, these safeguards could create new problems. For example, if particular words (such as “breasts”) are restricted to protect children from accessing inappropriate sexual content, this could mistakenly also exclude children from accessing age-appropriate content about bodily changes during puberty.

Many children are also very tech-savvy, often with well-developed skills for navigating apps and getting around system controls. Parents cannot rely exclusively on inbuilt safeguards. They need to review generated content and help their children understand how the system works, and assess whether content is accurate.

What risks do AI chatbots pose to children?

The eSafety Commission has issued an online safety advisory on the potential risk of AI chatbots, including those designed to simulate personal relationships, particularly for young children.

The eSafety advisory explains AI companions can “share harmful content, distort reality and give advice that is dangerous”. The advisory highlights the risks for young children, in particular, who “are still developing the critical thinking and life skills needed to understand how they can be misguided or manipulated by computer programs, and what to do about it”.

My research team has recently examined a range of AI chatbots, such as ChatGPT, Replika, and Tessa. We found these systems mirror people’s interactions based on the many unwritten rules that govern social behaviour – or, what are known as “feeling rules”. These rules are what lead us to say “thank you” when someone holds the door open for us, or “I’m sorry!” when you bump into someone on the street.

By mimicking these and other social niceties, these systems are designed to gain our trust.

These human-like interactions will be confusing, and potentially risky, for young children. They may believe content can be trusted, even when the chatbot is responding with fake information. And, they may believe they are engaging with a real person, rather than a machine.

How can we protect kids from harm when using AI chatbots?

This rollout is happening at a crucial time in Australia, as children under 16 will be banned from holding social media accounts in December this year.

While some parents may believe this will keep their children safe from harm, generative AI chatbots show the risks of online engagement extend far beyond social media. Children – and parents – must be educated in how all types of digital tools can be used appropriately and safely.

As Gemini’s AI chatbot is not a social media tool, it will fall outside Australia’s ban.

This leaves Australian parents playing a game of whack-a-mole with new technologies as they try to keep their children safe. Parents must keep up with new tool developments and understand the potential risks their children face. They must also understand the limitations of the social media ban in protecting children from harm.

This highlights the urgent need to revisit Australia’s proposed digital duty of care legislation. While the European Union and United Kingdom launched digital duty of care legislation in 2023, Australia’s has been on hold since November 2024. This legislation would hold technology companies to account by legislating that they deal with harmful content, at source, to protect everyone.