AI algorithms are starting to play a variety of roles in the digital ecosystems of children - being embedded in the connected toys, apps and services they interact with on a daily basis¹. Such AI systems provide children many advantages, such as personalised teaching and learning from intelligent tutoring systems², or online content monitoring and filtering algorithms that proactively identify potentially harmful content or contexts before they are experienced³. AI systems in games and entertainment services provide personalised content recommendations⁴, while social robots power the interactive characters in ways that make them engaging and human-like⁵. Going forward, AI systems will, in all likelihood, become altogether even more pervasive in children’s applications simply due to their sheer usefulness in creating compelling, adaptive, and personal user experiences⁶.

Yet, understanding the ways that AI-systems are being used in systems for children, and their harm and impact is still a new and emerging area of investigation. On one hand, there exists a growing body of literature (including research findings, codes of practice, and proposed legislation) characterising such AI harms across many kinds of applications (e.g. the White House’s Guidance for the Regulation of Artificial Intelligence⁷, and the EU’s proposed Artificial Intelligence Act⁸); and on the other, there is a complementary body of literature focusing on risks to children, which similarly includes primary research, proposed policies, and codes of practice, such as the UK ICO’s Age Appropriate Design Codes⁹ and UNICEF's AI policy for children¹⁰. These diverse efforts have created a confusing outlook for designers and practitioners to create concrete and safe designs for children.

Background

To establish the scope of our investigation we first aim to define what we mean by “AI for children''. Starting with “AI”, the OECD’s Recommendation of the Council of Artificial Intelligence defines AI systems as “machine-based systems that can, given a set of human defined objectives, make predictions, recommendations, or decisions that influence real or virtual environments”¹¹. This definition is cited by the proposed EU Artificial Intelligence Act (EUAIA)¹², as well as the 2020 UNICEF Policy Guidance for AI for Children¹³ and offers a convenient definition that remains independent of particular implementation or application. With respect to “AI system”, however there is some divergence among common use; some papers use the term “AI system” to mean the particular algorithm or subsystem that enables a particular AI-associated capability (e.g. learning, inference, or recommendation), while others consider a broader context, namely those components that enable a specific application-specific capability (e.g., voice recognition, face recognition, video content recommendation). In other contexts, an “AI system” refers to end-user systems that have such capabilities embedded within them (e.g. intelligent tutoring systems¹⁴). Here, we adopt the latter two, considering both the specific capabilities and entire end-user systems as AI systems.

Towards a Code for Age-Appropriate AI Design

The AADC of the UK ICO requires all online services to be used by children to appropriately safeguard children and support children’s rights¹⁵. This is grounded by UNCRC’s recognition that children’s rights in all aspects of their life should be guaranteed by appropriate legal protections¹⁶. Although the AADC codes provide strong guidance for ensuring children’s data to be used ‘appropriate to their age’, we argue that the application of age-appropriateness may require new thinking in the area of AI for children.

We believe that having a unified Age Appropriate AI framework, as a part of existing Age Appropriate Design efforts¹⁷, could help standards and regulatory bodies start to ensure that the complex, multi-dimensional and often difficult-to-anticipate long-term safety needs of children are met, as technology startups race to bring AI technical innovations to market¹⁸. Appropriately scoped, a regulatory codes and duties of care that are specific for designing AI for children could lower the barrier, and in some cases, incentivise innovators to address a set of core principles, including safety, fairness, inclusion, long-term impact/sustainability, and privacy.

What might we suggest about assembling a code for AgeAppropriate AI Design then ? We feel that such a code should be strongly connected to, and contextualised against both principles for safe and ethical AI, and those for child-centred and age-appropriate design:

Fairness, equality, inclusion and access is derived as a combination of the fairness & non-discrimination principles and that of universal inclusion, relating not only to discovering the needs of diverse groups but also ensuring that all children are treated fairly and equally. Participatory methods involving children from a variety of backgrounds and with different abilities have been applied in a variety of contexts (e.g.¹⁹), and should be an essential strategy in creating human-centred AI for children. Theoretical work in AI and fairness has focused on statistical approaches to ensure ‘black-box’ AI classifiers do not yield allocative harms that disproportionately impact particular groups²⁰. Work on fairness in a child-centred context, however, has thus far been scarce. Unfortunately, discovering and mitigating representative harms may be very difficult in practice due to their being highly contextual [67].

Transparency and accountability are often brought up together in the literature. Accountability requires identifying a chain of responsibility for system (mis)-behaviours, and we believe that this responsibility should include both algorithmic accountability that is derived from designs and social accountability that can be contributed to stakeholders, typically including the professionals or parents/guardians. Transparency is of great importance within this chain to make sure everything is easy to understand, and we believe transparency should not only be about the system itself, but also the accountability procedures, such as what to do if something went wrong.

Similarly, privacy and manipulation and exploitation are also closely related. To protect children’s privacy refers to two things - respecting their interpersonal privacy, examples including not showing detailed online activities of children to their parents²¹; and implementing systemic-level privacy, such as applying data minimisation and privacy-preserving techniques to prevent children’s data from being collected by third parties ²². In fact, misuse of data is one of the main mechanisms that manipulation and exploitation are built upon²³, and thus should be prevented appropriately to children’s age. That said, our findings also showed that the use of children’s personal information is sometimes essential for the functioning of many systems, medical diagnosis systems in particular, and a recent study with developers also suggested that while they want to respect children’s privacy, they have to make compromises due to limited monetisation options²⁴. Nevertheless, data collection should be strictly minimised for the purpose needed. On the other hand, it is often considered hard to determine and judge between ‘good’ nudging and ‘detrimental’ manipulation and exploitation²⁵, and although nudging practices can be used positively for children in applications such as engaging and motivating them in learning, transparency/accountability of such an approach or data minimisation principles are not always well-considered along.

The word ‘harm’ is centred to safety and safeguarding, that is to ensure systems would not do harm to children, as well as protect children from harm. On the other hand though, that has been a long debate on what indeed counts towards ‘harm to children’²⁶, and that ‘harmful content’ could be subjective and contextual, and thus individual to identify alone²⁷. Achieving personalised and contextualised assessment of system effectiveness would yield more informative outcomes, however, this would often require access to vulnerable children or specialised learning/medical environment, which can be a barrier non-trivial to overcome.

Finally, we posit that sustainability in a child-specific context should refer to supporting the long-term development of children, and in a way that considers the important aspect of meeting developmental needs of children. While such age-appropriateness consideration is vital not only for sustainability, but also across multiple aspects including safety, safeguarding, privacy, transparency; we argue that to anticipate/design for developmental needs may be difficult outside learning and education, where supporting theory and empirical evidence are less well-identified. We also found that the current evaluation criteria tends to be vague in terms of what counts as supporting children’s long-term development. Thus including children’s voices is increasingly recognised in both regulatory development²⁸ and empirical explorations²⁹.