key: cord-0523611-m33nrihc
authors: Lemaignan, S'everin; Newbutt, Nigel; Rice, Louis; Daly, Joe; Charisi, Vicky
title: UNICEF Guidance on AI for Children: Application to the Design of a Social Robot For and With Autistic Children
date: 2021-08-27
journal: nan
DOI: nan
sha: a96f6601873827706b00c4e3d327bcc27b28c582
doc_id: 523611
cord_uid: m33nrihc

For a period of three weeks in June 2021, we embedded a social robot (Softbank Pepper) in a Special Educational Needs (SEN) school, with a focus on supporting the well-being of autistic children. Our methodology to design and embed the robot among this vulnerable population follows a comprehensive participatory approach. We used the research project as a test-bed to demonstrate in a complex real-world environment the importance and suitability of the nine UNICEF guidelines on AI for Children. The UNICEF guidelines on AI for Children closely align with several of the UN goals for sustainable development, and, as such, we report here our contribution to these goals.

lower self-concept and these can impact education and other factors [13] .

There have been a range of studies that have implemented technology to assist autistic children in this regard [37] [40] [41] . However, these studies have often focused on deficit-based models and do not often include users in their design. Other strategies have also been used to support the well-being of autistic pupils. For example, recent work has highlighted that focusing on autistic intense or "special" interests in the classroom may be linked to improved wellbeing [42] . While the well-being of autistic groups is not well understood or defined, increasing awareness of the impact that well-being has on students' academic performance and their adult outcomes is well acknowledged [2] [14] . With greater attention applied to the well-being of autistic pupils, improved outcomes in areas of education and adult outcomes are often observed. Therefore, considering well-being from an early age and in a setting where this can be supported (classrooms) is worthy of investigation. Moreover, doing so in collaboration with autistic children and their teachers is paramount, if we are to be guided by their goals, ambitions and desires (for example, linking with 'special' interests). Technology provides one example of a way to help support the well-being of autistic groups in meaningful and impactful ways.

Within the field of autism and technology there are calls to better include and support autistic 'voice'. For example, Cascio and colleagues suggest that the "growth in autism research necessitates corresponding attention to autism research ethics, including ethical and meaningful inclusion of diverse participants" [7] . This suggests that ethics in this field is moving beyond typical ethical review processes (i.e. IRB approval / Ethical Review) and towards more meaningful / active engagement of diverse participants in our work. Coupled with this, the approach to methodologies (broadly speaking) has begun to move away from researching about autistic people, and started to move towards an approach that includes autistic people in the design, development and evaluation of research about them [18] . However, when considering the nature of autistic children, there can often be further ethical considerations [33] that include, locating ways for non-verbal individuals to contribute [26] , ensuring a gender balance, enabling participants to communicate in a range of ways (i.e. Picture Exchange Communication System; PECS), and prioritising their views and ways of communicating. Furthermore, there are calls for researchers to involve autistic voice and input even before research starts; building research that could be meaningful for them [30] [43] [28] . Within this project, we designed research that included a range of stakeholders, initially autistic pupils and some teachers, through to parents and an autistic adult (towards the end of the work). We also positioned the research with autistic voice central; they would be ones directing what the technology (Pepper the Robot) would do, where and why. In doing so we placed their lived experiences central, and their ideas for a robot in their school. This enabled and allowed their voices and input to direct what we did, which is what we next discuss.

C. The UNICEF guidance on AI for children This work takes place in the context of the release by UNICEF of a landmark report on artificial intelligence and children, entitled Policy guidance on AI for children [15] . This reports lays out 9 guidelines to design and build AI systems aimed at children. These are:

1) Support children's development and well-being; 2) Ensure inclusion of and for children; 3) Prioritise fairness and non-discrimination for children; 4) Protect children's data and privacy; 5) Ensure safety for children; 6) Provide transparency, explainability, and accountability for children; 7) Empower governments and businesses with knowledge of AI and children's rights; 8) Prepare children for present and future developments in AI; 9) Create an enabling environment.

These guidelines are based on the United Nations Convention on the Rights of the Child (UNCRC) which came into force by the UN General Assembly in the UK in 1989 and they and includes 54 articles that cover civil, political, economic, social and cultural rights that all children everywhere are entitled to [39] . Four of these articles play a fundamental role in realising all the rights in the Convention for all children and are known as 'General Principles' i.e. (i) non-discrimination (article 2); (ii) Best interest of the child (article 3); (iii) Right to life survival and development (article 6); and Right to be heard (article 12). Realising the need for the consideration of the UNCRC in the context of digital environment, the UN committee adopted Comment 25 [38] , which states that the rights of children will now apply online as they do offline. At the same time the United Nations International Communication Union (ITU) has launched a set of guidelines on child's online protection with recommendations for all stakeholders on how to contribute to the development of a safe and empowering online environment for children and young people [21] .

In a similar way, the recent developments in AI expanded the possibilities and the opportunities for children's wellbeing but they also resulted in further concerns regarding emerging risks in relation to children's rights. In this context, UNICEF published a report with the above-mentioned Policy Guidelines, which have several complementary focii: the children as individuals (e.g. (1), (8) ); the children at group/societal level (e.g. (2), (3)); technical underpinnings and regulations ((4), (5)); or the children's environment ((7), (9) ). Together, they build a comprehensive framework that impact many aspects of the AI system design. Importantly, they implicitly require the designers to 'step back' from the system itself, and adopt a much wider perspective, taking into account the end-users needs and circumstances, as well as the broad context and environment of use. While these guidelines are meant to facilitate research, development and policy regarding AI and Child's Rights, they only partially consider emerging issues that are related to robotics and the opportunities and risks that emerge in relation to the embodied nature of robotic artefacts for children. In addition, a children's rights-based approach rejects a traditional welfare approach to children's needs and vulnerabilities and instead recognises children as human beings with dignity, agency and a distinct set of rights and entitlements, rather than as passive objects of care and charity. When developing AI and AIbased robotic artefacts for autistic children, these principles are paramount. In the following sections, we discuss existing work on the intersection of ethical considerations for social robots and autistic children and we indicate the need for further systematic research towards this direction.

Research in the field of child-robot interaction has already indicated that the use of robots bring unique opportunities for autistic children and has a powerful impact on their behaviour and development mainly because of the embodied nature [4] , [10] , [22] ; however, an increasing body of research has started exploring the possible emerging risks and ethical considerations that should be addressed even from the design process of robotic interventions for autistic children [11] . One of the robot-specific characteristics that contributes to those concerns is their dynamic (not static) nature which allow robots' navigation into the human physical environment and the physical interaction with humans in a way that might have an impact on children's basic fundamental rights such as safety, privacy and autonomy. Systematic analyses of ethical considerations in research of robots for autistic children is still scattered and it usually appears as a reflection on existing practices such as in [11] and [27] . In principle, the discussions on the ethical considerations in research of robot for autistic children has mainly the form of anticipatory ethics, meaning the anticipation of how future technologies will be applied and what their consequences might be [5] , [23] . The focus of these discussions is mainly on harm prevention, data ethics issues, the robot autonomy and the transparency of the algorithms [11] , [24] . Especially for the design of robot-supported therapeutic interventions, there is still uncertainty regarding the degree of human supervision and the long-term effects of applications that understand the stages of the human condition and their integration into therapy [17] .

To address the emerging design challenges in robotassisted diagnosis and interventions, one of the methodological approaches that has been widely used is the participation of parents and autism clinicians and educators [1] , [20] , [36] . While in general stakeholders agree about the positive effects of robots in therapy for autistic children, they indicate that an approach of robot supervised autonomy would create more trust and would improve the quality of the therapy.

Although this body of research contributes in a substantial way to our understanding regarding the integration of trustworthy robots for autistic children, it seems that only a few studies include autistic children in the process of the design of robot-supported interventions for them. Considering the participation of autistic children in the design process is a challenging process which requires a delicate interplay between the pre-established broader ethical principles and guidelines with micro-ethics, especially for designs for marginalized children [34] . Spiel et al. highlight the importance of immediate interaction between researchers and marginalized children in participatory projects with insitu micro-ethical judgements (ibid.).

We build upon this line and we consider the policy guidelines on AI and Child's Rights as published by UNICEF as well as our reflections on our participatory design study with autistic children to re-examine the guidelines in the specific context of the design of a social robot with and for children. To our knowledge, this is the first systematic endeavour of the consideration of these guidelines in the context of social robotics and autistic children.

Building on interdisciplinary expertise (with the project's three Principal Investigators coming from robotics, education and architecture) and an existing collaboration with the UKbased SEN school The Mendip School (which hosts 144 autistic children), we co-designed a social robot to support the children (based on their ideas), and deployed it for a total duration of three weeks in the school. Indeed, our methodology has two main phases: the iterative and participatory co-design of the robot behaviours and the context for its use; a three-week in-situ deployment of the robot in the school; and the observations of its use and impact on the complex and dynamic context of the school.

The research methods employed during the first phase comprise of: (1) a half-day workshop at the school, with two focus groups with the target population, as well as unstructured interactions with other pupils (Fig. 1a) ; (2) a one-day workshop among the research team, with an external invited academic, expert in child-robot interactions and responsible AI. This workshop was focused on mindmapping the outcomes of the children focus groups; (3) a day with the robot embedded in the staff room to familiarise teaching staff with the robot's capabilities; (4) a one hour focus group with the school's teachers, where the results of the children's focus group were discussed and built upon and further insights from the teachers about the integration of Pepper into the school ecosystem; (5) a half-day workshop with an autistic academic, to better account for the autistic community perspective on this line of research and accordingly refine the project framing and interaction design.

Taken together, we found that the children felt Pepper could support their emotional and general well-being. The (a) One of the children focus groups: during one hour semistructured workshops, children could interact with the robot, and express their ideas about the robot's role.

(b) Physical location of the robot during the study, in a school corridor. In the foreground, one researcher passively observing the interaction. The research team also connected the children's input with the teachers and their approaches for teaching, learning and behaviour management in their classes/school. It soon became clear through the teacher's focus group that adopting the school's existing materials (emotional regulation sheets) would be an appropriate way to ensure pupils could interface with Pepper in a manner they would be familiar with (i.e. selecting an input that first asks how the pupil is feeling: happy, sad, angry, frustrated, tired, etc...).

One of the key outcomes of this co-design phase is the focus and framing of our research on the well-being of autistic pupils in their school rather than robot-supported cognitive development support. Indeed, contrary to previous research on robots for autism that focus on using robots to teach social skills to the children (eg [6] , [16] , [19] , [32] , [35] ), our focus groups have evidenced that more than social skills, there is a demand for social support and well-being. As a result, the interactions with the robot are unstructured (e.g. not scheduled), child-led and can be indifferently oneto-one or group-based. Also, unlike most of the previous research in this field, the robot is located in the school corridors, interacting autonomously with the children in a mostly unstructured way, rather pre-arranged in-classroom interactions. This gives more freedom to the children to choose when and how they want to interact.

The second phase took place in the school where the robot was placed in a communal area (see Fig. 1b ), and could be interacted with by the children (group of about 50 students, aged 13-16) at any time during the school day (8.45am-3pm). Specifically, no dedicated interaction time was scheduled for the pupils, as the researchers wanted to see whether and how robot usage patterns would emerge from within the school eco-system. As such, the robot has been used and observed in a naturalistic context. We argue that this indeed helps to form a better understanding of the actual impact of the robot on a school dynamics compared to structured face-to-face child-robot interactions.

The methods used during this phase comprise of observations made by the researcher while observing the robot from a distance, children's self-reported mood before and after interacting with the robot, quantitative measurements of the interactions (robot's logs), post-hoc questionnaires administered to the children, and post-hoc group interviews with the teachers (note however that the analysis of these observations is outside of the scope of this article).

Building upon the initiative of HRI2021 workshop on Child-Robot Interaction & Child's Fundamental Rights [8] , our project with The Mendip School attempts to explicitly address through its methodology the nine principles outlined in the UNICEF policy guidance.

We present hereafter how each of the nine guidelines are accounted for in our project and methodology.

The principle Support children's development and wellbeing is directly reflected in the aim of our work: we installed a social robot in a SEN school for 3 weeks, with the primary role of the robot being to provide social support and comfort to the pupils who requested it. The pupils' wellbeing and social needs were identified from the in-school focus groups with the children themselves and their teachers; the robot's eight basic behaviours (including story telling, dancing, playing calm music, cuddling, telling jokes) were identified during the focus groups as well, and available to the pupils during the study.

The inclusion of and for children is a fundamental aspect of the project: the robot is explicitly aimed at autistic children, a group that might otherwise be excluded of many AI-related technologies.

The children co-design the robot's behaviours with the researcher. As such, they are included from the onset in the creation of the AI system. The system is developed primarily for the children, and its behaviour and performance is only measured against the children's needs.

The study takes place in a very diverse environment, with a broad range of mental and physical abilities. In particular, some children are non-verbal, which could prevent them to interact with the system. To address this issue (as well as the technical limitations of speech technology in a school environment [25] ), we have designed an hybrid interaction mechanism, where the robot speaks to the children, and the children answer by pressing large, icon-based, buttons.

The location of the robot, in a communal space (school corridor), enables interaction with everyone, in a nondiscriminatory way. Interactions are child-led: because the children start themselves the interaction, the robot can not discriminate against sub-groups (due to eg detection issue of children in wheel chairs, etc.)

Finally, the robot's behaviours are co-designed between the children, the teachers and the researchers, taking the endusers voice into account at every level, and ensuring fairness and non-discrimination in the resulting robot capabilities.

Working with SEN children is ethically sensitive, and protecting the children privacy is essential; the robot is equipped with cameras and microphone, that could potentially be misused. While we sought the consent of both the parents and the children to take part in the study (and the children could choose not to interact with the robot), we additionally decided for the robot not to store any personal information (no videos, no audio). Only anonymous data is stored (like the position of children around the robot, but not their identities). Furthermore the robot is not connected to the Internet (preventing any possibility of malicious remote access).

The robot is a physical device, which could in principle cause harm. The robot is designed as a social agent, which might trigger undesired affective bonding from some children (especially vulnerable children).

The choice of the robot (Softbank Pepper) is based on its general, risk-assessed, safety to interact with human and children in particular. The height of the robot (1.2m) is similar to the children. The robot's arms are lightweight and compliant, meaning they can not cause serious injury, even in case of malfunction.

The psychological safety of the robot is investigated in the study; the on-site researcher and school staff were trained to recognise children's distress if/when it happened. As such we were able to respond immediately should anything happen. During our initial testing of Pepper in the school, and in consultation with the school, we devised ways to minimise any adverse effects of Pepper. For example, we de-briefed with teachers and school leaders every day with a view to reflect on, and if necessary, adjust our practices in the school. This included the teachers reporting anything adverse in their classrooms related to Pepper's presence.

One final consideration was that of an 'exit' strategy for Pepper. Autistic children can become tied to routine quickly [3] and removing Pepper, having become used to seeing Pepper, could have proven to be a problem for some of the children. In this regard, we ensured that Pepper said 'good-bye' at a school assembly and that pupils had a chance to see Pepper before the study ended. However, this issue does need careful and considered thought in work of this nature.

F. Principle 6 -Provide transparency, explainability, and accountability for children

The robot can be perceived as a 'magical being' by the children, hiding its decision making process behind interaction tricks.

What 'transparency' or 'explainability' means from the perspective of a child is however not entirely clear yet.

To shed some light on that question, we have exhaustively recorded every question asked by the children about the robot during the 3-week study, collecting 60+ questions. During the study the on-site researcher answered children's questions about the robot to the best of their ability in terms appropriate to the children's level of understanding. These answers and subsequent conversations were also recorded.

While still being analysed, this data should enable a better understanding of what an explainable and transparent behaviour means from the unique perspective of the autistic children.

Contributing to the robot's behavioural transparency, however, is the fact that the interaction is mostly child-led: the robot's behaviours are triggered and chosen by the children, who remain 'in control' of the interaction, ensuring a level of explainability and accountability.

Deploying an autonomous robot in a school for a meaningful period of time will raised awareness about the potential benefits of AI in a sensitive societal environment. It will also start the discussion with all stakeholders (from the children and their parents, to the Department for Education of the government) around the interplay between AI and children.

As a part of the project the university (UWE) produced a press release across several online platforms, including the university's website and TikTok. By sharing news about the project the public, and by extension government and businesses, were informed of the potential benefits of AI in this environment. We reviewed outgoing publications relating to the project to ensure that it was a fair and reasonable representation of the project's aims (to support the well-being of the children), the robot's role in the school, and its impact on the pupils.

The children are given the opportunity to freely interact with a robot for a meaningful period of time, experimenting with how AI and technology might impact their lives, and better understanding what they think would be beneficial (or not) to them.

The project will help prepare children for AI through exposure to these technologies during their educational experience in a school setting. As digital natives, this exposure can assist in familiarising children with technologies as they develop, and respond to the needs of the children. The use of codesign should further empower children to engage with the development of these technologies.

By embedding a robot in a rich, child-centered environment like a school, with a range of stakeholders able to interact with the robot (children, teacher, other school staff, parents), this study contributes to creating an enabling environment, where AI is not only an abstract concept, but a physical, situated system that each stakeholder can question and challenge. The behaviours of the social robot are designed to help empower and enable the children with their educational and emotional needs.

Our experiences, while positive, did reveal two key cases that need to be discussed. The first was a pupil who did not want to interact with Pepper at all. In fact they entered the school via a separate door to make sure they did not even see Pepper. This was managed by the school and the pupil's teacher. There was also a case of a pupil who was in a classroom adjacent to where Pepper was based in the corridor. This pupil did not like Pepper either, repeatedly stating "That stupid robot, I don't like it, when it is going?". So in these two instances managing the situation, and being carefully aware of it was paramount. In the case of the first child we found out that they are fearful of any digital technologies that do not work correctly (i.e. not specific to robots). In the case of the second pupil, we worked very closely with their teacher. We de-briefed with the teacher each afternoon (after school) to ensure that the pupil was not distressed or having their education impacted in anyway. While we offered in several occasions to interrupt the study, the school insisted for it to continue, and interestingly, we were informed by this pupil's teacher, that he was in class more (he would previously spend lots of time in the corridor) and engaged in learning in much more meaningful ways. So issues like this need (1) recognising (that not all pupils will like a robot like Pepper) and (2) managing carefully if/when they arise.

In another case we also found that some pupils, and one in particular, seemed to become emotionally attached to Pepper. She enjoyed spending time with Pepper listening to stories, calming music and also touching Pepper (stroking the head and holding hands, for example). As a result of removing Pepper at the end of the study, we had a follow up email from her parent (father), the day after the study ended in the school, who commented: "On Friday evening she [the pupil] came to me and said that she was sad. When I asked why she said that Pepper had left today. She went on to say that she will miss Pepper and hopes to meet again one day. This is a very strong reaction for [her] . through the project she has gone out of her way to tell us when she interacted with Pepper. Seeing Pepper quickly became the highlight of her days". While this project attempted to minimise potential distress from the robot's removal, this interesting response also raises issues for researchers who need to very carefully manage the emotional attachment some may have with a social robot over a long period of time.

The UNICEF guidelines align with several of the UN goals for sustainable development. In particular, UN goals 3 (Good Health and Well-being) and 4 (Quality Education) align with the UNICEF Principle 1 Support children's development and well-being outlined above, and UN goal 10 (Reduced Inequalities) with UNICEF Principle 3 Prioritise fairness and non-discrimination for children. With the overall agendas of both organisations closely aligned, our work informs and contributes to both sets of goals.

While relatively broad in its wording, the UNICEF guidance sets a high bar for the design of ethical and responsible AI systems, by forcing the designers to account for the wider context of use of the technology. This policy guidance did not have robotics as a specific focus, and consequently, one might question whether such guidance would be appropriate and applicable to embodied AI systems like robots.

While the scope of our study is modest (in total, 13 days of actual robot deployment in the school), it is also a challenging one: working with vulnerable populations (autistic children, some of them with additional cognitive, emotional and/or physical disabilities) in their naturalistic environment (the school) and over a period of time long enough to go past the initial excitement (novelty effect) and explore longer-term adoption, is far from routine.

In that context, we found that the UNICEF guidance aligned closely with our user-centered approach and provided a strong ethical framing for the work. We suggest that future work in the field of social robotics, and especially with children, carefully consider the UNICEF guidance in their ethical and practical approach of designing child-robot interfaces. In fact we suggest that the field should be led by users (in this context children) and that child-led-robotinteractions, designed with and for them, represents a major step in co-designing the future in this field.

To address the many implementation complexities, UNICEF invited governments and business sector to pilot the policy guidelines in various fields. Two of the pilot case-studies are closely related to our research: (i) the adoption of the guidelines for the Haru social-robot for typically developing children and (ii) the AutismVR Imisi 3D application that teaches people how to interact with autistic children. While results of the first case-study has contributed to the development of a framework on social robots and child's fundamental rights [9] , this was developed with the participation of neurotypical children and further long-term and systematic research is needed for its examination with autistic children.

Educators' views on using humanoid robots with autistic learners in special education settings in england

Students' perceptions of school climate as determinants of wellbeing, resilience and identity

Stress and emotional wellbeing of parents due to change in routine for children with autism spectrum disorder (asd) at home during covid-19 pandemic in saudi arabia

Are robots ready to deliver autism interventions? a comprehensive review

Anticipatory ethics for emerging technologies

Robotenhanced therapy: Development and validation of supervised autonomous robotic system for autism spectrum disorders therapy

Making autism research inclusive by attending to intersectionality: a review of the research ethics literature

Designing and developing better robots for children: A fundamental human rights perspective

Social robots and children's fundamental rights: A dynamic four-component framework for research, development, and policy

Long-term personalization of an in-home socially assistive robot for children with autism spectrum disorders

A survey of expectations about the role of robots in robot-assisted therapy for children with asd: ethical acceptability, trust, sociability, appearance, and attachment

Improving emotion regulation ability in autism: The emotional awareness and skills enhancement (ease) program

School experiences of students with autism spectrum disorder within the context of student wellbeing: A review and analysis of the literature

Picture my well-being: Listening to the voices of students with autism spectrum disorder

Policy guidance on ai for children (draft)

How to build a supervised autonomous system for robot-enhanced therapy for children with autism spectrum disorder. Paladyn

Your robot therapist will see you now: ethical implications of embodied artificial intelligence in psychiatry, psychology, and psychotherapy

Making the future together: Shaping autism research through meaningful participation

Follow the white robot: Efficacy of robot-assistive training for children with autism spectrum disorder

How to implement robots in interventions for children with autism? a co-creation study involving people with autism, parents and professionals

United nations international telecommunication union child's online protection guidelines

Modeling engagement in long-term, in-home socially assistive robot interventions for children with autism spectrum disorders

Ethics and technology 'in the making': an essay on the challenge of nanoethics

An architecture for safe child-robot interactions in autism interventions

Child speech recognition in human-robot interaction: Evaluations and recommendations

Including speaking and nonspeaking autistic voice in research

Robot enhanced therapy for autistic children: An ethical analysis

Assisting people with autism spectrum disorder through technology. Encyclopedia of Education and Information Technologies

The issue of prevalence of autism/asd

whose agenda? who knows best? whose voice?'co-creating a technology research roadmap with autism stakeholders

Prevalence of parent-reported asd and adhd in the uk: findings from the millennium cohort study

Robots for use in autism research

A process-model for minimizing adverse effects when using head mounted display-based virtual reality for individuals with autism

Micro-ethics for participatory design with marginalised children

NAO-dance therapy for children with ASD

ACM/IEEE International Conference on Human-Robot Interaction

Parents' views on using social robots for language learning

Emotional selfregulation of individuals with autism spectrum disorders: Smartwatches for monitoring and interaction

United nations general comment no. 25 (2021) on children's rights in relation to the digital environment

United nations general assembly, convention on the rights of the child

Improved socio-emotional and behavioral functioning in students with autism following school-based smartglasses intervention: Multi-stage feasibility and controlled efficacy study

Perseverations of the academy: A survey of wearable technologies applied to autism intervention

Autism, intense interests and support in school: from wasted efforts to shared understandings

A framework of evidence-based practice for digital support, codeveloped with and for the autism community