Project Overview

A social robotic system using a NAO robot for emotion education in children with autism spectrum disorder (ASD). The system integrates a NAO robot with a dual-interface architecture: one interface displays emotional stimuli for the child, while the other is a Graphical User Interface (GUI) for a human facilitator. This design balances the robot's autonomous operation with essential human oversight.

The therapeutic approach is based on a five-activity framework that incrementally increases the complexity of interactions. A session target four core emotions—happy, sad, surprised, and angry—through multimodal exchanges that include verbal, facial, bodily, and contextual emotional cues.

The system leverages several advanced technologies to facilitate these activities. ChatGPT/Whisper is used for adaptive conversation, while DeepFace is employed for facial emotion analysis. MediaPipe is used for real-time body pose recognition by monitoring key anatomical landmarks on the child. A critical component is the bespoke GUI for the facilitator, which offers granular control to initiate, repeat, or omit any part of the session, enabling tailored pacing for children with ASD who often benefit from repeated exposure to concepts.

I've implemented start and stop recording buttons, and a no response button as shown in Figure 3. The transitions between games are now smoother and more natural. And the story interactions now include all the engaging elements that were used in the previous games like facial expressions, body movements, and talk with Nao.