Pubblicato il

From Soloists to Symphony: The Evolution of Human-Robot Teaming

SPEME Lab is proud to share its latest research contribution, which pushes the boundaries of how humans and machines operate together.

In the newly published paper, “Exploring Multi-Agent Human-Robot Teaming Dynamics in a Musical Orchestral Context,” the team explores a critical paradigm shift: moving beyond simple Human-Robot Interaction (HRI) and Collaboration (HRC) toward true Human-Robot Teaming (HRT).

Why an Orchestra?

The authors identify the musical orchestra as the ultimate testbed for complex dynamics. In this setting, precision isn’t enough; the system requires “entitativity”—the state where individual agents (human and robotic) feel and act as a single, cohesive unit.Authored by Rashmi Chawla, Francesco Pupo, and Giancarlo Fortino, the paper proposes a conceptual review that defines HRT through the framework of the “4Cs”:

• Communication

• Coordination

• Cooperation

• Command/Control

By integrating multimodal inputs (visual, auditory, and tactile), this research lays the groundwork for multi-agent systems where robots don’t just work alongside us—they play the same symphony with us.

Abstract:

The traversing of Human Robot Interaction (HRI) to Human Robot Teaming (HRT) has seen huge transformation from human-robots working in collaboration to working as teammates. The progress in automation, network connectivity and HRI represent a transformative shift to embrace Industry 5.0 by integrating advanced technologies like AI, robotics and IoT to create a symbiotic ecosystem where human ingenuity and machine intelligence converge. This paper outlines a conceptual review, to provide a well-defined understanding of Multi-Agent HRT that paves a new path from simple HRI to more evolved HRC and emergence of HRT in the subsets of well-defined 4C’s combining multi modal communication, multitudinous sensory modalities viz. visual, auditory, and tactile inputs as a part of HRI systems. This study also explores the onset of MA-HRT framework for a musical orchestra system workflow to demonstrate key features such as coexistence, collaboration, control and cooperation.

Pubblicato il

MUSIC4D: When Music Meets Artificial Intelligence in the Heart of the Mediterranean

The MUSIC4D project, approved within the framework of the “Notice for the granting of funding for the internationalisation of AFAM institutions,” represents a global initiative that achieved first place in score and order of funding among the projects approved by the Ministry of University and Research. Coordinated by the Palermo Conservatory, the consortium involves all AFAM institutions in Sicily (Agrigento, Catania, Caltanissetta, Messina, Trapani) and Sardinia (Cagliari and Sassari), along with the University of Palermo (scientific lead: Prof. Antonio Chella) and the University of Calabria (scientific leads: Prof. Giancarlo Fortino, Prof. Francesco Pupo), for a total of 13 partners and nearly 6 million euros in funding.

The project, named “Music, Entrepreneurship, Creativity, and Digital Revolution: The Future of Performing Arts in the AFAM system (MUSIC4D: Music, entrepreneUrshIp, Creativity, for the Digital revolution),” aims to promote and enhance the popular musical culture of Southern Italy internationally, using advanced digital tools to create multi-sensory environments and innovative artistic platforms. Over the next two years, all activities will take place in globally strategic areas: Australia, Latin America, Vietnam, the United States, China, the Mediterranean, and the South Pacific (Fiji Islands, Polynesia, Micronesia).

As the University of Calabria, we chose to support the project with an advanced technological component that not only integrates music and artificial intelligence but also explores the possible applications of emerging technologies in the context of the performing arts. This initiative aims to develop solutions that can be used both in academic settings and in artistic events and immersive installations. Within the project, the focus is not only on the implementation of new technologies but, above all, on the ability to transmit the musical cultural heritage of Southern Italy, adapting it to digital formats that can be globally accessible.

The project aims to develop an integrated system that combines large-scale deep learning models, advanced sensory capabilities, and a high-performance robotic platform. Central to this vision is the use of a robot capable of analyzing and processing, in real-time, complex signals from the surrounding environment, such as facial expressions, body movements, postures, environmental parameters, and other emotional indicators.

The computational component is based on models of 7-8 billion parameters, designed to ensure a balance between processing power and adaptability in dynamic environments. These models will be trained on datasets specifically created to capture the emotional and behavioral nuances relevant in a performative context, with the aim of providing refined and reactive interpretations of human emotions.

From a hardware and robotic perspective, the NVIDIA Groot N1 platform will be utilized, which allows for advanced management of movement, perception, and autonomous control. The integration between the data collected by the robot’s sensors (vision, posture, sound, environmental parameters) and the outputs generated by the deep learning models will allow the system to continuously learn and adapt to interaction with the environment and the people present.

This approach will not only allow for conducting an orchestra in a sensitive and adaptive way but also for creating new modes of interaction between technology and the audience. The robot will be able to modulate its behavior, gestures, and communication based on the emotional context, making each performance unique. From a broader perspective, the system represents an innovative way to unite artificial intelligence, musical expression, and the enhancement of Italian cultural heritage, exploring new forms of narration and engagement through the dialogue between human, machine, and art.