How did this research come about?

Stevens: The project began in an honors music theory course at the University of Delaware. The students decided that they would like to make music listening more accessible to children with autism, inspired by a peer who had a sibling with autism who was never able to hear her performances. So, we partnered with Autism Delaware and a local library and generated the idea to create modular music that could be played on a specialized device that would allow children to modify the sounds as they listened.

Mauriello: I joined the project to fabricate the hardware and develop the software that drives the musical playback capabilities and responds to the interactions with children. Our hope is to use the data captured to provide insights composers can use to better develop music for this population.

How does this invention differ from what’s in the market?

Mauriello: Our device is more than a tool for playing music, it is about redesigning how we think about composing music. The device itself is a simple push-button controller, almost like a video game, that allows users to change the different tracks or ambient sounds — music, drum lines, melodies. It has various combinations that produce music in real time, based on pre-composed tracks that we’ve created.

Behind the scenes, our system software is logging these interactions, which helps us learn how the user experiences the music. Understanding how the music was experienced in the classroom, library or other settings can help composers modify that music so it is more accessible for people who may experience music differently. We’re also developing multiple controller devices to allow students to work together to change the musical layers and speeds, which builds soft skills like teamwork, patience and turn-taking.

Stevens: Our ultimate goal is to develop a device that can learn from the child and give them pre-selected options that might suit their needs at different times of day, in different locations, or depending on how the user is feeling. 

What have you been doing to translate your invention from the lab to the market?

Mauriello: We initially built hardware and software, and after sharing the music and device with parents and children, we found the system had some promise. So, we reached out to the Office of Economic Innovation and Partnerships (OEIP) to secure a provisional patent to protect our work. Today, we’ve deployed our technology at a community library and two Delaware schools in the Christina and Red Clay school districts. 

Stevens: We’ve seen children use our device and had parents observe their children having breakthroughs, including children who are nonverbal that began to communicate more effectively through gestures and expression. These initial studies have shown a real impact.

How have UD students been involved in this work?

Stevens: Undergraduate music students have been involved from the beginning, generating the idea to use music to help the autistic population, and then composing different modular arrangements of children’s songs using a specific compositional template. Graduate-level musicians are working to expand our library to create ambient-based music for older listeners with autism. 

Mauriello: Computer science students are often working on the physical hardware, from 3D-printing the enclosure that surrounds the listening device to developing small digital electronic and sensor systems that enable children to interact with our music. They are also developing the software, generating insights from the data and working collaboratively with other team members on protocols for interacting with children and teachers.

Stevens: One of the most exciting things has been to see the students on both sides of the project interact. For example, the music students now think about composition through the technological platform and data provided by the engineering side. And our engineering students are learning about different ways we might manipulate the sounds we're creating on the sound synthesis equipment for playback on the uCue device.

What are your future goals for the project?

Mauriello: We see potential for populations in physical therapy, in aging communities. For example, we could work with communities affected by memory loss, where slowing music down might be therapeutic and help individuals engage. Part of that work involves scaling up our ability to collect data, while expanding our physical devices to include online platforms that can be available beyond classrooms and individual settings. 

What kind of support and guidance have you received from the UD ecosystem?

Stevens: The UD ecosystem has served this project by creating a space in which innovation and community engagement go hand in hand. These foundational supports inspire the work that we're doing. At the same time, OEIP has helped us find opportunities to create partnerships with other community organizations and foundations to advance our work.

Mauriello: OEIP also helped us with networking, resources and potential licensing options and made the process easy for us to participate in, even though this is not our main role here at the University. We’ve also received seed grants from the Partnership for Arts and Culture in the College of Arts and Sciences, the Maggie Newman Health Fund in the College of Health Sciences and through the Institute for Engineering Driven Health.

What advice do you have for scientists who have not been involved in research translation?

Mauriello: Start early and engage with OEIP as soon as possible. Even if your long-term goals may not necessarily lead to a patent, it's good to protect the initial intellectual property before you disclose the invention through a publication or other public dissemination, to keep your options open as you develop your ideas.

Stevens: Remember that research happening across this campus is relevant to problems in the world around us. Speak to UD’s Community Engagement Initiative and find ways to interface with community members to identify ways your work can solve problems for others.

What do you find most rewarding about this work?

Stevens: As a child of a special education teacher, I grew up seeing and experiencing the world through the eyes of community members who have different abilities and experiences. This project has brought my life full circle, allowing me to make a positive impact while giving me the ability to hear music through the ears of children that I might not otherwise be able to understand in that way. 

Mauriello: For me, an interesting aspect of this project is being able to take computer science into the community. As researchers, we're often behind a computer screen, building software, working on the internet. This research allows us, along with our students, to work with the community and engage them in both music and technology design. Stepping out of the lab and working with people in the real world is very motivating.