Dance, Engineering professors team up to build AI model that can read dance movement
Dance carries knowledge about how people live — information about the dancer’s landscapes, communities and histories. However, AI systems that analyze human movement are trained almost entirely on everyday actions like walking, sports and gestures, recorded in Western labs with no information about the geographic or cultural origins of the movement.
An interdisciplinary project at The University of Texas at Austin, co-led by Dance Assistant Professor Sinclair Ogaga Emoghene and Engineering Assistant Professor Lillian Chin, will create an AI-driven large movement language model for dance. The team has been awarded a grant of $643,719 from the Humanities and AI Virtual Institute (HAVI) at the nonprofit Schmidt Sciences to support this research endeavor.
The project will utilize recordings from countries around the world to train a full model that treats dance as a kind of language. Just as language AI breaks text into words, this system breaks motion-capture recordings into small, reusable movement units the team calls “kinemes.” Each unit is tagged with data about the movement, including geography, climate and terrain, so the system can learn how place shapes the way bodies move.
Dance scholars have long observed that coastal, riverine and highland communities tend to move differently, but these ideas have mostly stayed anecdotal. This new dataset gives researchers a way to test these observations across dozens of traditions at once and potentially find connections invisible to human observation.
The team has already recorded and processed 155 dance recordings from 33 countries and built an online platform for exploring them. Over the course of Chin’s robotics research in the Chandra Family Department of Electrical and Computer Engineering, she has developed soft, 3D-printed sensor pads that capture how much force and pressure are exerted on the pads, information that cameras can’t see. In early work, the team reports finding a shared timing pattern in torso-and-limb coordination between Adzogbo dances from Ghana and Borno dances from northeastern Nigeria, hinting at historical connections along trans-Saharan trade routes.
“Embodied movement is among the oldest and most informationally dense forms of human knowledge, and we know that dance traditions encode centuries of accumulated understanding about environment, social organization, spiritual practice and aesthetic value,” said Emoghene. “Our work of documenting and codifying these dance movements helps preserve them as traditions change and the knowledge they carry disappears with them.”
On the technology side, no existing system connects motion models with geographic context. The team’s approach could apply anywhere data carries a sense of place. The project would release the first large culturally labeled motion dataset, but it also opens new possibilities as a model for archaeology and art history fields.
The grant would fund the next stage of their research through August 2029 and build on their existing data set by growing the collection to more than 500 recordings from more than 40 countries. The grant funding will enable the team to hire two graduate research assistants in Dance and Engineering; design and build a field-ready sensor for fieldwork, support international fieldwork in West Africa, South Asia, the Caribbean and East Africa; build and maintain an online analysis platform and host annual community workshops at UT Austin.
The research project is carried out with support from the Department of Arts and Entertainment Technologies and the Lab for Immersive Media (LIM), where motion-capture technologies allow for unique collaboration with faculty partners like department chair Michael Baker, Assistant Professor of Practice Kyle Evans and LIM manager Brandon LeJeune.
“We’re grateful to the Humanities and AI Virtual Institute for their support of our research,” said Chin. “This substantial investment in our work allows us to develop novel AI and sensor technology to expand our knowledge of human movement across the world.”
Emoghene and Chin are among 19 interdisciplinary teams selected to receive a collective $10 million from the Humanities and AI Virtual Institute (HAVI). The selected projects will use AI to expand the study of art, dance, history, music, literature and more across thousands of years of global culture, while helping researchers develop and test new AI capabilities. Selected from 501 proposals spanning 42 countries, the awards bring HAVI’s total funding commitment over the past two years to more than $21 million. Teams will openly share the tools and resources they develop, including code, model weights and benchmark datasets.
Related content
Wired for Rhythm: A year-long collaboration yields a unique performance that combines dance with next-generation sensing capabilities