Real-time following and recognition of time profiles

Gesture Follower by Frederic Bevilacqua and developed by Bruno Zamborlin. Visit the
IRCAM official webpage for details.

Examples

The follow demo takes advantages of the Gesture Follower in different use cases.

The gesture follower is used to select and synchronize prerecorded videos, following the dancer gestures. It uses data from inertial sensors worn on the wrists of the dancer.

The Gesture Follower is used in this example for real-time recognition of five pre-recorded gestures. As soon as one of these gestures is recognized, a specific gesture-driven audio engine is enabled among the follows:

Grainstick: based on the orientation of the phone, this audio engine selects the correlated segment of sound of an actual rainstick sample
Percussions: percussion samples are triggered anytime a ‘hit’ is detected
Shaking: this engine triggers with a fix tempo the portion of the loaded sound in which the intensity better corresponds to the energy of the gesture in that particular instant.
Circle: a sound loop is time-stretched based on the speed of the gesture. The speed is automatically detected by the Gesture Follower comparing the performance with the pre-recorded gesture.
Static positions: An audio loop is played whenever the phone assumes a static position for a while.

Read more

Continuous realtime gesture following and recognition
2010 – Bevilacqua, F., Zamborlin, B., Sypniewski, A., Schnell, N., Guédy, F., & Rasamimanana, N.
Gesture in Embodied Communication and Human-Computer Interaction, 73–84. Springer. Retrieved January 28, 2011  
[download pdf]

Online Gesture Analysis and Control of Audio Processing
2010 – Bevilacqua, F., Schnell, N., Rasamimanana, N., Zamborlin, B., & Guedy, F.
Musical Robots and Interactive Multimodal Systems. Springer Tracts in Advanced Robotics.