Research‎ > ‎

Vitual Keyboard: Real-Time Feet Tracking for the Control of Musical Meta-Instruments

Project type: Research Assistant Project
Duration: May 2006 - August 2006
Institution: University of Victoria, Victoria, BC, Canada
Group: Applied Computer Vision Algorithms Laboratory (ACVA)
Collaborators: Dr Alexandra B. Albu (superviser), Dept. of Electrical and Computer Engineering, University of Victoria
Dr Peter Driessen, Dept. of Electrical and Computer Engineering, University of Victoria
Dr Wolfgang Schloss, School of Music, University of Victoria

Description

The virtual keyboard tries to mimic the bank of foot pedals one may find on an organ. It is intended to be used with new musical instruments, as the Radio Drums. Some of these kinds of instruments require a foot control interface since both hands are already occupied when playing the instrument.

The virtual keyboard in itself is only a piece of wood on which color keys have been painted. It is observed by a cheap webcam and images are analyzed with computer vision algorithms in order to track the performer's feet and detect keys hits. A white marker is put on the tip of the performer's feet in order to get a reliable position for the feet tips. Since the positions and the layout of the keyboard are known to the algorithms, it is possible to detect which key has been hit and for how long a foot remained on that key. The following video sequence shows the use of the virtual keyboard. When a key is coloured green, it means that there is a foot over it, whereas the red color means the key has been hit.

Video

The following video shows the virtual keyboard in action.

Virtual Keyboard


Related Publications

The publications related to this project are listed below. For a complete list of my publications, see the publications page.

Frédéric Jean and Alexandra Branzan-Albu, "The Visual Keyboard: Real-time Feet Tracking for the Control of Musical Meta-instruments", Signal Processing: Image Communication, vol. 23, pp. 505-515, 2008. http://docs.google.com/uc?export=download&id=0B3TPx0P_KI4baHFueVd1MGtQSFU http://docs.google.com/uc?export=download&id=0B3TPx0P_KI4bcFh3V0Q5Vms2ODQ
Frédéric Jean, Alexandra Branzan-Albu, Wolfgang Schloss and Peter Driessen, "Computer Vision-based Interface for the Control of Musical Meta-instruments", In Proceedings of 12th International Conference on Human-Computer Interaction, (Beijing, China), July 22-27 2007. http://docs.google.com/uc?export=download&id=0B3TPx0P_KI4bbTNHQUZVQkFZRTg http://docs.google.com/uc?export=download&id=0B3TPx0P_KI4bLTBudWZyaEdrSm8