Virtual Keyboard: Real-Time Feet Tracking for the Control of Musical Meta-Instruments


Project Type

Research Assistant Project


May 2006 - August 2006


University of Victoria, Victoria, BC, Canada


  • Applied Computer Vision Algorithms Laboratory (ACVA)


  • Dr Alexandra B. Albu (superviser), Dept. of Electrical and Computer Engineering, University of Victoria

  • Dr Peter Driessen, Dept. of Electrical and Computer Engineering, University of Victoria

  • Dr Wolfgang Schloss, School of Music, University of Victoria


The virtual keyboard tries to mimic the bank of foot pedals one may find on an organ. It is intended to be used with new musical instruments, as the Radio Drums. Some of these kinds of instruments require a foot control interface since both hands are already occupied when playing the instrument.

The virtual keyboard in itself is only a piece of wood on which color keys have been painted. It is observed by a cheap webcam and images are analyzed with computer vision algorithms in order to track the performer's feet and detect keys hits. A white marker is put on the tip of the performer's feet in order to get a reliable position for the feet tips. Since the positions and the layout of the keyboard are known to the algorithms, it is possible to detect which key has been hit and for how long a foot remained on that key. The following video sequence shows the use of the virtual keyboard. When a key is coloured green, it means that there is a foot over it, whereas the red color means the key has been hit.


The following video shows the virtual keyboard in action.

Related Publications

The publications related to this project are listed below. For a complete list of my publications, see the publications page.

Frédéric Jean and Alexandra Branzan-Albu, "The Visual Keyboard: Real-time Feet Tracking for the Control of Musical Meta-instruments", Signal Processing: Image Communication, vol. 23, pp. 505-515, 2008.

Frédéric Jean, Alexandra Branzan-Albu, Wolfgang Schloss and Peter Driessen, "Computer Vision-based Interface for the Control of Musical Meta-instruments", In Proceedings of 12th International Conference on Human-Computer Interaction, (Beijing, China), July 22-27 2007.