Abstract

In this thesis, the design and implementation of an AR piano learning application prototype are described which is intended to run on a smartphone-powered Head-Mounted Display (HMD).
A unique approach for the detection of piano keys by analyzing the keyboard's natural features is presented, which lifts some restrictions of previous approaches.
It utilizes a novel monochromatic marker detection specifically designed for this task.
To achieve this, the tracking algorithm was implemented using native OpenCV and deployed as a self-sufficient library while the application prototype was created using Unity3D.
For the integration of the library into the prototype, low-level means of communication were implemented.
Finally, the system was successfully deployed on Android.
It is shown that a tailored natural feature tracking algorithm can be superior in terms of accuracy and performance compared to a generic, purely marker-based, solution.

Keyboard Detection Algorithm

Markers placed on the keyboard to mark specific keys are detected first and used to estimate the keyboard's orientation and key dimensions in the frame.

By scanning along defined lines with a width of 1 pixel, all keys can be detected.

As CPU-intensive image operations on a large amount of pixels can be avoided, the algorithm is very fast.

Compared to known marker based piano tracking approaches, the algorithm is faster and more accurate but fails in more environments than a fiducial marker tracking.

(Output of a successfull keyboard detection. The final system does not use markers of the same color.)

Full System

The Tracker Module is deployed as a C++ dynamic library (.dll or .so). It can be used by other applications as it is self-sufficient.

The Prototype was developed using Unity and integrates the tracking module. Although the communication between Unity and the library performs sufficiently well, it could be improved regarding efficiency.

The application is capable of loading, parsing and playing midi files and highlighting the corresponding keys on the piano.

For the positions of a key in the given frame, a 3D mesh can be constructed and positioned accordingly in the Unity scene.

The full system can run on a smartphone and therefore target a smartphone-powered HMD in the future.

Keyboard Tracking Demonstration Video

[ Slides Kickoff/Final (optional)]