Abstract

This thesis aims to solve the problem of recognizing user-performed actions in a Virtual Reality environment. Using the Unity game engine, a dataset of six actions was recorded by human beings. These six actions consisted of three movements, each performed with the left and right hands. The actions were chosen to be throwing, waving, and pointing. The dataset was constructed using the OpenVR Unity plugin API, which provided the position, velocity, and rotation of the VR headset and the handheld controllers. These were saved in a CSV and fed into a Neural Network with an LSTM (Long-Short-Term-Memory) Layer and three Dense Layers. After training the Neural Network multiple times with varying parameters, such as a higher Node count in the layers, adding Dropout layers, and a more extensive dataset, the Network was exported in an ONNX format and imported back into Unity, where the Barracuda framework was used to utilize the Neural Network. After importing the Neural Network, it can be fed live inputs. It then predicts the performed action and prints the result to the console. This paper was used as a proof of concept to show that action recognition in VR is possible. Possible topics that could be researched further include instructing a robot to perform actions based on the Neural Network's output or extending the Neural Network to support and recognize more actions.

Results/Implementation/Project Description

Conclusion

[ PDF (optional) ] 

[ Slides Kickoff/Final (optional)]