Abstract

As the number of applications employing virtual simulation environments has increased, so has the emphasis on the real-time integration of humans in these environments. Multiple solutions utilise motion tracking to grant humans control over virtual characters in order to achieve this integration. A remaining challenge with these solutions is to use few and easily accessible tracking devices while also simulating the user’s movements with high fidelity. In order to address this, the authors of QuestSim propose a novel method to track human movement using only the headset and the two controllers of a Virtual Reality device. Using these trackers, they train a Reinforcement Learning agent with motion data to control physically-simulated characters. Despite having only three trackers on the upper body, the agent achieves the unanticipated result of imitating the movements of the lower body. Therefore, we consider it crucial to corroborate these findings and investigate the capabilities and deficiencies of this method. Experiments conducted on our reimplementation confirm that QuestSim is capable of controlling the entire body of the character, but is limited to the variety of motions that it can replicate. Furthermore, additional modifications that we test reveal that the addition of feet trackers improves only slightly the quality of certain movements, whereas variations of the Reinforcement Learning reward refine the imitation significantly.

Project Description

In this thesis we study the method presented by the authors of QuestSim. Since neither the code nor the training data of QuestSim are available, we reimplement the algorithm and collect motion data based on the descriptions of the paper. With this we test if it is possible to control a physically simulated character using a VR-headset and two controllers as trackers. For our additional studies, we research how modifications of the reward, parameter variations and different motion types affect the quality of the motion imitation.

Implementation

Our implementation is divided in two modules. The first module is the physics simulation for moving the digital character and training the Reinforcement Learning-agent. We implement this simulation in Python and use Isaac Gym as the simulator. The second module is the VR application for tracking the poses of the VR-equipment and transmitting these poses to the simulation on real-time. For the development of the application we opt for Untity and we use the Pico4 as our VR-device. For the communication model between both modules we use the publisher-subscriber pattern. Here the VR application constantly publishes the poses of the device, while the simulation receives them and passes them as input to the RL-agent. Then, the agent decides which action to perform in order to move the virtual character.

Results

Conclusion

The experiments in our current implementation reveal that this method can control the entire body of the character despite of having just a small amount of trackers. Nevertheless, the movements cannot differentiate considerably from the training data and the imitation quality not always perform well. Moreover, the additional variations that we include demonstrate to increase the performance of the agent. In particular, PD-control and the modification of the reward function provide a better imitation.

[ PDF (optional) ] 

[ Slides Kickoff/Final (optional)]