Abstract

Physical embodiment in VR enables users to easily interact with a virtual world while realistically adhering to the physical framework of that world. Due to the countless amount of different hardware and software in the VR area, this process can be most efficiently realized using a modular, network-based approach. An existing implementation that uses the framework Ubi-Interact in Unity with C\# already shows that it is possible to implement such software. A new implementation uses Ubi-Interact in the web browser with Babylon.js and WebXR. This implementation uses the same structure as the Unity implementation and is, to some extent, compatible. Modules of both implementations can be replaced with their counterparts of the other implementation, demonstrating that Ubi-Interact is a suitable framework for physical embodiment. However, the web platform still has few suitable libraries, such as in the areas of Inverse Kinematics, which leads to incomplete and incorrect data. Some insights are found regarding precision and performance using an additional Ubi-Interact-based module for evaluation. In the outlook, further improvements and questions surrounding the implementation and the topic of physical embodiment are proposed.

Project Description

In physical embodiment, the following steps are being executed repeatedly and in parallel [1]:

1. Partial pose data from a user is obtained, for example, from hand tracking devices. The data is sent to stage 2.
2. Using data from stage 1 and Inverse Kinematics, a full pose is calculated to resemble the actual user's pose. The linear and angular velocities required to adjust a physical avatar are calculated using additional data from stage 3. These forces are being sent back to stage 3.
3. The forces resulting from stage 2 are applied to a physical avatar. The new pose is sent back to stage 2.

An implementation of this process already exists in C#/Unity. The goal of this thesis was to implement this process for a browser environment. This implementation should be as compatible to the Unity implementation as possible.

Implementation

The implementation split the task into multiple modules that are available on npm and are usable on their own:

1. Represents stage 1 of the physical embodiment process. Source code and online instance available at https://github.com/goldst/ubii-web-target-publisher
2. Represents stage 2 of the physical embodiment process. Can run both in an Ubi-Interact Device or Processing Module mode. Source code and online instance available at https://github.com/goldst/ubii-web-ik-force-computation
3. Represents stage 3 of the physical embodiment process. Source code and online instance available at https://github.com/goldst/ubii-web-avatar-applier
4. Inverse Kinematics library, which is a modified version of existing libraries. Source code is available at https://github.com/goldst/IK.ts
5. Main project which uses Babylon.js, WebXR, the three stage modules, and three options for physics engines. Source code available at https://github.com/SandroWeber/ubii-vr-physics-embodiment-babylonjs/tree/feature/leonard
6. Evaluation module which records published data to compare in- and outputs of any implementation. Source code and online instance are available at https://github.com/goldst/ubii-web-evaluation. Raw results available at github.com/goldst/ubii-evaluation-data

Results

This is a screenshot of the web implementation, each avatar representing one stage:

The implementation works. Some functionality from the Unity implementation is not completely available in the web implementation: The inverse kinematics library is limited in comparison to Unity's built-in inverse kinematics, so poses can be calculated less precisely and are incomplete regarding rotational data.

The three stages can be enabled or disabled, giving the software the possibility to run in combination with the Unity implementation. Some combinations worked well, producing viable outputs after publishing data from one implementation that is being interpreted by the other implementation. Other implementations did receive data irregularly or were not able to interpret the data correctly. Reasons include the inverse kinematics library and the browser possibly slowing down background tabs.

Conclusion

Even though the implemented prototype does not work well enough in every possible scenario yet, none of the problems are fundamentally unsolvable, so it was concluded that a browser implementation of the given process with Ubi-Interact is indeed possible and viable. Also, the use of Ubi-Interact devices for testing was highlighted, because the evaluation module uncovered problems that were not visible in the main application graphically.

Thesis:

Slides Kickoff: