The Idea
We want to integrate humans into a neuro-robotics simulation environment - meaning there is a physically simulated virtual environment in which a robot acts and learns. For the human to become part of the robot's virtual world a virtual body is necessary. This surrogate body needs to be controlled as a physical entity in all its movements. Controlling it happens through state-of-the-art consumer VR hardware used to track user movements. An intermediate processing step will then combine the user tracking data with the current state of the surrogate body to determine motion commands applied to the surrogate.
This picture shows the current body model inside an NRP (Neurorobotics Platform) experiment.
The Goal
Whenever the surrogate body needs to move - as it should replicate the movements of the human - certain forces need to be applied to individual body parts / joints. Control of these forces needs to be adequate, not too strong / weak. As human movement can change posture and direction quite quickly, these forces have to be adjusted with minimal latency. Additionally, local solutions that adjust for example only the force on a single joint according to a target position fail to incorporate the other movements up/down the kinetic chain and might therefore produce sub-optimal results.
Using different approaches (machine learning, motion models, etc.) , we would like to determine the necessary forces to move the remote body in a fast and precise fashion with a holistic approach of looking at the global target posture to determine individual local forces.
What's already there?
- Neurorobotics Platform with physics engine, web-based implementations of physics engines
- Rigged avatar model (with joint definitions)
- VR client in Unity3D
- Data communication plugin to allow movement commands from client (using VR hardware) to server (running the simulation)
Core topics
- choice of framework/tools (tensorflow, nengo, ...)
- implementation & evaluation of motion control mechanisms
- design of training procedure (random movements, animations, ...)
Applicable Skills/Interests
- Unity3D
- Web technologies (Three.js, Babylon.js, etc.)
- Virtual Reality