| Overview | Embodied Character |
|---|---|
This repository hosts the project conducted as a research assistant at the University of Glasgow under Dr. Mathieu Chollet. The main objective was to create a VR simulation using the Rokoko Smartsuit Pro II and Unreal Engine 5 to measure the degree of embodiment. |
 |
- Overview
- Useful Unreal Worlds
- Rokoko Studio
- Unreal Engine
- Live Link
- Virtual Reality
- Exporting to Other Projects
- Drift
- Mirror
There are a few unreal worlds worth looking at. These are:
- Tracking - This contains “Object tracker” use for tracking drift, find out more in the drift section.
- UE427_Natalia - This contains a metahuman that has been rigged to stream Rokoko Mocap.
- MixamoPipeline - This contains a few examples of rigged Mixamo characters using the Mixamo Pipeline with the “Animating the Character” section.
- VR_Testing - This contains the VR template world provided by UE. This also uses the Mixamo pipeline but uses the “Embodying the character” section.
Rokoko Studio is a 3D animation software designed for motion capture data recording, streaming, and editing. It integrates with Rokoko's mocap tools, allowing users to animate and retarget in real-time. The platform supports platforms such as Unreal Engine, Unity, and Blender.
Watch the Rokoko onboarding series to get acquainted with all the components.
This is covered in the Rokoko onboarding. To connect the suit plug the type-C cable into the hub and connect the USB to the computer. The process is the same for the gloves.
Once connected Rokoko should detect automatically. Configure the wifi settings, making sure the receiver IP is correct. This can be checked by typing ipconfig in the command line. connect to the IPv4 Address. Once configured the device manager should be as shown below. You can now unplug the device.
Sensors can be removed from the smartsuit so that it can be washed. To do this follow this tutorial. It takes aproximately 5 minutes to remove the electronics and 15 minutes to put them back.
-
Rokoko Livelink plugin for Unreal Engine.
-
Visual Studio should be installed with the relevant workloads
The guide below contains the necessary information to set up a project for both custom characters and metahumans. This video also provides a bone maps folder that contains bone maps for retargeting as well as a T-pose asset for metahumans that places limbs in the correct pose for retargeting.
Note: The workflow in Unreal 4 and 5 is the same. The project provided in the video can be converted to UE 5 and should work fine.
🎥 Watch this video for the full process 🎥
UE427_Natalia is a world that contains a metahuman and can also be downloaded from the video provided.
The "MixamoPipeline” UE world contains some example characters that are rigged using the Mixamo pipeline.
The "VR_Testing” UE world contains some example characters that are rigged using Mixamo pipeline as well as a “VR pawn” that is rigged to be attached to the character which follows from the “Embodying the character” section*.*
A Mixamo pipeline was designed so that different characters could be used easily. Mixamo was chosen as it provides a set of free characters as well as a simple way of rigging custom characters. The Rokoko Guide was used to develop this workflow.
The following steps will describe how to use any character from Mixamo to live-link to the Rokoko suit:
-
Visit mixamo and if you haven’t already make an account.
-
In the “Characters” tab choose any character.
-
In the “Animation tab” choose the T-Pose as this will allow rokoko to map the animations correctly.
-
Download.
-
Open the folder called “MixamoPipeline”.
-
In the content browser create a new folder with the name of your character in the “Characters” folder
-
Open that folder and import the .fbx file ensuring that:
- The Skeleton mesh is set to “None”
- "Use T0 As Ref Pose" is checked
- "Import Morph Targets" is checked
-
Import all.
-
Open the folder.
-
Right-click on the skeleton and “create > anim Blueprint” and open it.
-
Add 2 variables, “Rokoko Actor Name” and “Rokoko Bone Map” of types “Name” and “Rokoko Body Map Data > Class Reference” types respectively.
- Add the following elements to the AnimGraph:
- Compile this then set the “Rokoko Bone Map” to “MixamoCharacter_Bonemap” and compile and save again.
- Create a new actor and open
- Add a skeletal mesh component
- In the event Graph construct the following:
The cast should be set to the anim Blueprint you created previously
- Drag the Actor you created into the scene and set the anim class to the anim Blueprint you created and the mesh to the skeletal mesh.
-
In "VRTemplate>Blueprints,” make a copy of the VRpawn blueprint, open the “Viewport” and add a “SkeletalMesh” Component.
-
Set the Animation Mode to “Use animation blueprint”
-
Set the Anim class to the anim blueprint you created previously.
-
Set the skeletal mesh asset to the skeletal mesh of the character.
-
Go to the even graph and add to the end of the “Event BeginPlay”:
Note: The first node should cast to the anim blueprint that you just created.
-
Open the VRGameMode and set the default pawn class to the pawn you just created.
In Rokoko studio, open a scene. In the Livestream tab, open the Unreal setting, take note of the port number, and activate.
The Rokoko receiver should have the same port number.
Once ports are correct, in unreal engine, go to “Window > Virtual production > Live Link” then in the Livelink “+ source > Rokoko Studio Source”
Remember that the name used in rokoko studio should match the one set in the Mixamo pipeline.
This project was designed specifically for use with the Vive headset but should work with any headset that is Steam VR compatible
The best way to transfer assets from one project to another is to Migrate a folder.
Note: This must be exported to another “content” folder.
Tracking the drift is useful as it directly influences the person's sense of embodiment. With the Sensor Fusion 2.0 update , this should be reduced, however.
Made using C++, an object tracker can be located in the Tracking UE world.
Once placed in the world, the object tracker has two parts.
- Tracked object - Set to any object you wish to track
- Output CSVFile Path - currently set to “Saved/Logs/Tracking/” within the Unreal Engine world. This will be where the tracker outputs the object coordinates over time.
It will create a CSV file each day and append all the data collected that day to that file. This avoids creating a lot of files. To edit the “Object Tracker” Class you can open the “C++ Classes > Tracking > Object Tracker” and edit it. Make sure that you have the Visual Studio Integration Tool for UE 5.
Below is a a simple plot demonstrating the tracked object from the Tracking world.
This was done quickly to investigate using a mirror in unreal.
The tutorial: Create a Realistic Mirror was used.
To get the mirror working the project settings need to be changed to ensure the reflection works. Go into “Edit>Project Settings>Rendering” and check “Support global clip plane for Planar Reflections” and set the “Dynamic Global Illumination Method” to “Screen Space”, finally uncheck motion blur and restart the project for the effects to take place.
The material is simple, set to the following:
