Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
1 result
Name Last commit Last update
VRKinectReg
LICENSE
Readme.md
fixation1.jpg
fixation2.jpg
our_grid.jpg
Readme.md

Introduction

This project allows (1) registering point clouds with each other and (2) registering point clouds with an HMD / VR headset in Unreal Engine 4.

Tutorial Video

The Plugin (see VRKinectReg/Plugins/VRKinectRegistration/)

The registration implementation is based on our paper "Fast and Robust Registration and Calibration of Depth-Only Sensors" and is implemented as a standalone Unreal Engine 4 - plugin (see folder VRKinectReg / Plugins / VRKinectRegistration /).

The Project (see VRKinectReg)

The entire Unreal project uses the registration process of the plugin exemplarily with point clouds streamed in real-time from multiple Azure Kinects. The number of Azure Kinects connected to the PC is automatically detected and a corresponding number of AzureKinect actors are created and placed in the level when the level is started.

Required Grid

In order to perform registration without adjustments to the algorithms parameters, a grid of the following dimensions is required:

  • A total size of 44cm x 44cm
  • Consists of 6 bars horizontally and 6 bars vertically, each 4cm wide and 44cm long, which are as thin as possible, preferably not more than 1mm.

Our example grid: Our example grid

To registrate the point cloud with the virtual world (so that the point cloud and the hmd fits), it is also recommended to build a bracket that keeps the motion controller and grid at a fixed distance from each other during the registration process. For example, we use this: Our example grid Our example grid

DEPENDENCIES

The plugin

To run the project, PCL 1.9.1, Boost 1.6.8 and FLANN are required. We already included the Libraries in the Unreal Project (see /VRKinectReg/Plugins/VRKinectRegistration/ThirdParty/) to overcome barriers and build PCL and boost statically, but due to space reasons, we uploaded the libs separately here (the lib-folders in those both folders has to be copied into the PCL_1_9_1-folder, Flann-folder and the Boost-folder.

If you prefer newer versions, you also can build your own versions and fix the includes in the VRKinectRegistration.build.cs (VRKinectReg/Plugins/VRKinectRegistration/Source/VRKinectRegistration/VRKinectRegistration.Build.cs).

The entire project

To run the entire project, you need in addition to the dependencies of the plugin the AzureKinectSDK installed (we tested version 1.4.0 and 1.4.1. In case of other versions, you perhaps need to fix the files in VRKinectReg/ThirdParty/AzureKinectSensor).

Registration

The registration process is done by the RegistrationManager in the entire project.

To use the registration procedure for multiple Azure Kinects, you can for example:

  1. Run the project in simulation mode (or VR-mode, if you also plan to registrate the point cloud with the virtual world / hmd). The point clouds of the kinect will then displayed automatically for all Azure Kinects which are connected to the computer.
  2. Select the RegistrationManager in the Word Outliner
  3. Check the Property "Should Register Multiple Kinects".
  4. Move the grid slowly in the view volume of the multiple cameras around.
  5. Uncheck the property (3) when you are happy with the displayed registration result.

To use the registration of the point clouds with the world (so you can see the point cloud of your own in the vr headset):

First, keep in mind that you need both motion controllers of your HMD! Based on the position of the grid, the location of the motion controllers are estimated to get their location in camera space of the first kinect. That means, you have to hold the controllers in an exact position relative to the grid.

  1. Run the project in VR mode (if you did it not already for the registration procedure of multiple kinects)
  2. Define the offset of both controllers by editing the properties "Left Controller Offset" and "Right Controller Offset" of the RegistrationManager (value in meters, the default values assume that the motion controllers origin is exactly in the bottom corners of the grid).
  3. Check the Property "Should Register Kinect Into HMD".
  4. Hold the grid including the controllers in front of the first(!) kinect. This is the kinect which is simply named "AzureKinect", NOT "AzureKinect1" or "AzureKinect2". The RegistrationManager only uses the first kinect for this registration procedure.
  5. Uncheck the property (3) when you are happy with the displayed registration result.

If you want to use the plugin in another project, just look, how the RegistrationManager.cpp is implemented and uses the classes of the plugin.