3D Object Tracking

Keywords:
markerless tracking, structure from motion, KLT tracking, Augmented Reality
Author(s): Junming Peng (Jimmy), Martin Buchner, Ajin Joseph, Harald Wuest
Date: 2014-02-12

Summary: This tutorial will give you an overview on how to use Instant Reality's Structure-from-Motion (SfM) module for creating tracking models which can then be used for detecting and tracking a real 3D object without any additional markers.

Introduction

Instant Reality's computer vision framework is capable of tracking real 3D objects without additional markers. In order to achieve this, one must generate a tracking model of the desired object in advance. This can then be used for detecting and tracking the object inside your own application. Instant Vision uses a Structure-from-Motion (SfM) approach for generating a tracking model from a sequence of 2D pictures. This process might be hard to understand for non computer vision developers. Therefore, Instant Vision provides a graphical user interface where developers can just click on a series of buttons to create a 3D object tracking application with ease.

At first, this tutorial will give you a step-by-step guide of the procedures for building a tracking model using Instant Vision. This will be followed by a short description of the code structures. In the end, we will show you some of the results which you can reproduce with the attached paper house template.

Generating a tracking model

Please see images in sequential manner: PreparationStep.png -> FrameHistory1.png -> FrameHistory2.png -> PreparationStep.png.

The creation of a tracking model inside Instant Vision consists of four steps. First of all, one must record a feature point cloud of the object which should be tracked. After this, the features should be transformed into the coordinate system of your virtual objects. Then any unwanted features must be removed. Lastly, the final tracking model and all its components will get generated and saved. We will now describe these steps in greater detail.

1) Recording

Image: Figure 1: Recording and Saving step (PreparationStep.png)

Start Instant Vision. Then reference to Figure 1 and follow the numbers for doing the recording:

2) Calibration

Image: Figure 2: Calibration step (FrameHistory1.png)

Right now, the computed camera poses and features are in a coordinate system in which the first camera pose sits at the origin and looks along the negative z-axis. This is problematic because the tracking should work in the same coordinate system as your application. Otherwise, it’s very difficult to position the virtual objects which are used to augment the real scene. Therefore, it’s necessary to transform the tracking model into a better coordinate system. Inside Instant Vision, this is done by loading a 3D model and then telling the software which features correspond to which points on the 3D model. Please refer to Figure 2 and the following descriptions for doing this calibration:

3) Removing unwanted features

Image: Figure 3: Removing unwanted features step (FrameHistory2.png)

Instant Vision detects features on the whole images but the interesting object usually only covers a part of the images. The remainder of the images belongs to the background which is not constant in the most cases. Therefore, the features from the background would distract the tracking and should be removed. Please reference to Figure 3 and the following steps below on how to erase these unwanted features:

Image: Figure 4: Unwanted features removed (ErasedCompleted.png)

4) Saving the tracking model

With reference to Figure 1, we would like to conclude and finally create the tracking model for our application:

That pretty much covers the basic steps in creating the tracking model for a 3D object tracking application.

Code structures

If you open the “SfMReconTracker.pm” file, you will see that it’s quite complex. Therefore, we will briefly explain the most important parts in the following sub-sections.

Code: DOTTrackerAction

			<DOTTrackerAction category='Action' name='ActionPipe'>
				<Keys size='4'>
					<key type='' val='InstantVideosmall' what='image source, in'/>
					<key type='' val='World.Room.DOTDataModel1.DOTContainer' what='container with templates, in'/>
					<key type='' val='DOTPoseOut' what='pose, out'/>
					<key type='' val='IntrinsicsOut' what='intrinsic params of cam, in'/>
				</Keys>
				<ActionConfig ComputeDebugImage='1'/>
			</DOTTrackerAction>
		

The DOTTrackerAction is used to initialize the tracking by finding the previously recorded object inside the current image (InstantVideosmall). It uses binary image templates (World.Room.DOTDataModel1.DOTContainer) to achieve this. These templates were created and stored together with the respective camera pose during the generation of the tracking model. After a match for one template was found, the corresponding camera pose will get returned (DOTPoseOut). This pose is only a very coarse approximation of the real camera pose. Therefore, we refine this pose using feature based tracking (KLTTrackingAction).

Code: KLTTrackingAction

			<KLTTrackingAction category='Action' enabled='1' name='Action'>
				<Keys size='3'>
					<key val='CurrentImagePyramid' what='input image pyramid, in'/>
					<key val='World.Room.KLTModel1' what='KLT Model, in/out'/>
					<key val='' what='ExtrinsicData, in'/>
				</Keys>
				<ActionConfig MaxBTLevel='5' CalcError='1' CalcProjective='0' CalcProjectiveTrans='0' DontUpdateAffineHelpers='0' EpsBT='0.03' EpsF2F='0.03' ErrorThreshold='30' MaxBTIter='8' MaxIter='8' StoreAnalyserData='0' TrackOnlyNotTrackedFeatures='0'/>
			</KLTTrackingAction>
		

What the KLTTrackingAction does, is finding correspondences between features found in the previous frames and features found in the current frame. These correspondences can then be used to compute the camera pose. Overall, there are three KLTTrackingActions actively used:

  1. The first one works together with the DOTTrackerAction. The previously recorded (offline) features (World.Room.KLTModel1) are projected into the image using a coarse camera pose (DOTPoseOut). The KLTTrackingAction then computes the precise initial camera pose.
  2. The second KLTTrackingAction also uses the previously recorded features but the features are projected using the previous camera pose.
  3. The third KLTTrackingAction tries to track features which were added during runtime (online).
The offline and online features will later be merged into one container. This container will then be used to compute the current camera pose. The advantage of this approach is that you can move the camera around the whole scene as soon as the tracking was initialized.

Code: ShiThomasiCornerDetectAction

			<ShiThomasiCornerDetectAction category='Action' enabled='1' name='ShiThomasiCornerDetect'>
				<Keys size='3'>
					<key val='CurrentImageGreyU' what='input image, in'/>
					<key val='ProjectedPointsContainer' what='existing points, in, optional'/>
					<key val='NewPoints4OnlineRecon' what='detected points, out'/>
				</Keys>
				<ActionConfig CornerCandidatesGridHeight='8' CornerCandidatesGridWidth='8' CornerThreshold='0.02' FeatureDistance='20' MaxNumOfCornersPerGridCell='15' PatchSize='4' ScaleFactor='0.0001' WithDebugOutput='1'/>
			</ShiThomasiCornerDetectAction> 
		

As written before, we are adding new features during runtime. Those features are detected using the ShiThomasiCornerDetectAction. An alternative is the FastFeatureExtractAction. This Action is faster but produces slightly less reliable features than the ShiThomasiCornerDetectAction.

Code: DataSet:World

			<World key='World'>
				<TrackedObject key='Room'>
					<ExtrinsicData key='ExtrinsicData' calibrated="0">
						<R rotation="1 0 0 &#xA;"/>
						<t translation="0 0 0 &#xA;"/>
						<Cov covariance="1  0  0  0  0  0  &#xA;0  1.1  0  0  0  0  &#xA;0  0  0.9  0  0  0  &#xA;0  0  0  0.95  0  0  &#xA;0  0  0  0  0  0  &#xA;0  0  0  0  0  0  &#xA;"/>
					</ExtrinsicData>
					<KLTHistoryModel key='KLTModelOR'/>
					<Inline url='KLTModel.xml'/>  
					<Inline url='DOTDataModel.xml'/>	
				</TrackedObject>
			</World>
		

Information about the tracked object is stored inside the World. and load the tracking model which was generated after the “Create Tracking Models” button was clicked. The KLTModel.xml file contains all offline features and the DOTDataModel.xml file stores all binary image templates. If you take a look at these files using a text editor, you can see that they define the KLTModel1 and DOTDataModel1 keys. In case the keys are different, you might have to change them so that they match the keys from the PM-file.

Results After you have gone through this guide, you should be ready to create an object tracking application on your own. We have tagged a few screenshots of the sample application below which showcases the before and after tracking results and not to forget our little paper house template as a reference for the tracking and augmentation to take place ;) . This is just a reference object which you can use to test your tracking and you can definitely replace it with your own real 3D object. We have also showcased the possibility of 3D tracking and augmentation of the paper house using mobile devices (e.g. iPad mini). However, the documentation for the tracking on mobile devices will follow at another time. So stay tuned!

Image: Figure 5: Paper house template - Please cut only on the dotted lines and built the house by gluing the loose ends. Later glue the house to the center of the stone pavement
Image: Figure 6: Paper stone pavement
Image: Figure 7: Build paper house (the house in the background is just an old version of the paper house)
Image: Figure 8: Tracked paper house overlaid with a reconstructed 3D model of the house
Image: Figure 9: Tracked paper house augmented with animated 3D dinosaur and realistic soft shadows for the dinosaur and the paper house
Image: Figure 10: Using an iPad mini for tracking the paper house and augmenting it with an animated dinosaur 3D model
Files: