instantreality 1.0 - tutorial - Basic Marker Tracking

Basic Marker Tracking

Keywords:
vision, tracking, marker, augmented reality
Author(s): Mario Becker, Svenja Kahn, Michael Zoellner
Date: 2009-01-07

Summary: This tutorial shows you how to use the basic features of instant reality's vision module for marker tracking.

Introduction

instantvision is a set of visual tracking systems offering a variety of features such as simple marker tracking or markerless tracking (for example line trackers and KLT). The true power of the system lies in the ability to combine of several such tracking procedures, for instance using a line tracker for initialisation with an absolute pose and KLT for frame to frame tracking.

In this example we will focus on a simple marker tracking example using the VisionLib backend. First we will show how to track a single marker for a simple Augmented Reality application. Then we will extend this example by tracking a second marker.

This example works with all cameras which support DirectShow. If your camera does not support DirectShow have a look at the Camera Setup Tutorial and adapt the "source_url" in the instantvision configuration files of this tutorial.

IOSensor

The marker tracking is loaded like any other Instant IO device via an IOSensor. These are instantvision's fields:

VideoSourceImage (SFImage): Camera image
TrackedObject1Camera_ModelView (SFMatrix): Camera's modelview matrix
TrackedObject1Camera_Projection (SFMatrix): Camera' projection matrix
TrackedObject1Camera_Position (SFVec3f): Camera's position
TrackedObject1Camera_Orientation (SFRotation): Camera's orientation

Code: IOSensor

<IOSensor DEF='VisionLib' type='VisionLib' configFile='TutorialMarkerTracking_OneMarker.pm'>
    <field accessType='outputOnly' name='VideoSourceImage' type='SFImage'/>
    <field accessType='outputOnly' name='TrackedObject1Camera_ModelView' type='SFMatrix4f'/>
    <field accessType='outputOnly' name='TrackedObject1Camera_Projection' type='SFMatrix4f'/>
    <field accessType='outputOnly' name='TrackedObject1Camera_Position' type='SFVec3f'/>
    <field accessType='outputOnly' name='TrackedObject1Camera_Orientation' type='SFRotation'/>
</IOSensor>

Setting the virtual object's position and orientation relative to the camera

We add a virtual object (a yellow teapot) to the scene. Then we set its position and orientation relative to the camera by routing the translation and orientation from the IOSensor to the teapot. A second transformation translates and rotates the teapot relative to the marker.

Code: Transforming the virtual object relative to the marker

<Transform DEF='TransfObj1RelativeToCamPosition' translation='0 0 0' invert='TRUE'>
    <Transform DEF='transfObj1RelativeToMarker' translation='2.5 2.5 1.5' rotation='1 0 0 1.57'>
        <Shape>
            <Appearance>
                <Material emissiveColor='1 0.5 0' />
            </Appearance>
            <Teapot size='5 5 5' />
        </Shape>
    </Transform>
</Transform>

<ROUTE fromNode='VisionLib' fromField='TrackedObject1Camera_Position' toNode='Obj1' toField='translation'/>
<ROUTE fromNode='VisionLib' fromField='TrackedObject1Camera_Orientation' toNode='Obj1' toField='rotation'/>

Remark: To track only a single marker we could route the IOSensor's modelview matrix directly to the viewfrustum instead of setting the position and orientation of the virtual object. In this tutorial we use the position and orientation because the variant with the modelview matrix cannot be used to augment several markers with virtual objects. To use the modelview matrix instead of transforming the virtual object you would have to replace the routes from TrackedObject1Camera_Position and TrackedObject1Camera_Orientation with:

Code: Mapping the Camera's modelview matrix on a Viewfrustrum

<ROUTE fromNode='VisionLib' fromField='TrackedObject1Camera_ModelView' toNode='vf' toField='modelview'/>

With the camera's image in background we create an Augmented Reality scenario.

Code: Camera Image on Polygon BackGround

<PolygonBackground>
    <Appearance positions='0 0, 1 0, 1 1, 0 1' >
      	<TextureTransform rotation='0' scale='1 -1'/>
       	<PixelTexture2D DEF='tex' autoScale='false'/>
    </Appearance>
</PolygonBackground>
    
<ROUTE fromNode='VisionLib' fromField='VideoSourceImage' toNode='tex' toField='image'/>

Print the first marker (the one which looks like a "L") to augment it with a virtual yellow teapot.

Files:

Image: TutorialMarkerTracking_OneMarker.jpg

Tracking two markers

The InstantVision configuration file

Now let's add a second marker augmented with a blue teapot. First have a look at the InstantVision configuration file of the previous part of this tutorial (TutorialMarkerTracking_OneMarker.pm).

An InstantVision configuration file is an XML document. It has two main components: An "Action Pipe" with several actions and a "DataSet" where the data of the InstantVision module is stored.

We do not need to change anything in the ActionPipe to use several markers but let's nevertheless have a look at it. There are four actions in the ActionPipe of this example:

VideoSourceAction: gets the camera image
ImageConvertAction: converts the camera image from RGB to GREY
MarkerTrackerAction: the marker tracker
TrackedObject2CameraAction: creates a camera for every tracked object in the DataSet so we can transfer the data to the InstantPlayer

An Action usually has IN- and OUT-slots for incoming and outgoing data. The data routed to IN-Slots or OUT-Slots is identified by "keys". For example the VideoSourceAction has an OUT-Slot for which we use the key "VideoSourceImage". We use the same key for the IN-Slot of the ImageConvertAction to route the image from the VideoSourceAction to the ImageConvertAction. Actions can have an ActionConfig in which you can set several parameters of the Action. For example in the MarkerTrackerAction you can set whether to use a light invariant contour detection (ContourExtractor=0) or a silhouette extractor (ContourExtractor=1) and several other parameters. You can open a configuration file in InstantVision and have a look at the actions in the ActionManager to see a short description of the attributes.

Code: The ActionPipe

  <ActionPipe category='Action' name='main'>
    <VideoSourceAction__ImageT__RGB_Frame category='Action' enabled='1' name='VideoSourceAction'>
      <Keys size='2'>
        <key val='VideoSourceImage' what='image live, Image*, out'/>
        <key val='' what='intrinsic parameters to be modified, out'/>
      </Keys>
      <ActionConfig source_url='ds'/>
    </VideoSourceAction__ImageT__RGB_Frame>
    <ImageConvertActionT__ImageT__RGB_FrameImageT__GREY_Frame category='Action' enabled='1' name='ImageConvertActionT'>
      <Keys size='2'>
        <key val='VideoSourceImage' what='source image, in'/>
        <key val='ConvertedImage' what='target image, out'/>
      </Keys>
    </ImageConvertActionT__ImageT__RGB_FrameImageT__GREY_Frame>
    <MarkerTrackerAction category='Action'>
      <Keys size='7'>
        <key val='ConvertedImage' what='input image, ImageGREY*, in'/>
        <key val='IntrinsicDataPGRFlea8mm' what='IntrinsicDataPerspective, IntrinsicDatra*, in'/>
        <key val='World' what='World of TrackedObjects, World*, in/out'/>
        <key val='MarkerTrackerInternalContour' what='Contour, Contour*, out'/>
        <key val='MarkerTrackerInternalSquares' what='GeometryContainer of corner points, GeometryContainer*, out'/>
        <key val='MarkerTrackerInternalCorresp' what='internal use'/>
        <key val='MarkerTrackerInternalPose' what='internal use'/>
      </Keys>
      <ActionConfig ContourExtractor='0' MTASilThresh='140' MTAThresh='140' MTAcontrast='1' MTAlogbase='10' RefineCorners='0' WithPoseNlls='1'/>
    </MarkerTrackerAction>
    <TrackedObject2CameraAction category='Action' enabled='1' name='TrackedObject2Camera'>
      <Keys size='3'>
        <key val='World' what='world, World*, in'/>
        <key val='IntrinsicDataPGRFlea8mm' what='intrinsic CameraPerspective parameters, IntrinsicDataPerspective*, in'/>
        <key val='Camera' what='suffix string for the CameraPerspective, out'/>
      </Keys>
    </TrackedObject2CameraAction>
  </ActionPipe>

In the DataSet you can see an IntrinsicData element and a World with one TrackedObject.

Code: The DataSet

  <DataSet key=''>
    <IntrinsicDataPerspective calibrated='1' key='IntrinsicDataPGRFlea8mm'>
      <!--Image resolution (application-dependant)-->
      <Image_Resolution h='480' w='640'/>
      <!--Normalized principal point (invariant for a given camera)-->
      <Normalized_Principal_Point cx='5.0037218855e-01' cy='5.0014036507e-01'/>
      <!--Normalized focal length and skew (invariant for a given camera)-->
      <Normalized_Focal_Length_and_Skew fx='1.6826109287e+00' fy='2.2557202465e+00' s='-5.7349563803e-04'/>
      <!--Radial and tangential lens distortion (invariant for a given camera)-->
      <Lens_Distortion k1='-1.6826758076e-01' k2='2.5034542035e-01' k3='-1.1740904370e-03' k4='-4.8766380599e-03' k5='0.0000000000e+00'/>
    </IntrinsicDataPerspective>
    <World key='World'>
      <TrackedObject key='TrackedObject1'>
        <ExtrinsicData calibrated='0'>
          <R rotation='1 0 0'/>
          <t translation='0 0 0'/>
          <Cov covariance='0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0'/>
        </ExtrinsicData>
        <Marker BitSamples='2' MarkerSamples='6' NBPoints='4' key='MarkerOfTrackedObject1'>
          <Code Line1='1000' Line2='1000' Line3='1000' Line4='1110'/>
          <Points3D nb='4'>
            <HomgPoint3Covd Cov3x3='0  0  0  0  0  0  0  0  0  ' w='1' x='0' y='5' z='0'/>
            <HomgPoint3Covd Cov3x3='0  0  0  0  0  0  0  0  0  ' w='1' x='5' y='5' z='0'/>
            <HomgPoint3Covd Cov3x3='0  0  0  0  0  0  0  0  0  ' w='1' x='5' y='0' z='0'/>
            <HomgPoint3Covd Cov3x3='0  0  0  0  0  0  0  0  0  ' w='1' x='0' y='0' z='0'/>
          </Points3D>
        </Marker>
      </TrackedObject>
    </World>
  </DataSet>

We add a second TrackedObject (TrackedObject2) with the marker code (1000), (0110), (0000), (0000). This describes the second marker which has a white field in the first row and two white fields in the second row.

Code: The second TrackedObject

      <TrackedObject key='TrackedObject2'>
        <ExtrinsicData calibrated='0'>
          <R rotation='1 0 0'/>
          <t translation='0 0 0'/>
          <Cov covariance='0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0'/>
        </ExtrinsicData>
        <Marker BitSamples='2' MarkerSamples='6' NBPoints='4' key='MarkerOfTrackedObject2'>
          <Code Line1='1000' Line2='0110' Line3='0000' Line4='0000'/>
          <Points3D nb='4'>
            <HomgPoint3Covd Cov3x3='0  0  0  0  0  0  0  0  0  ' w='1' x='0' y='5' z='0'/>
            <HomgPoint3Covd Cov3x3='0  0  0  0  0  0  0  0  0  ' w='1' x='5' y='5' z='0'/>
            <HomgPoint3Covd Cov3x3='0  0  0  0  0  0  0  0  0  ' w='1' x='5' y='0' z='0'/>
            <HomgPoint3Covd Cov3x3='0  0  0  0  0  0  0  0  0  ' w='1' x='0' y='0' z='0'/>
          </Points3D>
        </Marker>
      </TrackedObject>

This is all we need to change in the InstantVision configuration file. Now we open the file TutorialMarkerTracking_OneMarker.x3d and add fields for the second object's translation and rotation to the IOSensor.

Code: Adding the fields for the second tracked object to the IOSensor

<field accessType='outputOnly' name='TrackedObject2Camera_Position' type='SFVec3f'/>
<field accessType='outputOnly' name='TrackedObject2Camera_Orientation' type='SFRotation'/>

Then we add the second teapot to the scene and set the routes needed for its translation and orientation.

Code: Adding the fields for the second tracked object to the IOSensor

<Transform DEF='TransfObj2RelativeToCamPosition' translation='0 0 0' invert='TRUE'>
    <Transform DEF='transfObj2RelativeToMarker' translation='2.5 2.5 1.5' rotation='1 0 0 1.57'>
        <Shape>
            <Appearance>
                <Material emissiveColor='0 0.5 1' />
            </Appearance>
            <Teapot size='5 5 5' />
        </Shape>
    </Transform>
</Transform>        

<ROUTE fromNode='VisionLib' fromField='TrackedObject2Camera_Position' toNode='TransfObj2RelativeToCamPosition' toField='translation'/>
<ROUTE fromNode='VisionLib' fromField='TrackedObject2Camera_Orientation' toNode='TransfObj2RelativeToCamPosition' toField='rotation'/>

We can now track both markers and augment them with a yellow and with a blue teapot.

Image: TutorialMarkerTracking_TwoMarkers.jpg

Files:

Comments

This tutorial has no comments.

Add a new comment

Due to excessive spamming we have disabled the comment functionality for tutorials. Please use our forum to post any questions.