New To The Forum? Click Here To Read The How To Guide. -- Developers Click Here.

Simple positional tracking with Kinect v2

dalootsdaloots Posts: 94
Brain Burst
edited November 2014 in General
Hi there,

I put together simple positional tracking using a Kinect v2*.
The new Kinect uses a time of flight method for the depth sensing. For this it emits a pretty strong IR light that produces a nice IR image. I stuck a small piece of reflective fabric to the front cover of the Rift and used the depth map to retrieve the location of the reflective marker.

The results are probably nowhere near those of the Crystal Cove prototype, but it's good enough to experiment with and couldn't be simpler. The new Kinect has a very wide field of view and a (preliminary) range of over 4 meter. Of course with the single marker setup, it loses tracking if you rotate your head more than 90° away from the Kinect.

Using the Kinect body tracking seems obvious, but I think this method has less latency, higher accuracy and the Kinect only needs to have line-of-sight to the marker.

http://www.youtube.com/watch?v=N46IWihUE1E


Cheers,
Davy

*Kinect for Windows v2 Dev Preview: This is based on preliminary software
and/or hardware, subject to change.
True 3D video for VR: http://q3d.quaternionsoftware.com

Comments

  • VRdopingVRdoping Posts: 112
    Hiro Protagonist
    This looks pretty sweet! How accurate does it feel when you're in the Rift?
    The demos I tried that used the Hydra for positional head tracking were not too accurate and didn't feel that good to me.
  • dalootsdaloots Posts: 94
    Brain Burst
    Hi VRdoping!
    It feels pretty good. Because I use a single pixel from the IR image for tracking, there is some slight jitter, so I use a lerp (smoothing) function of 100ms, which introduces some latency, but still pretty good. I have no way of accurately measuring the accuracy, but I think it's in the millimeter range.
    Since the Kinect tracks a point a few centimeter in front of the eyes, I should do some math to take the orientation of the headset into account for deducing the position of the actual eyes. But I'm currently too lazy to do that :D
    Cheers,
    Davy
    PS. thanks again for your help earlier ;)
    True 3D video for VR: http://q3d.quaternionsoftware.com
  • kojackkojack Posts: 3,649 Volunteer Moderator
    I wonder how much the Kinect 2 for PC will be when it's finally released to consumers?
    The dev version is $399 (and gives you 2 kinects, the dev one now and a consumer release one when it comes out).

    There's another 500 slots open for devs to buy one apparently.
  • dalootsdaloots Posts: 94
    Brain Burst
    As far as I know, they haven't released any information on pricing yet.
    I think the release is planned around the summer.

    You're right, there's another 500 spots available for developers. Register here:
    http://www.microsoft.com/en-us/kinectforwindowsdev/newdevkit.aspx
    True 3D video for VR: http://q3d.quaternionsoftware.com
  • mrjazzmrjazz Posts: 366
    daloots wrote:
    Because I use a single pixel from the IR image for tracking, there is some slight jitter, so I use a lerp (smoothing) function of 100ms, which introduces some latency, but still pretty good.
    After you have found the brightest pixel, I would compute a weighted average of a small area (for example 8x8 pixels) around this pixel. Perhaps this simple solution increases accuracy without having to do lerps.
    A more sophisticated method would be mean-shift tracking.
    Watch panoramic 3D photos, videos and do FPV-flying with LiveViewRift!
    viewtopic.php?f=28&t=11001
  • dalootsdaloots Posts: 94
    Brain Burst
    mrjazz wrote:
    After you have found the brightest pixel, I would compute a weighted average of a small area (for example 8x8 pixels) around this pixel. Perhaps this simple solution increases accuracy without having to do lerps.
    A more sophisticated method would be mean-shift tracking.
    Unfortunately it's a bit more complicated: the reflective piece itself doesn't yield a depth value (it's a blind spot for the depth sensor), so I first have to find pixels around it that do. But your point remains, I should use more pixels than 1 and I would probably no longer need the lerp.
    True 3D video for VR: http://q3d.quaternionsoftware.com
  • chaosgridchaosgrid Posts: 40
    Brain Burst
    hey,

    seems like the Kinect v2 has very stable and precise tracking..
    Did you come around testing the body tracking while using the Rift?
  • kevinw729kevinw729 Posts: 2,801
    Wintermute
    Good job @Daloots - I was just wondering, with the sale of the developers of the Kinect to Apple, will this still be an open platform?
    P6ftmuw.jpg
    ** New Book **
    "The Out-of-Home Immersive Entertainment Frontier: Expanding Interactive Boundaries in Leisure Facilities"
    https://www.amazon.co.uk/Out-Home-Immersive-Entertainment-Frontier/dp/1472426959
  • chaosgridchaosgrid Posts: 40
    Brain Burst
    Kinect v2 is developed by Microsoft in-house. It is a very open platform (MS Kinect SDK).

    PrimeSense, the developer of Kinect v1, is/was an Isreal-based company that has been bought by Apple.
  • SharkkuSharkku Posts: 574
    Art3mis
    I'd say the Kinect v2 will be the best way to put your body into the virtual world.
    DK1 - previously owned
    DK2 - 2014-08-26 RECEIVED (ordered 2014-04-24)
    CV1 - 2016-05-25 RECEIVED (ordered on release day)
    Touch - 2016-12-07 RECEIVED (pre-order with CV1)
  • Daloots,
    I had a few questions about your tracking in Kinect v2 using the IR and depth sensor. if you could shoot me and email at tylercork31@gmail.com I'd appreciate it. Also, I am an engineering student in college trying to implement this on a project.
  • dalootsdaloots Posts: 94
    Brain Burst
    Hi Tyler,

    I just sent you an email, but will also reply here for others that might find it useful.

    Here is the very quick & dirty (C#) code for the tracker. I consists of two parts: a "server" that extracts the position of one or two markers and sends it to a listener script for Unity.

    I've got two versions: one simple one for a single marker:
    https://dl.dropboxusercontent.com/u/722381/Kinect2/RiftPositionTracker.zip
    and the listener script for Unity:
    https://dl.dropboxusercontent.com/u/722381/Kinect2/UdpListener.cs.old (remove the .old of course)

    And a slightly more advanced one that uses two (one large and one small) markers on the same object for attitude estimation:
    https://dl.dropboxusercontent.com/u/722381/Kinect2/RiftPositionTracker2.zip
    and
    https://dl.dropboxusercontent.com/u/722381/Kinect2/UdpListener.cs
    The latter is demonstrated in this video:
    https://www.youtube.com/watch?v=3nermSaMpHA

    Good luck with the project!
    True 3D video for VR: http://q3d.quaternionsoftware.com
  • MoonShineVRMoonShineVR Posts: 1
    NerveGear
    Hello daloots,
    I am desperately need your help. I tried clicking the link you provided above, but it seems to be out of date. if you could shoot me and email at james2006gi@gmail.com I'd appreciate it.
    Thanks a lot!!!!!!!
  • pericoxpericox Posts: 1
    NerveGear
    edited September 13
    Hi Daloots,

    I am also wondering if there are still available to download those pieces of code. The links seems to be disconnected. I would really appreciate your help !! I am exactly trying to do the same marker tracking as you showed above. You can email me to zsd_a@hotmail.com.
    Thanks a lot in advance
Sign In or Register to comment.