MS Kinect as Input Device — Oculus
Welcome to the Oculus Developer Forums!

Your participation on the forum is subject to the Oculus Code of Conduct.

In general, please be respectful and kind. If you violate the Oculus Code of Conduct, your access to the developer forums may be revoked at the discretion of Oculus staff.

MS Kinect as Input Device

knchaffinknchaffin Posts: 49
Brain Burst
I have some MS Kinect for Windows devices along with the SDK. So far I have only used and studied the sample programs, but from what I see, I think the Kinect could make a very capable gesture control input to Oculus apps. I think it could be programmed to recognize arm and hand positions and movements in 3D space.

Comments

  • LaurentMTLaurentMT Posts: 57
    Hi,

    I'm also interested in investigating this kind of usage (Rift + Kinect).
    I know Kinect is not very popular in the VR community because of the latency but I still want to test it farther.

    I already started a project on the subject during my spare time. First results with Kinect alone are quite positive to my own opinion. Now, I have to wait for the arrival of my dev kit (surely not before may or june, since I'm a preorder) but I try to anticipate the eventual problems to do fusion of data coming from Kinect and Rift.

    If I can get some good results, I'll share work and sources with everybody.

    I'll keep you in touch.

    Laurent
  • dalootsdaloots Posts: 94
    Brain Burst
    Hi,

    +1 on being interested.

    I've done some stuff with the Kinect in the past. Latency has gotten better, but skeleton detection is still pretty slow. I think the depth map itself has much lower latency. Perhaps that could be used for relatively fast positional tracking in a restricted setup. Something like using the closest point as position.
    I'm a final-day kickstarter backer from Europe, so will be waiting some time for my dev kit as well...

    Cheers,
    Davy
    True 3D video for VR: http://q3d.quaternionsoftware.com
  • tbowrentbowren Posts: 19
    I recently got my Kinect working in Ogre, and my Ogre scenes to read the Oculus tracking data. Once I have the shader warping and viewport frustum adjustments working I will be able to put my oculus head on top of the kinect body. I am sure its horribly limited as to what I can do but I'm very anxious to try this out.
  • tobbytobby Posts: 7
    This guy has already done it:

    http://www.youtube.com/watch?v=R0-dsbeasgA
  • leelovellleelovell Posts: 14
    That video is cool.

    So much potential.
  • CheshyrCheshyr Posts: 74
    That was amazing. It's not that far off either, except perhaps the omni-treadmill. He had some good points.
  • LaurentMTLaurentMT Posts: 57
    Thanks for the video tobby ! I've had already seen a few things done by this guy (okreylos) via the openkinect community and for sure he's a good source of inspiration for VR.
  • LaurentMTLaurentMT Posts: 57
    I just did a quick test of a virtual body in Unity 4 with body tracking done by kinect.
    For this test, I've used this library. Basically, the library is a Unity wrapper for microsoft kinect sdk and it provides a class which wraps the skeleton tracking done by kinect.

    The result is fun.
    BHM1fReCAAELwLx.png:large
    But to be honest this simple solution (kinect skeleton tracking) won't provide very good results in current state (instability in hands detection, no rotation of arms and hands, ...). Anyway, I'll pursue my tests.

    I've also worked on a lean based navigation system using Kinect. The principle is quite similar to this demo using a wii balance board. Results seem quite promising but I still have to check with dev kit that it doesn't induce simulator sickness. :roll:

    For now, my thoughts are that a good (and affordable) set could be :
    - the rift dev kit for display and angular tracking of head,
    - a razer hydra for positional tracking of head (I think I'll test this solution)
    - a lean based navigation system :
      - wii balance board (cheaper) - kinect (more expensive but could provide some additional data for body tracking)
    - additional razer hydra used as a hand tool.
  • knchaffinknchaffin Posts: 49
    Brain Burst
    Michaeljohn, I have no idea what you are saying here.
  • svensven Posts: 339
    Hiro Protagonist
    it's a spam bot...
  • ZeRaxZeRax Posts: 12
    NerveGear
    Have you tested this:

    OpenNI (standard framework for 3D sensing) API and 3D hand tracking library and framework uses OpenCL or CUDA:
    http://www.openni.org/files/3d-hand-tracking-library
    http://cvrlcode.ics.forth.gr/handtracking

    zigfu development kit for Unity 3D works with OpenNI:
    http://www.openni.org/files/zdk-for-unity3d

    http://www.youtube.com/watch?v=N3ffgj1bBGw

    LaurentMT wrote:
    Hi,

    I'm also interested in investigating this kind of usage (Rift + Kinect).
    I know Kinect is not very popular in the VR community because of the latency but I still want to test it farther.

    I already started a project on the subject during my spare time. First results with Kinect alone are quite positive to my own opinion. Now, I have to wait for the arrival of my dev kit (surely not before may or june, since I'm a preorder) but I try to anticipate the eventual problems to do fusion of data coming from Kinect and Rift.

    If I can get some good results, I'll share work and sources with everybody.

    I'll keep you in touch.

    Laurent
  • LaurentMTLaurentMT Posts: 57
    I haven't tested these solutions yet but some could be interesting (the hand tracking is quite impressive).

    I had to pause this work for a while (unexpected long stay at hospital) but I will continue as soon as possible. At the moment, my priority is the lean based navigation system. The navigation works fine but I noticed a delay induced by skeleton detection : It takes around 20-30ms to have a skeleton returned by the ms kinect sdk.
    The library that I use calls the sdk into the game loop (in Update method). So, using this method FPS is around 30fps at its best which is far from the target 60fps required for VR.
    According to me, computation of skeletons should be parallelized with game loop in order to reduce this latency.
    I have started to rewrite the library to bypass this limit. Still a work in progress but I've done some progress.
  • knchaffinknchaffin Posts: 49
    Brain Burst
    I finally received notice that my Rift is being processed for shipping. So, hopefully I will be able to do some actual development with the Rift and my Kinects soon. I had to take a break from dreaming as I was getting too frustrated that I did not have a Rift in hand.

    I bought this for my personal use, but everyone in my scientific visualization research lab is anxiously awaiting me bringing my Rift to the lab.
  • derivenderiven Posts: 1
    I just got my Kinect in last night. Using zigfu's Kinect Unity3D package, it only took 15-minutes to get a demo up-and-running. I wanted to add the Razer Hydra; but for some reason the sixense drivers aren't playing well. (I think I need Unity Pro).

  • Looks awesome! You were able to lean back pretty good without losing balance/orientation, so I guess it felt quite real :lol:. How was the latency, did you notice it?
  • knchaffinknchaffin Posts: 49
    Brain Burst
    I received my Oculus Rift DK a couple of weeks ago. I then bought Unity Pro and have been having a blast doing C# scripted procedural geometries and outer space simulation scenes. I need to put the Kinect on my list of things to play with! I like what I am seeing here.
  • LaurentMTLaurentMT Posts: 57
    Hi everybody,

    Since my last post, I've made some good progresses.
    I've adapted the kinect library in order to keep a good framerate and to add a few things. For now, I've got : a lean based input controller, avatareering and a kind of "neck tracking" (in order to synchronize the position of the OvrCameras with the avatar). All this stuff is still very rough and need more work but it's already fun and provides quite good results.

    I received my Rift on monday. In order to test the integration of the Rift with Kinect, I thought it would be fun to get a kind of hoverboard controller (yes, like Marty Mc Fly :roll: ).

    Here's the result of my first tests.



    To be honest, the first sessions were like learning again to ride a bicycle or do skiing. But the very positive result is that even if the controller needs to be really improved, I feel less motion sickness during these tests than when I try some demos while seated.
  • LaurentMTLaurentMT Posts: 57
    Second test of the hoverboard controller in the wild.
    The environment is 25 square kilometers of moutains, rifts, cliffs, dunes, ...
    Perfect environment to cure your vertigo !
    It's also my first try to build a scene with Unity. I'm a real noob, so it's low poly and textures are :shock:

  • cyberealitycybereality Posts: 26,156 Oculus Staff
    Cool. Looking better.
    AMD Ryzen 7 1800X | MSI X370 Titanium | G.Skill 16GB DDR4 3200 | EVGA SuperNOVA 1000 | Corsair Hydro H110i
    Gigabyte RX Vega 64 x2 | Samsung 960 Evo M.2 500GB | Seagate FireCuda SSHD 2TB | Phanteks ENTHOO EVOLV
  • I'm developing an application which uses Unity3D + Oculus + Kinect (the Holy Trinity, FTW!), but when I use the Kinect the framerate gets limited to 30, which makes the Oculus experience subpar. It's not a performance issue. So my question is:

    How did you adapt the Kinect library in order to achieve a better framerate?

    Any help will be greatly appreciated!
  • I am curious has anyone tried this with Directx (Direct3D) instead, I am heading that way, would be happy to collaborate. DirectX although might be difficult to develop a high definition graphics game in (not saying its impossible, but time consuming) , but it might be better for 3D application design, which can be converted to good serious simulations, since Direct3D offers much more fine grained control.
    Or maybe I am doing something really stupid in which case please feel free to mock my approach around.
  • I'm using Kinect 2 with the Oculus. I haven't had noticeably issues with the Oculus performance with the Kinect going, and it runs over 60 fps (although I think the Kinect updates at 30). I am running a 4 Ghz Devil's Canyon and AMD R290. There is some lag with the Kinect, and the amount of lag is a trade-off with smoothing, but it all seems good enough to be really fun.
Sign In or Register to comment.