cancel
Showing results for 
Search instead for 
Did you mean: 

Accessing Video-Passthrough

Homunkoloss
Honored Guest
Are there any plans on letting Devs access the Quest's Video-Passthrough capabilities via SDK? I'd love to mess around with AR / MR Applications. I know that this is possible on the Vive Pro utilizing the SRWorks SDK but that requires to be wired. Having the same functionality but wireless would be a huge gamechanger for me.



8 REPLIES 8

JeffNik
MVP
MVP
I have been excited about this as well, but on the Rift S. I haven't heard from anyone about whether they will add this functionality to the SDK or not. I've found a few articles about concerns for privacy and that its  computationally expensive to provide the video feed.

MikeF
Trustee
been asked a hundred times, not going to happen

airmouse
Protege
Why not?

JeffNik
MVP
MVP

Wow! They're finally going to give us access to the passthrough stream to do AR! 

 

Oculus Quest v31 Adds Experimental Passthrough API (uploadvr.com)

Kind of.

There is no direct access to the stream. Instead you can choose to render passthrough layers, but have no access to the actual images. From the Oculus developer blog:

"We built Passthrough API with privacy in mind. Apps that use Passthrough API cannot access, view, or store images or videos of your physical environment from the Oculus Quest 2 sensors. This means raw images from device sensors are processed on-device."

https://developer.oculus.com/blog/mixed-reality-with-passthrough/

 

So you could do something like show the real life floor around a player, but not do things like fiducial marker tracking or depth mapping.

Yeah - regarding the marker tracking, I'm thinking the camera resolution isn't high enough to read the markers anyway, unless they are pretty large. But, if you look at the video where he draws the box in white lines, his arm occludes the lines behind it - I'm guessing there will be some sort of depth  capabilities, even if it's just handled internally by the API. In general, I wasn't expecting true, high rez AR capabilities... I know the hardware wasn't built for it (hopefully the next one will!). I'm just excited to finally get to play around with the basics - y'know... put a virtual dartboard on the wall, throw some virtual darts at it - and similar simple experiments. I have a ZED Mini camera that I mounted to the front of my Rift S, but there were so many issues trying to get it all to work together that I eventually stopped going down that road.

 

kojack
MVP
MVP

I've got a Zed Mini too, but haven't really used it for AR yet, I was mainly using it as a Kinect alternative for depth scanning. 🙂

Good call on the hand occlusion bit on that page. I wonder if that's a generic depth layer (which Quest can do a bit since it's needed to unwarp the passthrough view) or if they just used the hand tracking to generate an occluder.

 

Yep, fiducial trackers would need to be rather big, but there's also the chance of making a simple IR LED pattern that is visible further away.

I've done some long distance tracking testing of the Quest, such as walking 170m and back, the tracking drifted by 5.8m. Having some markers would help multiple headsets exist in one shared space by having a definite coordinate origin.

JeffNik
MVP
MVP

They just released the v31 Unity Integration package! Downloads (oculus.com)