There is no direct access to the stream. Instead you can choose to render passthrough layers, but have no access to the actual images. From the Oculus developer blog:
"We built Passthrough API with privacy in mind. Apps that use Passthrough API cannot access, view, or store images or videos of your physical environment from the Oculus Quest 2 sensors. This means raw images from device sensors are processed on-device."
So you could do something like show the real life floor around a player, but not do things like fiducial marker tracking or depth mapping.
Yeah - regarding the marker tracking, I'm thinking the camera resolution isn't high enough to read the markers anyway, unless they are pretty large. But, if you look at the video where he draws the box in white lines, his arm occludes the lines behind it - I'm guessing there will be some sort of depth capabilities, even if it's just handled internally by the API. In general, I wasn't expecting true, high rez AR capabilities... I know the hardware wasn't built for it (hopefully the next one will!). I'm just excited to finally get to play around with the basics - y'know... put a virtual dartboard on the wall, throw some virtual darts at it - and similar simple experiments. I have a ZED Mini camera that I mounted to the front of my Rift S, but there were so many issues trying to get it all to work together that I eventually stopped going down that road.
I've got a Zed Mini too, but haven't really used it for AR yet, I was mainly using it as a Kinect alternative for depth scanning. 🙂
Good call on the hand occlusion bit on that page. I wonder if that's a generic depth layer (which Quest can do a bit since it's needed to unwarp the passthrough view) or if they just used the hand tracking to generate an occluder.
Yep, fiducial trackers would need to be rather big, but there's also the chance of making a simple IR LED pattern that is visible further away.
I've done some long distance tracking testing of the Quest, such as walking 170m and back, the tracking drifted by 5.8m. Having some markers would help multiple headsets exist in one shared space by having a definite coordinate origin.