It's my personal opinion that AR and VR eventually should to merge into the same HMD, where pass through is the AR mode of the HMD. The major advantage of this is that we could get away from the current way AR does additive color. In AR, transparent is "black" and you don't actually get black, so if you ever have textures with a lot of darker colors, they would present very poorly in AR. In current generations of AR apps, you will only see brightly lit colors to get around this problem with additive light. I think the pass through AR would be the real fix. There's going to be some form factor problems for such a future device I think? From what I can see, the pass through image in the rift S is monochrome and a bit grainy, so it wouldn't be a very good AR solution. We'd really need an RGB camera which can handle multiple light conditions without grainy image quality in order to be viable as a reasonable candidate for pass through AR.
The Oculus sdk doesn't expose the passthrough video (probably for privacy reasons), same as you can't get access to the CV1 sensor video. So anybody trying to do it would need to intercept usb data and decode it. Most probably can't be bothered.
While its expensive, a Zed Mini would give much better results. I've got one, its pretty cool (I use it mostly for depth scanning though, I haven't done much with its AR)
Comments
From what I can see, the pass through image in the rift S is monochrome and a bit grainy, so it wouldn't be a very good AR solution. We'd really need an RGB camera which can handle multiple light conditions without grainy image quality in order to be viable as a reasonable candidate for pass through AR.
Hardware: Threadripper 1950x, MSI Gaming Trio 2080TI, Asrock X399 Taich
Headsets: Wrap 1200VR, DK1, DK2, CV1, Rift-S, GearVR, Go, Quest, Quest 2, Reverb G2