New to the forums? Click here to read the "How To" Guide.

Developer? Click here to go to the Developer Forums.

Access the Passthrough Camera via API

ArghzivalArghzival Posts: 10
Brain Burst
Itd be great if we could utilize the passthrough camera to create lightweight AR experiences. Even something like having the passthrough camera on as the backdrop while in Oculus Home. This would allow me to have multiple windows open at once (now possible in v15) giving me a seemingly limitless AR desktop experience!
Would also be great to allow developers to access and use the passthrough camera within apps.

Comments

  • BraydarBraydar Posts: 2
    NerveGear
    I would be horrified if they ever gave developers access to the entire video contents of my home. 
  • Umpa_PCUmpa_PC Posts: 835
    Neo
    I think they are the wrong type of cameras for that application - they are tracking cameras, that's why the picture from them is lousy.
    Oculus Rift S - Oculus Quest 128GB
    MSI trident 3 7RB-200UK Intel Core i5-7400 3 Ghz x2
    MSI GTX 1050 Ti (4GB) & MSI Aero GTX 1060 OC (6GB) & MSI Aero GTX 1070 OC 8GB
    16 GB RAM x2, 1TB HDD x2, 1TB SSD x2
    Windows 10 Home Edition Version 10.0.18363 Build 18363, Oculus version 17, Quest Version 17
    Fan Cooling by Zotac FireStorm - AfterBurner cause me problems.
  • kojackkojack Posts: 7,216 Volunteer Moderator
    Unfortunately they won't allow it, as it's a privacy issue. People freak out enough about having cameras on (or around) a headset that only Oculus has access to.
    If we had access to the cameras, it would be possible to do fiducial marker tracking, which would allow multiple Quests to do co-location games using a common coordinate system

    For passthrough AR it's better to use a dedicated addon like a Zed Mini (colour stereoscopic cameras that stick to the front of your headset). But it requires a Nvidia GPU to work, so can't run on a Quest.
    (Well, if you had two link cables it would work as a PC headset with AR, the Zed Mini also needs a USB-C cable as long as the headset cable)

    Author: Oculus Monitor,  Auto Oculus Touch,  Oculus Forum Tool,  Phantom Touch Remover,  Forum Dark Mode,  X-Plane Fixer
    Hardware: Threadripper 1950x, MSI Gaming Trio 2080TI, Asrock X399 Taich
    Headsets: Wrap 1200VR, DK1, DK2, CV1, Rift-S, GearVR, Go, Quest, Quest 2, Reverb G2
  • Umpa_PCUmpa_PC Posts: 835
    Neo
    And after all that, the quality would be crap right ?
    Oculus Rift S - Oculus Quest 128GB
    MSI trident 3 7RB-200UK Intel Core i5-7400 3 Ghz x2
    MSI GTX 1050 Ti (4GB) & MSI Aero GTX 1060 OC (6GB) & MSI Aero GTX 1070 OC 8GB
    16 GB RAM x2, 1TB HDD x2, 1TB SSD x2
    Windows 10 Home Edition Version 10.0.18363 Build 18363, Oculus version 17, Quest Version 17
    Fan Cooling by Zotac FireStorm - AfterBurner cause me problems.
  • frx089frx089 Posts: 1
    NerveGear
    @Braydar I do not understand the privacy concerns, is not the same of having an android phone and a camera? don't you have a camera app on your phone? 
  • JeffNikJeffNik Posts: 182
    Art3mis
    I just want to use the passthough background for an AR-like app ... I don't need accurate 3D data about my playspace, or even high quality video... I just want to see my real environment behind my 3D generated content that floats in the air in front of the user. In my opinion, the video distortion actually ADDs a "virtually generated" feel to the app. I understand the privacy concerns, but programmers are eventually going to get access to passthough video for AR apps on other hardware.
  • wuzpwuzp Posts: 516
    Neo
    Doesn't this forum have a "developer's side," with more qualified people that can answer the question?
  • BrizbaneBrizbane Posts: 1
    NerveGear
    JeffNik said:
    I just want to use the passthough background for an AR-like app ... I don't need accurate 3D data about my playspace, or even high quality video... I just want to see my real environment behind my 3D generated content that floats in the air in front of the user. In my opinion, the video distortion actually ADDs a "virtually generated" feel to the app. I understand the privacy concerns, but programmers are eventually going to get access to passthough video for AR apps on other hardware.

    I'd just like to have a way to pause what I'm doing, go to pass through and take a sip of coffee, smoke a cig, etc.  Without having to take the headset off. 
  • wuzpwuzp Posts: 516
    Neo
    Brizbane said:
    I'd just like to have a way to pause what I'm doing, go to pass through and take a sip of coffee, smoke a cig, etc.  Without having to take the headset off. 
    Not sure if the cigar[ette] being sucked into the unit by the cooling fan will be very helpful [or not].
  • EpicOculiEpicOculi Posts: 2
    NerveGear
    "They won't allow it." Umm excuse me but shouldn't that be a personal decision - just like allowing access to the microphone or photos? Put it under a privacy menu "Enable Passthrough In Apps" and we can enable or disable it ourselves ^^.
  • KNP54KNP54 Posts: 304
    Nexus 6
    What about point cloud data? I have this idea to develop a game that could use this data to create a virtual environment with what the Quest sees. Imagine walking in VR but with actual objects around your house!
    Specs: Delidded i7-8086k @ 5.0 GHz all cores, NZXT X52 cooler, 32GB G.Skill 3200MHz, ASRock Z370 Gaming-ITX/ac, Asus GTX 1080 Strix, Corsair SF600 platinum, Samsung 970 EVO M.2 NVMe 1TB, Samgung 850 EVO 500GB + 840 EVO 500GB in raid0, Seagate Ironwolf 10TB, Louqe Ghost S1, Asus PB287Q 3840x2160 @ 60Hz, Samsung SyncMaster 226BW 1680x1050 @ 60Hz. Oculus Rift S.
Sign In or Register to comment.