IMPORTANT:

All Oculus Quest developers MUST PASS the concept review prior to gaining publishing access to the Quest Store and additional resources. Submit a concept document for review as early in your Quest application development cycle as possible: https://developer.oculus.com/quest-pitch/

For additional information and context, please see "Submitting Your App to the Oculus Quest Store".
Welcome to the Oculus Developer Forums!

Your participation on the forum is subject to the Oculus Code of Conduct.

In general, please be respectful and kind. If you violate the Oculus Code of Conduct, your access to the developer forums may be revoked at the discretion of Oculus staff.

Mixed Reality Capture Tools (2.0) not working on Quest and test app crashes to ~1fps

Hi, I'm trying to set up the Mixed Reality Capture Tools to be able to record a demo with the Oculus Quest. I've managed to calibrate a USB camera and uploaded the config onto the device under my app's files/ directory and I have OBS set up with the 'Oculus MRC' fg/bg sources as documented. I can connect to the device via OBS once I start my application but when I do I seem to just get one frame grabbed by OBS and the app framerate suddenly drops down to about one frame per second, with no visible updates in OBS beyond the first frame.

I wonder if anyone else had any luck using the latest Mixed Reality Capture tools (Oculus_MRC_Tools_2.0/1.41.0_RC1, OBS 23.2.1, Unity Integration Asset 1.40) and can maybe give some hints for what I might need to do to get it working?

Comments

  • cly3dcly3d Posts: 69 Oculus Start Member
    This video was uploaded today: https://www.youtube.com/watch?v=p1c8D_l0upI   
    Hope its of help in some way.
  • ribrib Posts: 5
    NerveGear
    Thanks, it's a good overview of the technology and the setup process, though I think, afaikt, I did everything as described.

    One thing that caused me some trouble for a while was that I was using OBS 24.x.x to start with and it turns out that the current plugin binaries are tied to the OBS 23.x.x ABI which wasn't immediately obvious.

    I'll maybe take another stab and see if I have any luck.

  • Lazius001Lazius001 Posts: 8
    NerveGear
    Rib , i might be able to help but am stuck at the last hurdle, am not able to get my obs to connect the mrc.xml files.
    1.Do you save the camera file to mrc.xml after doing a chroma or after finish calibration only during camera calibration(so before clicking chroma)?

     1b.I assume we can close the camera calibration window after saving the mrc.xml file?

    2. To transfer mrc.xml file to quest, you put {youroculusdeveloper name}.{youroculusappnane}.files or .{youroculususername}.{beatsaber}.files. (example);

    3. When I open OBS after putting all the plugging folders in the root, I click the + in the source, I see OculusMrcSource, do you see that too or do you see Oculus MRC? 

    3b. The video capture source I assume was same as used during the camera calibration process, we don't use the screen (if left open) of the chroma from the canera calibration window from previous steps, correct? As mentioned I closed that window after saving the mrc.xml file as we are told to do chroma from obs, not calibration in previous steps, is that correct?

    4, after i apply the filters pad/crop and change the relative settings, my fg abd bg dont change blue or green, so am stuck there,

    4c, when i click connect after changing the ip in the OculusMrcSources, nothing happens.

    I tried a lot of variations and would really appreciate your help on this. Please let me know if you able reply here or wish a Skype or messenger chat, or discord.

    Thx 
    Laz
  • Lazius001Lazius001 Posts: 8
    NerveGear
    Just update to say, all works, but just still enable to connect obs to the quest only... (Plugin says Oculus MRC and have the blue and green in green after activation of plugin).
  • ribrib Posts: 5
    NerveGear
    Hey, I haven't gotten around to trying again yet, but a few notes...

    It looks like mrc.xml does store some chroma key info, so I suppose save as late as possible - though not sure why the device needs the chroma key of the camera?

    Yeah, you have to close the calibration window so OBS can access your camera. Note: It will be important to ensure you configure your videos source / camera in OBS with the same resolution as was used in the calibration tool.

    This is how I uploaded my calibration file:
    adb shell mkdir -p /sdcard/Android/data/co.realfit.RealFit/files
    adb push mrc.xml /sdcard/Android/data/co.realfit.RealFit/files

    Theoretically for BeatSaber I think it would go here:
    adb push mrc.xml /sdcard/Android/data/com.beatgames.beatsaber/files/

    Out of curiousity I tried connecting to BeatSaber too but it seems like they don't support Oculus Quest mixed reality yet ;-p (couldn't connect) - hopefully they will support it in an update at some point.

    In OBS the source I see is listed as 'Oculus MRC'. (Btw, I downgraded my OBS installation to 23.2.1 to get the plugin working for me).

    Yeah the chroma keying is done in OBS (I don't really know what the purpose of the chroma key functionality in the calibration tool is).

    For the fg/bg setup, can you maybe first confirm that if you look at the properties for the MRC sources then you see a wide rectangle that's blue on the left and green on the right? Essentially the cropping is cropping is cutting the left side of this source as the background and the right hand side as the foreground. If I just show the fg group I see green, and if I just show the bg group I see blue.

    I guess the fact that it doesn't connect implies that either implies that your mrc.xml is not in the correct location for the application to pick up and enable MR connections, or perhaps that application doesn't yet support this feature (which I think is the case for BeatSaber currently). Have you perhaps tried testing this with a custom Unity project with the 1.40 Oculus Integration?

    Hope some of those comments might possibly help. I'll try and take another stab myself at getting this working soon and try and share anything I learn.

    Best of luck!



  • Lazius001Lazius001 Posts: 8
    NerveGear
    Hey Rip, thanks for your reply.

    Yea I see fg/bg green/blue, all good there..i also downgraded and used 1.40 file set as 1.41 didn't work... The part am confused is where they tell us to put this
    adb shell mkdir -p /sdcard/Android/data/com.{yourorgname}.{yourappname}/filed
    
    So you mean, the name is any name we want, or facebook linked name... org name we create on oculus developer account
     what about the yourappname, what if I just wanted to record mrc content, not develop an app, how do you access the yourappname?, We put any name?
    
    For adb push, we suppose push to our quest, first step and second step, 
    according to instructions, we suppose keep same yourorgname.yourappname,
     then supposably suppose open app after finishing obs installation and then 
    click connect, should load beatsaber...so unsure why put the above
    com.beatgames.beatsaber
    
    someone on YouTube who posted beatsaber with quest...
    but unsure of authenticity or if he had private link to their account...or
    developed his own way to connect...
  • Lazius001Lazius001 Posts: 8
    NerveGear
    Here's his link 
  • Lazius001Lazius001 Posts: 8
    NerveGear
    Or you mean for yournameorgname.yourappname, we put the name_of_any_developer.theirappname of app we want to connect too, so not necessarily our nameorgname.our_own_app name if we dont have an app?

    Maybe that's where I got confused...
  • ribrib Posts: 5
    NerveGear
    hmm, curious, maybe BeatSaber does have mixed reality support (or they used something else like LIV if that maybe also supports Quest somehow?). I got the impression from the video linked above (from what the devs said) and from trying to connect to BeatSaber from OBS myself that it didn't yet ship with the necessary support.

    For the naming; it's the name that corresponds to the specific application that you want to connect to (that must have this feature integrated first). It relates to the physical location where the application is installed on the device. For BeatSaber I found that it was installed under /sdcard/Android/data/com.beatgames.beatsaber. If you make your own Unity project then the default name is com.DefaultCompany.NewUnityProject. If you go to the Player Settings and look under the Android settings there is a Package Name, where it can be changed.

    Whatever you do in Unity is only going to be related to recording your own developed application that you can side load onto the Quest. This new technology from Oculus for mixed reality with Quest requires all game developers to first integrate their MR technology into their game before it's possible to make recordings. Unfortunately it doesn't let you modify or record existing applications until their developers release an update with this extra functionality integrated. So for example the only way to record BeatSaber in mixed reality this way will be if they have released an update that enables this feature (and maybe they have already done that, but I didn't think they had yet).

    Hope that makes sense - sorry if I have any wires crossed though :)

  • XRAlXRAl Posts: 5
    NerveGear
    rib said:
    Thanks, it's a good overview of the technology and the setup process, though I think, afaikt, I did everything as described.

    One thing that caused me some trouble for a while was that I was using OBS 24.x.x to start with and it turns out that the current plugin binaries are tied to the OBS 23.x.x ABI which wasn't immediately obvious.

    Thank you! I was struggling for a while with 24.x not seeing the Oculus MRC camera. Seems like the official docs should clearly state the version needed.
  • Lazius001Lazius001 Posts: 8
    NerveGear
    Well until developpers integrated Oculus MRC, then I think its best, as a content creator for me to wait until the Oculus Link update is out in November then we can integrate with Steam as LIV is still working on wireless solutions and Mixcast also don't have wireless for now..

    Oculus Demo also showed Bestsaber on quest mrc, so not sure...

    I'll ask Oculus list of titles available, and see if we can share it as I can't find online.

    Thx for the replies, clarified all points.
  • XRAlXRAl Posts: 5
    NerveGear
    edited October 2019
    Yeah in the OC6 session they mention they are "working with" Beat Saber, Dead And buried 2 and super hot



    I tried copying my camera xml config to the com.beatgames.beatsaber/files location and it doesnt connect with OBS Studio and i ran a port scanner on there and also my own built app and can not even see the port listening and when running the below adb command for beatsaber I can clearly see its not implemented yet as it doesnt even try to start the camera capture in the app.

    When i check to see if its running on my own app I get:

    ./adb logcat -s OVRPlugin
    10-02 15:11:36.036 19860 19884 I OVRPlugin: [MediaManager] MRC camera found
    10-02 15:11:36.036 19860 19884 I OVRPlugin: [MediaManager] MrcConfig: enabled 1 hq_mode 0, frame 2560 x 1080 image 1920 x 1080 tcp_port 28734 debugcamera 0 bitrate 8000000
    10-02 15:11:36.036 19860 19884 I OVRPlugin: [MediaManager] Listening at 0.0.0.0 tcp_port 28734
    10-02 15:11:36.074 19860 19884 D OVRPlugin: [MediaManager] SetMrcActivationMode 0 listening 1
    10-02 15:11:36.817 19860 19884 I OVRPlugin: [MediaManager] deconstruct
    10-02 15:11:36.817 19860 19884 I OVRPlugin: [MediaManager] Listening stopped
    10-02 15:11:43.076 19860 19908 D OVRPlugin: Leaving VR mode.
    10-02 15:11:43.638 19860 19908 D OVRPlugin: Unregistering utility receivers.
  • osxiaoyuosxiaoyu Posts: 2
    NerveGear
    About Mixed Reality Capture Tools, only one program of skybox vr can be recognized in obs. Other applications cannot be recognized. How should I operate?
  • petergrahampetergraham Posts: 1
    NerveGear
    Today I tried for scene composition of MixedRealityCapture in OBS. I have been using a 1920 * 1080 resolution monitor for it. In OBS the output I get from sources like foreground, background and video capture I wasn't able to resize them without getting the output cropped/cut. On a 4K monitor, I am getting output as expected. Am I missing any option to resize the sources without getting the output cropped ?
  • LinsomontLinsomont Posts: 17
    NerveGear
    Thanks for sharing your views. I was looking for the answer to this question too and you guys provided me with some good ones. Thank you very much.
Sign In or Register to comment.