Quest, RenderDoc, Vulkan, and UE 4.24.2

[nkitten]

Hello,

In the past, our team has been able to get RenderDoc captures from the quest for graphics debugging and analyzing performance, but having recently upgraded to the Oculus 4.24 Unreal branch with Vulkan enabled, we get a crash when trying to perform a capture, even with a grey-box test level. RenderDoc connects to the Quest and successfully launches the process, it only crashes on the capture itself.  I've tried both the 1.6 stable and most recent (2020/4/26) nightly build of RenderDoc, with the same result.  Is there an older version I should revert to?  I also tried setting `adb shell setprop debug.oculus.zram 1` in case of memory issues, but it didn't help.  Here is the crash callstack we get from the logs:

Build fingerprint: 'oculus/vr_monterey/monterey:7.1.1/NGI77B/655140.4720.0:user/release-keys'

Revision: '0'

ABI: 'arm'

pid: 17902, tid: 18209, name: RenderThread 5  >>> com.Sanzaru.NewGame <<<

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xdc

    r0 8cf04100  r1 00000000  r2 00000000  r3 00000000

    r4 97d49f00  r5 8d569558  r6 0000003a  r7 8d56a208

    r8 b9231000  r9 e65c3620  sl fffffffc  fp b9407cb0

    ip e9047944  sp 8d5694c0  lr b99d6ff1  pc b9a2ab82  cpsr 680f0030

04-27 16:25:44.573 18303 18303 F DEBUG   :

backtrace:

    #00 pc 0008ab82  /system/vendor/lib/hw/vulkan.msm8998.so

    #01 pc 00036fed  /system/vendor/lib/hw/vulkan.msm8998.so

    #02 pc 00012a5d  /system/vendor/lib/hw/vulkan.msm8998.so (_ZN11qglinternal18vkBindBufferMemoryEP10VkDevice_Tyyy+28)

    #03 pc 00378ee5  /data/app/org.renderdoc.renderdoccmd.arm32-1/lib/arm/libVkLayer_GLES_RenderDoc.so

    #04 pc 002fe5f9  /data/app/org.renderdoc.renderdoccmd.arm32-1/lib/arm/libVkLayer_GLES_RenderDoc.so

    #05 pc 002fe289  /data/app/org.renderdoc.renderdoccmd.arm32-1/lib/arm/libVkLayer_GLES_RenderDoc.so

    #06 pc 0064456f  /data/app/org.renderdoc.renderdoccmd.arm32-1/lib/arm/libVkLayer_GLES_RenderDoc.so

    #07 pc 00301b79  /data/app/org.renderdoc.renderdoccmd.arm32-1/lib/arm/libVkLayer_GLES_RenderDoc.so

    #08 pc 004c4bc3  /data/app/org.renderdoc.renderdoccmd.arm32-1/lib/arm/libVkLayer_GLES_RenderDoc.so

    #09 pc 00117a27  /system/priv-app/VrDriver/VrDriver.apk (offset 0xd83000)

    #10 pc 001178ed  /system/priv-app/VrDriver/VrDriver.apk (offset 0xd83000)

    #11 pc 0010e8d3  /system/priv-app/VrDriver/VrDriver.apk (offset 0xd83000)

    #12 pc 000f753d  /system/priv-app/VrDriver/VrDriver.apk (offset 0xd83000)

    #13 pc 000f87f1  /system/priv-app/VrDriver/VrDriver.apk (offset 0xd83000)

    #14 pc 00004c8f  /system/priv-app/VrDriver/VrDriver.apk (offset 0xc0e000)

    #15 pc 0003dfb1  /data/app/com.Sanzaru.NewGame-1/lib/arm/libOVRPlugin.so (_ZN3OVR4Util22CompositorVRAPI_Vulkan11SubmitFrameEPK27ovrSubmitFrameDescription2_+48)

    #16 pc 000364e7  /data/app/com.Sanzaru.NewGame-1/lib/arm/libOVRPlugin.so (_ZN3OVR4Util15CompositorVRAPI8EndFrameEiRSt6vectorI20ovrpLayerSubmitUnionSaIS3_EEbPv+786)

    #17 pc 00023cd9  /data/app/com.Sanzaru.NewGame-1/lib/arm/libOVRPlugin.so (ovrp_EndFrame4+300)

    #18 pc 040383a4  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN9OculusHMD10FOculusHMD24FinishRHIFrame_RHIThreadEv+1212)

    #19 pc 0403da38  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN9OculusHMD14FCustomPresent25FinishRendering_RHIThreadEv+4272)

    #20 pc 0403c6d4  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN9OculusHMD14FCustomPresent7PresentERi+388)

    #21 pc 04a20ef4  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN15FVulkanViewport7PresentEP25FVulkanCommandListContextP16FVulkanCmdBufferP12FVulkanQueueS5_b+5256)

    #22 pc 049d7434  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN25FVulkanCommandListContext21RHIEndDrawingViewportEP12FRHIViewportbb+292)

    #23 pc 05ce5194  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN15FRHICommandList18EndDrawingViewportEP12FRHIViewportbb+112)

    #24 pc 06b05250  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN17FSlateRHIRenderer23DrawWindow_RenderThreadER24FRHICommandListImmediateR13FViewportInfoR23FSlateWindowElementListRK29FSlateDrawWindowCommandParams+20492)

    #25 pc 06b3541c  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so

    #26 pc 04a795b8  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN16FNamedTaskThread23ProcessTasksNamedThreadEib+3152)

    #27 pc 04a77f20  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN16FNamedTaskThread21ProcessTasksUntilQuitEi+108)

    #28 pc 05d562a0  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_Z19RenderingThreadMainP6FEvent+436)

    #29 pc 05db2f24  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN16FRenderingThread3RunEv+20)

    #30 pc 04b27780  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN22FRunnableThreadPThread3RunEv+164)

    #31 pc 04a7316c  /data/app/com.Sanzaru.NewGame-1/lib/arm/libUE4.so (_ZN22FRunnableThreadPThread11_ThreadProcEPv+80)

    #32 pc 00047d83  /system/lib/libc.so (_ZL15__pthread_startPv+22)

    #33 pc 0001a035  /system/lib/libc.so (__start_thread+6)

[dustanm]

Not a solution to your problem, but a possible alternative:  I'm using 4.24.3 and RenderDoc is working fine.  My RenderDoc version is 1.7 built from a56af589.

[MaxArch]

I think you could better email the creator of Renderdoc directly. I did that a while ago with another bug and he replied quite fast. A fix was online within a few days. Look for the contact info of his page https://renderdoc.org/

[nkitten]

I should've updated: I did contact someone working on RenderDoc at Oculus directly, and it turns out that (1) they've done some work to optimize RenderDoc memory usage on OpenGL ("low memory mode") which hasn't been ported to Vulkan yet, so it's simply less efficient to use Vulkan and therefore more likely to run out of memory, and (2) the Unreal Vulkan usage also potentially has has some leaks that makes it use more memory.  They did suggest reducing the backbuffer size to reduce memory using adb shell setprop debug.oculus.textureWidth 512 and adb shell setprop debug.oculus.textureHeight 512. In the end, I went back to OpenGL for the purposes of obtaining a capture, and just hope that can get me far enough.