Welcome to the Oculus Developer Forums!

Your participation on the forum is subject to the Oculus Code of Conduct.

In general, please be respectful and kind. If you violate the Oculus Code of Conduct, your access to the developer forums may be revoked at the discretion of Oculus staff.

Overhead/Gfx.WaitForPresent

braun0braun0 Posts: 44
edited April 2015 in Oculus Go Development
I am looking for some clarification and direction regarding some performance issues. My game is generally smooth, but often suffers from a stutter. I just enabled the profiler and got it setup so I can profile directly from the Note 4 while the game is playing. I immediately recognized that both Gfx.WaitForPresent and Overhead are causing spikes and killing my FPS.

Settings & version:
Unity v. 4.6.2
Oculus Mobile SDK 0.4.3
Mac OSX Mavericks
Fixed Timestep/Max allowed time = 0.166666
Vsync disabled
Quality settings set to Fastest (no shadows, Anti-Aliasing disabled, texture quality Eighth, Pixel Light Count = 0)
Other Settings (I have static & dynamic batching enabled)
I use Ambient Light (not a game component light source)

I believe I have followed all the settings rules outlined in the SDK documentation.

NOTE: All profiler data below if from the phone hardware (not my desktop).

Issue/Question #1: Gfx.WaitfForPresent
In my current build, this item in the profiler uses anywhere from 40%-90% of the CPU usage, with many occurrences pushing the FPS well below 60 (all by itself)

This example is from a static scene
https://www.dropbox.com/s/l23wtkxhbcbzkii/GFXandRender.png?dl=0

For a scene that falls into all the guidelines for Tris, Verts, Draw Calls (and quality setting turned all the way down) why is Gfx.WaitfForPresent eating up so many resources? As I understand Gfx.WaitfForPresent, it occurs when the GPU isn't rendering the frame fast enough for the CPU (causing the CPU to wait). Is this correct? What am I overlooking here?

I also tested an "empty" scene and Gfx.WaitfForPresent was also very high. I was very surprised that an empty scene couldn't even keep a consistent FPS over 60.

Issue/Question #2: Overhead
About once every 15-30 seconds the Overhead in the profiler spikes up to around 90% and it causes a huge stutter in the game. As I understand, Overhead can be caused by errors/warnings that appear in the console. When I run the game in Unity, I get no warning/errors in the console. However, when I run the game on the phone, I get about 14 warnings. All of them pertain to some value in various OVR scripts that are assigned by never used. Could all these warnings cause this spike in Overhead?

https://www.dropbox.com/s/y8npt5ohv06318n/overhead.png?dl=0


As a side note, when I play the game in Unity, the game as viewed in the "game" screen moves in slow motion. Meaning the player/game objects move very slowly compared to how fast the game actually is (when played on the phone). Has anyone else encountered this issue? Could this be a symptom of a Unity/SDK integration error?

Any input/advice would be greatly appreciated.

Comments

  • cyberealitycybereality Posts: 26,156 Oculus Staff
    In general, Gfx.WaitForPresent and Overhead spike when the CPU is waiting for the GPU to finish rendering work.

    Are other apps running? Does restarting the phone help? Do the SDKExamples we provide work?

    It might help to look at the GPU profile on PC for clues.
    AMD Ryzen 7 1800X | MSI X370 Titanium | G.Skill 16GB DDR4 3200 | EVGA SuperNOVA 1000 | Corsair Hydro H110i
    Gigabyte RX Vega 64 x2 | Samsung 960 Evo M.2 500GB | Seagate FireCuda SSHD 2TB | Phanteks ENTHOO EVOLV
  • braun0braun0 Posts: 44
    In general, Gfx.WaitForPresent and Overhead spike when the CPU is waiting for the GPU to finish rendering work.

    Are other apps running? Does restarting the phone help? Do the SDKExamples we provide work?

    It might help to look at the GPU profile on PC for clues.


    Cyber, I just ran a test using two of the SKD examples. For the most part, FPS fluctuates between 55-65 in the profiler. However, there is still a large spike once every 15-30 seconds. See the screenshots below.

    SDK Scene "boot":
    https://www.dropbox.com/s/bwtfba50536cabu/bootexample.png?dl=0

    SDK scene "FirstPerson_Sample":
    https://www.dropbox.com/s/c15ne5w224sra7a/FirstPersontest.png?dl=0

    Is 55-65 FPS normal for performance on these examples? Is it acceptable to have the scene drop below 60fps for any amount of time (no matter how short)? Could those large spikes in Overhead/Gfx.WaitForPresent be specific to my device? I am running stock, not rooted, and with no open programs (besides all the junk Samsung pre-installs, as I have tried to leave this phone as stock as possible for testing/development purposes).

    Thanks for your input/advice.
  • marchansenjmarchansenj Posts: 77
    Hiro Protagonist
    Cyber, I am experiencing the same issue.

    I went from getting 500+ fps to getting 40-60 fps in the unity editor. I also am seeing a spike every 10 seconds or so as braun0 described.

    When I deploy to the phone and profile over adb I see the same results only a bit worse.

    I have tried all of the SDK Examples and I am seeing Gfx.WaitForPresent at 96% with a frame rate of 40-65 with the spike every 10-15 seconds that brings it to 15fps.

    I had no noticeable FPS drop and never noticed the Gfx.WaitForPresent when profiling before I updated the runtime and SDK to Mobile 5.0 today; maybe its just a coincidence.

    The offending script is OVRManager.cs (which use to be OVRDevice)- If I disable that Script the Gfx.WaitForPresent goes away and the frame rate goes back to where it use to be 1000+. I see that the script when android targets 60 fps which is fine if Gfx.WaitForPresent wasn't gobbling up 90% of the resources and there wasn't the spike.

    I have wasted an entire day on this issue.

    Please if someone else has updated their SDK to the latest and is getting a high frame-rate and not seeing Gfx.WaitForPresent let me know so I don't chase a red herring.

    Can our JAM entry be built using the previous SDK and runtime? Or are you guys aware of an Issue? Maybe I am missing a setting?

    Thanks,
    Marc
  • HappySliceHappySlice Posts: 36
    Brain Burst
    I am experiencing the same problem on my Windows 7 machine. I was getting almost constant 60fps on my Note4. Then I upgraded to Unity 5.0.1 from 5.0.0f4 and also updated to the new Oculus runtime. I am still using the 0.4.3 SDK with all the default/suggested project settings.

    My GPU on my Windows machine is old but I see spikes from 60 to 30fps on the graph from rendering opaque meshes every so often making, even in a scene I made with only ~12k total triangles.

    Is there a way to profile the GPU directly from the Note4? I'm backing up my project now to update to the new mobile SDK. Hopefully more stuff doesn't "break" :(
  • HappySliceHappySlice Posts: 36
    Brain Burst
    Updated to Mobile 0.5.0 SDK and the performance is now almost constant 60fps again. My desktop version seems to run better too -- maybe it's just because the optimization I did for mobile though.

    I just had to delete some duplicate plugins for it to compile due to being unable to convert classes into dex format

    Mobile SDK 0.5.0
    5.0.1 Oculus Runtime
    Unity 5.0.1
    T-mobile Note 4 on Kit-kat
    Windows 7
  • HappySliceHappySlice Posts: 36
    Brain Burst
    I still get Gfx.WaitForPresent spikes on my Win7 machine. They coincide with GPU spikes that I don't understand. Even when my character stands still... Does anyone know how to reduce these GPU spikes? The level is low poly, with about 50 draw calls even though everything is a cube (different scales though). Is it all about draw calls? Do I need less draw calls?

    I'll have to wait until later to get time to post profiler screencaps from my Note4. Here are pics from the Unity5 Editor Profiler
    http://imgur.com/foLTaTo,B74cm9d,IbESlR0,CsVjDCA
    http://imgur.com/foLTaTo,B74cm9d,IbESlR0,CsVjDCA#1
    http://imgur.com/foLTaTo,B74cm9d,IbESlR0,CsVjDCA#2
    http://imgur.com/foLTaTo,B74cm9d,IbESlR0,CsVjDCA#3
  • marchansenjmarchansenj Posts: 77
    Hiro Protagonist
    I uninstalled and reinstalled the 5.0 runtime and 0.5.0 SDK following the documentation.
    I loaded the Cubes sample scene.
    On the editor I see 55-75 FPS and the Gfx.WaitForPresent at 95%:
    http://imgur.com/JNtdDAK

    On the Note 4 profiling over Wifi I now do not see Gfx.WaitForPresent but still do see spikes every few seconds that brings the phone down to 30fps and every 15 seconds or so brings the FPS down the 15.

    Overhead 83% spike down to 15fps:
    http://imgur.com/HNS9dvD
    Callback Coroutine 55% spikes to 30 fps:
    http://imgur.com/o2XBKm0

    Should I be upgrading to Unity 5 I wasn't planning on it but I am at a loss?
    Is there a player setting or something that would account for this strange behavior?
    I am going to guess that not everyone is seeing this problem or we would see more posts like this.

    Using Unity 4.6.3
    Note 4 T-mobile/kitkat
  • braun0braun0 Posts: 44
    Thanks to everyone else for added to this thread. It appears that at least a few of us are having this issue with Overhead/Gfx.WaitForPresent spikes that are killing FPS. Since my first post, I have updated to Unity 4.6.3 and Mobile SDK 0.5.0 - and I still have the same spikes.

    I have spent days attempting different settings/scenes and the spikes always occur (no matter how simple the scene is). I am hoping to at least get a response from Oculus regarding the spikes in the OVR demo scenes. Do they see anything similar when they run the demo scenes? Can they provide an example of what the FPS/profiler should look like in a particular scene?



    Also, since updating to SDK v0.5 this appears in the Console each time I run the game in Unity:
    OVRMoonlightLoader: Apply settings required for Android Mobile VR
    UnityEngine.Debug:Log(Object)
    OVRMoonlightLoader:.cctor() (at Assets/OVR/Moonlight/Editor/OVRMoonlightLoader.cs:51)
    UnityEditor.EditorAssemblies:SetLoadedEditorAssemblies(Assembly[])

    Is this script checking these settings and telling me they are NOT applied correctly? Is this script "applying" the settings for me? I am a bit confused by the use of "apply" vs. "applying". I have the settings below set properly in the Quality Settings section - so I am not sure what this notification is trying to tell me.

    PlayerSettings.defaultInterfaceOrientation = UIOrientation.LandscapeLeft;
    QualitySettings.antiAliasing = 1;
    QualitySettings.vSyncCount = 0;

    Any thoughts?
  • marchansenjmarchansenj Posts: 77
    Hiro Protagonist
    I figured something out. But I have only tested with the Sample scenes.

    The problem seemed to be trusting the UNITY PROFILER.
    When I hooked up the Oculus Remote Monitor I am seeing a consistent 59-60 fps with only one spike to 54 fps over a 5 minute period of running the Cubes Scene, much much better then what the Unity Profiler was telling me.

    Frame rate in the editor still sucks in 0.5.0 SDK with Unity 4.6.3 but that can be overcome by using an alternative camera for play testing when in the editor.

    Let me know if you guys figure out anything else.

    Thanks,
  • marchansenjmarchansenj Posts: 77
    Hiro Protagonist
    Hi Braun0,
    After uninstalling the runtime and reinstalling and reconverting my project to 0.5.0 SDK I am now seeing 1500+ FPS in the Unity editor again and a solid 60fps on the Note 4 using the Oculus Remote Monitor. This worries me a bit because I have no idea what I did different this third time around.

    Here is a screen capture of my Resolution settings:
    http://imgur.com/uDDtENt

    Here is a screen capture of my Other Settings under player settings:
    http://imgur.com/V1qw1Ro

    I wish you the best of luck.
  • chrispruettchrispruett Posts: 200 Oculus Staff
    Hi folks,

    It seems there's a lot of confusion about what Overhead or WaitForPresent means. When these items show up at the top of your profiler, it looks like they are "eating" all of your frame time. In fact, what these items represent is a stall in the pipe. They are a record of the empty space between frames that is caused when you miss the timing of one frame by spending too long in render.

    To be more specific, "Overhead" is generally the things that happen outside of Unity in the render loop. If we imagine the render loop looks like this:
    void drawFrame()
    {
       // do some stuff
       Unity.DrawFrame()
       // do some more stuff
    }
    

    ... then "overhead" is the time between the end of the last Unity frame and the beginning of the next one. It's the outer part of the loop that causes the frame to be drawn. Generally this value is small because there's not a lot of work going on outside of Unity, but occasionally a weird use case will cause it to spike. Generally it represents a constant overhead of drawing frames on Android and therefore can be ignored. If you really want to know what's going on in there, the Android TraceView tool can tell you.

    Gfx.WaitForPresent() represents the time that the CPU stalled out waiting for the GPU. Your screen updates at regular cycles of 60 frames per second. Every 16.6 ms, there's a point at which the frame back buffer is swapped with the front buffer so that the next frame can be displayed. If you're drawing very fast, say completing the frame in 5 ms, that means that the GPU has to wait another 11.6 ms before the frame you just drew can actually be displayed. The next frame can't be started until that happens, so the GPU stalls. If you're drawing slowly, say 20 ms, that means that you miss the first swap at 16.6 ms and then must wait 12 ms for the next one at 33 ms. Either way, the pipeline is stalled: the GPU is waiting on the hardware to get to a point where it can draw. WaitForPresent() measures this time.

    What this means is that WaitForPresent() by itself is not something you need to optimize away. What you should be looking at is what's going on in the rest of your frame that is causing the stall. If you're drawing at faster than 60 fps, this isn't much of a problem: the GPU will just spin over the extra time. If you're drawing slower than 60 fps, even by just a few ms, you can cause a stall that will effectively halve your frame rate because the GPU is missing the swap every time.

    To debug WaitForPresent stalls, the first thing you should do is turn off Multithreaded Rendering in the Player Settings. This will slow your frame down a lot, but it will also give you a much clearer picture of where the time is going. When you're done with optimization make sure to turn Multithreaded Rendering back on.

    With multithreading off, take a look at the profile and ignore WaitForPresent. You're probably not completing your drawing commands in less than 16 ms (and actually, you want to do it in ~12 ms to allow for distortion and TimeWarp overhead). You can also use the logcat profiler (check Enable Internal Profiler in the player settings and then connect the log to view it at runtime) to see a different view of where your time is going. Most common for Gear VR developers is to be CPU bound on draw call submission; if that's the case with you, it will show up as a large number of draw calls and a high CPU time. In that case, optimization is about improving the performance of static and dynamic batching by reducing the number of unique materials in your scene, marking static objects as static, etc.

    tl;dr: WaitForPresent isn't a thing that is eating your time, it represents your GPU missing the display refresh train. Solve it by making your frame time faster.

    RE: Unity 4 vs Unity 5 changes, I don't have details on perf issues with Unity 5, but the advice is the same: look at the performance profile of the frame itself, don't worry about tracking stall time.
  • dshankardshankar Posts: 14
    NerveGear
    If I understood you correctly, when CPU spends most of its time in Gfx.WaitForPresent and GPU spends most of it's time in Graphics.PresentAndSync, this implies that both the CPU and GPU finished a frame quickly and thus they are stalling, waiting to do the next frame.

    So it's a good thing that my app is running at nearly exactly 75fps, CPU & GPU taking roughly 13ms? Does OVR override Unity settings (where I've disabled vsync) and turned on 75fps vsync?

    (above numbers are desktop specific, but I'm targeting both desktop and mobile)
  • marchansenjmarchansenj Posts: 77
    Hiro Protagonist
    Wow, thank you Chris for the thoughtful and detailed response.
    It clarified a lot as to the meaning of Overhead and Gfx.Waitforpresent.

    I am still in the dark as to why my project and the sample SDK scenes were chugging horribly and now are flying but everything works great now.

    I hate to give out voodoo advice (but if your desperate); the only thing I can think of that I did differently on my third attempt was to fully uninstall the run time, reboot, and reinstall the new run time, the first time I just like the install shield uninstall the old run-time.

    I know it sounds like unplug your modem advice cause it is.... hell it could have been anything.
  • stan4oculusstan4oculus Posts: 15
    NerveGear

    With multithreading off, take a look at the profile and ignore WaitForPresent. You're probably not completing your drawing commands in less than 16 ms (and actually, you want to do it in ~12 ms to allow for distortion and TimeWarp overhead). You can also use the logcat profiler (check Enable Internal Profiler in the player settings and then connect the log to view it at runtime) to see a different view of where your time is going. Most common for Gear VR developers is to be CPU bound on draw call submission; if that's the case with you, it will show up as a large number of draw calls and a high CPU time. In that case, optimization is about improving the performance of static and dynamic batching by reducing the number of unique materials in your scene, marking static objects as static, etc.

    tl;dr: WaitForPresent isn't a thing that is eating your time, it represents your GPU missing the display refresh train. Solve it by making your frame time faster.

    First of all thank you Chris very much for explaining these...
    I disabled mutithreaded rendering and now profiler makes sense again.
    Just want to ask about few things I see there now.
    I can see that OVRManager.CallbackCoroutine() is eating now most of the frame time 12ms.
    Is this something that is normally happening on its own thread? Do I have to worry about it or it can be ignored when multithreaded rendering is off?
    Secondly, the overhead is taking 5ms which accounts for 15% of frame time, I don;t consider this as small take. does this include distortion and TimeWarp overhead ?
    Here is the profiler screenshot.
    https://www.dropbox.com/s/exo0y2f5lixve ... 1.png?dl=0

    Thanks
    Stan
  • chrispruettchrispruett Posts: 200 Oculus Staff
    Hi Stan,

    OVRManager.CallbackCoroutine means "draw the frame and wait for vblank," so it's the part that is normally running in a different thread when multithreading is used. If you make your draw time faster it will go down, so like WaitForPresent it's not a "cost" that is eating into your time.

    RE: Overhead, I agree that 5 ms is very large. I don't know what's going on there. Do you have gyro turned on? In this case, the best way to debug is to use the Android Traceview tool and try to dig into what's going on outside of Unity.

    http://developer.android.com/tools/debu ... acing.html
  • stan4oculusstan4oculus Posts: 15
    NerveGear
    RE: Overhead, I agree that 5 ms is very large. I don't know what's going on there. Do you have gyro turned on? In this case, the best way to debug is to use the Android Traceview tool and try to dig into what's going on outside of Unity.

    Hi again.. nope, gyro is off, I logged its state at runtime to make sure.

    I recorded trace as you suggested but I'm not any wiser from it ;o)

    here's how it looks
    https://www.dropbox.com/s/lrr7i61vouzm2 ... 9.png?dl=0

    95% of time is spent in unity and there are only 2 other significant areas
    1. is BinderProxy.transact called by Display manager reading display info. Related to this are probably getWidth(), getRotation(), getRealMetrics etc.
    2. seems to be processing main loop?

    Not sure what can I do about these.

    Thnaks
    Stan.
  • RazTOORazTOO Posts: 128
    braun0 wrote:

    Also, since updating to SDK v0.5 this appears in the Console each time I run the game in Unity:
    OVRMoonlightLoader: Apply settings required for Android Mobile VR
    UnityEngine.Debug:Log(Object)
    OVRMoonlightLoader:.cctor() (at Assets/OVR/Moonlight/Editor/OVRMoonlightLoader.cs:51)
    UnityEditor.EditorAssemblies:SetLoadedEditorAssemblies(Assembly[])

    Is this script checking these settings and telling me they are NOT applied correctly? Is this script "applying" the settings for me? I am a bit confused by the use of "apply" vs. "applying". I have the settings below set properly in the Quality Settings section - so I am not sure what this notification is trying to tell me.

    I am having the same issue. This effectively hinders me on making progress. Also Unity doesn't want to bake... instead freezex unity when baking. I followed the Mobile Integration guide and Best Practices. Can somebody help?
  • braun0braun0 Posts: 44
    Hey everyone.

    I have been able to improve performance, and rid my game of the stutter I was experiencing. From the various responses to my original post, I learned that Overhead/Gfx.WaitForPresent spikes can be attributed to many things. I can only share how I improved (not fixed) the issue, and hope it helps others.

    1) If you have any moving objects in your scene, it should have a rigid body. Most of my scenes have moving platforms and barriers, and I did NOT have a rigid body attached to them. Apparently Unity's physics engine works very differently with objects with/without rigid body.
    http://answers.unity3d.com/questions/7671/guidelines-for-using-rigidbody-collider-characterc.html

    2) Mark all non-moving game objects as "static" in the inspector: I had done this on some objects, but not all.

    3) If you use raycasting, change the layermask to "ignore raycast" (in the inspector) on all objects that have no need to interact with the raycast.

    I still have some Overhead/Gfx.WaitForPresent spikes, but they are reduced to the point where they are no longer an issue.
  • chrispruettchrispruett Posts: 200 Oculus Staff
    Stan,

    Your traceview looks clean. DisplayMetrics is indeed concerned with reading the metrics of the scene, and it's slow (memory allocation), but it's not hurting you badly here. Actually, it looks like your loop is pretty solid. In that case, I don't know what to attribute overhead to, but I suspect that resolving it is still just a matter of making your scene more efficient.
  • stan4oculusstan4oculus Posts: 15
    NerveGear
    Your traceview looks clean. DisplayMetrics is indeed concerned with reading the metrics of the scene, and it's slow (memory allocation), but it's not hurting you badly here. Actually, it looks like your loop is pretty solid. In that case, I don't know what to attribute overhead to, but I suspect that resolving it is still just a matter of making your scene more efficient.
    Yep, its probably related to scene complexity. We'll work on making it easier and overhead will probably go down with it.
    Thank you for looking into it.
  • HappySliceHappySlice Posts: 36
    Brain Burst
    I am still experiencing this huge spike in my profiler while running my android device. The game runs ~35-60fps on my Note 4. Yet in android mode on my computer, which is no prize (made in 2007), it runs at 120fps and shows spikes in the GPU, but nothing that drops that FPS below 80.
  • chrispruettchrispruett Posts: 200 Oculus Staff
    I think your FPS reporting is unreliable; Android games are limited to screen refresh rates, which are almost entirely 60 fps. The WaitForPresent pipeline stall can absolutely happen under non-VR conditions, it's just significantly more likely to occur when you draw the screen twice and your main cost is draw call submission (which you pay twice).

    Think of it this way: the train comes once an hour. You get there 5 minutes early every day. Then you move to a place that is twice as far away as before, and now it takes you 6 minutes to get to the station. So you're one minute late. But as a result, you missed the train and now you're an hour late. If you move 1 minute faster, you'll save yourself an hour (and a minute).
Sign In or Register to comment.