When running the VRC validator using the arguments "--test TestSubmitFramesWhenNotVisible --test TestSubmitFramesWhenVisible" using version 1.33 of the validator, the TestSubmitFramesWhenNotVisible fails with 2 frames submitted.
The strange thing is this same build used to pass both tests until a few days ago, before the latest Oculus Update on the 11th. We've also been working on the project, but removing the changes we made gives us the same results.
Strangely, I can trick the 1.33 validator into passing TestSubmitFramesWhenNotVisible by holding my finger over the HMD's sensor when the test starts, and removing it before the 5 second wait has elapsed. As far as we can tell, the manual version of this test passes just fine, the Oculus menu appears and gameplay pauses, no longer accepting input.
Here is the log showing results from both the 1.33 validator and 1.32 validator (both using 1.33 software however). 1.32 fails TestSubmitFramesWhenVisible since no frames are ever submitted at all apparently, but it used to pass before the update (holding my finger over the sensor in this case gives the same results as 1.33) . Any insight would be appreciated. This is the only issue holding up our submission.
============================= 2018-12-14 10:38:41 ============================== INFO: Starting TestSubmitFramesWhenNotVisible INFO: Waiting for the application to run for 5 seconds before testing begins... INFO: Starting test... INFO: Requesting the void... INFO: Number of texture swap chains committed when visible 34 INFO: Number of texture swap chains committed when not visible 2 ERROR: Committed a texture swap chain (called ovr_CommitTextureSwapChain) when application not visible INFO: Please refer to VRC Guidelines: https://developer.oculus.com/distribute/latest/concepts/vrc-pc-input-1/ INFO: Cleaning up... INFO: Test FAILED
INFO: Starting TestSubmitFramesWhenVisible INFO: Waiting for the application to run for 5 seconds before testing begins... INFO: Starting test... INFO: Number of texture swap chains committed when visible 902 INFO: Number of texture swap chains committed when not visible 0 INFO: Cleaning up... INFO: Test PASSED
INFO: Summary: INFO: TestSubmitFramesWhenNotVisible: Failed INFO: TestSubmitFramesWhenVisible: Passed INFO: Generated by OculusVRCValidator version 1.33.0.0 ============================= 2018-12-14 11:14:16 ============================== INFO: Starting TestSubmitFramesWhenNotVisible INFO: Waiting for the application to run for 5 seconds before testing begins... INFO: Starting test... INFO: Requesting the void... INFO: Number of texture swap chains committed when visible 0 INFO: Number of texture swap chains committed when not visible 0 INFO: Cleaning up... INFO: Test PASSED
INFO: Starting TestSubmitFramesWhenVisible INFO: Waiting for the application to run for 5 seconds before testing begins... INFO: Starting test... INFO: Number of texture swap chains committed when visible 0 INFO: Number of texture swap chains committed when not visible 0 ERROR: The application did not commit any texture swap chains INFO: Please refer to VRC Guidelines: https://developer.oculus.com/distribute/latest/concepts/publish-rift-app-submission/ INFO: Cleaning up... INFO: Test FAILED
INFO: Summary: INFO: TestSubmitFramesWhenNotVisible: Passed INFO: TestSubmitFramesWhenVisible: Failed INFO: Generated by OculusVRCValidator version 1.32.0.0
Some further info. If the HMD has been idle for a while before running the test, it shows no frames submitted for either test (so it passes NotVisible, but fails Visible). If the HMD had been active recently, I see the same results as the first test indicated above.
For anyone else having this specific issue, we did submit and managed to pass, so it would appear that the manual test should be considered over the validator's result.