cancel
Showing results for 
Search instead for 
Did you mean: 

Specular / Reflection mismatch in stereo!!

Infinite
Honored Guest
I was wondering if anyone at OculusVR have come across this strange bug with regards to specular / reflection mismatch in stereo generated from IBL. It's a horrid problem, where users are not able to focus on a reflection highlight or pure reflection simulation in the shader from a background.

With regards to reflections highlights rendered from IBL in Unity, in stereo we cannot converge on the specular highlight or reflection. I have a feeling there was a tweet from John Carmack (can't seem to find it 😞 ) regarding this phenomena, that no one seems to be able to address, as if it's an industry wide inherent rendering problem with stereo.

I've spoke with the guys at Marmost about this issue and here was their take on the subject:

I've had an experience similar to what you describe - not being able to focus on the reflected image quite right in scenes rendered in the rift (not just unity, but toolbag 1 and 2 and half life 2).

In the real world you'd get slightly different reflection images because each eye has a slightly different perspective on the shiny object, but you're right that in real life you can resolve images on such objects (at any distance including infinity). So I don't really buy " "" "'s explanation there - the problem i've experienced isn't insufficient reflection parallax but the opposite. I'll have to defer to him somewhat though since he has the goggles!


I think the same also happens in UE4.

Has anyone at Oculus come across a solution for this? It looks horrible on characters skin, you can feel it strain your eyes as you cannot converge on the highlight effect, even worse on reflection images.
9 REPLIES 9

Infinite
Honored Guest
Can anyone at Oculus or Unity confirm or describe what this issue is? Or how to solve it?

drash
Heroic Explorer
Hi Infinite, I tried to watch the video you linked but got a "Permission Denied. Sorry, there is no video here." message. So I may or may not know what you're referring to.

If you use a simplistic shader that does reflections based on a single reflection cubemap, the reflection is purely dependent on your overall view direction (same for both eyes) and thus you perceive a reflection that is off at infinite distance. The stock reflective shaders in Unity will give you this effect and it doesn't look right in VR. There's plenty of reflective shaders out there that do this right though.

There's also the matter of specular/rim/fresnel highlights that don't feel right, and looks a little different for each eye than your brain is expecting. I see this a lot in Senza Peso (on the rock walls in one of the first couple of parts, I forget which exactly). If the shaders recalculate the highlight for each eye, then it should look better, but this also depends on whether or not you're using a single reflection cubemap for your reflective colors.

I'm probably barking up the wrong tree though, huh?

Infinite
Honored Guest
Hi Drash,

Thank you kindly for the reply.

What you mention is exactly the issue but I wonder if this is a shader implementation issue, an engine issue, or SDK issue? Who can solve it? As Marmoset Co highlighted, they are aware of it but don't seem to know what to do about it, as it hasn't been fixed in the latest Toolbag 2.0 release. So I wonder if it's a core issue with stereo rendering?

I wish I could find John Carmack's tweet on the phenomena.

owenwp
Expert Protege
The problem with conventional environment maps is that they are effectively reflecting light from infinitely far away. That's fine for the sky, but for closer objects it looks obviously wrong.

Parallax corrected cube maps are one fair alternative, and Unreal 4 uses them extensively. Specular from point light sources should also be correct.

The ultimate answer is we need true real time global illumination to give us correct per-pixel reflections.

vrdaveb
Oculus Staff
"Infinite" wrote:
I wonder if this is a shader implementation issue, an engine issue, or SDK issue?

In most cases, it is a shader issue. The game needs to render the reflected image with the correct projection. That usually means an off-axis frustum. Unity's mirror shader seemes to do it right, but Shyshop might be doing something else.

Infinite
Honored Guest
"owenwp" wrote:
The problem with conventional environment maps is that they are effectively reflecting light from infinitely far away. That's fine for the sky, but for closer objects it looks obviously wrong.

Parallax corrected cube maps are one fair alternative, and Unreal 4 uses them extensively. Specular from point light sources should also be correct.

The ultimate answer is we need true real time global illumination to give us correct per-pixel reflections.


Thanks. "Parallax corrected cube maps are one fair alternative" this might be worth exploring further.

"vrdaveb" wrote:
"Infinite" wrote:
I wonder if this is a shader implementation issue, an engine issue, or SDK issue?

In most cases, it is a shader issue. The game needs to render the reflected image with the correct projection. That usually means an off-axis frustum. Unity's mirror shader seemes to do it right, but Shyshop might be doing something else.


Thanks for the reply. Would you be able to link to an example of the mirror shader you mentioned? Hopefully someone form Marmoset might be able to comment further. It would be amazing to find a fix for this issue.

Infinite
Honored Guest
To further add to the discussion some of the Marmoset Co guys have responded by email and would be interested to talk with Oculus or Unity further about this via Twitter, if the interest should arise.

I think this is a very important problem to solve and could have a profound impact on shader realism in VR:

I'm wondering if you've tried Skyshop's parallax corrected reflections (box projected skies) and if they help at all.

Also, as of Skyshop 1.08 or 09, I've converted to computing the eye vector as _WorldSpaceCameraPos - worldPos instead of relying on Unity's viewDir parameter. If this has fixed reflections, viewDir derivation in Unity's shader system is to blame. If this does not fix it, I'm willing to bet cameraPos is not configured properly for the two eyes.

If someone from Unity or Oculus could tell us how the two cameras are derived and which matrix the eye split is stored in, we'd have more clues to go off of.



Andres Reinot
Engineer - Marmoset


Twitter handle - @monkeyscience8

https://twitter.com/monkeyscience8

---------------------------------------------

Ah yes, this old problem. Andres and I spent a bit of time discussing it this morning. The conclusion is that we still aren't sure 😛 I can I think rule out a few things though, and I have one more guess.

One is that it's not an issue of parallax correcting the cube map (or rather, that's a different problem). Surfaces should be able to be rendered with infinite reflections without the painful artifacts we've seen. Proper parallax correction of cube environment maps would be a nice improvement for local reflections, but I don't think that's what we're dealing with here.

I think I can also fairly safely rule out some trivial bugs on our part regarding off-axis frustums or the like - we do very carefully use the correct camera positions from the SDK, as do many other apps, and the problem still remains in all of them. If you're using Skyshop in unity specifically, and there's some discrepancy between the unity shaders and ours, then there's likely a bug, and we'd like to hear more, as Andres says.

I have a reflective sphere on my desk, because I'm that kind of person. When I pick it up and hold it at arms length, and alternately close each eye, I can't tell much difference between the reflected images. If I move it much closer to my nose however and do the same, I can tell a large difference between each eye. With both eyes open its not possible for me to converge on features reflecting near the edges of the sphere (where there is the most distortion). This, qualitatively, is the same sensation I get with the headset on. Try it!

So, a new theory:

It's hard to resolve reflections when the object size/distance is comparable to your eye separation distance. This is because the distorted reflections vary so drastically from one eye to the other that your brain can't quite resolve it as a clear image. So I think one has to be careful noting the eye separation distance vs the distance to the subject - since one doesn't have focal cues with the headset on you have to be careful that you aren't "holding your subject against your nose" so to speak, as reflections are more sensitive than general parallax is to this effect.

Hope that's helpful,


Jeff Russell
Engineer, Marmoset


Twitter handle - @j3ffdr

https://twitter.com/j3ffdr


---------------------------------------------

I hope this will further the discussion to resolve the problem.

vrdaveb
Oculus Staff
"Infinite" wrote:
Would you be able to link to an example of the mirror shader you mentioned?

Here is one from Aras: http://wiki.unity3d.com/index.php?title=MirrorReflection2

Infinite
Honored Guest
Thanks for the link but it's a little confusing as that mirror effect is done via a script (doing all necessary calculations for the camera) rather than a shader, the shader just consists of pass applied to it for the reflection.

I hope the discussion might continue further with Oculus, Andres and Jeff via Twitter.

Having the ability to correctly converge or focus the reflection and specular highlights in stereo will look incredible in VR, at the moment it's headache inducing! and really breaks immersion.