New to the forums? Click here to read the "How To" Guide.

Developer? Click here to go to the Developer Forums.

Increasing Immersion of 360 Panoramas

kojackkojack Posts: 5,642 Volunteer Moderator
edited October 26 in General
While some consider them "not VR", viewing 360x180 panoramas on a VR headset is quite cool. Consumer 360 cameras are cheap and software like Hugin and PTGui let you turn normal photographs into panoramas.
There is a problem with panoramas though: depth.

Assuming you are using only one eye to view a panorama in VR, the image will be correct in all directions. Of course there won't be accomodation (focus), consumer VR headsets don't do that yet. The problem comes in when you have stereo vision.
Look around in a panorama using both eyes, everything looks ok. It's fairly flat, but ok. Now look at the ground. Oh crap, I'm flying in the air!
The standard way a panorama is rendered is by projecting it onto the inside of a sphere. This sphere is probably around 30m radius (that's where our stereo depth perception breaks down). This makes things around us look far away, which tends to suit most panoramas. But it also means the ground below us is 30m away! With stereo vision, we feel like we are way up in the air, even though the panorama was captured at head height.

One way to fix this is to use 3d panoramas (stereo images). This captures depth for the entire view. But these have issues too: you can't tilt your head sideways (otherwise your eye alignment won't match the panorama) and it's harder/more expensive to capture.

The technique I'm using in this demo is to add in ground plane geometry and project the panorama onto it, in addition to the usual sky sphere. From a fixed 3DOF view, a ground plane with correct projection will look the same, except you can see the depth is correct. You can tilt your head, the depth is dynamic instead of baked.



This ground plane does magnify another issue with panoramas: they are 3 Degrees Of Freedom only. Well, they don't have to be. In this demo you can change to a 6DOF mode. You can now walk around the panorama. Of course this distorts the view quite a bit. In particular any vertical surfaces projected onto the ground plane look distorted when your head isn't in the exact centre of the panorama space. But it still looks cool (especially if you are looking down, since the nearby floor is less distorted).

I've also added a 4DOF mode. In this mode, you can look around (3DOF), but also move up and down. Horizontal movement is disabled. You can bend down and get a closer look at the ground, with less distortion than the full 6DOF mode.


The demo includes 4 panoramas. Three are from the European Southern Observatory (their website has many awesome panoramas at crazy resolutions). The other is a Nvidia Ansel capture from the game The Witness.


Controls:
A - Toggle ground plane
B - Cycle to the next panorama
X - Cycle to the next degree of freedom. There are 3, 4 and 6 DOF available.
Right Thumbstick click - Reset position/orientation
Index Triggers - Spawn a cube
Grip Triggers - Grab a cube

No, I didn't add cube juggling just for the demo. This was built on a base project for my students, so it already had some hand/physics stuff in there. :)


Demo:
Rift/Rift-S version. Would probably work on SteamVR with a Vive too, but is untested.
https://github.com/rajetic/DepthPanorama/releases/download/v0.1/Depth.Panorama.Test.v0.1.Rift.zip
Quest version. You can upload it using Side Quest

How does it feel? Which degree of freedom mode do you like best? Did you ignore the panoramas and just juggle cubes?

Comments

  • kojackkojack Posts: 5,642 Volunteer Moderator
    edited October 26
    I guess no Rift owners like panoramas. :(

    So here's a Quest build you can sideload (works with SideQuest).

    The controls are the same, except the right thumbstick reset pos doesn't work for some reason (guess that Oculus sdk function isn't Quest compatible).

    I'd fix the controls for a Go version, but my Go is at work.



  • pjennesspjenness Posts: 686
    Neo

    SO..kojak Im interested in this topic as Im playuing around with similar idea.  Are you projecting this in the shader, or are you UVing the plane from the origin as 360o UVs and running the 360 on it as is in texture?
    What Engine are you using? Im interested in both techniques above, and currently using Unity and Maya...
    If there is a shader that just projects from origin or a point in a 360o manner onto any geo where can I find it?  Or what tool are you using to generate 360o based UVs on your geo?

    HOWEVER....

    Those links...as soon as I followed my antiVirus software wen crazy..and the "Downolad buttons they send you all over the place...wont be trying to download those again LOL

    -P
    Drift VFX Visual, Virtual , Vertical

    Want 970GTX on Macbook for good FPS?
    https://forums.oculus.com/viewtopic.php?f=26&t=17349
  • kojackkojack Posts: 5,642 Volunteer Moderator
    Fair enough. I've uploaded to Github instead (although the source isn't available right now, its using a VR code framework I developed for my students).

    The technique is based on a project I did for my masters degree last year.
    All projection is being done using shaders, with (0,0,0) at the location of the camera in the panorama. By adding a ground plane about 1.6m down, it picks up the panorama correctly projected. Well, that's assuming the original camera was 1.6m off the floor. That seems a pretty safe guess for the demo.
    It's currently Unity based. I'm not a Unity fan, but I already had code for it ready to use. That's why there's cube juggling, it was already there for my students.

    You can place other geometry into the scene and they will get covered too, adding more depth info. I was going to make a youtube tutorial on recreating a 3d scene from a single panorama photo in both Blender and Unity, but I didn't know if anybody cared. :)  (Hell, I was close to posting on Reddit)

    My original version had an editing interface that allowed you to place multiple ground planes and control their angles, so you could have sloping floors, including adjusting the player as they walk around to match. But the UI was pretty crap and it didn't work on mobile vr. So this new demo is simpler, but works.

    The same technique works with video too.

    The shader I'm using is:
    Shader "Kojack/PanoSpherical"
    {
    Properties
    {
    _MainTex ("Texture", 2D) = "white" {}
    }
    SubShader
    {
    Tags { "RenderType"="Opaque" }
    LOD 100

    Pass
    {
    Cull off
    CGPROGRAM
    #pragma vertex vert
    #pragma fragment frag
    // make fog work
    //#pragma multi_compile_fog

    #include "UnityCG.cginc"

    struct appdata
    {
    float4 vertex : POSITION;
    float2 uv : TEXCOORD0;
    };

    struct v2f
    {
    float3 world : TEXCOORD0;
    float4 vertex : SV_POSITION;
    };

    sampler2D _MainTex;
    float4 _MainTex_ST;

    v2f vert (appdata v)
    {
    v2f o;
    o.vertex = UnityObjectToClipPos(v.vertex);
    o.world = mul(unity_ObjectToWorld, v.vertex)*float3(-1,-1,1);
    return o;
    }

    fixed4 frag (v2f i) : SV_Target
    {
    float3 p = normalize(i.world);
    float r = sqrt(dot(i.world, i.world));
    float theta = atan2(i.world.z, i.world.x)/(3.14159265*2.0)+0.5;
    float phi = acos(i.world.y / r) / 3.14159265;
    return tex2Dlod(_MainTex, float4(theta,phi,0.0,0.0));
    }
    ENDCG
    }
    }
    }




  • pjennesspjenness Posts: 686
    Neo
    Thanks Kojack will give that a go.
    Drift VFX Visual, Virtual , Vertical

    Want 970GTX on Macbook for good FPS?
    https://forums.oculus.com/viewtopic.php?f=26&t=17349
  • pjennesspjenness Posts: 686
    Neo
    HIya

    So tried it out..works great.

    I think some extending from this point could be:
    1. Arbritrary worlds space origin..so you could have multiple 360 images in your scene and these around the place so that if you teleport to another location you can use a different image etc.
    2. Tying this into a multiple shader/shader so that you can mask this effect in for the local 2-3 metres and then blend into a texture(presumable that matches soemwhat.

    Anyways..this has already help me quite a bit as have been thinking about it myself had never tried.. thanks for the push forward.

    -P

    Drift VFX Visual, Virtual , Vertical

    Want 970GTX on Macbook for good FPS?
    https://forums.oculus.com/viewtopic.php?f=26&t=17349
  • kojackkojack Posts: 5,642 Volunteer Moderator
    The blending idea could be cool, I might give that a shot.
    I was thinking of doing some slight curving, so it's more like a flattened sphere.
     
    There was another technique related to this that I was working on, it was more involved though. The user would draw certain colours onto a copy of the panorama to mark changes between vertical and horizontal surfaces. Using some maths you can actually extract a fair amount of depth info from that, if you know the camera height. Then I'd use that data with a displacement shader on the sphere geometry.
    The main issue was it was time consuming and annoying to edit (in photoshop). I need a better editor for it, or make a VR mode editor (use the controller to draw lines to flag changes in surface direction).

    Thanks for giving it a try. :)
  • pjennesspjenness Posts: 686
    Neo
    Hiya. @kojack
    I have some hdr images I want to use on this also..Im using them in a skybox material successfully..however I cant seem to use them in this (or any general) shader due to the HDR.  
    Are you running into the same or have a way around this?

    -P
    Drift VFX Visual, Virtual , Vertical

    Want 970GTX on Macbook for good FPS?
    https://forums.oculus.com/viewtopic.php?f=26&t=17349
Sign In or Register to comment.