cancel
Showing results for 
Search instead for 
Did you mean: 

Increasing Immersion of 360 Panoramas

kojack
MVP
MVP
While some consider them "not VR", viewing 360x180 panoramas on a VR headset is quite cool. Consumer 360 cameras are cheap and software like Hugin and PTGui let you turn normal photographs into panoramas.
There is a problem with panoramas though: depth.

Assuming you are using only one eye to view a panorama in VR, the image will be correct in all directions. Of course there won't be accomodation (focus), consumer VR headsets don't do that yet. The problem comes in when you have stereo vision.
Look around in a panorama using both eyes, everything looks ok. It's fairly flat, but ok. Now look at the ground. Oh crap, I'm flying in the air!
The standard way a panorama is rendered is by projecting it onto the inside of a sphere. This sphere is probably around 30m radius (that's where our stereo depth perception breaks down). This makes things around us look far away, which tends to suit most panoramas. But it also means the ground below us is 30m away! With stereo vision, we feel like we are way up in the air, even though the panorama was captured at head height.

One way to fix this is to use 3d panoramas (stereo images). This captures depth for the entire view. But these have issues too: you can't tilt your head sideways (otherwise your eye alignment won't match the panorama) and it's harder/more expensive to capture.

The technique I'm using in this demo is to add in ground plane geometry and project the panorama onto it, in addition to the usual sky sphere. From a fixed 3DOF view, a ground plane with correct projection will look the same, except you can see the depth is correct. You can tilt your head, the depth is dynamic instead of baked.

ugwfk3f58erc.jpg

This ground plane does magnify another issue with panoramas: they are 3 Degrees Of Freedom only. Well, they don't have to be. In this demo you can change to a 6DOF mode. You can now walk around the panorama. Of course this distorts the view quite a bit. In particular any vertical surfaces projected onto the ground plane look distorted when your head isn't in the exact centre of the panorama space. But it still looks cool (especially if you are looking down, since the nearby floor is less distorted).

I've also added a 4DOF mode. In this mode, you can look around (3DOF), but also move up and down. Horizontal movement is disabled. You can bend down and get a closer look at the ground, with less distortion than the full 6DOF mode.


The demo includes 4 panoramas. Three are from the European Southern Observatory (their website has many awesome panoramas at crazy resolutions). The other is a Nvidia Ansel capture from the game The Witness.


Controls:
A - Toggle ground plane
B - Cycle to the next panorama
X - Cycle to the next degree of freedom. There are 3, 4 and 6 DOF available.
Right Thumbstick click - Reset position/orientation
Index Triggers - Spawn a cube
Grip Triggers - Grab a cube

No, I didn't add cube juggling just for the demo. This was built on a base project for my students, so it already had some hand/physics stuff in there. 🙂


Demo:
Rift/Rift-S version. Would probably work on SteamVR with a Vive too, but is untested.
https://github.com/rajetic/DepthPanorama/releases/download/v0.1/Depth.Panorama.Test.v0.1.Rift.zip
Quest version. You can upload it using Side Quest

How does it feel? Which degree of freedom mode do you like best? Did you ignore the panoramas and just juggle cubes?

Author: Oculus Monitor,  Auto Oculus Touch,  Forum Dark Mode, Phantom Touch Remover,  X-Plane Fixer
Hardware: Threadripper 1950x, MSI Gaming Trio 2080TI, Asrock X399 Taich
Headsets: Wrap 1200VR, DK1, DK2, CV1, Rift-S, GearVR, Go, Quest, Quest 2, Reverb G2
21 REPLIES 21

kojack
MVP
MVP
I guess no Rift owners like panoramas. 😞

So here's a Quest build you can sideload (works with SideQuest).

The controls are the same, except the right thumbstick reset pos doesn't work for some reason (guess that Oculus sdk function isn't Quest compatible).

I'd fix the controls for a Go version, but my Go is at work.



Author: Oculus Monitor,  Auto Oculus Touch,  Forum Dark Mode, Phantom Touch Remover,  X-Plane Fixer
Hardware: Threadripper 1950x, MSI Gaming Trio 2080TI, Asrock X399 Taich
Headsets: Wrap 1200VR, DK1, DK2, CV1, Rift-S, GearVR, Go, Quest, Quest 2, Reverb G2

pjenness
Rising Star

SO..kojak Im interested in this topic as Im playuing around with similar idea.  Are you projecting this in the shader, or are you UVing the plane from the origin as 360o UVs and running the 360 on it as is in texture?
What Engine are you using? Im interested in both techniques above, and currently using Unity and Maya...
If there is a shader that just projects from origin or a point in a 360o manner onto any geo where can I find it?  Or what tool are you using to generate 360o based UVs on your geo?

HOWEVER....

Those links...as soon as I followed my antiVirus software wen crazy..and the "Downolad buttons they send you all over the place...wont be trying to download those again LOL

-P
Drift VFX Visual, Virtual , Vertical Want 970GTX on Macbook for good FPS? https://forums.oculus.com/viewtopic.php?f=26&t=17349

kojack
MVP
MVP
Fair enough. I've uploaded to Github instead (although the source isn't available right now, its using a VR code framework I developed for my students).

The technique is based on a project I did for my masters degree last year.
All projection is being done using shaders, with (0,0,0) at the location of the camera in the panorama. By adding a ground plane about 1.6m down, it picks up the panorama correctly projected. Well, that's assuming the original camera was 1.6m off the floor. That seems a pretty safe guess for the demo.
It's currently Unity based. I'm not a Unity fan, but I already had code for it ready to use. That's why there's cube juggling, it was already there for my students.

You can place other geometry into the scene and they will get covered too, adding more depth info. I was going to make a youtube tutorial on recreating a 3d scene from a single panorama photo in both Blender and Unity, but I didn't know if anybody cared. 🙂  (Hell, I was close to posting on Reddit)

My original version had an editing interface that allowed you to place multiple ground planes and control their angles, so you could have sloping floors, including adjusting the player as they walk around to match. But the UI was pretty crap and it didn't work on mobile vr. So this new demo is simpler, but works.

The same technique works with video too.

The shader I'm using is:
Shader "Kojack/PanoSpherical"
{
Properties
{
_MainTex ("Texture", 2D) = "white" {}
}
SubShader
{
Tags { "RenderType"="Opaque" }
LOD 100

Pass
{
Cull off
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
// make fog work
//#pragma multi_compile_fog

#include "UnityCG.cginc"

struct appdata
{
float4 vertex : POSITION;
float2 uv : TEXCOORD0;
};

struct v2f
{
float3 world : TEXCOORD0;
float4 vertex : SV_POSITION;
};

sampler2D _MainTex;
float4 _MainTex_ST;

v2f vert (appdata v)
{
v2f o;
o.vertex = UnityObjectToClipPos(v.vertex);
o.world = mul(unity_ObjectToWorld, v.vertex)*float3(-1,-1,1);
return o;
}

fixed4 frag (v2f i) : SV_Target
{
float3 p = normalize(i.world);
float r = sqrt(dot(i.world, i.world));
float theta = atan2(i.world.z, i.world.x)/(3.14159265*2.0)+0.5;
float phi = acos(i.world.y / r) / 3.14159265;
return tex2Dlod(_MainTex, float4(theta,phi,0.0,0.0));
}
ENDCG
}
}
}




Author: Oculus Monitor,  Auto Oculus Touch,  Forum Dark Mode, Phantom Touch Remover,  X-Plane Fixer
Hardware: Threadripper 1950x, MSI Gaming Trio 2080TI, Asrock X399 Taich
Headsets: Wrap 1200VR, DK1, DK2, CV1, Rift-S, GearVR, Go, Quest, Quest 2, Reverb G2

pjenness
Rising Star
Thanks Kojack will give that a go.
Drift VFX Visual, Virtual , Vertical Want 970GTX on Macbook for good FPS? https://forums.oculus.com/viewtopic.php?f=26&t=17349

pjenness
Rising Star
HIya

So tried it out..works great.

I think some extending from this point could be:
  1. Arbritrary worlds space origin..so you could have multiple 360 images in your scene and these around the place so that if you teleport to another location you can use a different image etc.
  2. Tying this into a multiple shader/shader so that you can mask this effect in for the local 2-3 metres and then blend into a texture(presumable that matches soemwhat.

Anyways..this has already help me quite a bit as have been thinking about it myself had never tried.. thanks for the push forward.

-P

Drift VFX Visual, Virtual , Vertical Want 970GTX on Macbook for good FPS? https://forums.oculus.com/viewtopic.php?f=26&t=17349

kojack
MVP
MVP
The blending idea could be cool, I might give that a shot.
I was thinking of doing some slight curving, so it's more like a flattened sphere.
 
There was another technique related to this that I was working on, it was more involved though. The user would draw certain colours onto a copy of the panorama to mark changes between vertical and horizontal surfaces. Using some maths you can actually extract a fair amount of depth info from that, if you know the camera height. Then I'd use that data with a displacement shader on the sphere geometry.
The main issue was it was time consuming and annoying to edit (in photoshop). I need a better editor for it, or make a VR mode editor (use the controller to draw lines to flag changes in surface direction).

Thanks for giving it a try. 🙂
Author: Oculus Monitor,  Auto Oculus Touch,  Forum Dark Mode, Phantom Touch Remover,  X-Plane Fixer
Hardware: Threadripper 1950x, MSI Gaming Trio 2080TI, Asrock X399 Taich
Headsets: Wrap 1200VR, DK1, DK2, CV1, Rift-S, GearVR, Go, Quest, Quest 2, Reverb G2

pjenness
Rising Star
Hiya. @kojack
I have some hdr images I want to use on this also..Im using them in a skybox material successfully..however I cant seem to use them in this (or any general) shader due to the HDR.  
Are you running into the same or have a way around this?

-P
Drift VFX Visual, Virtual , Vertical Want 970GTX on Macbook for good FPS? https://forums.oculus.com/viewtopic.php?f=26&t=17349

Dally_Sparkles
Honored Guest

That's it.. you nailed it i have been looking at my 360 photos in the VR but it was just weird looking down couldn't work out what it was. I understand what you mean about the distortion towards the edges of your flat plane where it meets the sphere because you are looking at the image side ways. was just wondering what if instead of the flat plane what if it was a large radius sphere the ideal shape would be a ball with a bottom that sits flat see image attached360VR sphere?360VR sphere?

relaxculus
Expert Protege

What a difference! Thank you kojack.

It would be nice to see this function in mediaplayer like Pigasus, etc.

But warning, its not goog for every mono 360 pano, only from what was taken from around 1.5m, right? 🙂

So my drone shots doesn't need such correction at all!