Welcome to the Oculus Developer Forums!

Your participation on the forum is subject to the Oculus Code of Conduct.

In general, please be respectful and kind. If you violate the Oculus Code of Conduct, your access to the developer forums may be revoked at the discretion of Oculus staff.

360x180 CG Renders

ghostlyfughostlyfu Posts: 30
Brain Burst
I'm facing a problem when rendering any pair of equirectangular images from inside a 3D modelling package (Maya in my case, but no doubt the same issues apply to many different packages). The attached (side-by-side format) image is an example. I'm hoping the good folk on this forum can help.

I render out my panorama (I use the Lat-Lon Mental Ray lens plug-in) and it looks as you'd expect. Looking at it through the excellent VRPlayer, it works well. I then shift my camera to another position approx 6.5 'virtual' cm to the side of the first position and re-render. Again this panorama works fine on its own.

I was hoping that I could just place these x2 renders side-by-side (as they are in the attached example) to make a 3D version of the scene. Not so. What's happening is that the 'forward' view works fine but the depth gets reversed towards the peripheries of the image (which just looks bad) until the 'back' view is completely inverse (and so look pretty much OK again).

Logically I can see that the reason is that the 'Right camera' becomes the 'Left camera' as it renders the part of the view directly behind itself. However, I haven't got a clue how to fix it and output an equirectangular panorama that works in 3D. Help!
«13456

Comments

  • mediavrmediavr Posts: 234
    Art3mis
    There are a couple of ways of rendering spherical 3d panoramas ..
    see here for example
    http://paulbourke.net/papers/vsmm2006/vsmm2006.pdf
  • ghostlyfughostlyfu Posts: 30
    Brain Burst
    Many thanks mediavr - I'll chew through this with interest, I couldn't find anything which discussed CG render + 360 panoramas + stereoscopy. Any other info much appreciated
  • mediavrmediavr Posts: 234
    Art3mis
    When he talks about synthetic stereoscopic panoramas he is talking about CG renders. A recipe to do what you want is this - the article he describes this - with variants
    1. Set up a virtual camera rig with two parallel cameras say 3" apart.
    2. Configure each camera to render in equirectangular (spherical) mode -- except that you want each camera to render say 1 degree wide, 180 degree high strips -- your renderer will have to support partial equirectangular renders.
    3. Have the dual camera setup rotate 360 degrees on a point half way between each camera, rendering from each camera the 1 degree wide strips at each successive 1 degree increment.
    4. Now you have 360 L and R 1 degree wide, 180 degree high strips. They make very skinny pairs of stereo images.
    5. Join (concatenate) the L strips by assembling them side by side (you can use the Mosaic tool in Stereophoto Maker to do this). ditto the R strips.
  • ghostlyfughostlyfu Posts: 30
    Brain Burst
    Thanks again mediavr. I was on the wrong track initially because I was using the Metal Ray lens shader to give instant (and very nice) *monoscopic* 360 renders. But as the virtual camera doesn't move for this, the same technique couldn't be used for stereoscopic views.

    For anyone wishing to do the same, I'm starting to have better luck moving both virtual cameras (in tandem) around a common point, as suggested by mediavr. This is very similar to assembling 'real world' (photographic) equirectangular pans. So far I haven't needed to render the views in really thin strips (which I don't know how to do anyway).

    So rendering, say 90 degree views with overlaps and assembling these in PTGui seems to work. My next problem is that I can't get PTGui to any stitch two (L &R) pans in identical ways (even using batch mode - and yes I've been through the online tutorials!) so the 3D illusion doesn't work yet.

    Ah well, another day another technical Eiger for me to climb...
  • mediavrmediavr Posts: 234
    Art3mis
    You are doing things the hard way. Here is a more detailed recipe.
    1. Find a renderer for 3ds Max that will let you render partial equirectangular images. VRay I think can do it.
    http://www.pixelsonic.com/2011/04/360%C2%B0-in-3ds-max-with-vray-2/
    See where in the settings where it says "Camera type" and it is set to "spherical".
    See where it says "Override FOV". Set it to 1 degree here.
    Then you can render equirectangular (aka spherical) images from each camera of your rotating pair that are 1 degree wide and 180 degree high.

    2. Now you have 360 images from the Left camera say. Then open Stereophoto Maker.
    http://stereo.jpn.org/eng/stphmkr/
    Go File/Mosaic images
    Choose "Connect all files"

    3. Do the same thing with the images from the Right camera.
  • ghostlyfughostlyfu Posts: 30
    Brain Burst
    Thanks again - don't suppose you know of any partial equirectangular render plug-in for Maya? I can only find full panorama renderers. I did get stereophotomaker and it looks very promising.
  • ghostlyfughostlyfu Posts: 30
    Brain Burst
    I installed the demo of Vray for Maya but the (Spherical camera type/1 degree FOV) override doesn't seem to work as you describe. I end up with a very narrow horizontal strip when I connect the renders so perhaps Vray can't unhook the vertical and horizontal proportions to generate the required 'long strip' render sections?
  • mediavrmediavr Posts: 234
    Art3mis
    Well you might have to go back to rectilinear renders -- render each view 1 degree wide, 140 degree high
    Load them into PTGui (you might have to use PTGui Pro)
    Set the Panorama output to Equirectangular 1 degree by 180 degree.
    In the Render section set the output to "Individual layers"

    Join them with Stereophoto Maker

    Then you will have black nadirs and zeniths but the central zone should be ok

    Maybe some of the dome renderers here might be usable somehow
    http://thefulldomeblog.com/2013/06/28/fisheye-lens-shader-options/
  • mediavrmediavr Posts: 234
    Art3mis
    Well you might have to go back to rectilinear renders -- render each view 1 degree wide, 140 degree high
    Load them into PTGui (you might have to use PTGui Pro)
    Set the Lens information to FOV 1 degree, type Rectilinear
    Set the Panorama output to Equirectangular 1 degree by 180 degree.
    In the Render section set the output to "Individual layers"

    Join them with Stereophoto Maker

    Then you will have black nadirs and zeniths but the central zone should be ok

    Maybe some of the dome renderers here might be usable somehow
    http://thefulldomeblog.com/2013/06/28/fisheye-lens-shader-options/
  • cheerioboycheerioboy Posts: 69
    NerveGear
    mediavr! you're full of useful info! It just clicked to me, reading how you setup vray with a spherical camera but keep the fov at 1. This whole time I've been setting my camera to spherical with 360 fov, and then cropped on a strip of the view to render. This always caused me having to figure out the pixel overlap when using that stereo maker.

    can't wait to try this method out :lol:

    although this is still a very cumbersome approach.. I have renewed interest in using since nothing else is out there. I have been using single spherical renderings along with a depth map in vr player. Which gives you a quasi-acceptable view - but it's still not ideal - and even more difficult when working with GI. If not using a brute force approach you'd need to pre-calc the movement of the cameras to avoid any issues. Although I haven't extensively tested this. And it makes rendering 3d animation almost impossible... although now with your 1 fov trick I might be able to find another way to stitch the batch of slices.

    I'll keep prodding vray to update their stereo helper to work with spherical cameras :)

    cheers
  • cheerioboycheerioboy Posts: 69
    NerveGear
    So with this 1 degree of fov, what's a suggested resolution? I just tried 1 x 1080, which gave me a squeezed image (duh) - going to try 5 x 1080 to get something closer to 1920x1080.

    Is there no other software that will stitch/line up a sequence of frames into a full frame? I'm thinking AE, Nuke, some sort of compositing program there ought to be a way to automate it to make this feasible for animation. At least for now.
  • cheerioboycheerioboy Posts: 69
    NerveGear
    2 tests, both rendered with a spherical camera & 1 degree of fov

    the first was 1 pixel wide, 1080 high - the second was 5 pixel wide, 1080 high.
    next round is changing the fov to 5, for 5 pixels wide to see if that works...

    update: still struggling with how to up scale the 360x180 resolution

    My thinking was that since the camera was moving around 360 frames, 1 pixel wide each time, if I increased the frames around I could increase the width. So currently I'm rendering 1920 frames in a circle, with 1080 height... but that just looks like 360x180, resized. So the content looks blurry.

    Does anyone know the best resolution for a spherical image viewed inside the Oculus? Which I suppose will soon double when the new one arrives..
  • mediavrmediavr Posts: 234
    Art3mis
    Try rendering a 5 pixel wide by 900 pixels high strip -- at 1 degrees wide (5 pixels = 1800/360) then when you join the strips the end result should be 1800 by 900
  • NukemarineNukemarine Posts: 493
    Hiro Protagonist
    I'm no expert, but your 5x360 render looks like each slice was mirrored prior to stitching. For a curved surface it's creating a saw tooth pattern. This may also explain the blurriness in 1x360 image. Hopefully there's some setting that can make sure the slices are rendered left to right and the stitching is also left to right. For the 1x360, Is there a way to increase the frames with an even smaller FOV?

    By the way, I've viewed both images in the Oculus so the 360 feeling is great but you left a 5 degree black strip that shows up behind me on the 5x360 render, so make sure that's cut off in the rendered frame although this may be an artifact of rendering technique. Resolution for an Dev Kit Oculus is at around 6 pixels per degree of view, so a photo that's 1080x1920 should look fine for now if people keep a 110 FOV. For a 75 FOV that same photo will look blurrier because you're basically magnifying the image 1.5x or so.
  • cheerioboycheerioboy Posts: 69
    NerveGear
    thanks for the responses guys, I'm going to test them!

    since that post I thought I had nailed the sizing issue by rendering 1920 frames at 1 fov, 1 pixel wide x 1080 high. But it gave me quite a strange result. See attached. It looks like a resized version of the original small resolution image :P - the smaller two, which I did exactly as mediavr mentioned with the 360x180 render which I think came out right (haven't tested it).

    Also the comment nukemarine made about the direction of stitching got me thinking.. and I realized my cameras were rotating counter clockwise! So I wonder what effect that had on the setup...
  • cheerioboycheerioboy Posts: 69
    NerveGear
    mediavr, you're too damn good! I followed your recipe and got some beautiful renders. now I want to build a set of these that slowly gets more pixel dense until we can find the best resolution for the current oculus.

    1080x1920 is a good place to start. or is it 1920x1080?
  • NukemarineNukemarine Posts: 493
    Hiro Protagonist
    Wow! Now that worked great. I even took your Left eye and Right eye 5x900 renders, combined them in Paint as a SBS format and got decent stereoscopic viewing in VR Player. What's interesting is I didn't realize the front character's hair was sticking out way in front until I saw it in 3D.

    From testing, the spherical render is stereoscopic 3D starts to fail when you start to look up and down near the poles.

    Here's the image I merged:

    http://i.imgur.com/9Bj8L3a.jpg

    Anyway, the 5x900 looks much better than the 1x900 for whatever reason. I'm really impressed with the results.

    Now I'm wondering if such a technique can be done to take 360 degree screenshots in game. There's a big difference seeing just one angle and being able to see every angle of a location. Of course, after that comes the natural question: can we record game video in 360 degrees. My guess is that it can be done at a lower frame rate and resolution, but from my experience that's not a bad thing as viewing low res, low framerate in VR Player is still an enjoyable experience.
  • cheerioboycheerioboy Posts: 69
    NerveGear
    @nukemarine

    I'm going to guess it'll be difficult to take screenshots in-game at the moment. If it takes this long just to render a still image in a 3d programe, ie, Maya or 3dsmax. Unless the game is rendering the full 360 image and not just what you're seeing on the screens in your headset. Otherwise each time you hit a button to capture the image - a new camera would need to 'scan' the environment and save out. What would be even cooler is if the screen capture wasn't just an image but capturing the 3d environment, with baked textures and all.

    But you know how these things work, all in due time and we might be able to some cool programming wizardry to make ways to automate this process into an image captured in an instant.
  • j1vvyj1vvy Posts: 143
    Brain Burst
    It is a matter of incorporating the panoramic screen capture or panoramic video capture into the game engine. Game engines can already provide instant replays or screen captures it is just a matter of doing it with a different camera setup. The 360° s3D video would be too much to render in real time for most systems, but right after would be fine.

    I think the largest but still easy to encode video is as over/under in .mp4 in h264 in 4k video size. That would be 3072X3072 pixels same number of pixels as 4k's 4096X2304. Two panos of 360°X180° and 3072X1536 pixels each in spherical format.

    Using physical cameras to produce a similar result I would use as few cameras, or images, as possible to simplify workflow. But doing it with a game engine I would use as many cameras as possible change the camera for ever column of pixels. Set the camera to one pixel wide and a FoV of 360/width. Rotate the cameras 360/width each time.

    A prerendered 360° S3D video would not show the correct 3D when the viewer rolls his head and the zenith and nadir will result if very little S3D.
  • cheerioboycheerioboy Posts: 69
    NerveGear
    hey mediavr, do I have this right? I'm trying to take your resolution change and adjust it for various sizes --
    all sizes are 360 slices, at 1 fov

    1800x900
    - Set Output Size, Width: 5, Height: 900 - (5 pixels = 1800/360)

    HD'ish, 2160x1080 (1920x1080)
    - Set Output Size, Width: 6, Height: 1080

    4k'ish, 3240x1536 (3072X1536)
    - Set Output Size, Width: 9, Height: 1536

    I got it? I'm doing some internal tests but afraid to go higher res and have them be screwed up :oops:
  • NukemarineNukemarine Posts: 493
    Hiro Protagonist
    cheerioboy wrote:
    hey mediavr, do I have this right? I'm trying to take your resolution change and adjust it for various sizes --
    all sizes are 360 slices, at 1 fov

    1800x900
    - Set Output Size, Width: 5, Height: 900 - (5 pixels = 1800/360)

    HD'ish, 2160x1080 (1920x1080)
    - Set Output Size, Width: 6, Height: 1080

    4k'ish, 3240x1536 (3072X1536)
    - Set Output Size, Width: 9, Height: 1536

    I got it? I'm doing some internal tests but afraid to go higher res and have them be screwed up :oops:

    Cheerioboy, hope you don't mind me posting your work on the reddit page for Oculus. Looking forward to see what you can generate.

    I wondering a few things. One is have you been testing out Depth Map option which is discussed in this thread. I only found out about it today, but it seems to work great for VR Player. Edit: Seems I've been misunderstanding how effective depthmap can be. The more 3D you give it, the more artifacts you create. Sorry about that.

    Two is can you create an scene that has the grid layout, but with a 1.8 meter pole or character positioned further and further away at 3 meter intervals (maybe in a slight curved pattern so each object can be seen fully ). Then recreate this at different heights as the person turns around the scene. In front of me, all the characters are ground height, 45 degrees to my right, they're 1m off the ground, 90 degrees to my right they're 2m off the ground, etc. This then creates something visually that people can use to adjust their 3D viewer settings such as changing their FOV and horizontal seperation.

    Three, if you can do the above, can it be rendered in a Cube, Spherical and Cylinder layout in Mono, Over/Under, and Depth Map. My thinking is Stephane who created VR Player could use such renders for his Samples file which to me would be vastly superior to the current images he's using (cylinder view of a hallway, spherical view of a city street).
  • cheerioboycheerioboy Posts: 69
    NerveGear
    nukemarine, share away :)

    I'm now re-designing the scene based on your feedback - i have an array of human figures that go out into the distance, as well as spiraling up. Added some color shifting to make them easier to spot. I also displaced the ground to add something interesting to view there. Any other suggestions before I start rendering?

    I've attached where it's at, along with a depth map for you to check out. That post you linked is my post :) - I've been using the method of a single spherical render along with a depth map for several projects at work. As much as I hate the fact that it does a poor job, it's the easiest way to output images for client review within the short time frames that I'm given.

    I hope to perfect this slice method and start pushing out renderings to win more people over and hopefully development in cameras that automate this process!
  • NukemarineNukemarine Posts: 493
    Hiro Protagonist
    Thanks Cheerioboy. Just as I thought, I really over-estimated how effective depth maps could be in this. The 3D scene itself is really trippy and a little upsetting if you imagine as if you're being judged by mannequins. If you're able to post a Left Eye/Right Eye render like you did above it'll be great. I'm kind of tempted to learn how to do this myself now, though I don't know how well my laptop will pull it off.

    Here's the combined image for anyone that wants to try it out in VR Player, but the 3D Depth Map effect is not very good.
  • cheerioboycheerioboy Posts: 69
    NerveGear
    update! http://imgur.com/a/yj1y0
    edit: http://imgur.com/a/YsFc9 (precompiled a few - had trouble uploading the 8k image w/depth map)

    So I rendered both the left/right version at 8k, and then the single pano w/depth map. I figure with the largest size, you can scale it down in increments and save them out to determine what size is optimal for viewing with the Oculus. So I'd be interested in seeing how well the image holds up - 720hd, 1080hd, 4k & 8k -

    Something strange happened to the card floating below the view - in the left & right renders it rendered strange, while in the single pano you see the gradient of color on it properly... hmm.

    Are these left/right renders looking alright? I've wondered about whether the cameras need to diverge on a focal point. At the moment they both point straight ahead with no crossing. I think it's called parallel vs toe-in, like Paul mentions here, http://paulbourke.net/stereographics/stereopanoramic/.

    I've mentioned to VR Player that they might include a way to shift one view to create this toe-in, if it is indeed something needed for proper viewing.
  • NukemarineNukemarine Posts: 493
    Hiro Protagonist
    Cheerioboy,

    I checked all seven versions. The depth map setting does not seem good for this type of scene. As I know almost nothing about it, I don't know if it depth map works best with large objects up close with minimal or zero background. It worked with the flower in the VR Player sample folder and sort of worked with the teacup render.

    With the S3D, it looked great and had no problem seeing depth. There were rendering artifacts such as a purple line on the two-tone human, the reflection in the T-Rex seemed pixelated. There seemed little benefit after x1080, but I think that's the limitation of the developer kit resolution. Anyone that has as HD prototype or better will be able to say if it helps. I recommend that you save these as over/under both to help with image servers (might limitations on maximum dimensions for uploads) and to make the thumbnail easier to view (OU allows for larger thumbnails since it's scaled to the longest dimension).

    If anything, the S3D shows that this scene has too many humans in it beyond what's needed to determine depth. Perhaps just humans on the ground level that spiral out into further distances, but with lots of spacing inbetween so there's no overlap. Throughout the scene, like you did here, put in other objects for scale (except for a banana). People can do the left/right eye blink to see the offset.

    I also noticed the spherical layout of the image at the bottom of the render. Did you put that image there to help determine how the slicing is doing during rendering? It's a minor distortion compared to how cool the stereoscopic 3D in the rest of the scene looks.
  • cheerioboycheerioboy Posts: 69
    NerveGear
    nukemarine, for the depth map setting - did you try playing around with the max offset within the settings / effects ? You should be able to dial the offset to something view-able. But you're right, it's not effective in selling you on the depth.

    I think what works best for gauging the depth is setting an indoor scene with more objects that are closer to you - distinct near, mid, and far objects - I'll see what I can do going forward. I have plans for a side project that I'd like to render in stereo pano.

    I believe what sells depth the most is having subtle parallax shifting, as your head sways around you see tiny bits of parallax in your environment. I know with the new Oculus setup this would now be a possibility - I just don't know if there will be an easy way to setup a rendered scene to take advantage of it - maybe through having a stereo pair & depth maps to further distort the images based on head shifting.
  • cheerioboycheerioboy Posts: 69
    NerveGear
    I was viewing the images rendered and noticed that the ground right below the viewer does a bit of a twist. I'm guessing this is unavoidable? Will do a simpler test to show it - and see if it's only affected from the higher resolutions

    Too bad there's not a easy way to cycle through images with VR Player - I'd love to compare the left/right renders vs the depth map so really see the difference. Also interested in separating the eyes further. It might make sense to exaggerate the eye distance to create a more pronounced feeling of depth.
  • j1vvyj1vvy Posts: 143
    Brain Burst
    To see the differences open two instances of VRPlayer. Just Alt+Tab between them.

    What are the VRPlayer settings to see an over/under depth map shown with Anaglyph?
  • cheerioboycheerioboy Posts: 69
    NerveGear
    @ j1vvy

    Genius! I'll give the ol' alt+tab a go :)

    I'm not very familiar with anaglyph images and VR Player, but I don't believe you need a depth map. For an anaglyph image you'd set your media settings to: Mono / choose your projection preference / Anaglyph
  • j1vvyj1vvy Posts: 143
    Brain Burst
    cheerioboy wrote:
    @ j1vvy
    I'm not very familiar with anaglyph images and VR Player, but I don't believe you need a depth map. For an anaglyph image you'd set your media settings to: Mono / choose your projection preference / Anaglyph

    Although I am making content for the Rift, I don't have a Rift yet. I am using Anaglyph glasses to view the images in 3D.

    I think it is a bug in VRPlayer
    Anaglyph should be a viewing method under Device -> Layout But it is listed under Media -> Effect instead.

    My current method of looking at over/under with Anaplyph glasses is:
    For anaglyph glasses:
    Media -> format -> Mono
    > projection -> Sphere
    > Effect -> Anaglyph

    Device -> Layout -> Mono
    > Distortion -> None
    >Tracker -> Mouse

    Tools->Plugins->Effects->Anaglyph
    > Stereo mode -> Over / Under
    >Anaglyph type -> Red-Blue
Sign In or Register to comment.