cancel
Showing results for 
Search instead for 
Did you mean: 

Dynamic Room Modeling and Unreal ?

motorsep
Rising Star
Just saw this: https://developer.oculus.com/documentation/audiosdk/latest/concepts/release/ and wondering if support for dynamic room modeling is coming to Unreal 4.19 

Thanks beforehand
5 REPLIES 5

holosparkbiz
Explorer
Came here to post the same question. We use the Oculus spatializer including its shoebox reflection system and this looks like a great advance -- we'd love to have it working in Unreal.

Petroza
Heroic Explorer
Hi Folks,

We have a working implementation for UE4 but it's a bit hacky and so we need to clean it up and then package it. No ETA on that, but relatively straight forward to set it up in native code for a project which is probably the best way to go.

First grab the latest native package of the Audio SDK and replace the old DLLs and headers with the new ones in your unreal project. If you're using middleware then just make sure to grab the latest version of the plugin for that middleware version, and the header that comes with it.

Now you just need to set up the raycast callback and call ovrAudio_UpdateRoomModel once per frame. There is some coordinate and unit conversions which make this a little tricky.

The raycast function would look something like this (note I made a global getter for the World, this is a hack in your game you should be able to get the World in a more elegant way):
static const float DISTANCE_SCALE = 0.01f;

FORCEINLINE FVector ToUnrealVector(const ovrAudioVector3f& InVec)
{
return FVector(-InVec.z, InVec.x, InVec.y);
}

FORCEINLINE ovrAudioVector3f ToOVRVector(const FVector& InVec)
{
ovrAudioVector3f Out = { InVec.Y, InVec.Z, -InVec.X };
return Out;
}

static UWorld* GetMainWorld()
{
TArray<APlayerController*> playerList;
GEngine->GetAllLocalPlayerControllers(playerList);

return (playerList.Num() > 0) ? playerList[0]->GetWorld() : nullptr;
}

static void OculusAudioRayCast(ovrAudioVector3f origin, ovrAudioVector3f direction, ovrAudioVector3f* hit, ovrAudioVector3f* normal)
{
auto World = GetMainWorld();

if (World != nullptr)
{
FVector Start = ToUnrealVector(origin) / DISTANCE_SCALE;
FVector End = Start + (ToUnrealVector(direction) * 100000.0f) / DISTANCE_SCALE;

FHitResult Hit;
World->LineTraceSingleByChannel(Hit, Start, End, ECC_Visibility);

*hit = (ovrAudioVector3f)(ToOVRVector(Hit.ImpactPoint* DISTANCE_SCALE));
*normal = (ovrAudioVector3f)(ToOVRVector(Hit.ImpactNormal* DISTANCE_SCALE));
}
}

Then the update function that runs each frame needs to call ovrAudio_UpdateRoomModel, I've also added some debug drawing so you can visualize the room to make sure nothing went wrong. The most common issue is raycasts hitting things like player's own head or some UI elements which screws up the room modeling. If everything is working the virtual room should roughly line up with the actual room.


ovrResult Result = ovrAudio_UpdateRoomModel(OvrAudioContext);

auto World = GetMainWorld();

if (World != nullptr)
{
// draw room box
ovrAudioVector3f Position;
float Dimensions[3], Coefficients[6];
Result = ovrAudio_GetRoomDimensions(OvrAudioContext, Dimensions, Coefficients, &Position);

FVector Center = ToUnrealVector(Position) / DISTANCE_SCALE;
FVector Extent(Dimensions[2], Dimensions[0], Dimensions[1]);
Extent /= 2.0f; // half dimensions
Extent /= DISTANCE_SCALE;
Extent *= 0.99f; // Make it a little smaller to prevent z-fighting

DrawDebugBox(World, Center, Extent, FColor::Purple);

const int HIT_COUNT_MAX = 2048;
static ovrAudioVector3f Points[HIT_COUNT_MAX] = { 0 };
static ovrAudioVector3f Normals[HIT_COUNT_MAX] = { 0 };

// draw hit points
Result = ovrAudio_GetRaycastHits(OvrAudioContext, Points, Normals, HIT_COUNT_MAX);

for (int i = 0; i < HIT_COUNT_MAX; ++i)
{
FVector Position = ToUnrealVector(Points) / DISTANCE_SCALE;
DrawDebugPoint(World, Position, 10.0f, FColor::Green);
FVector LineEnd = Position + ToUnrealVector(Normals) * 3000.0f;
DrawDebugLine(World, Position, LineEnd, FColor::Emerald);
}
}


Anonymous
Not applicable
Hello,

I've been trying to get this to work but I couldn't even get project to include the right headers. We are using the FMOD plugin for Unreal so I tried to download the latest spatializer plugin and the include that comes with it as suggested. Not sure where to put it I placed it in the FMOD plugin's source folder as the dll goes in the plugin folder as well.

However the OculusSpatializerFMOD.h includes a OVR_Audio_DynamicRoom.h which I assume is from the Audio SDK but I have no idea where to download that as all downloads are just plugins or native integration for something else.

Kind Regards


Anonymous
Not applicable
I got it working with help from our Sound Designer Holley Gray talking with Peter Stirling directly.

First things first:
- We are using the spatializer for the Unreal 4 plugin of FMOD
- The OVR_Audio_DynamicRoom.h was missing from a previous release, it's been included in the new 1.24 one
- There is no .lib file so the dll needs to be loaded directly (making the header files themselves only useful for referencing)
- Our Sound Designer I haven't been able to check the results yet, but the debug traces look correct

That said, this is how we load the dll in Unreal, access the ovr methods and register the callbacks.

At the start I needed to wrap the stuff from the header into function pointer defines for the dll handles:
//Typedefs copied from the OculusSpatializerFMOD.h file
#include "OVR_Audio_DynamicRoom.h"
#include <stdint.h>

#ifndef OVR_RESULT_DEFINED
#define OVR_RESULT_DEFINED
typedef int32_t ovrResult;
#endif

//Handle to the dll
void* OVRAudioDLLHandle;

//Typedefs for the function calls, there are more methods exposed, but we only need those for now
typedef ovrResult(*_OSP_FMOD_GetRaycastHits)(ovrAudioVector3f points[], ovrAudioVector3f normals[], int length);
typedef ovrResult(*_OSP_FMOD_AssignRayCastCallback)(OVRA_RAYCAST_CALLBACK callback, void* pctx);
typedef ovrResult(*_OSP_FMOD_UpdateRoomModel)(float wetLevel);
typedef ovrResult(*_OSP_FMOD_GetRoomDimensions)(float roomDimensions[], float reflectionsCoefs[], ovrAudioVector3f* position);

_OSP_FMOD_GetRaycastHits OSP_FMOD_GetRaycastHits;
_OSP_FMOD_AssignRayCastCallback OSP_FMOD_AssignRayCastCallback;
_OSP_FMOD_UpdateRoomModel OSP_FMOD_UpdateRoomModel;
_OSP_FMOD_GetRoomDimensions OSP_FMOD_GetRoomDimensions;

//Raycast callback
static void OculusAudioRayCast(ovrAudioVector3f origin, ovrAudioVector3f direction, ovrAudioVector3f* hit, ovrAudioVector3f* normal, void* pctx);

Next comes the actual loading and assignemnt of the dll handles, we have a manger class so it's in the BeginPlay function of that. Error handling is a bit dodgy, you might want to be more careful 😉
FString DLLFilePath = FPaths::ProjectPluginsDir() + "/FMODStudio/Binaries/Win64/OculusSpatializerFMOD.dll";

//Load the dll
FString DLLFilePath = FPaths::ProjectPluginsDir() + "/FMODStudio/Binaries/Win64/OculusSpatializerFMOD.dll";

if (!FPaths::FileExists(DLLFilePath))
{
UE_LOG(LogTemp, Error, TEXT("DLL file does not exist %s"), *DLLFilePath);
return;
}

OVRAudioDLLHandle = FPlatformProcess::GetDllHandle(*DLLFilePath);
if (OVRAudioDLLHandle == nullptr)
{
UE_LOG(LogTemp, Error, TEXT("Could not get dll handle for %s"), *DLLFilePath);
return;
}

//Get all the needed function handles
OSP_FMOD_GetRaycastHits = (_OSP_FMOD_GetRaycastHits)FPlatformProcess::GetDllExport(OVRAudioDLLHandle, *FString("OSP_FMOD_GetRaycastHits"));
if (OSP_FMOD_GetRaycastHits == nullptr)
{
UE_LOG(LogTemp, Error, TEXT("Could not get proc handle for OSP_FMOD_GetRaycastHits"));
}

OSP_FMOD_AssignRayCastCallback = (_OSP_FMOD_AssignRayCastCallback)FPlatformProcess::GetDllExport(OVRAudioDLLHandle, *FString("OSP_FMOD_AssignRayCastCallback"));
if (OSP_FMOD_AssignRayCastCallback == nullptr)
{
UE_LOG(LogTemp, Error, TEXT("Could not get proc handle for OSP_FMOD_AssignRayCastCallback"));
}

OSP_FMOD_UpdateRoomModel = (_OSP_FMOD_UpdateRoomModel)FPlatformProcess::GetDllExport(OVRAudioDLLHandle, *FString("OSP_FMOD_UpdateRoomModel"));
if (OSP_FMOD_UpdateRoomModel == nullptr)
{
UE_LOG(LogTemp, Error, TEXT("Could not get proc handle for OSP_FMOD_UpdateRoomModel"));
}

OSP_FMOD_GetRoomDimensions = (_OSP_FMOD_GetRoomDimensions)FPlatformProcess::GetDllExport(OVRAudioDLLHandle, *FString("OSP_FMOD_GetRoomDimensions"));
if (OSP_FMOD_GetRoomDimensions == nullptr)
{
UE_LOG(LogTemp, Error, TEXT("Could not get proc handle for OSP_FMOD_GetRoomDimensions"));
}

//Register the callback
OVRA_RAYCAST_CALLBACK Callback = &OculusAudioRayCast;
OSP_FMOD_AssignRayCastCallback(Callback, this);


The tick method of our manager contains the update room model code from the post above. Just replace the methods with their OSP_FMOD_* counter part.
//Example call
ovrResult Result = OSP_FMOD_UpdateRoomModel(Wetness);

   
Lastly in EndPlay everything is getting cleaned up:
OSP_FMOD_GetRaycastHits = nullptr;
OSP_FMOD_AssignRayCastCallback = nullptr;
OSP_FMOD_UpdateRoomModel = nullptr;
OSP_FMOD_GetRoomDimensions = nullptr;

FPlatformProcess::FreeDllHandle(OVRAudioDLLHandle);
OVRAudioDLLHandle = nullptr;






jumbli
Heroic Explorer
Thanks @FreetimeCoder for posting this. I got the code running, but it doesn't sound right to me. Have you had good results?

I can see successful raycasts hitting appropriate walls, but the shoebox reverb model doesn't get close to walls when in long thin rooms. Instead they seem to use an average of the raycasts to define the shape.

I seem to find that using the dynamic room model I get some nice reverb but it hardly changes when the shape of the room changes and there are no distinct early reflections.

The video below shows a comparison between the dynamic mode and legacy mode. In the legacy mode I get strong early reflections from expected directions that highlight the shape of the room. 

@PeterStirling , can you please let me know if the shoebox reverb model should closely reflect the room size or is the behaviour in the video to be expected. I'm using the code above with FMOD and UE4.18 and the Oculus Audio Spatializer Plugin for FMOD 1.24.0 - thanks.

https://youtu.be/4mM5I1SP06o
Developer of Dimensional, The Relentless, Breath Tech, Jigsaw 360 View my dev blog at JumbliVR.com