cancel
Showing results for 
Search instead for 
Did you mean: 

FMOD DSP

Carandiru
Honored Guest
I'm not sure if the current aloha of the Oculus Audio SDK supports this with FMOD.
I tried adding to the tail of the FMOD DSP stack. I'm using the low-level portion of FMOD Studio (addDSP).

I was hoping I could do this, and apply it to dynamically generated sounds in FMOD.

Everything loads fine, oculus Plugin does get added to the DSP.

However, the final sound output sounds like a really creepy, dark sinister voice.

Hehe, so am I doing something wrong, or is this not currently supported with FMOD?

Thanks!
http://www.supersinfulsilicon.com/ supersinfulsilicon - software Home of the MaxVR Oculus Rift Video Player https://twitter.com/Carandiru
8 REPLIES 8

Petroza
Heroic Explorer
This is supported and tested with FMOD low level so it might be that you're doing something wrong. Can you post a code snippet where you load, create and add the plugin?

Anonymous
Not applicable
'creepy, sinister voice' might be a sample rate mismatch?

Carandiru
Honored Guest
Had another look at it today, tried to simplify a few things aswell. Still get that strange voice!
As far as samplerate, it's set to the default 48000hz, and I've also configured the audio control panel in windows to match. I've tried adding the DSP in parallel via a tutorial in the FMOD docs aswell (reverb example). Same result except mixed with the original sound + OSP sound. Original sound is correct, OSP is deep voice.

However it should be a serial operation, with OSP right in the main DSP chain (which is how I originally intended).
This is the code I have built for this:

I should also note that formatNew->nChannels is equal to 6 and formatNew->nSamplesPerSec is equal to 48000hz
for the test content I'm using to play (so there is no resampling).

I have also tried
m_FMODSystem->setSoftwareFormat(m_iFMODSampleRate, FMOD_SPEAKERMODE_5POINT1, 0);
which results in very incorrect sound, however I believe is should be set to FMOD_SPEAKERMODE_STEREO as this results in sound from the OSP. I believe the docs with the Oculus Audio SDK say to set it to Stereo aswell. Is this correct?

Here is the Initialization:

m_sSoundInfo = { 0 };
m_sSoundInfo.cbsize = sizeof(FMOD_CREATESOUNDEXINFO);
m_sSoundInfo.suggestedsoundtype = FMOD_SOUND_TYPE_RAW;
m_sSoundInfo.channelorder = FMOD_CHANNELORDER_WAVEFORMAT; // important!
m_sSoundInfo.channelmask = FMOD_CHANNELMASK_5POINT1_REARS;
m_sSoundInfo.numchannels = formatNew->nChannels;
m_sSoundInfo.defaultfrequency = formatNew->nSamplesPerSec;
m_sSoundInfo.format = FMOD_SOUND_FORMAT_PCM24;

FMOD_RESULT result = FMOD::System_Create(&m_FMODSystem); // Create the main system object.
if (FMOD_OK != result) {
LOGF(WARNING, "Create FMOD Main System FAILED(%d)", result);
return(E_FAIL);
}

int iSampleRate(DEFAULT_FMOD_SAMPLERATE);
result = m_FMODSystem->getSoftwareFormat(&iSampleRate, nullptr, nullptr);
if (FMOD_OK == result) {
m_iFMODSampleRate = iSampleRate;
LOGF(INFO, "FMOD Mixer SampleRate: %dHz", iSampleRate);
}
else {
LOGF(WARNING, "FMOD getSoftwareFormat FAILED(%d)", result);
return(E_FAIL);
}

unsigned int bufferSize(0);
result = m_FMODSystem->getDSPBufferSize(&bufferSize, nullptr);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD getDSPBufferSize FAILED(%d)", result);
return(E_FAIL);
}

if (!OSP_FMOD_Initialize(iSampleRate, bufferSize))
{
LOGF(WARNING, "OSP_FMOD_Initialize FAILED");
return(E_FAIL);
}

if (!OSP_FMOD_SetSimpleBoxRoomParameters(5.0f, 2.1f, 3.7f,
0.75f, 0.65f, 0.55f, 0.25f, 0.65f, 0.65f))
{
LOGF(WARNING, "OSP_FMOD_SetSimpleBoxRoomParameters FAILED");
return(E_FAIL);
}

m_FMODSystem->setSoftwareFormat(m_iFMODSampleRate, FMOD_SPEAKERMODE_STEREO, 0); // FMOD_SPEAKERMODE_STEREO Final Format Used (Headphones)

result = m_FMODSystem->init(32, FMOD_INIT_NORMAL/*FMOD_INIT_STREAM_FROM_UPDATE | FMOD_INIT_THREAD_UNSAFE*/, 0);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD Init FAILED(%d)", result);
return(false);
}

result = m_FMODSystem->loadPlugin(m_szOSPAudioPlugin, &m_hOSPAudio);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD Load Plugin OSP FAILED(%d)", result);
return(false);
}

result = m_FMODSystem->createDSPByPlugin(m_hOSPAudio, &m_dspOSP);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD Create DSP OSP FAILED(%d)", result);
return(false);
}

/*FMOD_ADVANCEDSETTINGS sAdvanced = { 0 };
sAdvanced.cbSize = sizeof(FMOD_ADVANCEDSETTINGS);
sAdvanced.resamplerMethod = FMOD_DSP_RESAMPLER_SPLINE;
sAdvanced.maxPCMCodecs = 1;
sAdvanced.HRTFMinAngle = 180.0f;
sAdvanced.HRTFMaxAngle = 360.0f;
sAdvanced.HRTFFreq = 4000.0f;

result = m_FMODSystem->setAdvancedSettings(&sAdvanced); // Initialize FMOD.
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD setAdvancedSettings FAILED(%d)", result);
return(false);
}*/

result = m_FMODSystem->getMasterChannelGroup(&m_channelgroup);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD getMasterChannelGroup FAILED(%d)", result);
return(false);
}

result = m_channelgroup->setPaused(true);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD setPaused FAILED(%d)", result);
return(false);
}

result = m_channelgroup->setVolume(0.9f);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD setVolume FAILED(%d)", result);
return(false);
}

// Plugin OSP
result = m_channelgroup->addDSP(1, m_dspOSP);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD addDSP OSP FAILED(%d)", result);
return(false);
}


And here is where I actually play the sound:


__declspec(restrict)FMOD::Channel * const FMODFilterManager::queueNextSound(CMediaBufferSampleWrapper const* const __restrict pSample, FMOD::Channel* const __restrict prevChannel, FMOD::Sound ** const __restrict curSoundSlot)
{
FMOD_RESULT result;
DWORD dwLength(0);
FMOD::Channel * __restrict newChannel;
char const* __restrict pBuffer(nullptr);

pSample->GetBufferAndLength(&pBuffer, &dwLength);

FMOD_CREATESOUNDEXINFO soundInfoCurLength(m_sSoundInfo);
soundInfoCurLength.length = dwLength;

result = m_FMODSystem->createSound(pBuffer, FMOD_OPENMEMORY_POINT | FMOD_OPENRAW | FMOD_IGNORETAGS | FMOD_LOWMEM | FMOD_UNIQUE, &soundInfoCurLength, curSoundSlot);

if (FMOD_OK != result)
return(nullptr);

result = m_FMODSystem->playSound(*curSoundSlot, m_channelgroup, true, &newChannel);

if (FMOD_OK != result)
return(nullptr);

if (nullptr == prevChannel)
{
result = PrimeFMODStream(&prevChannel, &soundInfoCurLength);
if (FMOD_OK != result) {
LOGF(WARNING, "PrimeFMODStream FAILED(%d)", result);
return(nullptr);
}
}

/* We want all channels to be sent as 5.1 instead of very end of DSP //
FMOD::DSP *channel_dsp_head;

result = newChannel->getDSP(FMOD_CHANNELCONTROL_DSP_HEAD, &channel_dsp_head);
if (FMOD_OK != result)
return(nullptr);
result = channel_dsp_head->setChannelFormat(FMOD_CHANNELMASK_5POINT1_REARS, 6, FMOD_SPEAKERMODE_5POINT1);
if (FMOD_OK != result)
return(nullptr);
result = newChannel->setMixLevelsOutput(1.0f, 1.0f, 1.0f, 1.1f, 1.0f, 1.0f, 1.0f, 1.0f);
if (FMOD_OK != result)
return(nullptr);*/

unsigned long long startdelay = 0;
unsigned int soundlength = 0;
float soundfrequency;
FMOD::Sound * __restrict playingsound;

//Get the start time of the playing channel.
result = prevChannel->getDelay(&startdelay, 0);
if (FMOD_OK != result)
return(nullptr);

//Grab the length of the playing sound, and its frequency, so we can caluate where to place the new sound on the time line.
result = prevChannel->getCurrentSound(&playingsound);
if (FMOD_OK != result)
return(nullptr);
result = playingsound->getLength(&soundlength, FMOD_TIMEUNIT_PCM);
if (FMOD_OK != result)
return(nullptr);

result = prevChannel->getFrequency(&soundfrequency);
if (FMOD_OK != result)
return(nullptr);

//Now calculate the length of the sound in 'output samples'.
//Ie if a 44khz sound is 22050 samples long, and the output rate is 48khz, then we want to delay by 24000 output samples.

soundlength = (unsigned int)((float)soundlength / soundfrequency * (float)m_iFMODSampleRate);

startdelay += soundlength; // Add output rate adjusted sound length, to the clock value of the sound that is currently playing*/

result = newChannel->setDelay(startdelay, 0); // Set the delay of the new sound to the end of the old sound
if (FMOD_OK != result)
return(nullptr);

/*unsigned long long tDelayStart = (pSample->GetStartTime() / 10000) * (m_iSampleRate / 1000);

result = newChannel->setDelay(m_ulStartDelay + tDelayStart, 0); // Set the delay of the new sound to the end of the old sound
if (FMOD_OK != result)
return(nullptr);*/

result = newChannel->setPaused(false);
if (FMOD_OK != result)
return(nullptr);

return(newChannel);
}
http://www.supersinfulsilicon.com/ supersinfulsilicon - software Home of the MaxVR Oculus Rift Video Player https://twitter.com/Carandiru

Petroza
Heroic Explorer
OK a few issues immediately pop out.

1. The spatializer is being added to the Master ChannelGroup, it should be at the other end of the signal path either on the Channel, or ChannelGroup if you have a couple of sounds that you want to play the same location. You need to create a DSP for each spatialized sound source.

2. The channel format is being forced to 5.1. That is probably the cause of the perceived lower pitch (6 channels of audio for 2 channels of output)



The code should look more like this (warning untested code)

Initialization:

FMOD_RESULT result = FMOD::System_Create(&m_FMODSystem);      // Create the main system object.
if (FMOD_OK != result) {
LOGF(WARNING, "Create FMOD Main System FAILED(%d)", result);
return(E_FAIL);
}

int iSampleRate(DEFAULT_FMOD_SAMPLERATE);
result = m_FMODSystem->getSoftwareFormat(&iSampleRate, nullptr, nullptr);
if (FMOD_OK == result) {
m_iFMODSampleRate = iSampleRate;
LOGF(INFO, "FMOD Mixer SampleRate: %dHz", iSampleRate);
}
else {
LOGF(WARNING, "FMOD getSoftwareFormat FAILED(%d)", result);
return(E_FAIL);
}

unsigned int bufferSize(0);
result = m_FMODSystem->getDSPBufferSize(&bufferSize, nullptr);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD getDSPBufferSize FAILED(%d)", result);
return(E_FAIL);
}

if (!OSP_FMOD_Initialize(iSampleRate, bufferSize))
{
LOGF(WARNING, "OSP_FMOD_Initialize FAILED");
return(E_FAIL);
}

if (!OSP_FMOD_SetSimpleBoxRoomParameters(5.0f, 2.1f, 3.7f,
0.75f, 0.65f, 0.55f, 0.25f, 0.65f, 0.65f))
{
LOGF(WARNING, "OSP_FMOD_SetSimpleBoxRoomParameters FAILED");
return(E_FAIL);
}

m_FMODSystem->setSoftwareFormat(m_iFMODSampleRate, FMOD_SPEAKERMODE_STEREO, 0); // FMOD_SPEAKERMODE_STEREO Final Format Used (Headphones)

result = m_FMODSystem->init(32, FMOD_INIT_NORMAL, 0);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD Init FAILED(%d)", result);
return(false);
}

result = m_FMODSystem->loadPlugin(m_szOSPAudioPlugin, &m_hOSPAudio);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD Load Plugin OSP FAILED(%d)", result);
return(false);
}


Play sound

__declspec(restrict)FMOD::Channel * const FMODFilterManager::queueNextSound(CMediaBufferSampleWrapper const* const __restrict pSample, FMOD::Channel* const __restrict prevChannel, FMOD::Sound ** const __restrict curSoundSlot)
{
FMOD_RESULT result;
DWORD dwLength(0);
FMOD::Channel * __restrict newChannel;
char const* __restrict pBuffer(nullptr);

pSample->GetBufferAndLength(&pBuffer, &dwLength);

FMOD_CREATESOUNDEXINFO soundInfoCurLength(m_sSoundInfo);
soundInfoCurLength.length = dwLength;

result = m_FMODSystem->createSound(pBuffer, FMOD_OPENMEMORY_POINT | FMOD_OPENRAW | FMOD_IGNORETAGS | FMOD_LOWMEM | FMOD_UNIQUE, &soundInfoCurLength, curSoundSlot);

if (FMOD_OK != result)
return(nullptr);

result = m_FMODSystem->playSound(*curSoundSlot, m_channelgroup, true, &newChannel);

if (FMOD_OK != result)
return(nullptr);

if (nullptr == prevChannel)
{
result = PrimeFMODStream(&prevChannel, &soundInfoCurLength);
if (FMOD_OK != result) {
LOGF(WARNING, "PrimeFMODStream FAILED(%d)", result);
return(nullptr);
}
}

/* We want all channels to be sent as 5.1 instead of very end of DSP //
FMOD::DSP *channel_dsp_head;

result = newChannel->getDSP(FMOD_CHANNELCONTROL_DSP_HEAD, &channel_dsp_head);
if (FMOD_OK != result)
return(nullptr);

unsigned long long startdelay = 0;
unsigned int soundlength = 0;
float soundfrequency;
FMOD::Sound * __restrict playingsound;

//Get the start time of the playing channel.
result = prevChannel->getDelay(&startdelay, 0);
if (FMOD_OK != result)
return(nullptr);

//Grab the length of the playing sound, and its frequency, so we can caluate where to place the new sound on the time line.
result = prevChannel->getCurrentSound(&playingsound);
if (FMOD_OK != result)
return(nullptr);
result = playingsound->getLength(&soundlength, FMOD_TIMEUNIT_PCM);
if (FMOD_OK != result)
return(nullptr);

result = prevChannel->getFrequency(&soundfrequency);
if (FMOD_OK != result)
return(nullptr);

//Now calculate the length of the sound in 'output samples'.
//Ie if a 44khz sound is 22050 samples long, and the output rate is 48khz, then we want to delay by 24000 output samples.

soundlength = (unsigned int)((float)soundlength / soundfrequency * (float)m_iFMODSampleRate);

startdelay += soundlength; // Add output rate adjusted sound length, to the clock value of the sound that is currently playing*/

result = newChannel->setDelay(startdelay, 0); // Set the delay of the new sound to the end of the old sound
if (FMOD_OK != result)
return(nullptr);

/*unsigned long long tDelayStart = (pSample->GetStartTime() / 10000) * (m_iSampleRate / 1000);

result = newChannel->setDelay(m_ulStartDelay + tDelayStart, 0); // Set the delay of the new sound to the end of the old sound
if (FMOD_OK != result)
return(nullptr);*/

// Plugin OSP
result = m_FMODSystem->createDSPByPlugin(m_hOSPAudio, &m_dspOSP);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD Create DSP OSP FAILED(%d)", result);
return(false);
}

result = m_channelgroup->addDSP(1, m_dspOSP);
if (FMOD_OK != result) {
LOGF(WARNING, "FMOD addDSP OSP FAILED(%d)", result);
return(false);
}

result = newChannel->setPaused(false);
if (FMOD_OK != result)
return(nullptr);

return(newChannel);
}

Carandiru
Honored Guest
Thanks for you help Peter,

I've tried you suggestions.

I have set everything to stereo, input audio is stereo and output audio is stereo format now.

I'm creating the DSP's after the sample has been created aswell, then adding the DSP to the channel.

Now I get some nullpointer exception inside the ovrfmod32.dll that I can't seem to find the cause for.
It happens when I addDSP if I do it after setPaused(false) on the channel, or if I addDSP before setPaused(false), it happens on setPaused(false).

I then swap the DSP for a simple low pass DSP to see if it's something I can isolate. However the lowpass DSP works fine.

I'm hoping you guys can add support for adding the Oculus Spatializer to a Channel Group instead of the actual channel.

Thanks again,
Jason
http://www.supersinfulsilicon.com/ supersinfulsilicon - software Home of the MaxVR Oculus Rift Video Player https://twitter.com/Carandiru

Petroza
Heroic Explorer
The input should be mono, and the output will be stereo. I just tested this on a channel group and it works correctly as long as the input is mono. If the input is stereo it sounds incorrect (but doesn't crash). We can down-mix to mono internally to support stereo input but there is no benefit over mono input. Can you try setting it up with mono input (or just calling DSP::setChannelFormat(0, 1, FMOD_SPEAKERMODE_MONO), and see if that works? If it's still crashing can you post your updated code?

Carandiru
Honored Guest
"PeterStirling" wrote:
The input should be mono, and the output will be stereo. I just tested this on a channel group and it works correctly as long as the input is mono. If the input is stereo it sounds incorrect (but doesn't crash). We can down-mix to mono internally to support stereo input but there is no benefit over mono input. Can you try setting it up with mono input (or just calling DSP::setChannelFormat(0, 1, FMOD_SPEAKERMODE_MONO), and see if that works? If it's still crashing can you post your updated code?


Thanks again Peter,
What I'm ultimately trying todo:
Audio Input is 5.1, split the 5 channels into 5 mono inputs, skip LFE.
Hookup OSP to each 5 mono inputs.
Mix output of OSP DSP to final DSP Head, into final 5.1 Output

I have it working, so that I'm sending mono only now to the OSP, and working off of the FMOD DSP Per Speaker Example as a reference. However, I've lost "stereoness, or surroundness". Everything that plays from any channel sounds like it comes from one position.

I use a TEST 5.1 AC3 file to verify sound correctness. Does the Left, Right, LFE, Rear Left, Rear Right, moving etc.

I believe I'm missing some important step, so I have attached / linked the code I have built for FMODFilterManager.h / cpp

If you could have a look and tell me what I'm doing wrong that would be great.
I'm guessing this is the only way to take a 5.1 sound, split into 5 mono channels, then spatial the audio per speaker?
The incoming audio is from 5.1 Movies / etc, so this is the requirement I have to meet.

** I should note I have also tried disabling all FMOD_3D, and related functions, same result
** I have also set FMODSystem::setSoftwareFormat(default, FMOD_SPEAKERMODE_STEREO, 0), same result

Commenting out :
//result = pdspOSP->setChannelFormat(channelMask, 1, FMOD_SPEAKERMODE_5POINT1);
or
//result = pdspOSP->setChannelFormat(0, 1, FMOD_SPEAKERMODE_MONO);

Restores the correct speaker sound output, however can't use the Oculus Spatializer, "incorrect deep sound"
So test with a different DSP, such as lowpass instead, and verify speaker sound output.

http://www3.telus.net/~j.tully/FMODFilterManager.zip

Thanks again,
Jason
http://www.supersinfulsilicon.com/ supersinfulsilicon - software Home of the MaxVR Oculus Rift Video Player https://twitter.com/Carandiru

Petroza
Heroic Explorer
Hi Jason,

This is advanced usage of the FMOD API, the best route would probably start small and build it up. The requirements for our plugin are: mono in, stereo out. The problems you're running into are because of the weird channel formats. The major issue here is splitting a 5.1 signal into 5 mono signals, the best place to get help with that would be from FMOD directly. They have email support and a Q&A area.