This is about 4.23 (since 4.22 is very buggy with the various audio plugins).
What exactly can someone currently do with Ambisonics in UE4?
My case description. I’ve been thrown into a VR project where there is a room and a musical instrument playing in the middle of the room. I’ve done the recording on location using a surround technique and then converted to Ambisonics AmbiX, but without the Z channel (so Z is silent in my 4-ch wav). The multi-mic setup was fixed in a position close to the front side of the musical instrument. That day I also made a verbal position check by having a person speaking around the mic, for instance saying Front, Front Left, Left, Rear Left, Rear, etc. the full circle around the mic. This all converted to binaural sounds fine in my DAW and I can rotate the sound field by turning my view. Then I throw this in UE4 and try to experiment with Steam audio and Resonance audio plugins. I set the sound to play directly in front of my view and the localization sounds normal. But if I turn my view around the sound becomes suddenly almost mono, panned to the left or right, losing all the realistic width that it should have in a VR project. As far as I know, I have set all settings in all assets correctly (hrtf, binaural, etc).
My (ideal) plan is to use the Ambisonics material recorded in the room and somehow from it get a better simulation of the sound propagating around the room. With my ambisonics material I have in theory a 360 degree circular sound field representation of the room. If I had the height channel it would be a sphere but for my case it is a circle, which is still fine for the application. But how to implement this in UE4?..I know this is all a newborn field and I don’t expect a super realistic representation but at least with the current recording techniques and methods it should be much better than what we could do in the past years with simple stereo recordings (talking about game engines at least).
Ok last attempt here. I significantly narrow down my initial post to this:
UE4 claims to support various audio formats (mono, stereo, surround, Ambisonics) and binaural reproduction for headphones. Using mono or stereo formats it is obvious that the reproduction is pseudo-spatialized (calling it 3d stereo?). But when you use surround or Ambisonics formats you are supposed to get a fuller around-the-room sound experience. I.e. as you move your head around the sound source you should hear more or less of the front/rear/etc audio channels. So how can someone use either of these formats in UE4 in order to get this result in a VR project? If you can point to some documentation or ideally tutorial I would be so grateful.
After lots of experimentation I’ve come to this point. Using mono or stereo wav files the sound direction works as expected (turning your head left of the source the sound becomes louder to the right ear etc.) but using Ambisonics files the effect is inverted which is of course wrong. But if you walk to the other side of the source and pan your head then it sounds correctly. It is like there is only one fixed orientation for the sound on the whole map ignoring the player position. This happens with Oculus and Resonance. With Steam Audio all ambisonics sounds are just static.
The best way of playing ambisonic files (up to 3rd order) in UE4 is using the Renderering SDK from Facebook Audio 360 Spatial Workstation. Not long ago they updated the Unreal plugin and so far for me it’s been working great in 4.23. It does soundfield rotation correctly according to camera orientation:
In my experience of working in VR, ambisonics are mainly useful for ambiences which don’t have to deal with the finer points of rotation, distance etc. like an air conditioning hum, or background traffic ambience etc. They behave oddly and I don’t think (though I could be wrong) that they’re supported in the way you’re trying to use them.
Mono files do spatialise really well, and you have to use mono for a point source that is then spatialised. The standard thing to do is simply to use a mono file, and then have it spatialised ambisonically within the engine.
I’m trying to install the FB plugin but on this step:
In Windows Explorer, navigate to the project folder. Right click on the .uproject file and choose the option “Generate Visual Studio project files”.
I get the following error and I’ve been totally stuck there:
Running C:/Program Files/Epic Games/UE_4.23/Engine/Binaries/DotNET/UnrealBuildTool.exe -projectfiles -project=“C:/Users//Desktop/aaa/xxx.uproject” -game -rocket -progress -log=“C:\Users\Desktop\aaa/Saved/Logs/UnrealVersionSelector-2019.09.28-22.47.17.log”
Discovering modules, targets and source code for project…
WARNING: Trying to build an enterprise target but the enterprise directory is missing. Falling back on engine components only.
While compiling C:\Users\Desktop\aaa\Intermediate\Build\BuildRules\xxxModuleRules.dll:
c:\Users\Desktop\aaa\Plugins\fb-audio360-ue4\Source\Audio360\Audio360.Build.cs(47,17) : error CS0151: A switch expression or case label must be a bool, char, string, integral, enum, or corresponding nullable type
c:\Users\Desktop\aaa\Plugins\fb-audio360-ue4\Source\Audio360\Audio360.Build.cs(73,17) : error CS0151: A switch expression or case label must be a bool, char, string, integral, enum, or corresponding nullable type
ERROR: Unable to compile source files.
Ambisonics indeed works better for ambient sounds since it’s hard to make it work for point sources that are intended for the user to approach and listen to from different distances, but I don’t believe it behaves oddly (unless not properly setup). Ambisonic sound fields do rotate and that is what actually produces amazing results paired with a VR headset since you can then simulate a diffuse field as it happens in reality. When I experience VR that neglects this (i.e. leaving the ambient sound completely non-responsive to head rotation) I notice it and find it to diminish the illusion of presence. This is what ambisonics is great for. I’ve recorded ambisonic soundscapes using a 2nd order microphone (OctoMic) and listening to them in VR, with proper headtracking and rotation, really gives you that “gut” intuitive response that makes you feel like you are in that place, including very defined sources at a distance (e.g. a duck flying and quacking and hitting branches a few meters away).
Ah! Yes. That’s an API change in 4.23. I fixed that by replacing the switch occurrences with chains of if-else-if statements in Audio360.Build.cs. I’ll share that if needed.
I don’t understand what you mean by “spatialized ambisonically”. I don’t see how this is possible if you don’t have an audio file that has been originally Ambisonics. Maybe you mean “spatialized binauraly”?
I just built it freshly from the latest Rendering SDK and fixed the compilation errors using UE4 4.24. There are two files that need to be changed:
fb-audio360-ue4\Source\Audio360\Audio360.Build.cs both switch statements need to be replaced by If-else statements
fb-audio360-ue4\Source\Audio360\Private\Audio360Module.cpp the ModuleManager.h include needs to be replaced by “Modules/ModuleManager.h”
This is the whole thing built for 4.24 (but should also work in 4.23 and not for 4.25 preview since it requires other changes). Just put it in your Plugins folder and it should work (or recompile if using 4.23)
I was able to build and use this for a game a while back, however I am coming back to this again and running into an issue I don’t believe I hit when I first ran this. I am getting this error when build with Oculus 4.24.1. Wondering if you’ve seen this or know what the fix might be?
ParallelExecutor.ExecuteActions: In file included from …/Shipping/Audio360/Module.Audio360.cpp:2:
ParallelExecutor.ExecuteActions: C:/UnrealEngine-oculus-4.24.1/Engine/Plugins/fb-audio360-ue4/Source/Audio360/Private/Audio360Module.cpp(241,61): error: declaration shadows a local variable [-Werror,-Wshadow]
ParallelExecutor.ExecuteActions: InitializeAudioEngine(Key), [Key](TBE::AudioEngine* EnginePtr) {
ParallelExecutor.ExecuteActions: ^
ParallelExecutor.ExecuteActions: C:/UnrealEngine-oculus-4.24.1/Engine/Plugins/fb-audio360-ue4/Source/Audio360/Private/Audio360Module.cpp(235,33): note: previous declaration is here
ParallelExecutor.ExecuteActions: TSharedPtrTBE::AudioEngine& EnginePtr = TBEAudioEngines.FindOrAdd(Key);
ParallelExecutor.ExecuteActions: ^