Jimmy pointed this comment out to me – I didn’t see it in the forum post here, apologies for the late reply!
Yeah, we decided to intentionally decouple the analysis product of synesthesia from the USoundWave asset on purpose.
The idea is that you may want to load the analysis file independent of the associated audio file. Or, conversely, you may want to hear a single file but use many audio assets purely for analysis and never load or play them ever. For example, you might break up a piece of music into different stems or filter or process the audio file differently depending on what you want to analyze. Then at runtime you play a single music file that is the mix you want to hear, but use any number of synesthesia analysis files you want in whatever way.
Incidentally Synesthesia isn’t the first analysis feature we added. We DO have a built-in analysis thing in our USoundWave asset – we’ve had it for over a year! Check out USoundWave properties and look for the “Analysis|FFT” and “Analysis|Envelope” categories. You have been able to have baked FFT and envelope data with sound waves for a while. We use this on Fortnite for driving our “audio reactive” emotes. I talked about it at my GDC talk in 2019. I implemented this baked analysis thing in about a 3 days before a hard deadline when it was determined by important people that the emotes for a particular release/promotion needed reactivity from audio.
We learned from hard experience that baking the analysis with the audio file was a BAD idea. Having the baked product IN the audio file is bad news and we’ve regretted it since. So when we set about making a more serious audio anlaysis plugin (and after we had hired some serious DSP people who know this domain very well), we intentionally decoupled it as I described.
As for the best way to organize the data, it’s up to you. We treat BP as a real scripting language with native UE4 unlike WWise where things are more “blackbox” and “turnkey”. It does mean there’s a bit of onus on learning and utilizing BP to build the thing you want to build. Basically there’s a kind of intermediate API layer where we make the thing do the thing that is really hard (like all the work of DSP analysis) and present it in a way where the typical BP scripter can go to town building really cool stuff. To do otherwise would be to fundamentally restrict the application of the tech and limit use-cases. We could make some turn-key example content or plugins but usually we’re pressed for time as we are developing these tools for use with a very fast-moving (production-wise) game. We’ve used Synesthesia in exactly the use case I described for the Fortnite live event with Travis Scott for example. We’re using it for a bunch of stuff for Party Royale, etc. We have some very capable technical sound designers and a dedicated gameplay audio programmer who are working on Fortnite-specific tools and integrations.
To give you some more direct and concrete advice. If you wan to make something like a straightforward thing that drives stuff based on analysis files, I’d say that what you suggested is definitely a way to go. Make a BP struct (or BP class) that has the sound you want to play and the analysis files you want to associate and write some methods or functions to play the sound and then trigger visuals or do whatever delegates/callbacks you want to interact with other systems (gameplay, physics, vfx, etc).
Finally, on this topic, you may be more interested in real-time analysis vs baked analysis in general. There is yet another method for analysis you can try out with both AudioComponents and with Submixes. Both allow you to do realtime analysis of audio amplitude envelopes, which it sounds like you are interested in. They don’t do the perceptual loudness stuff that Synesthesia does, but it does use the same envelope follower DSP algorithm that would be used in a dynamics processor (compressor, etc). Those are all reatlime and very very easy to hook into things like driving visuals (lighting strikes, etc).
I’m looking in the code and I don’t see these files. NDI stands for “Niagara Data Interface” so my assumption is these are related to the Niagara work. I knew this was going to be confusing, but the realtime visualization stuff we did for the Niagara Data Interface stuff wasn’t done in the Synesthesia Plugin. It’s technically not part of Synesthesia though it uses some of the same DSP analysis algorithms, albeit in realtime. The NDI stuff was done very late before 4.25 released and we didn’t have time to package it within Synesthesia, which is where we intend to put analysis features, real-time and non-realtime. I want to refactor it at some point but it’s been deprioritized.
I’ll ask the people who worked on the NDI stuff if they know about these asset types you are referring to.