Is the new Audio Synesthesia realtime or baked? I’m hoping realtime…
Im assuming its real-time and its safe to assume that events are fired from the audio-thread to the game thread at the nearest or next frame.
edit: I have no clue, didn’t even know about this plugin before this
It looks like you add a new asset type and associate it with a sound, so it’s likely baked. Not entirely sure what the setup is supposed to be though…
I noticed this too and thought the same, but I’m hoping there might be a way to swap that audio asset out in game. If anyone has any ideas how to make use of this feature I would be really interested to know.
Hey Everyone, I’ve been the main man working on Audio Synesthesia. Super glad to see you all interested! I want to give you some quick notes on what’s available in 4.24 and what we have planned for the near future.
Audio Synesthesia In 4.24
Audio Synesthesia is in a pretty early beta stage. As it stands in 4.24 there will be support for baked analysis (aka Non Real-time) and it will generate:
- perceptual loudness over time
- constant Q over time (Constant Q is like a spectrogram except that the frequencies are spaced more like notes in a scale)
- audio onsets over time
The analysis assets keep references to the sounds that created the underlying analysis data, but there’s nothing that forces them to be used together. In blueprints you can pick whichever asset you’d like to use as a data source and sync it to whatever sound you’d like, or not sync it to any sound at all.
Audio Synesthesia In The Future
Audio Synesthesia in the future will absolutely support real time analysis for any algorithms that can be done in real time. That means loudness and constant Q will for sure be in there. Possibly onests will make it to with some modifications from the baked version.
It will also have some usability updates and new analysis algorithms.
On my short list of analyzers to add are
- Beat Grid - This is only a non-real-time analyzer, but it’s great for syncing animations to musical clips. It will fire off regularly spaced blueprint events just like you were tapping your foot to the music.
- Peak partial tracking - This will break down a sound into a handful of 100 or so tones with amplitudes and trajectories
- A regular old spectrogram
If you have any things you’d like to see in future version, let me know!
2 Likes
I’m in the business of music visualizations (for live DJs) and there are 2 features that I could really use:
- Real-time analysis with events dispatched to Blueprints so I’m able to respond to amplitudes in a range of frequencies that I’m interested in.
- Being able to analyze whatever is playing on the user’s sound card. That means anything that is currently playing, from any application. This can be accomplished using loopback mode in WASAPI, a UE4 demo of which can be found here. (If you want to build the demo project, you’ll need to wrap any calls to Windows headers in #include “PreWindowsApi.h” and #include “PostWindowsApi.h”). I was thinking this could be implemented in a similar manner to the AudioCaptureComponent but instead of using a microphone input, it would use WASAPI loopback.
Those are two critical features for me at the moment. @saunassa Thanks for your response, much appreciated.
hello @saunassa nice road map. Is there somewhere a doc that explains how to use this ? Could nt find a way to test it … and since it s baked, how would u sync it with a cue ? tks 
Not realtime as well. For example if I want to get sample of existed audio file at specific time (loudness at specific frequencies, spectrum). Currently we have baked data, BUT can’t access to it more or less direct way. If I can sample audio file, then I can ‘predict’ in do something in advance before event happen.
I think the Synesthesia plugin is ‘just’ non-realtime(NRT) for now?
Here’s a quick preview of what I’m doing with it - everything is baked and available! Some of the analysis tools has “get [analysis type] at time” nodes as access points in BPs…To bake that for myself, I made a little machine in construction script that incrementally reads the data every few milliseconds, and puts values+times into some dataset. https://youtu.be/Vi6MfUjyRpc
Question about the constantQ bit! Tried to make it have more than 48 bands(added octaves, so tried 60, 72…), but there doesn’t seem to be any updated values coming out of the extra bands. It self-reports having more bands, but each band seems to be stuck at 0 value.
I also tried changing starting frequency, but couldn’t really see a difference then either.
Doing changes works again if I create a new ConstantQNRTSettings asset and hook that up to the existing related assets. So maybe just that one old settings asset I was using for a while had been corrupted or something.
@ArthurBarthur I’m glad the ConstantQNRTSettings worked out for you. My guess is that it was either a blip that occured when calculating the ConstantQNRT or some edge cases with the settings. For future reference, you can run into a situation where the ConstantQ is looking for frequencies that don’t exist in the audio file.
For instance, if you have a sample rate of 48kHz, the maximum frequency in your audio file is 24kHz. Then, if you have some settings that look at 100 bands, your upper bands may go beyond 24kHz.
@cannabis.cod3r Just a heads up, real-time analysis is in the works. As far as sampling audio off the sound card, it’s definitely doable but requires you to do the routing yourself. I think the setup is
- Route your system audio output in your OS to a virtual sound card.
- Then have that virtual sound card send the audio to a mic input in UE4.
- Mute output of UE4 audio submix to avoid digital feedback.
I haven’t tried that solution myself, but I’ve heard it work for others.
Not yet, holding out for realtime analysis.
@saunassa Thanks for the tip, but re-routing virtual sound cards seems like a burden to put on end-users. I still think an audio capture component with WASAPI loopback mode would be ideal (for Windows obviously).
Let me know if you want some user testing and/or Niagara visualizations for a demo (although I’d have to run it by my employer first).
Hey guys - I’m following the documentation here: Audio Synesthesia | Unreal Engine Documentation
I can’t for the life of me get the example to work - I can’t seem to find a version of the “Get Normalized Channel Loudness at Time” node that’s blue with Exec pins
I’m new to blueprints so I’m sure it’s something obvious in retrospect hahaha - could anyone help me out?
@JohnnyHalcyon It’s likely you haven’t set the variable type. Click on the “Loundness Analyser” variable and check the Details panel. Be sure the type is set to LoudnessNRT.
Hi,
This plugin would be very useful, but unfortunately it is not capable of using USoundWave at runtime.
I would really like to use this plugin, since it is very well designed and made, do you know when it will be possible to use it for USoundWave, which were not included in the editor, but during the game (in real time)?
I wonder if anyone can help me, I have made a Niagara system that reacts to any ambient sound in the level, this works great at runtime but when I try to record in sequencer it goes out of sync because the audio plays in realtime whereas the video take just over realtime to process some moments and everything goes out of sync. Is it at all possible for me to bake the audio so that it won’t go out of sync when recording, and if so, how on earth would I do it? Any help at all would be so very much appreciated