Is the new Audio Synesthesia realtime or baked? I'm hoping realtime...
Announcement
Collapse
No announcement yet.
Audio Synesthesia
Collapse
X
-
edit: I have no clue, didn't even know about this plugin before thisLast edited by ArthurBarthur; 11-07-2019, 04:09 PM.Try my procedural music tools: Arthurs Audio BPs
Comment
-
Originally posted by cannabis.cod3r View PostIt looks like you add a new asset type and associate it with a sound, so it's likely baked. Not entirely sure what the setup is supposed to be though...
Comment
-
Hey Everyone, I've been the main man working on Audio Synesthesia. Super glad to see you all interested! I want to give you some quick notes on what's available in 4.24 and what we have planned for the near future.
Audio Synesthesia In 4.24
Audio Synesthesia is in a pretty early beta stage. As it stands in 4.24 there will be support for baked analysis (aka Non Real-time) and it will generate:- perceptual loudness over time
- constant Q over time (Constant Q is like a spectrogram except that the frequencies are spaced more like notes in a scale)
- audio onsets over time
Audio Synesthesia In The Future
Audio Synesthesia in the future will absolutely support real time analysis for any algorithms that can be done in real time. That means loudness and constant Q will for sure be in there. Possibly onests will make it to with some modifications from the baked version.
It will also have some usability updates and new analysis algorithms.
On my short list of analyzers to add are- Beat Grid - This is only a non-real-time analyzer, but it's great for syncing animations to musical clips. It will fire off regularly spaced blueprint events just like you were tapping your foot to the music.
- Peak partial tracking - This will break down a sound into a handful of 100 or so tones with amplitudes and trajectories
- A regular old spectrogram
If you have any things you'd like to see in future version, let me know!
- 3 likes
Comment
-
I'm in the business of music visualizations (for live DJs) and there are 2 features that I could really use:- Real-time analysis with events dispatched to Blueprints so I'm able to respond to amplitudes in a range of frequencies that I'm interested in.
- Being able to analyze whatever is playing on the user's sound card. That means *anything* that is currently playing, from any application. This can be accomplished using loopback mode in WASAPI, a UE4 demo of which can be found here. (If you want to build the demo project, you'll need to wrap any calls to Windows headers in #include "PreWindowsApi.h" and #include "PostWindowsApi.h"). I was thinking this could be implemented in a similar manner to the AudioCaptureComponent but instead of using a microphone input, it would use WASAPI loopback.
- 1 like
Comment
-
Originally posted by saunassa View PostIf you have any things you'd like to see in future version, let me know!
Technical Artist https://www.youtube.com/user/rendermanpro/videos
Comment
-
I think the Synesthesia plugin is 'just' non-realtime(NRT) for now?
Here's a quick preview of what I'm doing with it - everything is baked and available! Some of the analysis tools has "get [analysis type] at time" nodes as access points in BPs..To bake that for myself, I made a little machine in construction script that incrementally reads the data every few milliseconds, and puts values+times into some dataset. https://youtu.be/Vi6MfUjyRpc
Last edited by ArthurBarthur; 12-10-2019, 03:21 AM.Try my procedural music tools: Arthurs Audio BPs
Comment
-
Question about the constantQ bit! Tried to make it have more than 48 bands(added octaves, so tried 60, 72..), but there doesn't seem to be any updated values coming out of the extra bands. It self-reports having more bands, but each band seems to be stuck at 0 value.
I also tried changing starting frequency, but couldn't really see a difference then either.Try my procedural music tools: Arthurs Audio BPs
Comment
-
Haha, all audio guys in one topic. Did you try it with Niagara? @cannabis.cod3rTechnical Artist https://www.youtube.com/user/rendermanpro/videos
Comment
-
Originally posted by ArthurBarthur View PostQuestion about the constantQ bit! Tried to make it have more than 48 bands(added octaves, so tried 60, 72..), but there doesn't seem to be any updated values coming out of the extra bands. It self-reports having more bands, but each band seems to be stuck at 0 value.
I also tried changing starting frequency, but couldn't really see a difference then either.Try my procedural music tools: Arthurs Audio BPs
Comment
-
ArthurBarthur I'm glad the ConstantQNRTSettings worked out for you. My guess is that it was either a blip that occured when calculating the ConstantQNRT or some edge cases with the settings. For future reference, you can run into a situation where the ConstantQ is looking for frequencies that don't exist in the audio file.
For instance, if you have a sample rate of 48kHz, the maximum frequency in your audio file is 24kHz. Then, if you have some settings that look at 100 bands, your upper bands may go beyond 24kHz.
Comment
-
cannabis.cod3r Just a heads up, real-time analysis is in the works. As far as sampling audio off the sound card, it's definitely doable but requires you to do the routing yourself. I think the setup is
1. Route your system audio output in your OS to a virtual sound card.
2. Then have that virtual sound card send the audio to a mic input in UE4.
3. Mute output of UE4 audio submix to avoid digital feedback.
I haven't tried that solution myself, but I've heard it work for others.
Comment
Comment