Announcement

Collapse
No announcement yet.

Audio Synesthesia

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Audio Synesthesia

    Is the new Audio Synesthesia realtime or baked? I'm hoping realtime...

    #2
    Im assuming its real-time and its safe to assume that events are fired from the audio-thread to the game thread at the nearest or next frame.

    Comment


      #3
      edit: I have no clue, didn't even know about this plugin before this
      Last edited by ArthurBarthur; 11-07-2019, 04:09 PM.

      Comment


        #4
        It looks like you add a new asset type and associate it with a sound, so it's likely baked. Not entirely sure what the setup is supposed to be though...

        Comment


          #5
          Originally posted by cannabis.cod3r View Post
          It looks like you add a new asset type and associate it with a sound, so it's likely baked. Not entirely sure what the setup is supposed to be though...
          I noticed this too and thought the same, but I'm hoping there might be a way to swap that audio asset out in game. If anyone has any ideas how to make use of this feature I would be really interested to know.

          Comment


            #6
            Hey Everyone, I've been the main man working on Audio Synesthesia. Super glad to see you all interested! I want to give you some quick notes on what's available in 4.24 and what we have planned for the near future.

            Audio Synesthesia In 4.24

            Audio Synesthesia is in a pretty early beta stage. As it stands in 4.24 there will be support for baked analysis (aka Non Real-time) and it will generate:
            • perceptual loudness over time
            • constant Q over time (Constant Q is like a spectrogram except that the frequencies are spaced more like notes in a scale)
            • audio onsets over time
            The analysis assets keep references to the sounds that created the underlying analysis data, but there's nothing that forces them to be used together. In blueprints you can pick whichever asset you'd like to use as a data source and sync it to whatever sound you'd like, or not sync it to any sound at all.

            Audio Synesthesia In The Future

            Audio Synesthesia in the future will absolutely support real time analysis for any algorithms that can be done in real time. That means loudness and constant Q will for sure be in there. Possibly onests will make it to with some modifications from the baked version.

            It will also have some usability updates and new analysis algorithms.

            On my short list of analyzers to add are
            • Beat Grid - This is only a non-real-time analyzer, but it's great for syncing animations to musical clips. It will fire off regularly spaced blueprint events just like you were tapping your foot to the music.
            • Peak partial tracking - This will break down a sound into a handful of 100 or so tones with amplitudes and trajectories
            • A regular old spectrogram

            If you have any things you'd like to see in future version, let me know!

            Comment


              #7
              I'm in the business of music visualizations (for live DJs) and there are 2 features that I could really use:
              1. Real-time analysis with events dispatched to Blueprints so I'm able to respond to amplitudes in a range of frequencies that I'm interested in.
              2. Being able to analyze whatever is playing on the user's sound card. That means *anything* that is currently playing, from any application. This can be accomplished using loopback mode in WASAPI, a UE4 demo of which can be found here. (If you want to build the demo project, you'll need to wrap any calls to Windows headers in #include "PreWindowsApi.h" and #include "PostWindowsApi.h"). I was thinking this could be implemented in a similar manner to the AudioCaptureComponent but instead of using a microphone input, it would use WASAPI loopback.
              Those are two critical features for me at the moment. saunassa Thanks for your response, much appreciated.

              Comment


                #8
                hello saunassa nice road map. Is there somewhere a doc that explains how to use this ? Could nt find a way to test it .... and since it s baked, how would u sync it with a cue ? tks

                Comment

                Working...
                X