Hello all i am brand spanking new to this UE4 world, im not a game creator but a film maker looking to create film/videos with UE4. I want to start off by having assets/characters/shapes or things move based on audio files. Can someone point me in the right direction for a tutorial or manual link. Im not looking to do anything extravagant at the moment perhaps a simple box moving to a simple drumbeat. Any help would be appreciated.
Im a Music Composer/Producer of 20 years with a wide range of genres I like to create. Some strange stuff some mainstream. Hip-Hop, Rock, Jazz, Orchestral, EDM I just love music and want to put Visuals to it as well.
I’m a creation fanatic and get my kicks on making things out of pure thought, I can also help with Audio related things and provide audio related services. Im also quite a techy geek so for me delving into the UE4s massive system is labor of love that I have no problem getting my hands dirty in.
Video editing and Compositing is my new venture and UE4 is right on time for me.
Hey, I dunno exactly how you’d go about implementing such a system but it’s definitely doable.
I think you need to figure out just some of the technical details about how you’re going to go about syncing audio to visuals.
Is your plan to compose visuals to a specific track like a music video? Or are you trying to get something procedural? So it’ll do a certain action when an event occurs like a kick drum triggers a mesh to flatten and reform as a random example.
I think you’d be best off trying to find some way to work with MIDI data. I don’t have much experience working with MIDI outside of DAWs but I imagine you could find a way to export MIDI from a track and use that data to call functions in a blueprint script.
It really depends on how crazy you want to go with this. You could build a whole system that takes in data in MIDI or audio form and creates the visuals on the fly. Or if you’re planning on just the music video type route you could basically just ‘transpose’ your MIDI data into some kind of visual event by hand. I was just thinking you could also do something like build a sampler out of blueprints, I did this with a console application a few weeks ago, so you create functions with parameters that are like, play this synth in C at octave 3 or whatever, and when you do that, draw this character sequence. So you just make a function to play various different sounds, and then make a script to act as your sequencer that calls those functions. It actually sounds kind of fun to try out in UE.
Whatever you do, if you have programming experience, great. If not you should learn to work with blueprints.
I’d recommend FMOD for this, you can add markers to the FMOD studio timeline and receive callbacks in Cpp or Blueprint when it has reached these markers. If you use their Cpp API you’d be able to do a whole lot more too. Definitely your best option without writing your own.
Hmm, you just opened a door there. I seen a mention of midi while researching UE4, I may have to look into that down the road. I’m thinking the audio visualizer may be where I need to start for simple object triggering, hoping that the output data can be sent to shapes/characters etc.
I did install Fmod and only got as far as getting the audio files into the timeline. Once I got back to UE4 I was noob scratching. Had to get to bed by then. The markers option sounds good, will look into that. Thanks guys
Read through the FMOD manual and the UE4 integration guide and you’ll be flying in minutes. I used it for the first time the other day and think it’s great. I’m not aware of any midi libraries/plug-ins for UE4 although suspect they exist. I opted FMOD due to the precision of the timers, the on tick event and ms precision timers just don’t cut it. Although this is only important if you were triggering audio rather than moving object etc…
I will do just that. There is plugin that allows the use of a midi controller in ue4, I turned it on and while I was creating a test song in my DAW, I noticed the midi controller was not available and showed in use by another program. I haven’t tried to actually use it in UE4 yet, but it was definitely active there. Thanks again, and the timer precision observation puts a smile on my face. FMOD it is.
My guess is that you’re actually looking to utilize this little plugin in Unreal Engine 4:
It’s called the Sounds Visualizations Plugin and it provides you with two BluePrint items: Calculate Frequency Spectrum and Calculate Amplitude which both allow you to extract sound information to use as meaningful variable data which can then be applied to all kinds of things–like the size or location of a cube.
Yeah, for the moment I gave up on FMOD after reading the manual. The Sound visualizer is where I’m currently at and clueless. Tried following one guys video who seem to be the only one with any info on YouTube in regards, once I can get my head wrapped around its functions I think I can get this accomplished. Any other tutorials or info on the sound visualization plugin would be helpful.
Here’s where i’m at thus far, managed to get data flowing from the array output, now i just need to convert this to modify a property in the viewport, in this instance i need to control the light intensity! The sound itself is a house bassdrum, so when it plays i want the light to turn on then off when there is no sound. seems easy enough.
I got some headway with this, I started another topic ‘Audio Visualization 101’ so it would be easier for people to Find and eliminate too much needless Google searching, so Ill continue there, and since i’m a noob, it should also be helpful for those who haven’t a clue as i figure it out, thanks for all the reply’s
