Darkdrift: generative/procedural audio experiment

Spent a few hours throwing this together recently after some discussion regarding the new audio features coming to 4.16, and haven’t had time to work on it since. But figured I’d drop this here as a simple example of gameplay/input-based generative audio using the new realtime audio synthesis plugin (available now in the Epic launcher as 4.16 Preview 1):

Every time the accelerator is pressed, it plays a random selection from among 3 notes (C, E, F) using a 3-oscillator subtractive synth, piped through a slow lowpass filter sweep depending on how long the accelerator is held, as well as a bit of distortion and delay.

Good things are coming with this tech – can’t wait for 4.16!

I’m sure better learning material will be forthcoming after 4.16 is released, but for reference in the meantime, here’s how I did it (I’m sure there are better ways):

1. The new audio engine won’t be available until 4.16, so download and compile a recent Master or Promoted branch of UE4 from GitHub is available now in 4.16 Preview 1 in the Epic launcher:

2. Enable the new audio engine by adding the applicable line to any platform-specific engine config files for your project (see 4.16 preview release notes for syntax):

3. Under Editor Preferences -> General -> Experimental -> Audio, enable the “Show Audio Mixer Data” option This is no longer necessary!

4. Enable the new audio plugins:

5. Profit! Now the fun starts… add a ModularSynth component to a blueprint (the main Pawn in this case). All parameters can be left default, but in this example I’ve added a reference to a Source Effect Preset Chain asset which contains some additional effects:

6. On the Pawn’s BeginPlay event, define some basic parameters for the synth. You only need a minimum of 1 oscillator, but in this example I’ve defined 3 for a nice, thick sound (note individual oscillators indexed 0, 1, and 2 for later reference). And finally “start” the synth so that it can be used (simply setting the component to Auto-Activate doesn’t seem sufficient):

7. On any forward input, if a note isn’t already triggered, then start one. This note will continue until a command is sent to stop the note.

In this example, I have defined an array of integer semitones [0,4,5] for the notes C, E, and F, from which I’ll randomly select a note to play each time the accelerator is pressed. Also notice that the NoteOn function references a Note parameter indexed 0 (zero), so we could potentially be starting and stopping many separate notes (up to the Voice Count defined for the Modular Synth component) with separate calls to NoteOn/NoteOff, each referenced by their index (although the Note parameter appears to be a float, so surely I’m missing something).

NB: This example actually sends 3 potentially different random notes (a new random integer is pulled for each reference) to each of the 3 oscillators, due to a logic error. (To send the same note to all 3, I should have called RandomIntegerInRange only once and assigned it to a temp variable…) But I like the variably discordant sound of how this turned out :wink:

8. To raise the lowpass filter frequency based on speed: on each Tick event, get the current velocity and apply the ratio of that versus max velocity to the ModularSynth’s filter frequency.

In this example, the variable SpeedCamMax is a number slightly higher than the max velocity the pawn can reach in-game (due to wind resistance), so the filter frequency will be set between 50 Hz and 12 kHz depending on current velocity:

9. And finally, stop the note whenever the accelerator is released (this is checked on every tick).

Note that in this example, it’s only turning off 1 note: the Note parameter 0 (zero), which we referenced when starting the note. In the example video this single note sounds like a chord due to me erroneously sending different random semitones to each oscillator on each NoteOn :wink:

Apologies for the spaghetti code, hopefully this helps clear things up for someone!

Amazing! Thanks for this!!

Hey Acatalept!

It’s awesome you found all the set functions for all the Modular Synth parameters! However, if you’d like to set them all in one go, you can with the Set Preset function:

It’s just a data structure, so you can also create variables, arrays, etc. of presets.

I managed to sneak in some changes to 4.16 right before I was told to put my pencil down – you don’t need to do the “show audio mixer data” step anymore. I made it so it automatically figures out to unhide editor stuff for when you run with the audio mixer. I hated that checkbox and just overcomplicated things. I originally did it because of order-of-init issues with when things get loaded in editor.

This is awesome man, thanks for the write up!

Also, check out the modular synth preset bank object. There’s native support for creating, managing, banks of presets. We used this for the GDC demo to make keyboard/bass/pads/lead synth presets.

I knew I was missing something :wink: I’d seen references to presets, but didn’t quite get how to use them. Thanks for the tip!

Edited my steps, thanks for keeping things simple :wink:

Thanks both for all your hard work on the new audio tech!

Hello!

Can you tell me how to hook up GranularSynth I am trying second day now.
This took me only 4 hours :slight_smile:

Lovely, acatalept - and thanks for the detailed notes!

Not sure if you’ve explored the preset/preset bank object further, but do you know if there’s any way to store settings for more than 2 oscillators? My workaround for now is to store everything minus my 3rd osc’s settings in a preset, then get osc3 settings out of a struct to update the modular synth right after the preset is loaded…but it’d be nice to have everything stored in one place.