New Audio Engine Audio Capture frequencies and note recognition

Hello, we are working on a student VR project which is due in 2 months.
The main mechanic of the project is voice manipulation and visualisation.
The new Audio Engine seems great but with lack of documentation and zero experience in sound programming it’s unclear how to achieve what we need,
so if you could point us into the right direction it would be great!

  1. The first thing we would like to do is to control real time parameters of a particle system based on players voice coming from VR mic (its color is based on musical notes and its shape is based on different frequencies amplitudes).
    As I can see Audio Capture provides interpolated envelop value which seems not enough for what we are trying to achieve as far as I understand.
    The question is how do we get the frequencies from Audio Capture and how do we calculate which note was it?

  2. We want to pre-record the voice, apply sound fx to it and then visualise it using the logic from step (1).
    As I can see from this demo:
    the recording and fx part should be possible but I could not find documentation or demo project on how to do it.
    Could anyone please provide a link or something that shows how to do it please? Or tell us which component to use to record the sound and how to apply fx to it.

Another question would be similar to 1): how can we get the frequencies and calculate which note were played from the processed sound?

So if to summaries we are after some kind of GetNote and GetFrequenciesByRange nodes regardless from which source the sound is coming from live or pre-recorded.
And also a documentation about how to record and manipulate the sound using new Audio Engine.

Your help is greatly appreciated!!!

Hello. It took a bit a while for moderators to approve this post so we’ve managed to do a little progress since.
By playing with the new audio engine we’ve figured it out that you can use Submix recording to record your voice and save it as a sound wave asset.
And we can do it pretty much every tick getting real-time audio feed in sound wave format.
Now using Sound Visualisation plugin we are able to calculate the frequency spectrum from both ‘live’ or pre-recorded voice to drive particles shape.

So the only thing left to figure out is how to get the musical note from the spectrum or the sound wave.
We’ve also noticed that the new Audio Engine has GetMIDIPitchFromFrequency function which gives us MIDI note from a frequency,
which means that what we actually need to find is a frequency in Hz.

As far as we understand the sound wave gives us multiple frequencies combined in a single wave.
The questions are:

How do we know which one we should use to get the pitch from?
Is it the loudest one? Or is there some averaging formula exists?
How do they recognize pitch in games like Guitar Hero or similar karaoke apps?
And how do we get the frequencies in Hz rather their amplitudes in dB in UE4?

Thanks in advance!

As nobody replied I’ll just post an update of what we’ve end up doing for now if it helps to anybody.

We’ve found a simple algorithm of finding fundamental pitch described here:
But to get this working you need FFT output of at least 8192 samples to get rough approximation of the pitch.
There are couple of issues with this. The array is quite big to go through real time and will affect the frame rate significantly so FFT should be able to work within provided frequency range to perform binary search. The second one is that Sound Visualisation plugin CalculateFrequencySpectrum (not sure which FFT they are using there) is limited to 512 samples which is not nearly enough to find the pitch.

We could not find anything in unreal that would let us to implement it. So we end up using Pure Data (PD) and OSC plugin for live pitch recognition.

It would be awesome if new Audio Engine provides more data for audio visualisation in the future!!! Best regards!