How to get Volume from AudioCapture

Development Audio
, ,
1

Hey there,

I’m trying to implement a feature whereby I listen to the player’s mic, and turn the resulting input into a 0...1 value, so that I can animate the player character’s mouth opening and closing in response to the mic input.

I’ve found the AudioCapture libs and I’m doing this:

// Called whenever new audio data is available
void OnAudioGenerate(const float* InAudio, int32 NumSamples)
{
 AudioBuffer.Append(InAudio, NumSamples);
}

 // Create new Audio Capture instance
 UAudioCapture* AudioCapture = NewObject<UAudioCapture>();
 AudioCapture->AddGeneratorDelegate(FOnAudioGenerate(&OnAudioGenerate));

 // Start capturing the audio
 AudioCapture->OpenDefaultAudioStream();
 AudioCapture->StartCapturingAudio();

However, I can’t figure out how to turn this float* data into a useful per frame 0...1 Volume value. Presumably there would need to be a way to adjust the mic sensitivity too.

I haven’t been able to find any examples online of doing this - only playing back the captured audio (which I don’t need to do).

Can anyone help?

Cheers,

Ricky

[Attachment Removed]

2

You could consider putting an AudioCaptureComponent on the character, route its output to a dedicated Submix, and use the submix’s envelope follower (AddEnvelopeFollowerDelegate or “On Submix Envelope” in Blueprint, after StartEnvelopeFollowing). This should give you a smoothed 0 to 1 amplitude per channel with attack/release configured on the submix, and it’s Blueprint-accessible. The downside is the mic gets rendered to output unless you mute/route the submix away from the master.

If you eventually want actual visemes rather than a single jaw-open value, the amplitude approach caps out fairly quickly, that’s where you’d look at frequency-band analysis (AddSpectralAnalysisDelegate) or a dedicated lip-sync solution. What the spectral analyzer gives you that the envelope follower doesn’t, is the “AutoRange” option. You have to set that to “true”, but that will help have it automatically adjust to the loudness/sensitivity of someone’s mic.

[Attachment Removed]

3

Thank-you! I’ll give all this a try!

[Attachment Removed]

4

Hey there,

I’ve been trying to do this in c++. The code (below) all compiles fine, but I don’t ever get into the delegate functions. I’ve both tried the Envelope Follower and the Spectrum Analyzer routes..

There must be something I’m missing, but not sure what. (It seems a bit strange that AddEnvelopeFollowerDelegate

and AddSpectralAnalysisDelegate expect ‘BP’ functions: I notice for example that there is also a ‘FOnSubmixEnvelope’, but I don’t see a method on USoundSubmix to bind that).

I’ve put my earlier code back in too, and I still get into the OnAudioCapture, so I believe my mic should be working okay..

Ricky

// in PlayerCharacter ctor:
 
AudioCaptureComponent = CreateDefaultSubobject<UAudioCaptureComponent>(TEXT("AudioCaptureComponent"));
 
 
void APlayerCharacter::BeginPlay()
{
	Super::BeginPlay();
 
	USoundSubmix* AudioCaptureSubmix = NewObject<USoundSubmix>(this, "AudioCaptureSubmix");
	AudioCaptureComponent->SoundSubmix = AudioCaptureSubmix;
	
	AudioCaptureSubmix->StartEnvelopeFollowing(this);
	FOnSubmixEnvelopeBP AudioCaptureSubmixDelegate;
	AudioCaptureSubmixDelegate.BindUFunction(this, FName("OnAudioCaptureEnvelope"));
	AudioCaptureSubmix->AddEnvelopeFollowerDelegate(this, AudioCaptureSubmixDelegate);
	AudioCaptureComponent->Start();
	
	// Start recording
	Audio::FMixerDevice* MixerDevice = FAudioDeviceManager::GetAudioMixerDeviceFromWorldContext(GetWorld());
	MixerDevice->RegisterSoundSubmix(AudioCaptureSubmix);
	MixerDevice->StartRecording(AudioCaptureSubmix, 0);
	
	FSoundSpectrumAnalyzerDelegateSettings SpectrumAnalyzerDelegateSettings;
	FOnSubmixSpectralAnalysisBP SpectrumAnalyzerDelegate;
	SpectrumAnalyzerDelegate.BindUFunction(this, FName("OnAudioCaptureSpectrumAnalyze"));
	MixerDevice->AddSpectralAnalysisDelegate(AudioCaptureSubmix, SpectrumAnalyzerDelegateSettings, SpectrumAnalyzerDelegate);
 
	// Mute echo
	AudioCaptureSubmix->SetSubmixOutputVolume(GetWorld(), 0);
 
 
 
 
 
void APlayerCharacter::OnAudioCaptureEnvelope(const TArray<float>& Envelope)
{
	FString Debug = "";
	for (int i=0; i<Envelope.Num(); i++)
	{
		Debug += FS(Envelope[i])+" ";
	}
	TRACE1(Debug);
}
 
void APlayerCharacter::OnAudioCaptureSpectrumAnalyze(const TArray<float> Magnitude)
{
	FString Debug = "";
	for (int i=0; i<Magnitude.Num(); i++)
	{
		Debug += FS(Magnitude[i])+" ";
	}
	TRACE1(Debug);	
}

[Attachment Removed]

5

Hi Ricky,

One thing to check. You will need to call:

FAudioDevice::StartSpectrumAnalysis(....)
 
FAudioDevice::StopSpectrumAnalysis(...)

and do the same kind of thing with:

FAudioDevice::[Start/Stop]EnvelopFollowing(...)

Let me know if this helps.

[Attachment Removed]

6

Hi Ricky. Apologies for the delay on responding.

I think you need to register the submix before starting the envelope following and adding the delegate. Since the submix is not registered at start envelope following, it starts the follower on the main submix instead, but you are attaching the delegate to the created submix which isn’t doing the envelope following so no delegate gets called. Moving lines 20 and 21 to just below line 10 in your code should help that delegate start firing on the correct submix.

The spectrum analyzer is initialized after the submix is registered, thus delegate functions are called. However, the settings parameter is left uninitialized, and the analyzer skips processing without those. Fill out the settings defined on line 24 with some default values and you should see the array of magnitudes fill up. For example:

FSoundSpectrumAnalyzerDelegateSettings SpectrumAnalyzerDelegateSettings;
SpectrumAnalyzerDelegateSettings.UpdateRate = 15.f;
 
FSoundSubmixSpectralAnalysisBandSettings Band;
Band.BandFrequency = 1000.f;   // Hz
Band.QFactor = 2.f;
Band.AttackTimeMsec = 10;
Band.ReleaseTimeMsec = 500;
SpectrumAnalyzerDelegateSettings.BandSettings.Add(Band);
 
// add one FSoundSubmixSpectralAnalysisBandSettings entry per frequency band desired
  
MixerDevice->AddSpectralAnalysisDelegate(AudioCaptureSubmix, SpectrumAnalyzerDelegateSettings, SpectrumAnalyzerDelegate);

The magnitudes returned correspond to the bands added in the settings

[Attachment Removed]

7

Thanks Richard, this worked a treat!

(fwiw I think it would be super useful to put working example code like this into the docs - especially for things like this, where there isn’t tons of info online about the correct way to do it)

[Attachment Removed]

8

Thanks for the feedback, Ricky. I will definitely pass this suggestion on to the Product Team.

[Attachment Removed]

9

Hey Richard,

I was doing it for Envelope Following, I believe: On line 13 in the code above, I do:

AudioCaptureSubmix->StartEnvelopeFollowing(this);which does this:

void USoundSubmix::StartEnvelopeFollowing(const UObject* WorldContextObject)
{
	if (FAudioDevice* AudioDevice = GetAudioDeviceFrom(WorldContextObject))
	{
		StartEnvelopeFollowing(AudioDevice);
	}
}

.. however I still don’t get into my EnvelopeFollower delegate method.

for Spectral Analysis the situation is different: it turns out I wasn’t calling StartSpectralAnalysis on my Submix, so I just added:

AudioCaptureSubmix->StartSpectralAnalysis(this);.. and now I do get into my OnAudioCaptureSpectrumAnalyze delegate method.

However, the TArray<float> Magnitude is empty.

My guess is that maybe I need to provide some sensible values to the FSoundSpectrumAnalyzerDelegateSettings struct (including perhaps some entries in the BandSettings

array). However, I can’t find any guidance on what those might look like: is there a helper method to provide sensible defaults somewhere?

thanks,

Ricky

[Attachment Removed]