I’m trying to implement cross platform voice chat in my project and for the most part it currently seems to be working quite well. The voice data is pulled from the VoiceCapture interface it is then encoded using Opus and sent across the network, where it is then received and processed and decoded (again using Opus) by remote players.
The only problem I am having is that whenever more than one remote player speaks at the same time the sounds keep cutting in and out sometimes it’s quite terrible with random ‘popping’ noises appearing from somewhere.
I did a stat grab when this was happening and the AudioThread seems extremely high. I should also note that the audio components created for receiving voice sounds are attached to a players multiplayer representation and have attenuation and spatialization settings applied to them.
I’ll post the code below for when I am receiving the voice data and queuing the audio using USoundWaveProcedural, can anybody see any problems with anything I’m doing here that might cause this?
FPhotonRemoteTalkerDataStream* QueuedData = RemoteTalkersList.Find(SenderID);
if (QueuedData == NULL)
{
//check the player has been loaded into the game
if (!CheckValidPlayer(SenderID))
return;
//if we do not have a remote talker established create one here
RemoteTalkersList.Add(SenderID, FPhotonRemoteTalkerDataStream());
QueuedData = RemoteTalkersList.Find(SenderID);
}
//don't play thr voice if the player is muted
if (QueuedData->isMuted)
return;
//update their last seen status
QueuedData->LastSeen = FPlatformTime::Seconds();
uint32 UncompressedDataSize = MaxUncompressedDataSize;
// de compress the data to its original raw PCM
VoiceDecode(Data, Size, UncompressedData.GetData(), UncompressedDataSize);
if (UncompressedDataSize <= 0)
{
UE_LOG(LogOnline, Log, TEXT("Received 0 Bytes for Voice"));
return;
}
// Generate a streaming wave audio component for voice playback
if (!QueuedData->AudioComponent->IsValidLowLevel() || QueuedData->AudioComponent == NULL || QueuedData->AudioComponent->IsPendingKill())
{
QueuedData->AudioComponent = CreateVoiceAudioComponent(VOICE_SAMPLE_RATE);
//attach the component to the players headset for spatialisation
PhotonInterface->AttachAudioComponentToHeadset(SenderID, QueuedData->AudioComponent);
if (QueuedData->AudioComponent)
{
QueuedData->AudioComponent->OnAudioFinishedNative.AddUObject(this, &APTPhotonVoiceInterface::OnAudioFinished);
QueuedData->AudioComponent->Play();
}
}
if (QueuedData->AudioComponent != NULL)
{
//manually update the sound location so attenuation is correct
EUpdateTransformFlags flags = EUpdateTransformFlags::None;
ETeleportType TeleportType = ETeleportType::None;
QueuedData->AudioComponent->OnUpdateTransform(flags, TeleportType);
USoundWaveProcedural* SoundStreaming = CastChecked<USoundWaveProcedural>(QueuedData->AudioComponent->Sound);
if (SoundStreaming->GetAvailableAudioByteCount() == 0)
{
UE_LOG(LogOnline, Log, TEXT("VOIP audio component was starved!"));
}
//PS4 does not support the default decompression type so set it here
SoundStreaming->DecompressionType = DTYPE_Procedural;
//add the voice data to a sound que
SoundStreaming->QueueAudio(UncompressedData.GetData(), UncompressedDataSize);
}
However I don’t believe this is my issue. I’m not using Online subsystems to get the voice data or for multiplayer. I’m using third party source for networking and the IVoiceCapture interface for voice capture, I’ve debugged the sending of packets and none of them are ever missed (it’s at least very rare)
My problem seems to lie with the actual playing of the sounds themselves.
I understand, sorry. I figured you were using what’s included with UE4.
How often are you getting the UE_LOG(“VOIP audio component was starved!”)?
I am curious if the root of the problem isn’t the sounds being received but maybe the audio thread causing the issue because it isn’t able to play it correctly; you mention you hear popping.
Have you tried playing these sounds either on their own thread or outside of the audio thread? I’m honestly not sure if all AudioComponents are handled in the audio thread or if you can play them without it.
Lastly, the audio system had a overhaul recently, have you tried 4.16? Again, I am not sure at the extent up the updates to the system but it might be worth exploring.
Sorry I can’t be more help. My experience with the system is limited to debugging through it when exploring VOIP issues and not actively using it.
I do seem to get that log every now and again but not enough for me to suspect that’s what is causing the popping.
I’m in the middle of moving the sounds being played and the voice being captured to their own thread now as I came to the same assumption when I noticed the problem wasn’t as bad on the non VR version of the project. I’m guessing this is because both the CPU and GPU threads are significantly lower in non VR.
Thanks for the reply, I’ll let you know how I get on.
I put the voice capture stuff onto it’s own FRunnable thread and even played the sounds onto a thread away from the audio thread. Neither seemed to make any difference.
When playing online with just one other player the networked voice is really clear. Almost immediately after just one more player joins the game, all remote voice connections start to pop and cut out.
It’s a really strange issue, after more checks there is definitely no voice data being lost over the network, so I believe you are correct that somewhere is struggling to play all these sounds at similar times.
My assumption is that the audio is transmitted to the server and then routed to all the clients. Maybe its a replication issue of trying to replicate those audio components down to all the clients?
I’m using Photon for networking, therefore there is no replication being used on anything. I’ve debugged the sending of the voice packets several times and none seem to be being lost. Are there any debug commands I can run to investigate further? I’ve tried turning on LogAudio and LogXAudio2 but there is nothing in them logs to suggest there is a problem.
I have never used Photon and didn’t know what it was until just now, when I looked it up, so I cannot offer any support with it.
I’m a little confused by what you mean when you say there is no replication on anything because how will the clients have any data if you aren’t replicating anything?
Since you’ve debugged the sending of the voice packets, have you tested receiving them?
All data is sent through Photon events right now including the movement of all actors etc that would usually be handled using replication if we were using UE4 networking.
Apologies if I wasn’t being clear. I’ve debugged both sending and receiving the voice packets and a very minimal amount (usually zero) are ever lost. It seems to just have trouble playing multiple USoundWaveProcedural audio components.
Creating individual static mesh actors for each remote player than attaching an Audio Component to each one. A USoundWaveProcedural is created with each Audio Component and when new voice packets are received they are queued on each players procedural sound wave.
When the audio is coming down to the clients, is it directly going from the server side down to the static mesh actors? Or, do you route through the Pawn/Controller to that Actor?
Also, I have never used the SoundWaveProcedural class but there are a couple things that stand out.
QueueAudio( ) expects the BufferSize to be divisible by 16 bits.
GeneratePCMData( ) can reset the entire AudioBuffer if bReset has ever been set to true; which I don’t know where but could be an issue if mismanaged.
Serialize( ) is calling USoundBase::Serialize( ) on purpose - with a comment - but I do not know if maybe the SoundBase has been updated or changed and USoundWave might not be handled as gracefully as it could be. This is even more odd because it looks like you can serialize the audio data from the class but there are comments about “never pushing compressed data to SoundWaveProcedural”, so maybe there is an issue of trying to play compressed sound data?
Lastly, this isn’t a huge class to debug through. I would step through and make sure certain conditions and asserts are clearing, especially during times where your audio is coming out bad.