Playing a 2D Sound in a custom thread

[444_Nuits]

Hi,

I’m creating a demo game showcasing my plugin that can imitate voices based on models.

To do that I record and transform the audio using JUCE through this function that runs in its own thread :

void AAudioController::audioDeviceIOCallbackWithContext(const float* const* aInputData, int aNumInputChans, float* const* aOutputData, int aNumOutputChans, int aNumSamples, const juce::AudioIODeviceCallbackContext& context)
{
    (void)context;
    int index, numActiveInChans = 0, numActiveOutChans = 0;
    int numOutsWanted = 2;
    const int numInsWanted = 1;


    for (index = 0; index < aNumInputChans; ++index)
      if (aInputData[index] != 0)
        inChans[numActiveInChans++] = (float*)aInputData[index];

    while (numActiveInChans < numInsWanted)
      inChans[numActiveInChans++] = mEmptyBuffer->getWritePointer(0, 0);

    for (index = 0; index < aNumOutputChans; ++index)
      if (aOutputData[index] != 0)
        outChans[numActiveOutChans++] = aOutputData[index];

    index = 0;
    while (numActiveOutChans < numOutsWanted)
      outChans[numActiveOutChans++] = mEmptyBuffer->getWritePointer(++index, 0);

    juce::AudioSampleBuffer input(inChans, juce::jmin(numInsWanted, numActiveInChans), aNumSamples);
    juce::AudioSampleBuffer output(outChans, juce::jmin(numOutsWanted, numActiveOutChans), aNumSamples);

    for (int channel = juce::jmin(output.getNumChannels(), input.getNumChannels()); --channel >= 0;)
      output.copyFrom(channel, 0, input, channel, 0, aNumSamples);
  
    float* leftOChannel = output.getWritePointer(0);
    float* rightOChannel = output.getWritePointer(1);

    float *leftIChannel = input.getWritePointer(0);

    int blockSize = output.getNumSamples();

    std::vector<short> bufferIn(blockSize);
    std::vector<short> bufferOut(blockSize);

    convertFloat32toInt(leftIChannel, blockSize, (char*)bufferIn.data(), false);

    if (_handle == -1)
    {
    }
    else
    {
      std::vector<short> transformedAudio;
      InBufferInt.insert(InBufferInt.end(), bufferIn.data(), bufferIn.data() + blockSize);
      imi_trans_transform(_handle, (imi_sample_t*)bufferIn.data(), (imi_sample_t*)bufferOut.data());
       
      //fill the outBufferInt with the bufferOut
      outBufferInt.insert(outBufferInt.end(), bufferOut.data(), bufferOut.data() + blockSize);

      float* bufferOutFloat = new float[blockSize];
      convertInttoFloat32((char*)bufferOut.data(), blockSize, bufferOutFloat, false);

      // Write bufferOutFloat to the output audio channel2
      for (int channel = 0; channel < juce::jmin(output.getNumChannels(), 2); ++channel)
        output.copyFrom(channel, 0, bufferOutFloat, blockSize);

      // If the number of output channels is greater than 2, clear the additional channels
      for (int channel = 2; channel < output.getNumChannels(); ++channel)
        output.clear(channel, 0, output.getNumSamples());
    }
}

This works fine but it doesn’t generate any sound in the level and just play the audio through the callback of my output devices I’d like to change that by capturing the audio data transformed here :
imi_trans_transform(_handle, (imi_sample_t*)bufferIn.data(), (imi_sample_t*)bufferOut.data()); (so bufferOut)

and playing it in another thread but I don’t really know how to continuously get the buffer data and play it even though I’ve already played sound using a float buffer like that :

Audio::FSoundWavePCMWriter Writer;
Audio::FSampleBuffer TransformedBuffer(AudioBuffer.data(), AudioBuffer.size(), 2, 32000);
TransformedSoundWave = Writer.SynchronouslyWriteSoundWave(TransformedBuffer, nullptr, nullptr);
UGameplayStatics::PlaySound2D(GetWorld(), TransformedSoundWave);

If anyone could give me some help through this project I’d be grateful :))

[444_Nuits]

Here’s what I’ve already done :

I created a FRunnable thread running this function : (AAudioController is the class where my Juce thread runs on)

FAudioProcessingThread(AAudioController* controller, AAudioCaptureActor *actor) : mController(controller), AudioCaptureActor(actor), Thread(nullptr), bRunning(false), bStopThread(false) {}

virtual uint32 Run() override
{
		while (bRunning)
		{
			if (bStopThread) {
				UE_LOG(LogTemp, Warning, TEXT("Stopping Audio Thread"));
				break;
			}
			AudioCaptureActor->DequeueAudioData(mController);
  }
  return 0;
}

Here’s how I fill my TQueue AudioQueue:

//fill the audio queue with the output buffer
std::lock_guard<std::mutex> lock(AudioQueueMutex);
AudioQueue.Enqueue(bufferOut);
//unlock the mutex
std::lock_guard<std::mutex> unlock(AudioQueueMutex);

bufferOut being the short vector output from my imi_trans_transform result

Here’s how I dequeue it and play sound from it :

void AAudioCaptureActor::DequeueAudioData(AAudioController* controller) const
{
  if (!controller)
    return;
  if (controller->AudioQueue.IsEmpty())
    return;

  std::vector<short> audioData;
  if (controller->AudioQueue.Dequeue(audioData))
  {
    // Lock the mutex before accessing the queue
    std::lock_guard<std::mutex> lock(controller->AudioQueueMutex);
    // Create USoundWave and play audio
    USoundWave* SoundWave = NewObject<USoundWave>(GetTransientPackage());
    Audio::FSoundWavePCMWriter Writer;
    Audio::FSampleBuffer TransformedBuffer(audioData.data(), audioData.size(), 2, 44100);
    SoundWave = Writer.SynchronouslyWriteSoundWave(TransformedBuffer, nullptr, nullptr);
    UGameplayStatics::PlaySound2D(World, SoundWave);

    // No need to unlock the mutex or clearing the audio data explicitly; it will be unlocked when lock_guard goes out of scope
  }
}

Unfortunately at some point the Tail of my queue is a nullptr so when I check if the queue is empty it throws me an exception :((

[444_Nuits]

Up !

I Did some stuff :

I removed the FRunnable class and decided to rplay the sound in the controller class

In my AAudioController class, I instantiate a std::thread to read and play from a sound file and in my callback thread I just write in the file so it looks like this :

When creating my AAudioController class, I call StartRecording() at the end :

	void StartRecording()
	{
#if WITH_EDITOR
			file = fopen("C:\\Users\\PC\\Desktop\\StageSofiane\\output.snd", "wb");
#else
			FString FullPath = *FPaths::ProjectContentDir() + FString(TEXT("imitation/RecFile.snd"));
			file = fopen(TCHAR_TO_ANSI(*FullPath), "wb");
#endif
		stopPlaybackThread = false;
		playbackThread = std::thread(std::bind(&AAudioController::PlaybackThread, this, _world));
	}

Here’s how I play the sound in my thread :

void AAudioController::PlaybackThread(UWorld * world)
{
  while (!stopPlaybackThread)
  {
    std::vector<float> audioChunk;
    {
      std::unique_lock<std::mutex> lock(playbackBufferMutex);
      playbackBufferCondition.wait(lock, [this] { return !playbackBuffer.empty() || stopPlaybackThread; });
      if (stopPlaybackThread && playbackBuffer.empty())
        break;
      audioChunk = std::move(playbackBuffer.front());
      playbackBuffer.pop();
    }

    // Create USoundWave and play audio
    USoundWave* SoundWave = NewObject<USoundWave>(GetTransientPackage());
    Audio::FSoundWavePCMWriter Writer;
    Audio::FSampleBuffer TransformedBuffer(audioChunk.data(), audioChunk.size(), 1, SAMPLE_RATE);
    SoundWave = Writer.SynchronouslyWriteSoundWave(TransformedBuffer, nullptr, nullptr);
    UGameplayStatics::PlaySound2D(world, SoundWave);
  }
}

Here’s how I write in the file :

void AAudioController::audioDeviceIOCallbackWithContext(const float* const* aInputData, int aNumInputChans, float* const* aOutputData, int aNumOutputChans, int aNumSamples, const juce::AudioIODeviceCallbackContext& context)
{
...
 fwrite(bufferOutFloat, sizeof(float), blockSize, file);

 {
   std::lock_guard<std::mutex> lock(playbackBufferMutex);
   playbackBuffer.push(std::vector<float>(bufferOutFloat, bufferOutFloat + blockSize));
 }

 playbackBufferCondition.notify_one();
...
}

And at the destruction of my class I call StopRecording():

void StopRecording()
{
	{
		std::lock_guard<std::mutex> lock(playbackBufferMutex);
		stopPlaybackThread = true;
	}
	playbackBufferCondition.notify_one();
	playbackThread.join();

	if (file != nullptr)
	{
		fclose(file);
		file = nullptr;
	}
}

This seems to work but the sound played looks strange, it seems that not all the samples are played

Here is the comparison between the file read (bottom) and the sound played (on top) :

I don’t have enough knowledge to figure out something between and what could fix this issue

[444_Nuits]

Here’s the buffer I pass through the soundwave to produce the sound

And here’s the sound I play :

[444_Nuits]

Alright I’ll give up

I may have found an other way by using a voice chat and changing its input source from the output of my callback function using an audio router

I don’t think playSound2D is appropriate to play a short sound continuously