Help on writing android voice module?

Development Audio
, , , ,
1

I’m currently successfully using the Voice Chat feature available through OnlineSubsystemNULL on PC and I’d like to extend the same functionality to my Android clients.

Though OnlineSubsystemNULL seems to work also on Android, the voice subsystem is currently not implemented on this platform (so I cannot neither send or receive voice chat when connected in a multiplayer session).
I’d like to help writing the missing code, but since I’m quite new to the UE source code, I would need some guidance.

In particular:

  • Is the opus encoder / decoder
    available on android? Trying to use
    it (e.g. enabling Android in
    VoicePrivatePCH.h) causes linker
    errors, what shall I modify to link
    this lib also to Android?
  • If I wanted to write a VoiceModule for Android what libs may I use?
    OpenAL? other?

Thanks, any help would be appreciated!

2

Any feedback? In particular, how can compile and link lib opus in the source code?

3

libopus is currently supported on Windows, Mac, and Linux in UE4. I haven’t tried making it for Android, but these steps may work.

There is an android.mk for libopus here:
https://android.googlesource.com/platform/external/libopus/+/master

Compile a libopus.a and libspeex_resampler.a for ARM and optionally x86.
Create an opus-1.1/Android/ARMv7 directory and optional opus-1.1/Android/x86 directory and copy the .a files into them.

Making the following addition to libOpus.build.cs in Engine/Source/ThirdParty/libOpus:

else if (Target.Platform == UnrealTargetPlatform.Android)
{
			PublicAdditionalLibraries.Add(LibraryPath + "/Android/ARMv7/libopus.a");
			PublicAdditionalLibraries.Add(LibraryPath + "/Android/ARMv7/libspeex_resampler.a");
			PublicAdditionalLibraries.Add(LibraryPath + "/Android/x86/libopus.a");
			PublicAdditionalLibraries.Add(LibraryPath + "/Android/x86/libspeex_resampler.a");
}
4

Thank you, that was exactly was I looking for! Everything worked like a charm and I’ve copied the static libs in the folder you said, but when I try to build UE4Game for Android I get the following error, is there any further configuration I should do?

MBP:UnrealEngine-Fork xxx$ ls Engine/Source/ThirdParty/libOpus/opus-1.1/Android/ARMv7/libspeex_resampler.a

Engine/Source/ThirdParty/libOpus/opus-1.1/Android/ARMv7/libspeex_resampler.a

MBP:UnrealEngine-Fork xxx$ ls Engine/Source/ThirdParty/libOpus/opus-1.1/Android/ARMv7/libopus.a

Engine/Source/ThirdParty/libOpus/opus-1.1/Android/ARMv7/libopus.a

MBP:UnrealEngine-Fork xxx$ ./Engine/Build/BatchFiles/Mac/Build.sh 
UE4Game Android Development
Setting up Mono
Engine/Build/BatchFiles/Mac/SetupMono.sh: line 32: [: /Users/xxx/Documents/Unreal: binary operator expected
Building UBT...
	 0 Error(s)
Building UE4Game...
2015-04-24 14:54:56.271 defaults[41608:4942887] 
The domain/default pair of (/Users/xxx/Library/Preferences/com.apple.dt.Xcode, IDEBuildOperationMaxNumberOfConcurrentCompileTasks) does not exist
Running command : Engine/Binaries/DotNET/UnrealBuildTool.exe UE4Game Android Development -deploy 
Compiling Native code with NDK API 'android-19'
Performing 1 actions (8 in parallel)
[1/1] clang++ UE4Game-armv7-es2.so
clang: error: no such file or directory: 'ThirdParty/libOpus/opus-1.1/Android/ARMv7/libopus.a'
clang: error: no such file or directory: 'ThirdParty/libOpus/opus-1.1/Android/ARMv7/libspeex_resampler.a'
-------- End Detailed Actions Stats -----------------------------------------------------------
Cumulative action seconds (8 processors): 0.00 building projects, 0.00 compiling, 0.00 creating app bundles, 0.00 generating debug info, 0.11 linking, 0.00 other
UBT execution time: 5.43 seconds
5

Ok, done with compiling lib opus (even though with the problems above, I had to use absolute paths in libOpus.build.cs to link with of course is just ok for working on but not for a pull request), done with a basic VoiceModuleAndroid.cpp that initialises the encoder and decoder.

Now I need to work on the capture part. How do I integrate this with the existing engine audio infrastructure?
I’ve seen that for playback the initialisation of the engine happens in AndroidAudioDevice.cpp. I suppose the HW initialisation should be up to the engine, and done just once. How do I properly get a reference to this instance of the engine?

6

Ok, capture part mostly done and working!

Pending issues:

  • Building lib opus: I had to rename the opus-1.1 folder to “jni” in order to have ndk-build working, plus I had to move the static libs manually. I would need some help on fixing and automatising this in the UE4 source code
  • As noted in the comment above, I had to use absolute paths in libOpus.build.cs in order to being able to link the libs.
  • On Android, I had to implement the OnSoundWaveStreamingUnderflow callback in order to have audio playback working (underflow would cause audio to stop). Currently I’m filling up the missing data with zeros and it kind of works, but probably there are better solutions (e.g. have the callback wait for enough data before proceeding). This issue also affects audio streaming on OS X (here it doesn’t stop the streaming, but causes weird stutters). Discussed this with Josh Markiewicz, I can provide more info here if needed.
  • There is a crash in SoundWaveStreaming.cpp due to concurrent access to an array. More info about this [here

I’m available for any further info, please let me know if more details are needed.

Android - Crash in SoundWaveStreaming - Audio - Epic Developer Community Forums

7

Hi devel.mad. I’m trying to get this to work on my android as well. I just need simple mic detection for noises, no play back or anything else. Have you gotten this working/can you help me out? What did you do to get to the point where you could read mic input from android?

8

Still working on it :slight_smile: What you are trying to achieve shouldn’t be too hard, you have to use OpenSL in Android for capturing are playing back audio.
Here is an article with a working example on how to use it:
https://audioprograming.wordpress.com/2012/03/03/android-audio-streaming-with-opensl-es-and-the-ndk/

9

Hey devel.bmad, i am sorry i am no help to your problem however i am trying to implement voice chat on PC alone, but am finding it difficult to find info on for unreal engine 4. I thought that maybe you might have had it working for PC initially and then worked on getting it sorted for android, and if so, maybe you could give me some information on how you got voice chat on pc working (even just links to any help if you used any) , it would be much appreciated. Thanks a lot.

10

Hi devel, I am trying to implement the voice capture part right now as well and have no direction of where I should go. Do you mind giving me a bit help on that? Also would be great to have the Opus library since I was unable to compile it.

Thanks

11

Hi FrankKappaPride, as concern the Opus library I used the following steps:

  • Started from the opus-1.1 folder in
    UnrealEngine/Engine/Source/ThirdParty/libOpus

  • modified the android.mk linked by Chris Babcock above to compile
    speex_resample.a

  • renamed temporary the folder opus-1.1 in “jni” as a workaround for
    an ndk compilation problem

  • ndk-build opus

  • ndk-build speex_resampler

  • copied the static libs libopus.a and libspeex_resampler.a in
    Engine/Source/ThirdParty/libOpus/opus-1.1/Android/ARMv7/

  • modified libOpus.build.cs adding:

     else if (Target.Platform == UnrealTargetPlatform.Android)
 	{
 		PublicAdditionalLibraries.Add(LibraryPath + "Android/ARMv7/libopus.a");
 		PublicAdditionalLibraries.Add(LibraryPath + "Android/ARMv7/libspeex_resampler.a");
 	}

This was never checked into GIT because I needed some guidance about how to properly build the lib with the UE4 source code (I build manually the libs and copied them in the appropriate folders).
You can find the files I used here:

As concerns the capture part, my implementation of the VoiceModuleAndroid.cpp is here:
https://drive.google.com/open?id=0B7SpUfi4PH7aS1VMcHQ4cmpWbnM

It’s perfectly working, but the code is not very clean. The code was partially derived by these tutorials:
https://audioprograming.wordpress.com/2012/03/03/android-audio-streaming-with-opensl-es-and-the-ndk/
https://audioprograming.wordpress.com/2012/10/29/lock-free-audio-io-with-opensl-es-on-android/

It would be great if you or some of the audio engineers from Epic (@MinusKelvin? are you listening?:slight_smile: ) could clean this up and integrate it in one of the future official releases of UE4. Let me know if you need more details.

12

Hi devel, thanks for the reply! Opus worked out really well :slight_smile: . Two things, the one you posted is VoiceModuleMac.cpp, are they using the same one or just a mistake? Also, do you play the soundwaveprocedural to play the raw PCM data? It is causing me an error and I’m not sure if you had the error before. I mentioned it in this thread :USoundWaveProcedural crash on android - Audio - Unreal Engine Forums.

13

Sorry, my bad. This is the right file: https://drive.google.com/open?id=0B7SpUfi4PH7aZEVXaGxKeXdGanc

This was tested in 4.9, some modifications may be needed for 4.11, let me know how it goes.
I didn’t have to use soundwaveprocedural to play the data, this was handled directly by VoiceEngineImpl.cpp (haven’t checked if this changed in in 4.11 though)

Also, if you found a way to properly integrate the generation of the opus libraries please share :slight_smile:

14

Hi devel, thanks for the help! You literally saved my day :slight_smile: I will test the capture part out on Monday!

I believed that the opus library is just a normal static library compiled with ndk tool chain. I found the command here: GitHub - xuan9/Opus-Android: Build Opus lib with Android ndk-build. After that I moved the .a file to the right place and modified the build.cs accordingly and everything seems to be working. I should probably do a pull request with the Android version soon enough.

15

Hi, I tested the Android voice module and seems like it’s working. I just have a question:

uint32 MaxRawCaptureDataSize = 100 * 1024;
		TArray<uint8> RawCapture;
		RawCapture.AddUninitialized(MaxRawCaptureDataSize);
		uint32 NewVoiceDataBytes = 0;
		EVoiceCaptureState::Type MicState = VoiceCapture->GetCaptureState(NewVoiceDataBytes);
		if (MicState == EVoiceCaptureState::Ok && NewVoiceDataBytes > 0)
		{
			VoiceCapture->GetVoiceData(RawCapture.GetData(), NewVoiceDataBytes, NewVoiceDataBytes);
			UE_LOG(LogTemp, Warning, TEXT("Getting data with %d bytes"), NewVoiceDataBytes);
			sendSound(RawCapture.GetData(), NewVoiceDataBytes);
		}

Is this the correct way of using the capture? I did create a voice capture class and called the start function. Also, I see that the readin valie are set to 2048 bytes every single frame. Is it intended?

Thanks

16

Hi, cannot remember exactly why I was using those parameters. I had been struggling a bit with underflows and audio artefacts and those values were a good compromise, but I’m not sure this is the best way to have it working. Using smaller values by the way would probably reduce the latency quite a bit.

17

Sorry to bother again, I am having troubles getting audio playing. I looked the voiceengineimpl.cpp and in fact it is using the soundwaveprocedural. When I use it it gives me an error very deep in AndoirdAudioSource.cpp. Do you have any idea how the OpenSL->Enqueue thing works in there?

I think It may be the same issue of what you discussed above about filling with zero and a callback. May you explore a little more on that?

Thanks

Edit: nevermind I got it working now :slight_smile:

18

Hi Senken, you can find some detailed instructions in this thread: How can I make in game voice communication? - Audio - Epic Developer Community Forums

Let me know if this woks for you.

19

Hi FrankKappaPride, sorry for not replying you before. Which was the issue? I had an issue in AndoirdAudioSource.cpp related to buffer underflows, but that was fixed in 4.9 if I remember well.

20

Hi FrankKappaPride, all, haven’t checked the git repos for a while, but I have seen the android codec is not yet integrated in the main distribution. If somebody could help me with polishing my implementation (the opus lib compilation in particular) I’d be glad to open a pull request. Is there anybody else who has already opened a request?