I would like to merge your changes in this Github commit to make audio work on iOS. Are those all the changes that are needed, or are there more? I am only interested in the MTAudioTap stuff for now, but some of your other changes look interesting, too. I’ll look at those later.
Also, would you be able to submit a pull request that contains only the MTAudioTap related changes? If I can see just those changes, I might be able to get it in for 4.18. Thanks!
HI gmpreussner,
I had to hack the audiomixer a bit also - the buffer sizes didn’t match in some cases, it didn’t handle deactivating the app and reactivating (neither did the media framework), and as mentioned elsewhere in this thread it was hardcoded for USoundWaveProcedural. Not sure a clean PR can be made at this time (i.e for IOS and Mac). If you only care about Mac and you don’t care about the audio mixer then I think you just need the changes in AvfMedia. However, tbh the media framework shortcomings on Mac are the least of the problems. It’s highly unusable due to the editor being unstable (crashes several times a day for me - slate and cooking bugs) and takes forever to build due to generation of DSYM files (5 to 10 minutes after changing a single line of code). We only use mac to deploy to ios so I can’t spend time dealing with the myriad issues there.
Oh, I also had to hack AvfMediaPlayer a bit to get it to not die on remote urls.
Sorry you’re having so much trouble with the Editor. We actually just started an initiative to get the macOS Editor stable and to feature parity with Windows. It will take a while, but we’re actively working on it.
Could you point me to the commits that make the required changes in AudioMixer?
Also, MediaSoundWave has been removed and replaced with a procedural sound component. I may not even need your changes, but I’d like to take a look anyway.
Right, I’m aware - which is why i haven’t bothered to clean this up much or at all. Also unfortunately some changes to AvfMedia were submittted to the 4.17 release branch which I never bothered to merge since they weren’t relevant to us.
Hey!
These changes to the AudioUnit backend look good and we’re in the process of checking these out now. The multi-suspend and div by zero case on init are particularly concerning. Surprised we missed these during testing, I’ll see if we can add more rigorous mobile testing for audio. Right now QA is having to do double-duty testing (old and new audio engine on 9 platforms) and think this just slipped through the cracks.
I have a question on the change: I’m not a iOS expert, but did you run into the case where the callback requested bytes change between callbacks? The docs imply its possible but we weren’t able to find a case where it happened. It would be useful to know how you triggered that case so we can figure out how to deal with it. Also, your code to handle this doesn’t deal with buffer truncation. If we don’t fully consume a generated buffer in a given callback, I’d expect to need a loop here in the next callback and then call ReadNextBuffer() once we fully consume the previously generated buffer if we don’t have a submitted buffer ptr or we’d get underruns/discontinuities.
We appreciate the work and apologize for it not being perfect out of the box yet.
We opted to send it out in 4.17 under experimental vs waiting until 4.18. It’s not internally exercised (or fully tested really) as we’re not using this brand new backend on any of our projects yet. Of course, this is why it’s released as “experimental”. UE4 releases are constant and juggernaut-like and as you can imagine, its hard to roll out a huge system refactor (especially when it’s just one programmer working on it while also supporting 6 internal games and many licensees) because the work spans many releases. We also prefer getting works-in-progress out to devs and licensees ASAP so they can give feedback while its being worked on (and help us shake out any bugs). Hopefully it won’t be too many more releases before we take it out of experimental.
I’ll add you as a the github pr credit so you should get a shout out in our next release. I’m going to try and push to get these fixes in a 4.17.2 hotfix if its not too late.
Minus_kelvin,
Sorry, this was just a quick hack to get something running. IIrc I was still seeing errors in setting the size of the buffer (setPreferredIOBufferDuration) even after avoiding the divide by zero. That may be the root of the problem. I can’t say we’ve tested this much either at this point. I’ll let you know if we run into more problems. Obviously if it wasn’t so onerous to do UE4 development on Mac/IOS I could offer more help - and your QA would probably be a lot more productive too. So the editor improvements mentioned by gmpreussner are welcome - Please don’t forget to include the ios deployment as part of that! While I was at google we worked with Jack P and B to get the android deployment sorted - IOS needs similar love.