Download

Seeing and using your breath in VR - free source code

The source code (Unreal Engine 4.7) for my “Breath Tech Demo” is now available for free download. It demonstrates how to detect the players breath using a microphone and show it in VR with a direction and velocity to allow interaction.

See article for links

I’ve included some rough notes below on the various approaches I investigated. I’m new to game development so don’t assume these are the best approaches, but hopefully they may save someone a little time or inspire other new developers to experiment.
Please note, I’m very busy working on my own project so can’t offer any support with the code.

Sampling
Samples captured by a small C++ routine using UE4 voice capture API. API details found at quoteunquotestudio.com - thanks for sharing. I’m using the minimum 8Hz sample rate. Nothing more is required for breath detection. Check for new samples every tick. I calculate the amplitude being reached by at least 40% of the sample, compare it to the calibration data and create a 0 to 1 based velocity.

Each tick
Do a sphere trace to find actors / components of a Wind object type within a couple of meters. Raise an event on the actor so they can process the wind.

Showing breath in cold air or bubbles when underwater
Spawn new sprite emitters for each part of the breath and send them off on a forward vector based on the velocity at that time.

Dust particles
Dust particles are spawned as actors with physics enabled so I can simply apply a directional force.

Blowing dust off an object
Use a mask to hide the dust where it has been blown away. I used a SceneCaptureComponent2d and some hidden geometry painted with decals to create the mask. This could be done much better and smoother using C++, updating the texture for the mask directly and using the mask to calculate the size of the dust cloud when blown.

Candle Flame
A static mesh using a dynamic material instance with world position offset adjusted based on breath velocity.

Cloth
Use a skeletal mesh and NVIDIA APEX SDK to create the cloth. I use a WindDirectionalSource to manipulate the cloth based on the rotation of the hmd and breath velocity. I couldn’t find a way to make the wind source only affect a single actor. Ideally there would be “wind importance volumes” attached to each wind source so we could define the reach and fall off.

Bottle
Just uses a collision box and plays a sound when found by a sphere trace.

Low oxygen area
Note that you can’t detect an inhale. You can play a sound on an exhale, then assume an inhale when nothing is detected. Play around with affecting the volume based on the velocity, or the decay based on the duration of the exhale.

edited:to correct formatting

I love that this exists!
Thanks for your work it looks really impressive and would be a neat part of the immersive experience. I look forward to using it asap.
Cheers!
Matt Hermans

oh, that’s so creative !

I suppose something is missed in the source code?
949ee5cbd91df7a120addc6b0ddc23d20204a831.png

I mention in the included notes.txt file that the software makes use of FMOD for sound.
You can either install their free plugin for Unreal Engine http://www.fmod.org/download/
Or you can change the code to use native Unreal Engine functionality. I’ll change my article to make it clearer.

Hope that help.

Would this work for speech also. with MMO games at the moment most of the chatting is done via the chat screen, But with this connection with mic input wouldn’t you be able to connect the mic input to speech and have the sound projected out of the mouth. So you could have a normal conversion in VR.

It’s not something I’ve tried, but the Unreal API’s are their for voice chat in games - so i should work OK. Using the 3d positional audio and making the avatars mouth move would make it pretty good.

Jumbli, I hadn’t got around to test your breath for VR code. But I just tried it and it was fun as hell. I love the cold breath, the best part was speaking and the the cold air showing my breath make a great experience. now I wish I had a snow level in my game to show it. Hmmm. I might add a dungeon and make it damp and cold, just for this.

The only down fall as the Fmod. Norton detected as a virus and broke it, even though I restored it. So the sound stopped working.

Great Work, I wonder what you may try next.

This looks great.

Has anyone tested it with the Gear VR and got it working by any chance?

this is just awesome! Do you see any way to modify your aproach to actually record the voice Input from the microphone and integrate it as a soundwave in the game? I’m looking for way that players can leave audio messages for other players in a multiplayer game. Any help and hints much appreachiated.

thanks a lot!
andtheand

It sounds possible. The code I used gives you the raw sample data, so you should be able to buffer it up and store it. The next problem would be to load and play the data. I came across the following post which may be useful: Custom Music Player

couldn’t you use the same as video recording. But only play the audio. That’s just a rough guess

Can you record video at runtime? I’ve not come across that. Ideally you would only record audio so that you don’t have the overhead of storing the video.

there are some guys who implemented webcam streaming or even twitch streaming into a level, but I haven’t seen any recording of this kind of input yet. Sadly I’m a total c++ noob, so it will take some time for me to figure this out. Thanks for the help anyway!

Hi, Jumbli,

This is a really neat project. I was trying to implement this but I’m glad I stumbled upon this post . Your project has saved me quite a few hours. Do you know which version of the FMOD plugin your project is compatible with?

Thank you

Hey jumbli,

I’ve downloaded your breathing examples and I had to update it to 4.12.5. (Although I had to nerf the interactive and the airless areas because the blueprints were broken but they were beyond my immediate need.). There was some macros syntax changes in the C++ that were needed (“FriendlyName” to “DisplayName”). After that, I was kind of expecting it to work. I tested the environment and when I went to calibrate the mic, the runtime didn’t register a mic. The .ini values are set up.

I have a mic that does work in a bunch of games and in skype so it’s not the hardware.

Somewhere between the C++ and my mic, there’s configuration that I’m missing or something… Opinions? Common issues, I should look out for?

Jeremy