Here are my findings on what I’ve discovered in trying to do volume rendering in UE4. Here are the approaches I’ve tried. All use some kind of UV indexing into a 2d texture exactly how @<a href=“https://forums.unrealengine.com/member.php?u=82295” target="_blank">r</a>yanB describes it. I’d derived all that stuff myself with modulus etc so want to check out if ryan’s new nodes in 4.13 (they missed 4.12) are doing the same but faster. I havn’t bothered with the two z interpolations but aware that it was needed, its just been on the back burner. Check out my youtube videos for all the things ive tried.
Using gpu particles to fill a bounded area that index into the 2d texture.
Pros, easy to set up and get a lot of particles to fill the space. works with fourier opacity maps by default to get self and environment shadowing.
Cons, particles are placed randomly (good/bad? bit of both), breaks up patterns not consistent placement though, density is not adapted to how many particles there are (could link) but hard to relate density to space between particles.
Half angle slicing - required c++ plugin to generate procedural mesh that renders back to front to avoid deferred renderer transparency order issues. @<a href=“https://forums.unrealengine.com/member.php?u=82295” target="_blank">r</a>yanB someone at epic has the code I believe if you want to look over it). Could probably be done in BP but worried about performance.
Pros - good even distribution of geometry, can relate number of slices to density in material so changing number doesn’t lower/raise apparent density.
Cons - Had serious issues with fourier opacity maps and bounds of mesh. @<a href=“https://forums.unrealengine.com/member.php?u=404” target="_blank">DanielW</a> wasn’t sure of cause but thought it might be a bug. imho opacity maps don’t give a good look for volumes anyway even when they dd work with particles. required inputting a light angle from scene to change slice direction. This method is prone to a ‘pop’ when you go beyond the angle threshold. a large number of these items causes the game thread to slow down because of generating that much vertex data and uploading to GPU. probably don’t want that many anyway.
Ray marching. nice results
Pros - everything what @<a href=“https://forums.unrealengine.com/member.php?u=82295” target="_blank">r</a>yanB said
Cons - Can be expensive with shadows as said before but read below…
no video for that yet.
So now my mind has been coming back to this and I want to allocate some time to finishing what I think will be a good solution. Basically the idea is to combine slicing with raymarched shadows. The cost will be as ryan said O(Density Samples + Shadow Samples) as the initial density is taken care of by the layering of the geometry and becomes a simple compositing thing for the renderer to just do. Each pixel will then do it’s own shadow samples with raymarching. From my earlier ray marching, the initial denity step was pretty fast (on an nvidia 770 gtx). The Shadows samples slowed it right down so I was limited to ~16 as discovered also by ryan. Breaking it down into two linear cost steps would really make things doable. I’m kind of excited about it actually.
In terms of content, I was using maya and rendering maya fluids through an orthographic camera over 512 frame or so and moving the clipping planes every frame with 1/512 width. then storing that in 2d texture made with texturepacker. I even was able to pack animation in a texture. I think I was able to fit 32 frames of 128^3 voxel data into an 8k texture. With 3 or 4 channels that could be 96 or 128 frames of monochrome (the indexing math is harder/more expensive but doable). That was this…
You should also be able to remove the normalizes since you are just testing the sign of the result and that won’t change with vector length. ( maybe thats the same thing Jenny’s link pointed out but I didn’t watch it). Also, you could try moving that calculation to the CustomUVs so its all done on the vertex shader. It might save something. It might add some distortion to the result but should be ok.
Re: dopiken:
Everything I have read about half angle slicing also talks about using geometry slices but from what I can tell, you don’t need the geometry slices and you could treat it just like any other ray marcher as long as you structure the trace to do it in the right order where it alternates between lighting and density samples in a way that shares them. Maybe its just to make it a bit faster to index into the correct locations?
From my understanding, the biggest drawback to half angle slicing over regular ray marching is that it doesn’t scale super well since you would need two more buffers for each instance in the world since each would have a different slice angle from different viewing angles. Technically shadow volumes have the same limitation where you’d need a different volume for each instance that has a different rotation but you could at least share un rotated instances.
The nice thing about regular old ray marching is that even though it is slow, it is fairly simple and doesn’t have many issues with scaling other than the brute force cost.
I am unable to understand how I can speed it up using the general form. I have to generate the point vector anyways, so I am not sure what I will save. Could you maybe show me an example? If I had a screenshot or example code I might understand it better. Thanks.
Custom UVs? How can I use those? I read this, but I dont see how I can access the custom UV, nor what exactly it does.
RE: dokipen
I am looking through your stuff now. Thanks a lot. There is a lot to read up on, but it is so far very helpful. As for your offer, I would be honoured to work with you, but there are two small problems. First, I was unable to find your email and I am not sure I know what gchat is. Second, I am quite new to the implementation and this kind of usage of UE4. As such I am not sure how much I can actually be of help. If you still would not mind I would love to cooperate. That way I will learn more and I will be able to create a better system.
Custom UVs means performing operations on the vertex shader instead of the pixel shader. Go through and read that whole page you linked carefully, it explains all of it including how to use it.
I’d really like to hear your ideas about how to share the samples to mimic the slicing in a raymarch. I havn’t seen that done anywhere before. although I cant quite see how that would be done in a pixel shader unless there is some kind of buffer used in multiple passes.
Thats true if you are using the half angle slicing technique. What I am proposing is to not actually do half angle slicing for the shadows but do slicing for the density pass only (in that case you don’t even need half angle, just slice towards the camera) and then do the shadows in raymarching. If raymarching, you have do do shadow samples inbetween every density sample. If I do the slicing the density sample is done up front and overdraw and shadow samples are the main cost. It seems like in that case the shadow samples would be very similar to doing just a density sample. This matches with the big 0 cost like you say.
I’m close to testing. I’m actually first re-creating my slicing code in blueprint (mainly out of curiosity to see how it performs, especially when it comes to nativizing it and also because I want to be able to share this around without users having to compile plugins/modules etc). I’m up to the point where I’ve detected start and end intersection points and have written the box/plane intersection function. Now I just need to build the vertices and indexes.
That should work. You are basically talking about flipping the shadow volume behavior. The one downside to that method is it will require another large render target to store the density slices. It will have to be the same size as the original volume texture to avoid losing resolution. Also you would require a separate sliced volume texture for each instance in the world so it wouldn’t be as flexible as regular ray marching. But the good news is you will be able to compute the shadows in parallel instead of as nested steps and it should avoid the shadow volume edge bleeding artifacts.
Layering your slices will bring back some of the cost as overdraw so it is hard to predict exactly how that might perform compared to regular ray marching. It won’t be as fast as half angle slicing though because it is still not sharing the shadow samples between slices which means that the deeper samples will cost more than the ones near the edge of the volume, and there is more re-peat steps being taken.
I think shadow volumes will be a bit easier to start with so I may try an experiment with them soon. Basically you just precompute the light energy received for each voxel which is costly once, but from then on its just single sample reads to get it.
Can you compute shadows with this approach? If you precompute the light intensity for each voxel you still need to compute how much is obfuscated by other filled voxels. Or are you thinking about just shading of the voxel itself viewed in a vacuum?
Shadow volumes are one of the most basic and common forms of lighting and shadowing volumetric effects.
No, it is not shading the voxel in a vacuum, it takes into account the loss of transmission through all other voxels that are visible from the current voxel towards the light. if you had rotated instances, each rotated instance would have to have its own shadow volume. If they were the same rotation then they could all share one.
I haven’t read through all the replies here, but I did implement volume rendering in UE for our VR experience, Allumette. I wrote a shader that ray marched some voxel grids I exported from Houdini. I baked in the lighting, so there are no shadow rays to march, which made it feasible. The cloudscape was really really big so I had to use some tricks with empty space traversal in order to get it running at framerate with a reasonable number of steps. I presented this at DigiPro this year: http://dl.acm.org/citation.cfm?id=2947699
If you can’t get access to that paper, send me a message, and I can try to get you a preprint of the paper.
Also, if you are interested in volume rendering as a topic, I recommend checking out some of the notes from these SIGGRAPH courses (I helped out the first year in 2010). http://magnuswrenninge.com/productionvolumerendering
Also, Magnus Wrenninge (who helped organized those courses) wrote a book on volume rendering that has a ton of good info:
Yeah you are right about the cost. as I was going to sleep last night it dawned on me that even though I would be splitting the density into the geometry slices and shadows ray traced, it still has to shade each pixel for each slice which is a large surface area vs doing a raymarch on each pixel on a bouding volume. The cost is probably the same.
I wasn’t even thinking about shadow volumes as I have no idea how to do that in the engine. it probably involves engine code modifications? I was trying to think of ways to use the new drawmaterialtorendertarget as a way to do this. Does that issue a command to be queued on the GPU? Is that also what the 2d canvas render capturing does (I guess you figured out that stuff from the flipbook/imposter BP tools you made). If those BP nodes just issue a command and you can exclude/include certain actors, would proper half angle slicing work by doing this…
create mesh section for one slice -> issues command?
2d render to render target from light angle -> issues command?
render next slice using 2d texture ditto
do that for all slices, etc Does all the commands then get run on the gpu in that order for one render frame?
ooh @, I think drawMaterialToRenderTarget could work for precomputing shadow volumes. the material that drew the unlit quad into the rendertarget would do the tracing (with the light direction as a parameter) and then the actual volume material reads that in during raymarch. I think that’s what you are getting at right?
we could then do two rendertargets for odd and even voxels (red/black similar to gaus seidel red black solvers) on alternate frame and blend between frames to get temporal sampling with a 1-2 frame delay. half the cost again (with twice the render targets)
I think I took a pretty different approach from you guys, and I wanted to share the rough outline of my process and some images. The basic idea is that I utilize 3d textures with my own loader, and ray march them in a post process shader.
Create a big OpenVDB grid in Houdini
Light in Houdini, and bake the color into rgb. I did channel lighting (r=key light, g=scattering, b=enviornment light)
Export it as a simple format in dense grid fashion
Modify the engine to load it as a 3d texture
Ray march the volume in a post process shader which samples the texture in a few steps:
1 - Initialize ray start positions according to the shell of the cloud to avoid empty space
2 - Ray march and sample the rgba texture. rgb are 3 baked in lighting passes mentioned in (1). a is the density
3 - Remap lighting to give a nicely lit appearance
There are a lot more details in the paper I mentioned, but I got this running nicely at 90fps on a Vive/Rift and 60fps on PSVR. By doing the rgb channel lighting in Houdini, I was able to get a bunch of lighting conditions like at sunset. I also played with some hacks doing dynamic lighting. Let me know if you have any questions.
Thank you so much for this. This will probably help me get to grips with volume rendering. I am new to the techniques and concept related to volume rendering so I appreciate resources like this. The course notes are huge, so thank you for that as well.
BTW, I am able to open and read the paper you wrote, but I am not able to download it and print it. It is a short paper so it does not matter that much, but I thought you might want to know. Maybe I can download it if I am on my university’s network. Will update after I have tested that.
Holy wow, I am completely blown away (cloud pun intended) by all of this…
This is so high above my skill level, but I’m bookmarking this thread and will read with intent each update
@dpenney These are very impressive results, thanks for sharing!
I am curious as to how you implemented termination of your rays. Usually this is done by rendering the back faces of the bounding geometry into a separate render target in a separate pass, but so far I have not seen a possibility to define custom render passes in UE unless you use some workaround like the Scene Capture 2D attached to the main camera. From the figures in your paper it looks like you only terminate when the ray opacity is saturated, but at the fringes of your clouds these would raymarch until the end of your scene?
Great results gpenny! That is similar to how I did the metaballs for the protostar demo by starting the raytrace with proxy geometry that was a bit larger than the actual spheres.
Re: deathray, basically you just need to use inverted polygons and solve a ray equation for where the box would have intersected the volume. Then you can pre calculate the number of steps through the volume and remove all math or branching to ensure the ray stays within the volume (you dont need to check every iteration when you know it will stay within due to precaulation of ray size and step count). Doing that alone saved almost 30% of the total time in my tests.
This is simple for things like boxes or cylinders but not as simple for gpennys method of arbitrary shapes. You could try to fit a box tightly around each cloud perhaps.
Termination of rays can occur anytime density goes over a certain threshold (which is the same as transmission going under some threshold).
