Plugin for Raymarching (Volume Rendering), Medical Data visualization and Labeling

This plugin is deprecated in favor of an improved version!
New and improved plugin can be found here.

Hi guys.

I’d like to share a plugin that I created as part of my master’s thesis at the Technical University of Munich (great uni btw, no tuition, great research and industry connections etc.).

The plugin is mainly for Raymarching true volume textures. We implemented some nice raymarching shaders based on Ryan Bruck’s plugin, but using actual Volume Textures (since those weren’t yet implemented when he was making his plugin).

I’m not gonna write too much about it here, as I wrote a pretty extensive guide about it on the GitHub page, so just a short list of things that the plugin can do:

**Raymarching true Volume Textures with a transfer function **- since our use-case was medical data visualization, we only support loading single-channel data which is then converted into a RGBA value by a transfer function

Raymarching together with labels - you can add a second volume texture with label information and the labels will be color-displayed over the medical data

Writing spheroids into volume textures - this we used to actually write labels into a volume texture, we only needed to label spheres, but it should be pretty self-explanatory how to label different shapes

Precomputing illumination using a Light volume texture - because our assumption was that the lighting wouldn’t get changed too often, we use a separate volume texture for storing light information at each voxel of the data-volume.

**Invoking compute shaders **- this plugin is a great learning tool for anybody trying to use compute shaders to perform any kind of calculations on the GPU. The documentation for this is non-existent and most other plugins for this are obsolete by now so they don’t work out-of-the-box, we use compute shaders for the illumination calculation and labeling.

**Loading 8-bit MHD files **- that’s a format used for storing the volume texture of some medical scans

Some utility functions for Volume Textures - as volume textures aren’t used too much in the engine by now, we added some blueprint utility functions for them, such as creating volume texture assets from raw uint8 arrays, creating persistent volume textures (ones you can save and load in the editor), clearing textures and so on

Limitations

The main limitation is that it requires a custom engine build, because there is currently no way to create Volume Texture assets that are UAV-targettable (and we need RW access to the texture in compute shaders, so we needed this). We could’ve created the underlying resources manually, but then we couldn’t pass them nicely into the material editor…

If you like the plugin and would like to see it being usable on the default engine build, upvote our pull request here.

Another limitation is that since I was a bit too enthusiastic about blueprint programming, a lot of stuff that could’ve been done in C++ isn’t and would have made more sense there. Ah well, live and learn.

Only compatible with the **release **branch at our custom engine repo here
https://github.com/TheHugeManatee/UnrealEngine

Or if you already have a 4.22 source build, just copy the changes from the pull request and save yourself a download.

Demo
Since a picture is worth a thousand words and the video is 300s at 60fps, here’s roughly 18 MWords worth of information on the final project :slight_smile:

[SIZE=14px]Unfortunately, the final project contains some corporate IP and we use a paid plugin in it (VRIntegrator, great plugin btw), so we cannot release the whole thing.

UPDATE: Before, Github LFS Quotas were screwing up the checkout of the plugin, now everything works and it can be checked out.[/SIZE]

Since UE constantly changes, this is now also obsolete (as of 4.24) :smiley: And since I’ll be working on a project that will use some of the code from this plugin, I’ll be updating it to be 4.24 compatible in the coming weeks/months.

Planned changes:

  1. use RenderGraph instead of RHI Immediate Commands to (maybe?) get some performance boost Rendering Dependency Graph | Unreal Engine Documentation

  2. Get rid of the need for a custom engine build! Have a couple ideas on how to do this, as the only reason we’re using a custom build is being able to use VolumeTexture assets whose resource is targettable as a UAV

  • possibility one - Bypass the material editor by handling raymarched volumes in a separate render pass (since Epic made that easier in 4.22, I’ve heard https://www.youtube.com/watch?v=qx1c190aGhs)
  • possibility two - Since the textures Resource is public, maybe just hardcore create a new RHITexture3D and swap the resource (which would be a super ugly hack)
  • possibility three - pray every day that Epic gets to my 18 months old pull request in the coming months and just implement the change

So yeah, probably separate render pass it is :wink:

Stay tuned.

Cheers, Tommy.

Nice plugin and nice Master thesis I love that, I have a question is it good for Medical Images? and how can I use different frames of images?

Alright, I figured out a way to do this without a custom engine version and have a full cleaned-up project almost ready! Just need to update a readme and make a tutorial video for it and it’s gonna get uploaded tomorrow/the day after.

It’s coming!

Cheers, Tommy.

New and improved version of this plugin can be found here. I won’t be updating this thread or plugin anymore, the new one is superior in many ways :slight_smile:

1 Like

New what? You mean the Volumetric features inside Unreal natively in 4.25?

Have you looked in the forum post I linked? You can find the answer there pretty easily.