TBRaymarchPlugin - Volume Rendering (Raymarching) Plugin for Unreal Engine

Development Pipeline & Plugins Plugins
1

Hey guys, I’d like to share a project that I’ve been working on the last couple of months, it’s a massive improvement over my previous raymarching plugin that can be found here.

The whole thing can be found on my github and works out of the box (even contains the scan that you can see in the screenshots).

Long story short - it lets you render volumetric data (mostly used for medicinal visualization, but hey, whatever data you got, we will render).

Some screenshots to quickly grab your attention:
On the left you can see the volumetric rendering and on the right is a cut through the original data volume.

Render1.jpg
Render1.jpg1035×453 111 KB

render2.jpg
render2.jpg1022×489 114 KB

Both of these are rendered from the same CT scan, just using a different transfer function.

Features

  • Works out of the box with binary UE 4.26
  • Volume raymarching for arbitrary UVolumeTexture textures (true 3D textures, no flipbooks).
  • .dcm (DICOM), .mhd and .raw file import into volume textures.
  • Precomputed Volume illumination with unlimited number of light sources implemented in compute shaders using concepts from Efficient Volume Illumination with Multiple Light Sources through Selective Light Updates (2015) by Sundén and Ropinski, implemented using URenderTargetVolume and compute shaders.
  • Uses color curves for transfer function definition
  • Fully integrated and functional within UE editor viewport.
  • Windowing support (google DICOM Window center / Window Width or watch my youtube description for an explanation of what windowing does)
  • Basic menus for manipulating the volume
  • Basic VR support and example map

Limitations

  • Raymarched volume doesn’t cast or receive shadows info to/from the scene, it only self-shadows
  • We use a very simple (but fast) raymarching and illumination algorithm with no specular, refraction or scattering
  • Algorithm is already a bit dated and implementation leaves a lot to be desired as for efficiency. It is however, good enough for real-time applications with several lights and large (512^3 or more) volumes.
  • Currently do not support persistent (saveable) 32bit grayscale textures, so saving is limited to G8 or G16

If you want to give it a shot, just clone the repo, generate VS files, compile and play with it in the editor or in play mode. Everything should work out of the box, there is an extensive readme on my github for usage details and I went crazy with commenting my code, so everything should be pretty clear.

For a showcase video where I show off all the things you can do with this, here’s my 40 minute rant about it - new one should be coming soon, as this one is already kind of outdated.

Here’s a discord channel I made for the project:

Any suggestions, criticisms, bug-reports, pull-requests etc. are more than welcome!

Cheers, .

Edit: updated info to reflect that DICOM loading is now possible.

2 Likes
2

Hi !

Amazing work.

I’m trying to use this to generate 3d model from an MRI. The data is in DICOM format. I read your comments on the subject on your Github repo. I want to write the missing code in order to be able to read DICOM. What would be faster, convert DICOM to MHD and then use your plugin as is, or write the DICOM reader right into the Plugin code?

I have downloaded pydicom and been playing around, some tips would be great !!!

Thanks

Alejandro

3

Heya Alejandro, glad to see that this might be useful to people :slight_smile:

I should actually refactor the plugin, so that it’s not tied to MHD as much, it’s gonna be a simple inheritance structure, so you can just extend the asset for loading any kind of medical data.

But for now, if you can get the info that is needed in the MHDAsset from the DICOM series, create a new MHDAsset and just set the members with whatever you read from the DICOm, it will work just fine.

I think the cleanest solution for the DICOM loading is making a standalone ITK plugin (just a plugin wrapping a built ITK library with some convenience functions) and then using that to load the tags and binary data from my plugins. This would have the added benefit of not marrying the DICOM loading part to the rest of the raymarcher plugin, so people could use that for other stuff.

The only main challenge I had when reading the DICOM series myself straight in C++ was getting the 3rd party library built as .dll or .lib packaged into a plugin and forcing UE to load it and be usable from other plugins, I remember that being a pain.

Once I had that, the parser itself wasn’t too hard, as ITK or Imebra SDKs can read the DICOM tags/data pretty easily.

It’s been a year since I was making the DICOM loader, so I don’t remember why, but in the end I went with building a static library + headers instead of a .dll, which still feels like a hacky solution, but was a lot simpler.

I’d recommend ITK, as Imebra has pretty restrictive licensing (doesn’t really matter as long as this is non-commercial, but you never know).

If you run into a wall with the .dll wrapping plugin, I can give you more info on how to do it with the static .lib solution.

Some sample code using ITK to parse :

https://github.com/InsightSoftwareCo…ImageWrite.cxx

I haven’t actually used pydicom myself, I just recently came across a master’s thesis where the guy was doing something very similar to my plugin and loading the data with pydicom. The downside there is that it will only work in-editor, unless you use the 3rd party python-during-runtime plugin (default Epic’s python in UE only works in-editor).

Cheers, .

5

DICOM loading is now live in the repo :slight_smile:
Feel free to give it a shot.

Cheers, .

6

Hi .

I m drag and dropping a .dcm file that contains an image stack of a series of images into the content browser, but unreal fails to recognize this file format. Shouldn’t .dcm files also work with this plugin?

7

Is it a single file that contains a whole stack? Or a stack of .dcm files that together make up the full image?

The reader can only handle the second case as of now.

1 Like
8

Hi ,

I first used photoshop to stack all the .dcm files as a series of rows and coloumns and then exported it as a single .dcm file. Then I tried to drag and drop this single file into unreal’s content browser and it failed to import.

So I tried a new way and tried to drag and drop the .dcm files as they are into the content browser and they also don’t import.

Then I tried a third way of stacking these .dcm files in photoshop as layers on top of each other and export that as a single .dcm file and it still doesnt work.

I’m sorry but I desperately need your help. This may seem trivial for a lot of people, but for 3d artists such as myself its a bit frustrating. Can you please tell me how to make the stack of .dcm files that you mentioned?

Thanks in advance

9

What I can tell you is that my reader can read a folder full of .dcm files, where each file is one slice of the bigger volume.

I have zero experience with photoshop, or creating the DICOM files myself, I always just read data gotten from CT/MRI scanners that use this format. Maybe try searching “convert XXX to DICOM stack”.

Or export your volumes as .mhd, that should work fine.

10

Hi, ,

This plugin is wonderful. This plugin impressed me.
I’m trying CT data downloaded from 3D Slicer.

Rendering results was flipping right and left not same as 3D Slicer. Top and bottom is same as 3D slicer. On you posted photo the liver appears to be on the left side different with normal.
I think this problem is fatal to medical use.
Please check and fix it.

CTACardio_3DSlicer_and_UE4
CTACardio_3DSlicer_and_UE41693×774 167 KB

11

Thanks for the catch.

I sincerely hope nobody is using this plugin for surgery planning (yet) :smiley:

Now that I think about it, I never compared my renders to any industry-standard MHD/DICOM visualizers, could be that this happens for all data or it could be just for some specific MHD/DICOM images that have some tags that I don’t know about.

Does it happen to all data you tried? Does it happen with .mhd or .dcm?
My implementation of the DICOM loader is very experimental now.

Could you share the data? There could be a thousand reason why this happens and I need more details.

EDIT : Seems like you’re right and everything is flipped. I find it hilarious that nobody (me included) noticed until now…

12

Hi , thanks for the reply.

One more question please ? Is it possible to use this plugin reconstruct volumes that are animated in any way?

13

Hi ,

thanks for all your work on this.

I have one question: I wondered if there was a reason you decided to write your own volume renderer rather than writing a plugin that prepared the CT data for Unreal’s own volume renderer?

Thanks,

14

Hi , this looks fantastic. I’d like to use it in an Unreal 5 project, yet the library doesn’t build in UE5. Do you have any plans for upgrading it to fit the latest version of Unreal? Happy to contract you if you’d be willing to work with us.

15

Is there any way I could access the transfer function Alpha channel and modify it in blueprint?

16

Hi,
I am using this plugin as the basis for a final year engineering project. However, when I try to build and compile the code, I get a WaitMutex - FromMsBuild error. Is there maybe any reason as to why, or any possible solution?