Announcement

Collapse
No announcement yet.

TBRaymarchPlugin - Volume Rendering (Raymarching) Plugin for Unreal Engine

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    [PLUGIN] TBRaymarchPlugin - Volume Rendering (Raymarching) Plugin for Unreal Engine

    Hey guys, I'd like to share a project that I've been working on the last couple of months, it's a massive improvement over my previous raymarching plugin that can be found here.

    The whole thing can be found on my github and works out of the box (even contains the scan that you can see in the screenshots).
    https://github.com/tommybazar/TBRaymarchProject

    Long story short - it lets you render volumetric data (mostly used for medicinal visualization, but hey, whatever data you got, we will render).

    Some screenshots to quickly grab your attention:
    On the left you can see the volumetric rendering and on the right is a cut through the original data volume.

    Render 1




    Render 2




    Both of these are rendered from the same CT scan, just using a different transfer function.


    Features
    • Works out of the box with binary UE 4.25
    • Volume raymarching for arbitrary UVolumeTexture textures (true 3D textures, no flipbooks).
    • .mhd and .raw file import into volume textures.
    • Precomputed Volume illumination with unlimited number of light sources implemented in compute shaders using concepts from Efficient Volume Illumination with Multiple Light Sources through Selective Light Updates (2015) by Sundén and Ropinski
    • Subclassed Unreal's UVolumeTexture to allow access to the resource as a UAV in compute shaders.
    • Uses color curves for transfer function definition
    • Fully integrated and functional within UE editor viewport.
    • Windowing support (google DICOM Window center / Window Width or watch my youtube description for an explanation of what windowing does)
    • Basic menus for manipulating the volume
    • Basic VR support and example map (coming very soon - mid-November 2020).
    Limitations
    • Raymarched volume doesn't cast or receive shadows info to/from the scene, it only self-shadows
    • We use a very simple (but fast) raymarching and illumination algorithm with no specular, refraction or scattering
    • Algorithm is already a bit dated and implementation leaves a lot to be desired as for efficiency. It is however, good enough for real-time applications with several lights and large (512^3 or more) volumes.
    • Currently do not support persistent 32bit grayscale textures, but plan on investigating that possibility soon(ish).
    • My NDA with my current employer doesn't allow me to publish the DICOM loading part (as I wrote that code under contract), which would truly make this a one-stop-shop for medical visualization in Unreal.

    If you want to give it a shot, just clone the repo, generate VS files, compile and play with it in the editor or in play mode. Everything should work out of the box, there is an extensive readme on my github for usage details and I went crazy with commenting my code, so everything should be pretty clear.

    For a showcase video where I show off all the things you can do with this, here's my 40 minute rant about it:




    Any suggestions, criticisms, bug-reports, pull-requests etc. are more than welcome!

    Cheers, Tommy.
    Last edited by tommybazar; 11-12-2020, 12:09 AM.


    Plugin for Volumetric Rendering of medical data in UE

    #2
    Hi Tommy!

    Amazing work.

    I'm trying to use this to generate 3d model from an MRI. The data is in DICOM format. I read your comments on the subject on your Github repo. I want to write the missing code in order to be able to read DICOM. What would be faster, convert DICOM to MHD and then use your plugin as is, or write the DICOM reader right into the Plugin code?

    I have downloaded pydicom and been playing around, some tips would be great !!!

    Thanks

    Alejandro

    Comment


      #3
      Originally posted by A.Boccardi View Post
      I'm trying to use this to generate 3d model from an MRI. The data is in DICOM format. I read your comments on the subject on your Github repo. I want to write the missing code in order to be able to read DICOM. What would be faster, convert DICOM to MHD and then use your plugin as is, or write the DICOM reader right into the Plugin code?
      Heya Alejandro, glad to see that this might be useful to people

      I should actually refactor the plugin, so that it's not tied to MHD as much, it's gonna be a simple inheritance structure, so you can just extend the asset for loading any kind of medical data.

      But for now, if you can get the info that is needed in the MHDAsset from the DICOM series, create a new MHDAsset and just set the members with whatever you read from the DICOm, it will work just fine.

      I think the cleanest solution for the DICOM loading is making a standalone ITK plugin (just a plugin wrapping a built ITK library with some convenience functions) and then using that to load the tags and binary data from my plugins. This would have the added benefit of not marrying the DICOM loading part to the rest of the raymarcher plugin, so people could use that for other stuff.

      The only main challenge I had when reading the DICOM series myself straight in C++ was getting the 3rd party library built as .dll or .lib packaged into a plugin and forcing UE to load it and be usable from other plugins, I remember that being a pain.

      Once I had that, the parser itself wasn't too hard, as ITK or Imebra SDKs can read the DICOM tags/data pretty easily.

      It's been a year since I was making the DICOM loader, so I don't remember why, but in the end I went with building a static library + headers instead of a .dll, which still feels like a hacky solution, but was a lot simpler.

      I'd recommend ITK, as Imebra has pretty restrictive licensing (doesn't really matter as long as this is non-commercial, but you never know).

      If you run into a wall with the .dll wrapping plugin, I can give you more info on how to do it with the static .lib solution.

      Some sample code using ITK to parse :

      https://github.com/InsightSoftwareCo...ImageWrite.cxx

      Originally posted by A.Boccardi View Post
      I have downloaded pydicom and been playing around, some tips would be great !!!
      I haven't actually used pydicom myself, I just recently came across a master's thesis where the guy was doing something very similar to my plugin and loading the data with pydicom. The downside there is that it will only work in-editor, unless you use the 3rd party python-during-runtime plugin (default Epic's python in UE only works in-editor).

      Cheers, Tommy.
      Last edited by tommybazar; 11-23-2020, 10:52 PM.


      Plugin for Volumetric Rendering of medical data in UE

      Comment

      Working...
      X