I am exploring an idea of mapping mass data in UE4. I was wondering what the best way of doing something like this would be.
For example, if I have a data-set of several million points (xyz/rgb/scale) representing some specific data, and I want to map them in space - what would be an efficient way of doing this?
Can individual GPU particles be spawned at precise locations? Or is spawning a bunch of camera facing triangles be just as efficient?
Start with Blueprints. I spent two weeks trying to do this in C++ and got nothing except headaches and crashes. Check out some of the procedural tutorials for the basics. The first 20 minutes of this showed me the way. https://www.youtube.com/watch?v=mI7eYXMJ5eI
This might not be entirely UE4’s fault, I’ve been working with C++ for less than a year, and dealing with new libraries is a big weakness. Still, it’s clear that UE4 has been designed to support Blueprints first-and-foremost. Manipulating an object took most of a week to get working well, but it wasn’t too difficult. Doing that and more (object spawning and manipulation) with Blueprints took a few hours, and was quite easy.
I still haven’t found a great solution for spawning lots of objects with different color materials, all defined at runtime - but Blueprints seem to be the way to go there, too.
I tried unsuccessfully to find Particle creation and manipulation through code (which is super easy in Unity), though by the time I switched to Blueprints, I had set that tactic aside, so I’m not sure if that’s easier to accomplish with BP. Depending on how your several million points break down (e.g. is it several million total, broken down into XYZ, RGB, Scale.XYZ or is it several million points, each with their own XYZ, RGB, and scale) it might be too much for UE4 to handle - unless you’ve got a really powerful machine. Particles would almost certainly be more efficient. If you just want tris, I’d look into Panda3D - which is an ordeal in itself, but has significantly less overhead for simply rendering several million of them. To be sure, I’m not promising it will be easier than UE4, but it will almost certainly be more efficient for doing the same thing.
This also might help for the UE4 route. Several million data points might be a little much for excel - creating a csv file automatically from your data should be less of a problem. Whether or not your machine can run the results well, that’s something else.
You could try with GPGPU, if I’m not mistaken there should be a plug-in that integrates OpenCL into UE4, I never tried it but it could be worth looking into it. The basic idea is to use OpenCL buffers to manipulate your points, those buffers reside on the video card memory and can be shared with the graphics API. Problem is, UE4 hides graphics API so I have no idea how difficult can be to make them see each other.
On the rendering side, particles are quite expensive, when you have millions of points overlapping you will end up with terrible overdraw and fill-rate issues. For you problem it might be better using something like a Sparse Voxel Octree and do ray-casting for rendering. But I am shooting in the dark here, could you explain more what you need to do?
p.s. You could give a shot to a custom procedural mesh creation, basically you convert your point-cloud into some kind of vertex representation and end up with a single actor with a huge geometric content. I am not aware is UE4 can clip, at least partially, the geometry outside the camera frustum to speed up the rendering process.