Point Cloud Library in UE4

[Mell_D]

I tried to include PCL into my project. I need some methods to process my point cloud like filtering, smoothing, downsampling and triangulation.
Unfortunately I always get the same error when using some function of PCL. For example at the following filter method:

#include <pcl/filters/radius_outlier_removal.h>
#include <pcl/filters/conditional_removal.h>

void AVoxelGenerator::GeneratePointCloud(TArray<FVector4> Voxels)
{
	 pcl::PointCloud<pcl::PointXYZI>::Ptr Cloud(new pcl::PointCloud<pcl::PointXYZI>());
 	 pcl::PointCloud<pcl::PointXYZI>::Ptr Cloud_filtered(new pcl::PointCloud<pcl::PointXYZI>());

	 Cloud->width = Width;
	 Cloud->height = Height;
 	 Cloud->is_dense = true;		
	 Cloud->points.resize(Voxels.Num());

	  for (int i = 0; i < Voxels.Num(); i++)
	  {
		 Cloud->points[i].x = Voxels[i].X;
		 Cloud->points[i].y = Voxels[i].Y;
		 Cloud->points[i].z = Voxels[i].Z;
 		 Cloud->points[i].intensity = Voxels[i].W;
	 }

	pcl::RadiusOutlierRemoval<pcl::PointXYZI> Ror;
	Ror.setInputCloud(Cloud);
	Ror.setRadiusSearch(20.0);
	Ror.setMinNeighborsInRadius(5);
	Ror.filter(*Cloud_filtered);

	pcl::PCDWriter Writer;
	Writer.write<pcl::PointXYZI>("Table_inliers.pcd", *Cloud_filtered, false);
}

When I debug the method everything works fine (The Writer writes the file and saves the estimated points). But at the end of the method before jumping to the next method I always get an error at MallocTB::Free().
First-chance exception at 0x00007FFB2AB60080 (UE4Editor-Core.dll) in UE4Editor.exe: 0xC0000005: Access violation writing location 0x000000293E868050
Unhandled exception at 0x00007FFB2AB60080 (UE4Editor-Core.dll) in UE4Editor.exe: 0xC0000005: Access violation reading location 0x000000293E868050

I really tried a lot but I don't know why this error occurs... . It seems that there are some problems between memory handling in UE4 and memory handling in the PCL framework? I would be grateful if someone can help me.

Is it possible that there is a problem with using boost::shared_ptr in UE4? "Ptr" is of this type.

[Cyxx]

Hi Mell_D
I setup a small PCL plugin project and tried your code.
Except the Ror filter everything works fine here without exception and the pcd file is written.
From the name "AVoxelGenerator" I guess that you derive from AActor, or?
Is this actor always available? Because all UObjects can be garbage collected:
https://wiki.unrealengine.com/Garbag...ction_Overview
https://wiki.unrealengine.com/Garbag...ory_Allocation

Which version of PCL, Boost and VStudio do you use?
I used UE4.9, VS2013, Boost 1.59 and the actual PCL code from the repository.

Attached the code from my test class:

PCLFilter.h:

#pragma once
#include "Engine.h"
#include "GameFramework/Actor.h"
#include "PCLFilter.generated.h"


UCLASS()
class AVoxelGenerator : public AActor
{
 	GENERATED_BODY()
	
  public:	
 	AVoxelGenerator();

 	void GeneratePointCloud(TArray<FVector4> Voxels);
};

---
PCLFilter.cpp:

#include "PCLPrivatePCH.h"
#include "PCLFilter.h"
#include "AllowWindowsPlatformTypes.h"
#include "pcl/point_types.h"
#include "pcl/filters/radius_outlier_removal.h"
#include "pcl/filters/conditional_removal.h"
#include "pcl/io/pcd_io.h"
#include "HideWindowsPlatformTypes.h"

AVoxelGenerator::AVoxelGenerator()
{
	
}

void AVoxelGenerator::GeneratePointCloud(TArray<FVector4> Voxels)
{
  pcl::PointCloud<pcl::PointXYZI>::Ptr Cloud(new pcl::PointCloud<pcl::PointXYZI>());
  pcl::PointCloud<pcl::PointXYZI>::Ptr Cloud_filtered(new pcl::PointCloud<pcl::PointXYZI>());

  Cloud->width = Voxels.Num();
  Cloud->height = 1;
  Cloud->is_dense = false;		
  Cloud->points.resize(Voxels.Num());

  for (int i = 0; i < Voxels.Num(); i++)
  {
    Cloud->points[i].x = Voxels[i].X;
    Cloud->points[i].y = Voxels[i].Y;
    Cloud->points[i].z = Voxels[i].Z;
    Cloud->points[i].intensity = Voxels[i].W;
  }

  // ...

  pcl::PCDWriter Writer;
  Writer.write<pcl::PointXYZI>("d:/Table_inliers.pcd", *Cloud, false);
  //Writer.write<pcl::PointXYZI>("d:/Table_inliers.pcd", *Cloud_filtered, false);	
}

Regarding your boost::shared_ptr:
Did you try using a reference instead? Like:
pcl::PointCloud<pcl::PointXYZI> Cloud;

Best regards
Heiko

[Mell_D]

Hi Heiko,

thank you very much for your answer and sorry for my late reply.

Yes, my class derives from AActor and yes, this class is always available. I use UE4.9, VS2013, Boost 1.57 and PCL 1.7.2 (I used the all-in-one-installer from http://unanancyowen.com/?p=1255&lang=en for VS2013, Win64).
And yes I tried to use a reference instead of boost::shared_ptr but that didn't work. Which problems did appear with your RoR filter? Is it too slow?

If I comment the filter method out my code works well too. Otherwise the filter method works well but at the end of my function I suddenly get that error. I tried other filters too (e.g Voxelgrid) but it's always the same error.

Best regards
Melli

[Cyxx]

Hi Melli
thanks for your additional infos. I just wanted to let you know that I switched to your Boost and PCL version,
activated the ROR filter again and now I get a similiar exception as you:
Unhandled exception at 0x000007FEE757D983 (UE4Editor-Core.dll) in UE4Editor.exe: 0xC0000005: Access violation writing location 0x0000000000000010.
Debugger stops at the end of FMallocTBB::Free method.
In the stack frame I see that the destructor of class Filter and then the dispose method of Boost class sp_counted_impl_p is called before.
I will now try to debug deeper.

As a quick dirty workaround you can declare Ror as global.
Just move the line outside the function:
pcl::RadiusOutlierRemoval<pcl::PointXYZI> Ror;

Best regards
Heiko

[Mell_D]

Hi Heiko,

yes the error arises after the sp_counted_impl.hpp. I just tried to uncomment the "delete x" in the file boost/core/checked_delete.hpp (... dangerous(!) haha...).
So that works but what does it mean for the problem? Unreal wants to delete the pointer again after Boost does it and that's why we get the access violation error? I can prevent Unreal from doing this only if I declare Ror as a global?
Not so pretty but it works! Thank you very very much! :-) I really spend a lot of time to understand what's going on here...

Best regards
Melli

[Erik Karlsson]

Hi,

I'm new to unreal and trying to use PCL. How did you get it to work? What has to be done before you can write "#include <pcl/..."?

I hope it's ok that I borrowed the thread!


Best regards
Erik

[as3ef2th1]

Hello,

Hi,

I also would like to import points cloud in the engine.
Are you still working on your plugin, to be honest my C++ knowledge is not that good ?

I guess everyone want at least two things : get point cloud in the engine and get the dynamic LOD to work inside the engine (you know, the real time decimation from octrees and camera position).

Best regards,

[qbhust]

Hello,

Hi,

I also would like to import points cloud in the engine.
Are you still working on your plugin, to be honest my C++ knowledge is not that good ?

I guess everyone want at least two things : get point cloud in the engine and get the dynamic LOD to work inside the engine (you know, the real time decimation from octrees and camera position).

Best regards,

Hi Did you find solution ? I am also working on the same issue.

[as3ef2th1]

Hi Did you find solution ? I am also working on the same issue.

Not yet unfortunately. Maybe when I have more time I will look into this seriously...

[plangton]

Is anyone still interested in cpu based pointcloud ? I can make a basic plugin for you guys.
I was testing the possibilities after reading this thread and it is not too hard to create, draw and update points in ue4

[Neoplanetz]

Hi plangton. I'm also interesting using PCL library in UE4. Is it possible to get point cloud data from depth sensor like Kinect v2 and draw and display in UE4 realtime?
If It can be, Can you share your idea or plugin code? thanks you~

[plangton]

Hi plangton. I'm also interesting using PCL library in UE4. Is it possible to get point cloud data from depth sensor like Kinect v2 and draw and display in UE4 realtime?
If It can be, Can you share your idea or plugin code? thanks you~

I have done that with Kinect but for that it's the best if you do it with GPU particles or in materials with worldpositionoffset. the reason I am telling this is that Kinect v2 will send you 30 fps and each frame consists of 512*424 pixel data this is a massive amount of information to be process all in cpu and better to be shared between cpu and gpu.
So, here are my suggestions
1 - have a plane with 512*424 vertices, get each frame as a depth texture, send it to a material that apply the depth value to worldpositionoffset z position and apply that material as a dynamic material to your plane. ( this will set the Z but not the exact x&y for that you need more calculation to get world (x,y,z) from depth texture which is in (u,v,z) basically you will need to calculate x,y using u,v and camera intrinsics

2 - it is mostly like solution one except that you will set everything to a gpu particle. you can adjust it to have exactly 512*424 number of particle and set the position of each separately using the texture. I once saw another thread explaining this solution extensively

[hausfrau87]

hey guys,

this might be helpful: https://answers.unrealengine.com/que...#answer-679343