We are working on a Research Project to replicate a real world environment to a virtual world using reality capture for VR simulations.
We have created a 3D model from photogrammetric data of one of our rooms in Oakland University. We are planning to integrate Lidar Scans to the photogrammetric model.
We captured some data using Unitree Go2 Robot.
Specifically we used the 4D Lidar L1 present on the Unitree.
The data from the Unitree robot is collected in a ROS2 Bag in the format “Filename.db3”.
We have written a python script that would extract x,y,z and intensity data from the ROSBag and save it to CSV.
We have used the cloud compare tool to convert the CSV data to E57. Later, we also used Faro Scene to convert the E57 Files to Ordered Point clouds.
We have around 42 scans of the rooms with each scan having over 1 Million points.
When we imported those E57 files in Reality Capture, none of them were being imported onto the tool as it was showing an error. The error in the console was being shown that the imported files were not scans.
To test a different approach, we have Exported Unordered scans from the Faro Scene tool and imported onto Unreal Engine. Surprisingly, all of the unordered scans were imported without any error. Approximately 142 LSP’s were generated by the RC. But, after the alignment phase, only 6 or 7 rigs were being aligned and the model generated is poor.
To create the photogrammetry model, we have followed the guidelines provided by Reality Capture in several youtube videos. The Photogrammetry model we have is captured using a DSLR, Canon EOS10 camera.
Settings: Aperture: f/9, ISO< 1000 ( between 800-1000), Shutter Speed - 1/10 sec. This is an indoor environment. The files are imported in CR3 Raw format.
Query 1: We have not found any Conference paper or research paper that uses ROSBag to collect Lidar Data and then use that in Reality Capture. We are trying to automate the Data collection using that Robot. The files are saved in DB3 format. How can we properly convert the DB3 format data to E57 files so that they can be Imported correctly onto reality capture and properly aligned?
Query 2: The lidar we are using would give us the data in uncolored format. Does this affect the Model generation?
Query 3: Do we just import the E57 files along with the photogrammetry model and Align them or is it best to generate the Lidar model and Photogrammetry model individually and then integrate them?
Query 4: As Reality Capture cannot detect Plain surfaces, textureless surfaces and reflective areas, we are trying to overcome that by collecting Lidar Data and integrating them with the photogrammetry model. Would lidar be able to solve those issues or is there any other method?
Query 5: Do we need to do the registration of the scans in any other tool and then import them to RC or should we let Reality Capture do the registration? We are tying to automate the entire pipeline from capturing data to generate the 3D model. Considering this, what do you suggest is best in this scenario?
I’d like to upload some attachments, but the forum is allowing me to upload.(new user)