I already have a lot of experience in photogrammetry and 3d rendering. A client asked me to transform a moving subject into an animated 3d model, to be imported into a virtual environment. I am already equipped with greenscreen, cameras and camcorders. I have no idea how to detect the moving subject in 3d.
Hi Marco,
what type of object do you need to model?
In this case you need to capture each step of moving and model each this step. Then you can merge these models into one movable model. Like it is described in this video: https://www.youtube.com/watch?v=ZWWYwEJ_UeA
I can add other details.
Client wish to transform a singer into an animated 3d model, to be imported into a virtual environment where the fans with virtual devices like Oculus, can go near the artist while singing.
So in this case you will need to have a camera rig (something like this: https://www.capturingreality.com/FullBody-Scanning-with-Backface). Then you need to capture him during movement/singing. As for movie, you will need 24 fps and create a model from each frame. Then you need to put these models to one movable alembic model which I suppose can be used in virtual environment.
I’ve found a simplier and accurate way that is motion capture + full-body 3d scan (artec scanner or videocamera 4k) . Model rigging with mocap data in unity/unreal and then export to virtual environment.
Now the point is about exporting the right model to unity/Unreal, with right texture…any workflow/advise for that?
Here is our simplified workflow that will not produce any texture artifacts assuming you have a precise alignment (which can be an issue from what I can see in the screenshots).
(Optional)Postprocess your images in your favorite SW like (Lightroom, Capture One, etc) and do not use any lens profiles (we need images with lens distortion on - distorted images)
Import these inputs RealityCapture
Scale the scene with the distance constraint tool or Import GCPs, FlightLogs
Align images and troubleshoot alignment until it gives it most accurate results (with the Ray of Sight and Inspection tool)
Create a reconstruction region
Use normal/ high detail reconstruction (create a mesh)
Filter out large triangles, unimportant parts with the mesh selections and filtering selection tool
Use the smoothing tool to smooth out the noisy parts if there are any (using mesh selections)
Use the Clean model tool to automatically repair any topology defects
Optionally: Use the lasso selection to only simplify parts that require it, do not simplify the whole model, otherwise, you can lose a lot of detail (this step depends on the type of dataset)
Optionally: Use the simplification tool in 50% increments to get to a suitable triangle count without losing any detail and name the model as a high-poly model. After each step, verify if you are not losing detail or sharp edges.
NOTE: don´t forget to save the project after a large calculation to not lose any work
Create a UV unwrap with fixed texel size setting by selecting the high-poly model (cleaned one) and creating the UV unwrap texture (optionally, you can fill in the unwrap textures with the checkerboard to verify the UV unwrap layout in 2D and 3DS and search for any UV related issues)
Then Simplify the model to create a low-poly version (without reprojecting textures setting enabled because this would reproject texture for every low-poly iteration). When satisfied with the low-poly model quality.
Process of simplification:
Use the lasso selection to only simplify parts that require it, do not simplify the whole model, otherwise, you can lose a lot of detail (this step depends on the type of dataset)
Use the simplification tool in 50% increments to get to a suitable triangle count without losing any detail and name the model as a high-poly model. After each step, verify if you are not losing detail or sharp edges.
Filter out large triangles, unimportant parts with the mesh selections and filtering selection tool
Use the smoothing tool to smooth out the noisy parts if there are any (using mesh selections)
Use the Clean model tool to automatically repair any topology defects
NOTE: don´t forget to save the project after a large calculation to not lose any work
Create a UV unwrap with fixed texel size setting by selecting the low-poly model and creating the UV unwrap texture (optionally, you can fill in the unwrap textures with the checkerboard to verify the UV unwrap layout and search for any UV issues)
Then click on the Texture reprojection tool in the Reconstruction tab and select high poly as a source model and low-poly as a target model. Enable normal map generation and click on reproject (change the reprojection distance or sampling if you will see any issues)
Then just export the model with these settings:
Note: Also, by selecting Unreal Engine transformation preset model will be automatically scaled and rotated to be correctly displayed in Unreal Engine
After importing the model to Unreal Engine don´t forget to check two of these settings to read the 16-bit normal texture map correctly in Unreal Engine.
Also don´t forget to check all the texture materials if diffuse maps and normal maps are correctly connected to the master material (and optionally you can tweak roughness, specular, and AO channels too by creating a more complex material)
This should give a good understanding of how it works and hope this was helpful to you!