Proper Way to do Camera-based AR with a Webcam

I’ve been struggling for some time to find the “right” way to do camera-based AR on PC. What I would like to do is take a camera image from a webcam and draw augmentations on it, while also using the image somehow for processing like object tracking etc… A starting point for me would be to replicate the standard Marker Tracking things that use Vuforia etc. to augment a 3D object onto a tracked location in 3D space.

I know I can do this by for example using OpenCV to get and process the camera image. To create the AR view, I would attach a plane to the camera actor that would be always at the far plane of the camera. This would then have a material which would sample the uploaded camera image in a texture.

However, it feels like with all the facilities that must be in the engine for supporting AR modes on Android/iOS, as well as all the stuff for virtual production, there should be something simpler here?

I can’t for the life of me find the right starting point for doing this properly without hacky half-■■■■■ solutions, so does anyone have some suggestions there?

Hi, you’re right, using mobile AR can be a better solution. No need for markers.
A PC webcam is bound to a cable or built in a monitor. A smart phone or tablet can take care of all tracking AND transmit the camera image wirelessly, that’s another plus.

If you want to capture the image from the device, just make an AR project and look for a way to stream the content to a pc.
(it’s easy on ios devices, i am not very familiar with android)