Hello,
I’m trying to create a VR experience that uses Quest 3 passthrough, but I need an image tracking system to be used as an anchor (stick like a QR-code to an object in real life so it can be tracked) is there a plugin that can do this functionality?
It does not have to be an image tracking, I just need to track the object. I just assumed that the best way to do so is by using image tracking.
Thanks.
Nope you can’t do that with any engine or any plugin if you want to work only with Quest devices without any accessories and it isn’t about engines at all.
Quest device family doesn’t have camera raw buffer access feature.
Even if you integrate ZXing, OpenCV or TensorFlow, you can’t read camera data thus can’t process it to decode it.
Meta forbids that access because so-called privacy reasons.
Couple days ago I talked with a team from Meta who works on “Quest 3 CAD Parts in MR / CAD Viewer” video and gave them some suggestions about CAD viewing.
After main subject, I said couple words about QR reader API. They said, they could look at it but it was not an official meeting. So, there is no guarantee.
In order to partially solve that, I can suggest 3 options.
I developed crossplatform Intel RealSense wrapper for UE5. If you attach a D435i or D455 to Quest with USB C, you can get RGB data and process it with ZXing in FRunnableThread and you can get decoded QR. But that sensor doesn’t have auto-focus feature. So, it can’t detect QR if they are not big enough. Also it is highly susceptible to lightning conditions and camera angles.
(Btw. distance calculation works good)
Developing a LibUVC wrapper for Unreal Engine 5.
I will work on that in couple months. This will allow to attach normal webcams to Quest 3. I suggest Logitech Brio or Insta360 Link like high end cameras. Also this plugin can open Epson Moverio and Vuzix development possibilities.
Using a phone to read read/detect real world object
//
Or you can use Pico Enterprise or HTC Vive.
Even Apple Vision Pro won’t allow us to access that buffer. (But probably they will give a detection api through ARKit)
First thanks for the detailed reply, I appreciate that.
One note: I feel like I formed my question in the incorrect way, just to be clear I want to track a real life object, the user can see his environment using the passthrough and interact with the objects … in a Mixed reality form more than VR.
nope, you formed and ask your question correctly. There is no need to change words.
VR, AR, MR, Spatial Application and lepciron who wears skirt… they are all the same thing.
You have an Android Application but with different UI/UX designs.
In order to “track, decode, detect” any object in “real” environment, you need two things.
1- Camera buffer (not Unreal’s cameras. I mean HMD’s cameras or RGB sensors. What ever you call)
2- An image processing library which can process that buffer to give a meaning.
Your purpose can be anything. Decoding, tracking, etc.
Library integration is easy. There are lots of sample for that. But if you don’t have camera buffer ? what will you process ? And Quest devices DON’T allow us to access their camera buffer. So, there is nothing to process.
Let’s see your use case.
You have one real life object in real world and one digital twin or equivalent digital data in your scene and they have to be somehow synchronized (movement, informations, etc.). In order to achieve your goal, you have to “detect” that object and this requires camera buffer access. Because you have to process that view to detect your target “real world object”
On the other hand, even if you just try to use USceneCapture2D component to capture your whole MR environment from engine after printing passthrough view to SkySphere and try to process it, you will get only black screen. Because Quest’s passthrough plane is a low level stuff.
thank you so much for the clarification, have a nice day.
Btw. Pico and HTC Vive are standalone devices if your customer’s reason is portability.
I work on simulation projects and one my project was creating an MR machine part catalog. Users could read QR code of machine and they will see CAD file in front of their eyes while they can see real world environment (for example a machine on factory zone)
We have similiar use cases. You wanted to detect an image or object and I want to detect QR code. Both of them are artifacts outside of digital world and both of them require image processing.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.