Hello, myself and a few others are developing a developer toolset to be used for video and recording. We are using OpenCV to accomplish a variety of tasks. The issue which is giving us the most trouble is the sharing of the video data.
Context:
We are targeting both Unity and UE4, and we have implemented OpenCV via a dll which can be used with both engines, with some engine specific implementations here or there. Each engine has its own solution for capturing a live recording. Our current solution to utilize openCV is to save a frame at some point(s) to disk, then provide the path to the image to generate an OpenCV mat to be used accordingly.
Problem:
Writing to disk is slow and it prevents us from doing things in realtime with CV such as pose tracking. Since the engine owns the connection to the video hardware, we cannot open the video directly from openCV. So we want to convert the video frame data to an InputBuffer type In memory instead of disk for faster streaming to openCV.
What we’ve tried:
we tried using an intermediary struct to carry the pixel data, which technically worked. However the process of converting pixels to our struct, then converting again to an InputBuffer is even slower than writing to disk (painfully so) and is not viable, especially at 4k or 60fps video.
I’ve spent much time on google trying to find or craft a solution, however I am at a loss. Any advice how to best accomplish video processing with both OpenCV and UE4 would be very appreciated.