I’m working on a project to package commonly used OpenCV classes, and functions into UObjects that can be used in, and extended with Unreal Engine 4’s blueprint system. This is my first real project in either Unreal Engine, or C++ but I think I’ve come a long way in the past few days. You can look over my work at my github: OpenCVProject · GitHub
I’d appreciate any feedback, and I hope to distribute this in the future as a plugin. As current, the only drawback is that you will have to manually copy the OpenCV dll’s into your project’s binary folder for it to run. Additionally, the OpenCV binaries are not included on my git, you will have to download thoes yourself (OpenCV_310.dll,lib and OpenCV_310d.dll,lib) In future I would like to harness the unreal build system to actually build the OpenCV dll’s for your project, but that is all very pie in the sky.
As current, the demonstration content is essentially a reimplementation of the OpenCV integration into Unreal tutorial found Here, however this time it is primarily implemented in blueprint where possible, and the C++ is attempted to be as minimal to bridge between Unreal Blueprints and OpenCV as possible.
I’ve updated the public Git recently, because I have a small issue. The problem is in the CVTexture Object. The CVTexture object, which doesn’t really wrap anything OpenCV related, is an implementation of a dynamic texture whose texture data can be updated with the contents of a cv::Mat. The UpdateTextureRegions function (which is essentially copied from UTexture2D) crashes inconsistently in some cases. I imagine it’s because of the asynchronous behavior of the call. It might be because there was enough time between it’s executions for the then following operation of the Dynamic Material’s Texture being updated to reference the texture data potentially mid update? I’m not entirely sure what the crash IS but the crash report says that it happens mid way through the operations of the asynchronous function while executing bytecode. I have addressed it so far by switching the order of operations such that the Dynamic Material Texture variable is updated before CVTexture::UpdateTextureRegions is called giving it the time between frames to resolve. This seems to have addressed the problem.
Still I am unhappy that I’ve created nodes that crash Unreal. What can I do to avoid this crash state or is this why UpdateTextureRegions is excluded from access in the first place?
Did you make sure to allocate the parameters for UpdateTextureRegions correctly? It’s a bit misleading, as it sort of looks like it will just take any old values, but really wants you to allocate a couple of them before passing because it tries to free them, I think.
That said it’s probably not that, because it would likely crash on any call if that were the problem.
All the parameters are being allocated correctly as far as I can tell, as it does work sometimes. The UpdateTextureRegion function does not free anything unless the last argument is ‘true’. The problem I experience appears to be decided per session. Meaning that when I boot up unreal editor, and test the level for the first time, if it works, it will continue to work each time I test it. If I start a new session, it’s a toss up if it’ll work or not, and I don’t understand why.
For example, Attached is a report for a crash which I experienced this morning after restarting the editor, moments after the scene was working properly. Following the crash, I started the project again, and it works now. No modifications took place.
I have updated the Git. I added a tonne of sanity checks, and made the project more closely resemble the tutorial code, which has routed out the crashing issue. It also appears that the problem persists (in that the cv::VideoCapture.read() call does not succeed) if I relaunch unreal before a prior session has fully exited out. There is about a 10 to 30 second window where Unreal is still running doing cleanup. If you start a new session during that time, something doesn’t get loaded, and causes my project to crash unreal. It may be the case that the previous sessions static reference to UCVVideoCapture is holding onto the webcam, and so the new session’s reference to UCVVideoCapture cannot have access to the camera… but I don’t see how my code is doing that. Since the cv::VideoCapture constructor calls cv::VideoCapture.release(), and is called by the UCVVideoCapture destructor. And the UCVVideoCapture destructor must be called by the instance of UCVVideoCapture used in gameplay.
Anyway, I guess that’s “solved.” Now in the worst case it just does nothing, but doesn’t crash. In this case there doesn’t seem to be much I can do from inside unreal to ensure that the cv::VideoCapture can read the specified device. Still a silent fail is better than a catastrophic fail.
Project resurrection! Looking through my github stuff it seems like this was the most popular so I’m going back into it.
I’ve revamped this quite a bit from last time using what I’ve learned so far about how unreal interacts with your C++ code, and how it handles objects. As a result, I’ve moved to an implementation where each OpenCV object is put into a struct, and accompanied by Blueprint Libraries which do the nodes, this way I’m leaving the object creation and destruction up to Unreal to handle, and the results are much cleaner Blueprints, and less instability. I’ve reimplemented what I had done earlier in blueprints using the new blueprint library and so far so good. However presently the implementation does not render the captured cv::mat to a texture. The issue appears to be how the texture was being updated, calling the render function from within blueprint appears to be unstable, so I’m trying to find a more acceptable means to get a dynamic texture rendered. At the moment, I’m thinking about making an actor component which will take care of creating and updating a transient texture with data placed into it’s buffer, but I am unsure if this will circumvent the issue or not. Any input would be greatly appreciated.
Hey, this is really interesting. Can you explain a bit more about the potential of this plugin. I saw the original wiki at an earlier point but got a bit stuck with it. Do you think I’d be able to capture from a blackmagic device? I’m really interested in going the other way too - I’d like to be able to send the render window to a blackmagic card for broadcast quality output, do you think this is a likely outcome? I’ll download your plugin when I can and give it a whirl. Thanks!
Right now, all that is supported is reading from a camera or other video input device into a matrix. I’m working on making a safe visualization of a matrix content which was the cause of trouble earlier. Depending on how much more development occurs with this project you will be able to do things that OpenCV itself can do. OpenCV employs graphic rendering libraries to visualize processes and algorithms taking place over visual input, but for the most part OpenCV is primarily about reading and analyzing video input. It’s unlikely this will help you pipe unreal game play video to a capture device. However this could potentially do OpenCV fueled analysis over video content if it can be rendered to a mat. I’m working on a way to get a mat into a texture, it shouldn’t be too difficult once that is done to get a texture into a matrix.
