Has anyone tried to simulate computer vision with AI agent in Unreal?


I am looking for approaches that would allow an NPC to recognize 3d objects in a virtual environment. I doing a final paper for software engineering specialization and would like to try an approach that uses computer vision inside a game environment. Do anyone know of something similar to this?

Like have a terrain with primitive 3d objects and an agent for npc ai that should behave differently for each kind of primitive shape, it should recognize the primitive shapes like cube, sphere, torus, pyramid.

It would be very easy to set something like that up, but if you want it to truly recognize the shape and not just be told what it is then that’s much more complicated. For an easy method you can tell the AI what object it is based on a number of different things, proximity, tracing vision, etc. But that’s not the same thing as recognizing what they are.

There are a few ways you can achieve something like this, depending on what you want to achieve.

If you’re going for a proper human-like vision technique then your best bet would be to render the objects to texture from the AI’s point of view and perform an image recognition algorithm on it. Something like a SIFT or SURF algorithm (though I think those are patented, so check the licenses before you implement them).

To make it easier, you can render the objects against a black background so you don’t have to deal with background noise.

Darthviper107: Yes I intend to recognize the shape. I am initially thinking in using an geometric hash table. The main challenge is to recognize the same object considering different rotations, sizes and positions. Considering an individual object preserves its proportions.

Thanks for the suggestions, I will work on this during this weekend and get back to this thread with the idea refined :slight_smile:

Neoptolemus: I believe I will follow your tip to start against a black background and then I will iterate to check how far I can get once it works. I will also take a look on the Sift and sure algorithms.

The challenge I want to work on is recognizing pre registered primitive objects. These objects will be registered in a geometric hash structure. So that the algorithm should be able to identify these objects in an environment with a set of variations like size, and axis, also being able to register new objects using a geometric hashing algorithm.

I appreciated your support. Thanks!