I have to say this is one of the more interesting use cases for the engine I’ve heard of so far :).
If I’d just think a bit about your first idea, to launch the engine as a subprocess to your own and then using IPC to control the environment, I don’t see any insurmountable obstacles to getting this to work, although you could probably not leverage everything the engine has to offer. For example, any static lighting would probably be hard to make work since that generally requires you to first build your level, light it and then build the lightmaps for the level, which is an extremely expensive operation and not something that you would do during run time. Considering the info you have given me, I’m guessing you guys own some type of visualization software, like architecture visualization or scenario simulation or something like that. If this is indeed the case, this is probably something that you would want to have.
One alternative to this is making all lighting dynamic. If you were to go this route, you would not suffer the light building cost to the same extent. There are however other features, that like the static lighting requires design time processing, like Mesh Distance Fields or content processing etc.
The biggest downside with this is probably an additional performance cost, but I don’t really know to what extent.
As I’m assuming you’re mostly interested in placing static mesh actors with high quality materials and then manipulating a camera around that scene, I get the feeling that if you could solve those two problems, I think the rest might be pretty straightforward. If you want interactivity / AI / triggers and all that other good stuff, there is nothing I know of that cannot be created/controlled from C++. You could easily make everything you need on the UE4 side of the IPC channel and then spawn them on demand.
But then there is the question if this is a good idea, or if there is a better one. I think if I was tasked with solving this problem, I would much rather reverse the relationship.
That is; I would host our application inside the engine somehow. I would design and create all the blueprints, meshes, materials, particle systems and base levels I needed in UE4, and then I would write a small plugin or just project source that would parse my business domain files and populate the scene in accordance with them. If our app had a preexisting GUI, you could still make it work easily by either just sharing the data models through IPC, or by using file watchers. Trying to bypass the content processing and creation features in the engine just seems like a lot of unnecessary work to me
But then again, I know nothing of your system!
I definitely think what you are suggesting is possible, but you might face both logistical, feature related and legal problems to have a chance at making it work decently well.
My recommendation is to invert the relationship, using the full feature set of UE4 in conjunction with your current application through a common data model.
To answer your leading question though; Yes. I think that UE4 is a good fit for any project that aims to have impressive visuals, from prototyping to full products. Since you have the full source that is also fully C/C++ compliant, you can basically make it work with any setup you can imagine.**
I hope you found this useful in some way.