Building a digital twin of a robot

Hey Guys,

Starting this discusion to get input, feedback or tips for my digital twin research project. My goal is to use unreal engine for engineering simulation purposes. These simulations will help to make FMEA studies and design optimasations of production lines and factories easier and faster to iterate. These simulations would then preferably be used as HMI for daily operations.

I am a freelance engineer and all to often I see projects were there is miscommunication due to lack of visualisation of the problem. Gaming engines offer tremendous opportunities to help better connect with different departments to build production lines and machines. After some reading I came up with a plan to test out and help customers build better machines faster and cheaper.

The plan for now:

  • Create basic CAD objects using Shapr3D(Siemens parasolid powered tool, intuitive and good for fast designing)
  • Import the CAD into unreal engine (Datasmith import tool)
  • Setup blueprints which allow PLC communication that will control machine movement in unreal engine (this will be the hard part as I believe custom plugin will be needed)

For the first phase of the plan I will model a KUKA IONTEC, a 6 axis robot.

Here is a shot of the cast iron baseplate I modeled:

Next step is to import the machined part using datasmith and see what happens.

I will post my progress in this discussion.

Kind regards,

Karel

Finished the simplified base pivot cast, next up is the import to unreal and start assembling baseplate.

I am looking for good tutorials on best practice to build the robot blueprint, if you know some please send me a link.

Kind regards,

Karel

For a robot arm I would focus on implementing two joint ik systems for segments.
It also depends if the plc controller would pass in specific motor rotations,or if would it just be start / stop / rotation direction signals.

You also need to identify the protocol of the PLC (does it use CIP).
Does it communicate via rs232 / com / lan etc

Then build a plugin to bridge with the protocol (probably need a handshake to establish connection) and then translate the information to unreal.

1 Like

Hi 3dRaven,

Thanks for your reply, do you have some starting material I can watch to learn more about this? There is a lot of info out there and it is a bit overwhelming.

As for the PLC control, the idea is to use IOT devices that send data packages with specified output. The IOT device would be on the machine capturing the output data. This way I avoid linking protocols to unreal engine directly.

Azure cloud would be my preferred way to link different devices to different CAD as production lines often have multiple suppliers.

When tackling physical interfaces (like mbus or direct engine controls) I usually go by documentation provided by the parts manufacturer (haven’t done PLC’s yet but other hardware interfaces)
I’m guessing there will be more inbuilt watchdog’s that need to be handled so accidents don’t occur on assembly lines.

I’d see what standards Kuka uses. Seems they offer clients a HMI: KUKA.HMI zenon DEV

but I’m not sure if it’s isolated to a flashy user interface or if it has an api behind it.

You could try contacting them directly to see if they have an SDK on request.

It might also be behind the download page but it requires registration
https://www.kuka.com/en-us/services/downloads?terms=Language:en:1&q=

Jup, it always depends on the manufacturer of the machine/robot hence i’m not keen on linking it directly into unreal. I’m not after direct engine controls, these are handled through KUKA API.

At one of my clients the KUKA is integrated into a main program (PLC) through API. My goal for this project would be to just extract and send preprogrammed positions and would prefer to do this via the cloud as this will allow the application to be mobile. It is also easy to simulate and no need for an actual robot to test the unreal setup.

I have been searching around the forum and youtube to find out how best to tackle this and understood I need skeletal mesh. Found out I need a new step between building the CAD and importing to Unreal. Seems like blender software will be a necessary step. Complete noob here… :joy:

Kind regards,

Karel

Hi Guys,

Just a small update on the project before the Christmas holidays start. As I discovered I will need to make a blender step to create the FBX file to import skeleton mesh for the robot.

I completed the simplified model for the axis 2-3 arm:

I already tried some assembling in blender to get to know the tool.

Next steps will be to complete the simplified robot model and create the FBX file with skeleton to start animating in unreal.

Continued my research on how to create IOT link through azure and guessing this will be the hard part. If any of you have experience or have some pointers feel free to drop your insights here.

Enjoy the holiday!

Much Love,

Karel

Hope you guys enjoyed the holidays, I sure did!

Getting into unreal engine and seeing how a project can get big, fast, I decided to import a robot model with skeletal mesh from Sketchfab:

KUKA Robotic arm

This model allows me to skip the steps of creating skeletal mesh for now as this is not my target for the project.

Currently trying out stuff with the control rig and got my first animation out of it! Quite proud about it although it’s simple stuff.

KUKA Robot animation

I will keep trying things out with the control rig.

Hi Guys,

Another small update, finished the basic animations and decided to make the scene more factory minded.

The blue wire running down the robot still needs setup/fixing. The tutorials on control rig mention some bone IK structures I could use to animate the wire.

small animation video here below:

Robot animation test 2

more to come,

Cheers!

1 Like

Hi Guys,

Just another small update after some more playing around in the control rig setup.

Below you can find a target that sets the base rotation based of the position of the target.

Initial position:

New target:

The angle of the rotation is calculated in the blueprint to set the bone:

If you have an other method of getting this done, please share.
Was pretty hard getting here and learned much about the engine in the process :sweat_smile: :pray:

will keep working on the robot, next up is the actual IK arm!

Cheers,

K

Hi Guys,

Back with a small update, have managed to understand the transform system better and got the base transform working. Made a small video of the result:

Check the video!

Now I got the hang of the transform system, I will complete the arm and post the progress.

Kind regards,

Karel