I suppose the question is how to get UE4 to communicate with MATLAB, correct? Does it need to happen real-time or would it be through some interim file/process on the disk?
I wouldn’t do it through a .exe shell extension as for one it limits it to a single platform (Windows), plus it might need elevated privileges to setup/run.
I did not look into MATLAB API so that’s something you need to check, but with UE4 you can add plugins to communicate with external events/processes or use existing things (perhaps OSC comes to mind, not sure if applicable here).
Maybe if you give us more details, we could come up with a way of integration?