It is quite possible to do what you are asking. There is a documentation on the subject in
UnrealEngine/Engine/Documentation/Source/Programming/Rendering/MaterialExpressions/
Functionality of the nodes is implemented in HLSLMaterialTranslator
Alternatively, you could make a function in MaterialTemplate.usf and call it from the custom node.
There is absolutely nothing wrong with custom node, apart from being unable to return something larger than float4. ( be sure to vote here), so I have no idea what part of it is causing you so much pain.