materials are both pixel/fragment and vertex shaders combined, In order for all engine effect to work properly UE4 has centered shader code and materials are extending this code. Extra HLSL code is generate from nodes you placing. You can manipulate vertex shader by inputing “World Position Offset” which will manipulate position of vertices.
Now i don’t really understand what you want to do, for me it sounds like you want to apply effect to what render to texture captured, you can aplly that in material that add the texture and based on texture data you can add effects, alternatively you can add post processing to capture camera, but then you can only manipulate pixels. Here is how you do post processing material:
If you mean oyu want to write custom HLSL you need to search how to write usf files:
I never dives in to this so not sure what it’s limitations are definitly is less limited then materials themselves, but you still most likely be forced to use material to apply you shaders in usf. You can also apply HLSL code in Custom node:
You should use usf as a last resort, the material system guaranties that everything will work right, so i think best option for you is first option to apply effect with material that uses captured texture aspecially that you want to play with vertices too.