Alright found the issue, got it from looking at the code used in this repo: heyx3/ExtendedGraphicsProgramming at bc613297401a08fa373d7322255aa93c9e4a17be
Also good read for this information from the same person: Advanced Graphics Programming in Unreal, part 5 | by William Mishra-Manning | Jun, 2025 | Medium
Anyway, so the issue is here
FMaterialPixelParameters MaterialParameters = MakeInitializedMaterialPixelParameters();
FPixelMaterialInputs PixelMaterialInputs;
// This is the call to the material graph
CalcMaterialParameters(MaterialParameters, PixelMaterialInputs, SvPosition, true);
You need to set the UV coordinates, which get set with the interpolator shenenigans but I did it this way
FMaterialPixelParameters MaterialParameters = MakeInitializedMaterialPixelParameters();
MaterialParameters.TexCoords[0] = SvPositionToViewportUV(SvPosition);
FPixelMaterialInputs PixelMaterialInputs;
// This is the call to the material graph
CalcMaterialParameters(MaterialParameters, PixelMaterialInputs, SvPosition, true);