Updating after a while to let everyone who has this problem that this is the solution!
Thank you Ryan, you are my saviour!
I actually never realised that GI bounces could affect shadows that much, well Ambient Occlusion simulates shadows at edges and everyone who knows lighting and rendering good would have known this. Poor stupid me!
Thank you once again, I owe you one!