Sphere collision produces inconsistent results based on input

I’m trying to replicate Cities:Skylines camera controls and behavior. Here is my actor setup:

	RootSphere = CreateDefaultSubobject<USphereComponent>(TEXT("Root"));
	RootComponent = RootSphere;

	RootSphere->InitSphereRadius(50);
	RootSphere->SetCollisionProfileName(TEXT("CameraPawn"));
	RootSphere->SetEnableGravity(false);
	RootSphere->SetLinearDamping(0);
	RootSphere->SetAngularDamping(0);
	RootSphere->BodyInstance.bLockXRotation = true;
	RootSphere->BodyInstance.bLockYRotation = true;
	RootSphere->BodyInstance.bLockZRotation = true;
	RootSphere->SetSimulatePhysics(true);

	// Other components

“CameraPawn” profile was made by me and it only blocks WorldStatic and does nothing else. Then I wrote the following code inside Tick:

		FVector Movement = FVector(MovementInput /* FVector2D */, 0) * DeltaTime * Speed;
		Movement = CurrentCameraAngle.RotateVector(Movement);

		AddActorWorldOffset(Movement);

This properly prevented the actor from going through simple flat landscape. However, in C:S you can also zoom using the mouse wheel and I tried to replicate the same thing:

		const float Input = ZoomingInput * ZoomMultipler;
		const FVector Direction = CurrentCameraAngle.Vector();

		AddActorWorldOffset(Direction * Input);

This works as expected with exception of that it passes though the landscape. I’ve tried many things - enabling sweeping, ccd, collision thickness on landscape, batching AddActorWorldOffset calls together - nothing helps. The physics engine seems to completely ignore second approach to modifying position.

Also the steps aren’t even big enough to just “jump” through the landscape - it is possible to stop at a point where the sphere is half above, half below/inside the landscape and the engine still doesn’t attempt to force it away. Also this problem appears only when moving directly down - “zooming” at angle properly triggers blocking collision. Also sometimes colliding with landscape by zooming causes the actor to endlessly fly upwards…

I’m completely lost - any suggestions are welcome

P.S. I’ve omitted most of actor code for sake of question brevity but I’m sure that it works correctly. I can post it if it’s needed