ASIS project position to screen based on camera/connect id

Development Platform & Builds
, ,
1

Hello,

Currently when using the ASIS plugin, we ran into an issue where we couldn’t properly get screen projection of items in the world to the 2d coordinates. This was due to the fact that in cases where the camera was not the camera at index 0(player viewport), the rendering context changes, and that info isn’t easily accessible from outside the plugin.

We solved by adding a function within ASIS to project in multiview scenarios only. The function is

bool UAndroidSingleInstanceServiceBPLibrary::ProjectAsisExtraViewWorldToScreen(
UCameraComponent* Camera,
const FVector& WorldPosition,
FVector2D& OutScreenPosition)
{
OutScreenPosition = FVector2D::ZeroVector;

#ifdef WITH_MULTIWINDOW
using namespace UE::ASIS;
using namespace UE::MultiWindow;

if (Camera == nullptr) return false;

// 1. Find FClientId for this camera by scanning RegisteredClients
const FClientId* FoundClientId = nullptr;
for (const TPair<FClientId, TWeakObjectPtr<UObject>>& Pair : RegisteredClients)
{
if (Pair.Value.IsValid() && Pair.Value.Get() == Camera)
{
FoundClientId = &Pair.Key;
break;
}
}
if (!FoundClientId) return false;

// 2. Look up the most recent FAttachId for that client
const FAttachId* FoundAttachId = ClientIdToLastAttachedId.Find(*FoundClientId);
if (!FoundAttachId) return false;

// 3. Look up the bound view handle
TVariant<FViewportClientRegistration, TUniquePtr<FViewportClientBinding>>* ViewHandle =
BoundViewHandles.Find(*FoundAttachId);
if (!ViewHandle || !ViewHandle->IsType<FViewportClientRegistration>()) return false;

// 4. Ask the registration for its projection data (same source the renderer uses)
FSceneViewProjectionData ProjectionData;
ViewHandle->Get<FViewportClientRegistration>().GetProjectionData(ProjectionData);

const FMatrix VP = ProjectionData.ComputeViewProjectionMatrix();
const FIntRect Rect = ProjectionData.GetConstrainedViewRect();

// 5. Project
const FVector4 Clip = VP.TransformFVector4(FVector4(WorldPosition, 1.0f));
if (Clip.W <= KINDA_SMALL_NUMBER) return false;

const float InvW = 1.0f / Clip.W;
const float NDCx = Clip.X * InvW;
const float NDCy = Clip.Y * InvW;

OutScreenPosition.X = Rect.Min.X + (NDCx * 0.5f + 0.5f) * static_cast<float>(Rect.Width());
OutScreenPosition.Y = Rect.Min.Y + (1.0f - (NDCy * 0.5f + 0.5f)) * static_cast<float>(Rect.Height());

return true;
#else
return false;
#endif
}

The request has 2 parts:

  1. Is this the proper way to go about solving this project world to 2d coordinates for a multiview camera.
  2. Is it possible to have this functionality baked into the ASIS plugin, expanded to not care about attach index, so just callable on any camera, and it will return the proper projection points.

Thank you!

[Attachment Removed]

2

Hello J-F,

FSceneView already has ProjectWorldToScreen API. Instead of make own calculation, you could try this function.

bool UAndroidSingleInstanceServiceBPLibrary::ProjectAsisExtraViewWorldToScreen(
UCameraComponent* Camera,
const FVector& WorldPosition,
FVector2D& OutScreenPosition) 
{
...
ViewHandle->Get<FViewportClientRegistration>().GetProjectionData(ProjectionData);
return FSceneView::ProjectWorldToScreen(
	WorldPosition,
	ProjectionData.GetConstrainedViewRect(),
	ProjectionData.ComputeViewProjectionMatrix(),
	/*out*/ OutScreenPosition);
#endif
 
return false;
}

Could you please validate it on your side?

If it resolve your request, I will add this API to 5.7 version.

Also will check how we could address this request in UE 5.8 version.

Regards,

Yevgen

[Attachment Removed]

3

Oh wasnt aware of that function, I will give it a try and let you know.

[Attachment Removed]

4

As mentioned in the meeting, this function does work and provides the proper projection. Thank you!

[Attachment Removed]

5

The api was added to ASIS plugin. Aunmar will provide latest pluign

[Attachment Removed]