VisionOS - Communication with UESwift.Swift

Development Platform & Builds
, ,
1

I’m digging into bridging SwiftUI and Unreal Engine in order to take advantage of what SwiftUI can bring to an Unreal VisionOS application (in particularly where eye gaze and UX interactions come out of the box which with eye gaze of example, simply isn’t present in Unreal VisionOS implementation).

Is it possible to get some clarity on how this is currently being implemented as UECppToSwift.h appears to be working some magic as if I define a function in that file and have the implementation somewhere else (like in my own custom plugin), this function is automatically exposed so that it can be called from UESwift.Swift quite trivially:

extern “C” void MySwiftCallingUnrealFunction()

I am however struggling to properly understand why this is happening and how I could move this definition to a custom plugin rather than an engine plugin and what may be needed to get this to work (as when I move this definition outside of UECppToSwift.h, the function call immediately crashes the application where SwiftUI can’t call the specially named function - suggesting there is something special with UECppToSwift.h which I don’t properly understand.

I’d also like to understand how to go the other way and call SwiftUI from Uneal C++. It appears that this can be done by creating an objective c .m file that can be used as an intermediary however this file needs to be able to call functions on the SwiftUI file and I don’t appear to be able to generate a bridging header for UESwift.Swift in the build process in order to do this. I’ve managed to get from Unreal C++ -> ObjectiveC .m file - it’s the link between an ObjectiveC file and UESwift.Swift which I can’t seem to get working.

It also appears from some of Apple’s recent developer documentation that it may be possible to skip ObjectiveC bridging entirely and enable some kind of Swift interop with C++ however these settings don’t appear evident in the VisionOS Xcode project that is generated for the Unreal project - which may be because the UESwift.Swift is only part of a larger Xcode project?

https://www.youtube.com/watch?v=MixEUoUQyNU

So in summary:

  • SwiftUI -> Unreal C++ - why does this work with functions defined in UECppToSwift.h and how can I do something similar without having to modify this file - ideally being able to enable this kind of interop from a project C++ plugin.

  • Unreal C++ -> SwiftUI - how can this be achieved given the Xcode project is substantially more complex than most typical code examples online and where the Xcode project is a bit more bespoke

It goes without saying that I’m working fully with a source build for this endeavor.

1 Like
2

Hello, sorry for the delay but I was working on exactly this so I have much better answers than I did a few days ago.

First i’ll disclaim that I am far from a swift expert, but I did manage to get unrealc++ to swift communication working. I’ll paste the github cl when it comes through in a few hours. But first I have any ugly text dump of my knowlege of the two paths as currently implemented:

######################################

SwiftMain.swift to Unreal C++

SwiftMain.swift

KickoffWithCompositingLayer

Launch.Build.cs

SwiftInteropHeader = Path.Combine(ModuleDirectory, “Private/Apple/LaunchCPPToSwift.h”);

// This is where we store the header that exposes cpp to swift. As you can see it is hard coded by name. See the next step.

UEBuildModuleCPP.cs

if (InputFiles.SwiftFiles.Count > 0 && Rules.SwiftInteropHeader != null) …

// This is where we use the SwiftInteropHeader. It is passed into the swift compilation as an argument. I’m not sure if one can pass multiple headers, or if one would include headers into one big header to pass in.

// This is also where we are using the SwiftFiles we found. It looks like we just scoop them up along with cpp files and other source files in UEBuildModuleCPP.cs. I imagine any swift files in directories that cpp files could exists in will be found, but i have not yet tried it.

LaunchCPPToSwift.h

void KickoffWithCompositingLayer(CP_OBJECT_cp_layer_renderer* Layer);

// The header where we define the functions we want to call from swift.

LaunchIOS.cpp

void KickoffWithCompositingLayer(CP_OBJECT_cp_layer_renderer* Layer)

{

…

GSwiftTriggerEvent->Trigger();

}

// This is the definition of one of the functions exposed to swift. I happened to be interested in that event, but there were a few more lines above. Anyway fully in unreal c++ at this point.

######################################

Unreal C++ to SwiftMain.swift

// This is all new code I wrote last week.

LaunchIOS.cpp

include “../Apple/SwiftMainBridge.h”

UE::SwiftMainBridgeNS::ConfigureImmersiveSpace(ImmersiveStyle, UpperLimbVisibility);

// This is where I am calling from unreal c++ to eventually get into swift. My parameters here are simply Int32 values.

SwiftMainBridge.h

// Copyright Epic Games, Inc. All Rights Reserved.

#pragma once

// Bridge functions for communicating from C++ to SwiftMain.swift.

// See LaunchCPPToSwift.h for an example of going the other way, from SwiftMain.swift to C++.

#if PLATFORM_VISIONOS

namespace UE::SwiftMainBridgeNS

{

void ConfigureImmersiveSpace(int32 InImmersiveStyle, int32 InUpperLimbVisibility);

}

#endif // PLATFORM_VISIONOS

// Declaring my bridge functions. This probably isn’t strictly necessary, but I like ‘hiding’ the objective c from the rest of unreal.

SwiftMainBridge.cpp

// Copyright Epic Games, Inc. All Rights Reserved.

include “SwiftMainBridge.h”

#if PLATFORM_VISIONOS

// This import is the autogenerated “bridging header” created by xcode because SwiftMain.swift contains @objc tagged items.

// Look in there if you need to understand the objective c interface better.

import “SwiftMain-Swift.h”

// All our calls come in from unreal threads, so we dispatch_async to get the into Apple’s main thread where SwiftUI updates must happen.

void UE::SwiftMainBridgeNS::ConfigureImmersiveSpace(int32 InImmersiveStyle, int32 InUpperLimbVisibility)

{

dispatch_async(dispatch_get_main_queue(), ^{

[SwiftMainBridge configureImmersiveSpaceInImmersiveStyle :InImmersiveStyle inUpperLimbVisibility:InUpperLimbVisibility];

});

}

#endif // PLATFORM_VISIONOS

// So here we are calling the objective c.

"SwiftMain-Swift.h

Autogenerated objc corresponding to the @objc items from SwiftMain.swift

// This ‘just worked’ for me, as soon as I added an @objc to SwiftMain.swift this header appeared in a build intermediate directory. Perhaps other .swift files would do the same? Might just have to look for a file who’s name fits this pattern (eg ending in -Swift.h).

SwiftMain.swift

// Bridge class for calling into this file from c++

// Data comes in from SwiftMainBridge.cpp and goes out via the swift notification system to SwiftUI elements.

@objc class SwiftMainBridge: NSObject {

[Content removed] inUpperLimbVisibility: Int32) {

NotificationCenter.default.post(name: .configureImmersiveSpace,

object: nil,

userInfo: [“immersiveStyle”: inImmersiveStyle, “upperLimbVisibility”: inUpperLimbVisibility])

}

}

extension Notification.Name {

static let configureImmersiveSpace = Notification.Name(“configureImmersiveSpace”)

}

// This ‘bridging’ class isn’t strictly necessary. I like how I have a nexus on the cpp side and on the swift side that match. Using the notification system and the userInfo dictionary for parameters is presumably not the most efficient path, but it is very loosely coupled. One ought to be able to send information to other swift files which know nothing about this swift file.

SwiftMain.swift (on the SwiftUIView):

.onReceive(NotificationCenter.default.publisher(for: .configureImmersiveSpace)) { notification in

guard let userInfo = notification.userInfo else { return }

if let styleValue = userInfo[“immersiveStyle”] as? Int32 {

// Receiving the notification on my window and starting to use the userInfo dictionary:

3

As context specifically I am reading some config file values in the c++ side and sending them over to swift so that the immersive space is opened in .full or .mixed and with the specified hand visibility). So this is all happening pretty early in engine startup, but in principle it should work the same way later.

The pre-existing path is how we block engine startup until the swift window is dismissed and we launch the immersive space.

4

Re UESwift vs SwiftMain: there was a refactor there changing the file name. I think you are correct.

In theory this swift/cpp setup should be the same on ios as visionOS. I think, by default anyway, we aren’t using it on ios. Only on visionos. I was specifically working on launching the immersive space, so I made my change visionos specific.

Re: outside the engine.

Yes, I’m not exactly sure how to tie the swift files in engine and out of engine together. Would the notificaiton mechanism be enough? Not sure.

5

This is a link to my change in the dev-5.6 branch. It should go into the 5.6 branch some time in the next few days, I cannot predict exactly when. it is also in ue5-main

https://github.com/EpicGames/UnrealEngine/commit/f18882ccfa7333ab7c1c6db1126aa54d38c78512

1 Like
6

There is a fair bit of potential in building on this bridging functionality as it may open up some ways of bridging support of some native RealityKit features that may not readily be exposed through OpenXR

7

Yes, I can definitely see why you are interested in it, given the current restrictions.

1 Like
8

Thanks - this is a really helpful start - could I check if there are any differences calling to/from UESwift.swift which is part of the Vision Pro implementation or is the process pretty much the same for iOS and VisionOS?

Actually - coming back the iOS side of things, does this also mean there would be a way to layer SwiftUI over an Unreal instance running on iOS too?

[EDIT] - Looks like you’ve been refactoring the Vision Pro launch process as UESwift.swift now appears to be SwiftMain.Swift in Engine/Source/Runtime/Launch/Private/Apple

rather than Engine/Platforms/VisionOS/Source/Runtime/Launch/Source/

Would I be correct in assuming that on 5.5 I should be referring to UESwift.Swift and 5.6 onwards it will be SwiftMain.Swift?

Would be amazing to find a way of linking outside of these engine dirs in such a way that a non-engine plugin could contain its own Swift code that could hook back in.

1 Like
9

Thanks Jeff - this is really really helpful - am building dev-5.6 from GitHub now on my dev Mac to look into this

1 Like