PixelStreaming for macOS M1 Max, UE5.2+

PhDittmann · June 6, 2023, 3:59pm

I’d like to use PixelStreaming with UE5.2+ on Mac Book Pro with M1 Max chip.

I started the SignallingWebServer and the packaged game with ./MyUnreal5.app/Contents/MacOS/MyUnreal5 -AudioMixer -PixelStreamingIP=localhost -PixelStreamingPort=8888 -AllowPixelStreamingCommands=true, but http://localhost:80 says 'No streamers connected.:

Where to go next? Does it require implementing Apple’s Video Toolbox?

Steps until now:

I made the project compiling and packaging by added "Mac" to PlatformDenyList of existing GPU vendor codecs:

UnrealEngine/Engine/Plugins/Media/PixelStreaming/PixelStreaming.uplugin

	// ...
	{
		"Name": "WmfMedia",
		"Enabled": true,
		"PlatformDenyList": [ "Linux", "Mac" ]
	},
	{
		"Name": "AVCodecsCore",
		"Enabled": true,
		"PlatformDenyList": [ "Mac" ]
	},
	{
		"Name": "NVCodecs",
		"Enabled": true,
		"PlatformDenyList": [ "Mac" ]
	},
	{
		"Name": "AMFCodecs",
		"Enabled": true,
		"PlatformDenyList": [ "Linux", "Mac" ]
	},
	// ...

For the SignallingWebServer in PixelStreamingInfrastructure, I made sure, that all the bash files have UNIX line endings (specified at checkout or using git attribute *.sh text eol=lf).
I copied NodeJS from homebrew to the appropriate locations in PixelStreamingInfrastructure/SignallingWebServer/platform_scripts/bash/node/. If you don’t do so, the script tries to install and run a linux version.
Start SignallingWebServer via sh ./PixelStreamingInfrastructure/SignallingWebServer/platform_scripts/bash/Start_SignallingServer.sh.
Start packaged game via ./MyUnreal5.app/Contents/MacOS/MyUnreal5 -AudioMixer -PixelStreamingIP=localhost -PixelStreamingPort=8888 -AllowPixelStreamingCommands=true
Visit http://localhost:80 with a WebRTC capable browser.

EDIT 1: Interesting folders:

Engine/Plugins/Experimental/AVCodecs/AMFCodecs
Engine/Plugins/Experimental/AVCodecs/NVCodecs
Engine/Plugins/Media/HardwareEncoders
Engine/Plugins/Media/HardwareEncoders/Source/EncoderAMF
Engine/Plugins/Media/HardwareEncoders/Source/EncoderNVENC
Engine/Plugins/Media/PixelStreaming
Engine/Source/Runtime/AVEncoder
Engine/Source/ThirdParty/AMD/Amf
Engine/Source/ThirdParty/NVIDIA/CUDA
Engine/Source/ThirdParty/NVIDIA/nvEncode

EDIT 2: VideoToolbox

VideoToolbox can be used by adding the following snippet to your *.Build.cs:

if (Target.Platform == UnrealTargetPlatform.Mac)
{
	//PublicFrameworks.Add("VideoToolbox"); // or
	PublicFrameworks.AddRange(new string[] {
		"VideoToolbox"
	});
}

Samples can be found in:

Engine/Plugins/Media/ElectraPlayer/Source/ElectraPlayerRuntime/ElectraPlayerRuntime.Build.cs
Engine/Plugins/Media/AppleProResDecoderElectra/Source/AppleProResDecoderElectra/AppleProResDecoderElectra.Build.cs
Engine/Plugins/Media/ElectraCodecs/Source/ElectraDecoders/ElectraDecoders.Build.cs

To enable more PixelStreaming modules, we have to compile WebRTC for Mac first.

EDIT 3: WebRTC

# https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html#_setting_up
% git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
% export PATH="`pwd`/depot_tools:$PATH"

# https://gn.googlesource.com/gn/+/master/README.md
# Download: https://chrome-infra-packages.appspot.com/dl/gn/gn/mac-amd64/+/latest
% mv ~/Downloads/gn-mac-amd64 gn
% export PATH="`pwd`/gn:$PATH"

brew install ninja

# https://webrtc.github.io/webrtc-org/native-code/development/
# https://github.com/tmthecoder/WebRTC-macOS
% mkdir webrtc
% cd webrtc
# fetch --nohooks webrtc
% fetch --nohooks webrtc_ios
% gclient sync
% cd src

# https://chromiumdash.appspot.com/branches
# 4664 -> fails: sprintf, NSString
# 5414 -> builds, but cannot be linked
# 6045 -> builds, but cannot be linked
% WEBRTC_VERSION=5414

% git checkout -b $WEBRTC_VERSION refs/remotes/branch-heads/$WEBRTC_VERSION
% gclient sync -D
% gn gen out/${WEBRTC_VERSION}_mac_arm64 --args='target_os="mac" target_cpu="arm64" is_component_build=false is_debug=false rtc_libvpx_build_vp9=false enable_stripping=true rtc_enable_protobuf=false'
% ninja -C out/${WEBRTC_VERSION}_mac_arm64 mac_framework_objc
# for multi-platform additionally:
#% gn gen out/${WEBRTC_VERSION}_mac_x64 --args='target_os="mac" target_cpu="x64" is_component_build=false is_debug=false rtc_libvpx_build_vp9=false enable_stripping=true rtc_enable_protobuf=false'
#% ninja -C out/${WEBRTC_VERSION}_mac_x64 mac_framework_objc

% cd ..
% cp -R src/out/${WEBRTC_VERSION}_mac_x64/WebRTC.framework WebRTC.framework
% lipo -create -output WebRTC.framework/WebRTC src/out/${WEBRTC_VERSION}_mac_arm64/WebRTC.framework/WebRTC
# for multi-platform:
#% lipo -create -output WebRTC.framework/WebRTC src/out/${WEBRTC_VERSION}_mac_x64/WebRTC.framework/WebRTC src/out/${WEBRTC_VERSION}_mac_arm64/WebRTC.framework/WebRTC

% ls -l WebRTC.framework 
total 43752
lrwxr-xr-x  1 user staff        24 Nov 16 11:27 Headers -> Versions/Current/Headers
lrwxr-xr-x  1 user staff        24 Nov 16 11:27 Modules -> Versions/Current/Modules
lrwxr-xr-x  1 user staff        26 Nov 16 11:27 Resources -> Versions/Current/Resources
drwxr-xr-x  4 user staff       128 Nov 16 11:27 Versions
-rwxr-xr-x  1 user staff  22397224 Nov 16 11:27 WebRTC

% otool -L WebRTC.framework/WebRTC
WebRTC.framework/WebRTC (architecture x86_64):
	@rpath/WebRTC.framework/WebRTC (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/AVFoundation.framework/Versions/A/AVFoundation (compatibility version 1.0.0, current version 2.0.0)
	/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1690.5.4)
	/System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1971.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 61.0.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1228.0.0)
	/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 1000.0.0)
	/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/VideoToolbox.framework/Versions/A/VideoToolbox (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo (compatibility version 1.2.0, current version 1.5.0)
	/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
WebRTC.framework/WebRTC (architecture arm64):
	@rpath/WebRTC.framework/WebRTC (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/AVFoundation.framework/Versions/A/AVFoundation (compatibility version 1.0.0, current version 2.0.0)
	/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1690.5.4)
	/System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1971.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 61.0.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1228.0.0)
	/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 1000.0.0)
	/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/VideoToolbox.framework/Versions/A/VideoToolbox (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo (compatibility version 1.2.0, current version 1.5.0)
	/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)


% zip -ry WebRTC.framework.zip WebRTC.framework/*
# Copy WebRTC.framework.zip to i.e. $UnrealEngine/Engine/Source/ThirdParty/WebRTC/${WEBRTC_VERSION}/Frameworks/Mac/WebRTC.framework.zip

WebRTC.build.cs:

	protected virtual bool bShouldUseWebRTC
	{
		get =>
			Target.Platform.IsInGroup(UnrealPlatformGroup.Windows) ||
			(Target.IsInPlatformGroup(UnrealPlatformGroup.Unix) && Target.Architecture == UnrealArch.X64) ||
			Target.Platform == UnrealTargetPlatform.Mac;
	}

	//Config switch to use new WebRTC version 5414 (M109)
	protected virtual bool bShouldUse5414WebRTC
	{
		get =>
			Target.Platform == UnrealTargetPlatform.Mac ||
			false;
	}

	public WebRTC(ReadOnlyTargetRules Target) : base(Target)
	{
		// ...

			else if (Target.Platform == UnrealTargetPlatform.Mac)
			{
				PlatformSubdir = "Mac";
				string FrameworkPath = Path.Combine(ModuleDirectory, "5414", "Frameworks", "Mac", "WebRTC.framework");
				System.Console.WriteLine(FrameworkPath);
				
				PublicFrameworks.AddRange(new string[] {
					"AVFoundation",
					"CoreGraphics",
					"CoreMedia",
					"OpenGL",
					"Foundation",
					"ApplicationServices",
					"AudioToolbox",
					"CoreAudio",
					"VideoToolbox",
					"CoreVideo",
					"CoreFoundation",
					"QuartzCore",
					"Metal",
					"MetalKit",
					"AppKit"
				});

				// searches for FrameworkPath, FrameworkPath+".zip", FrameworkPath+".ZIP"
				PublicAdditionalFrameworks.Add(
					new Framework(
						"WebRTC",
						FrameworkPath
					)
				);
			}

The archive will be unzipped to $UnrealEngine/Engine/Intermediate/UnzippedFrameworks

------ Building 3 action(s) started ------
[1/3] Link [Apple] UnrealEditor-PixelCapture.dylib
ld: framework not found WebRTC
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Total time in Parallel executor: 0.09 seconds
Total execution time: 7.99 seconds

** BUILD FAILED **

Travis_Ashcraft · October 31, 2023, 2:59am

Did you ever figure this out?

PhDittmann · November 2, 2023, 4:36am

At the moment I’m using PixelStreaming only for Windows and Linux, because it looks quite challenging to implement all this.

Travis_Ashcraft · November 2, 2023, 3:13pm

Did you get it to compile and open the project with out errors? My biggest issue atm is that I can not make a Mac build due to the pixelstream plugin.

PhDittmann · November 3, 2023, 9:55am

All changes I made are described above: disabling important modules, which won’t run on Mac anyway. I’m using ue5-main as base.

MWillWallT · November 9, 2023, 12:46am

Hi guys!

At this stage, Pixel Streaming does not support Mac, however it is in our roadmap!
No promises of course, but we are targeting Pixel Streaming Mac support for UE 5.4.
Keep an eye on UE5/Main in Github over the next few months if you want an early peek at it.