Using PixelStreaming Module to stream SceneCaptureCamera

Development Programming & Scripting
, , , , , ,
1

This is the current setup:

ACameraSensor::ACameraSensor() {
    RootComponent = CreateDefaultSubobject<USceneComponent>(TEXT("RootComponent"));

	RenderTarget = CreateDefaultSubobject<UTextureRenderTarget2D>(TEXT("RenderTarget"));
	RenderTarget->TargetGamma = 100.0f;
	RenderTarget->RenderTargetFormat = ETextureRenderTargetFormat::RTF_RGBA8_SRGB;
    RenderTarget->InitAutoFormat(1920, 1080);

	SceneCapture = CreateDefaultSubobject<USceneCaptureComponent2D>(TEXT("SceneCapture"));
	SceneCapture->SetupAttachment(RootComponent);
	SceneCapture->TextureTarget = RenderTarget;

	SetActorEnableCollision(false);
}

Here we initialise a camera with a render target and camera.

void ACameraSensor::StartStreaming(FString& InId) {
	if (IPixelStreamingModule::IsAvailable() && !IsStreaming) {
		Id = InId;
		SceneCapture->Activate();
		TSharedPtr<FPixelStreamingVideoInputRenderTarget> SceneVideo = FPixelStreamingVideoInputRenderTarget::Create(RenderTarget);

		CameraStreamer = IPixelStreamingModule::Get().CreateStreamer(Id);
		CameraStreamer->SetSignallingServerURL("REDACTED");
		CameraStreamer->SetVideoInput(SceneVideo.ToSharedRef());
		CameraStreamer->StartStreaming();
		IsStreaming = true;
	}
	else {
		UE_LOG(LogTemp, Error, TEXT("Failed to start streaming, either module not loaded or camera is already streaming"));
	}
}

Here when we start streaming the camera, we pass in the name we want to stream it under, activate the SceneCapture and streamer. This works really well and produces the output.

However, the issue comes when I end the game. If I end the game, it stops but then a segmentation error comes from inside the PixelStreamingVideoInputRenderTarget, my guess is that at this point the resource has come null but it doesn’t check this.

This is how I stop the camera:

void ACameraSensor::StopStreaming() {
	if (IsStreaming) {
		CameraStreamer->StopStreaming();
		SceneCapture->Deactivate();
		IsStreaming = false;
	}
}

void ACameraSensor::EndPlay(const EEndPlayReason::Type EndPlayReason) {
	StopStreaming();
}

All the properties apart from the CameraStreamer are marked as UProperties so should not cause this. This is the exact line the segmentation fault occurs at.

if (FTexture2DRHIRef Texture = Target->GetResource()->GetTexture2DRHI())

Here are the logs:

[2024.01.29-14.09.47:123][619]LogWorld: BeginTearingDown for /Game/VehicleTemplate/Maps/UEDPIE_0_VehicleAdvExampleMap
[2024.01.29-14.09.47:123][619]LogPixelStreamingSS: Closing websocket to SS ws://REDCATED
[2024.01.29-14.09.47:123][619]LogHttpServerModule: Stopping all listeners...
[2024.01.29-14.09.47:123][619]LogHttpListener: HttListener stopping listening on Port 7080
[2024.01.29-14.09.47:123][619]LogHttpServerModule: All listeners stopped
[2024.01.29-14.09.47:131][619]LogTemp: Warning: Service deregistered successfully
[2024.01.29-14.09.47:131][619]LogService: Stopping service...
[2024.01.29-14.09.47:135][619]LogWorld: UWorld::CleanupWorld for VehicleAdvExampleMap, bSessionEnded=true, bCleanupResources=true
[2024.01.29-14.09.47:139][619]LogSlate: InvalidateAllWidgets triggered.  All widgets were invalidated
[2024.01.29-14.09.47:139][619]LogContentBundle: [VehicleAdvExampleMap(Standalone)] Deleting container.
[2024.01.29-14.09.47:140][619]LogPlayLevel: Display: Shutting down PIE online subsystems
[2024.01.29-14.09.47:141][619]LogSlate: InvalidateAllWidgets triggered.  All widgets were invalidated
[2024.01.29-14.09.47:529][619]LogKafka: Error: Error consuming message -185
[2024.01.29-14.09.47:529][619]LogKafka: Display: Exited consumer
[2024.01.29-14.09.47:532][619]LogSlate: Updating window title bar state: overlay mode, drag disabled, window buttons hidden, title bar hidden
[2024.01.29-14.09.47:532][619]LogAudioMixer: Deinitializing Audio Bus Subsystem for audio device with ID 2
[2024.01.29-14.09.47:579][619]LogUObjectHash: Compacting FUObjectHashTables data took   0.28ms
[2024.01.29-14.09.47:599][620]LogPlayLevel: Display: Destroying online subsystem :Context_1

Any guidance on how I could solve this is greatly appreciated as I am clueless to why this shouldn’t work. Many people have replicated this and it works fine in blueprints.

2

This is the full stack trace of the error as well:

Caught signal 11 Segmentation fault

libUnrealEditor-PixelStreaming.so!FPixelStreamingVideoInputRenderTarget::OnEndFrameRenderThread() [/home/elecsys/UnrealEngine/Engine/Source/./../Plugins/Media/PixelStreaming/Source/PixelStreaming/Private/PixelStreamingVideoInputRenderTarget.cpp:28]
libUnrealEditor-PixelStreaming.so!TBaseRawMethodDelegateInstance<false, FPixelStreamingVideoInputRenderTarget, void (), FDefaultDelegateUserPolicy>::ExecuteIfSafe() const [/home/elecsys/UnrealEngine/Engine/Source/Runtime/Core/Public/Delegates/DelegateInstancesImpl.h:532]
UnrealEditor-Linux-DebugGame!void TMulticastDelegateBase<FDefaultDelegateUserPolicy>::Broadcast<IBaseDelegateInstance<void (), FDefaultDelegateUserPolicy> >() const [/home/elecsys/UnrealEngine/Engine/Source/Runtime/Core/Public/Delegates/MulticastDelegateBase.h:254]
UnrealEditor-Linux-DebugGame!FEngineLoop::Tick()::$_10::operator()(FRHICommandListImmediate&) const [/home/elecsys/UnrealEngine/Engine/Source/./Runtime/Launch/Private/LaunchEngineLoop.cpp:6107]
UnrealEditor-Linux-DebugGame!TEnqueueUniqueRenderCommandType<FEngineLoop::Tick()::EndFrameName, FEngineLoop::Tick()::$_10>::DoTask(ENamedThreads::Type, TRefCountPtr<FGraphEvent> const&) [/home/elecsys/UnrealEngine/Engine/Source/Runtime/RenderCore/Public/RenderingThread.h:209]
UnrealEditor-Linux-DebugGame!TGraphTask<TEnqueueUniqueRenderCommandType<FEngineLoop::Tick()::EndFrameName, FEngineLoop::Tick()::$_10> >::ExecuteTask(TArray<FBaseGraphTask*, TSizedDefaultAllocator<32> >&, ENamedThreads::Type, bool) [/home/elecsys/UnrealEngine/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:1265]
libUnrealEditor-Core.so!FNamedTaskThread::ProcessTasksNamedThread(int, bool) [/home/elecsys/UnrealEngine/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:758]
libUnrealEditor-Core.so!FNamedTaskThread::ProcessTasksUntilQuit(int) [/home/elecsys/UnrealEngine/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:648]
libUnrealEditor-RenderCore.so!RenderingThreadMain(FEvent*) [/home/elecsys/UnrealEngine/Engine/Source/./Runtime/RenderCore/Private/RenderingThread.cpp:411]
libUnrealEditor-RenderCore.so!FRenderingThread::Run() [/home/elecsys/UnrealEngine/Engine/Source/./Runtime/RenderCore/Private/RenderingThread.cpp:562]
libUnrealEditor-Core.so!FRunnableThreadPThread::Run() [/home/elecsys/UnrealEngine/Engine/Source/./Runtime/Core/Private/HAL/PThreadRunnableThread.cpp:25]
libUnrealEditor-Core.so!FRunnableThreadPThread::_ThreadProc(void*) [/home/elecsys/UnrealEngine/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.h:187]
libpthread.so.0!UnknownFunction(0x8608)
libc.so.6!clone(+0x42)
3

I am still having this issue, any advice on where to start to look?

Some more context in PC Specs:

OS: Ubuntu 20.04
GPU: Graphics Card Nvidia 4090
CPU: 13th Gen Intel(R) Core™ i9-13900
RAM: 64GB DDR5

4 
#include "Sensor/IceSimCameraSensor.h"

AIceSimCameraSensor::AIceSimCameraSensor() {


    RootComponent = CreateDefaultSubobject<USceneComponent>(TEXT("RootComponent"));
	SceneCapture = CreateDefaultSubobject<USceneCaptureComponent2D>(TEXT("SceneCapture"));
	SceneCapture->SetupAttachment(RootComponent);
	SceneCapture->CaptureSource = SCS_FinalColorLDR;
  	SceneCapture->PrimitiveRenderMode = ESceneCapturePrimitiveRenderMode::PRM_RenderScenePrimitives;

	RenderTarget = CreateDefaultSubobject<UTextureRenderTarget2D>(TEXT("RenderTarget"));
  	RenderTarget->CompressionSettings = TextureCompressionSettings::TC_Default;
  	RenderTarget->SRGB = false;
  	RenderTarget->bAutoGenerateMips = false;
	RenderTarget->bForceLinearGamma = true;
	RenderTarget->TargetGamma = 2.2f;
  	RenderTarget->AddressX = TextureAddress::TA_Clamp;
  	RenderTarget->AddressY = TextureAddress::TA_Clamp;
	RenderTarget->InitAutoFormat(1920, 1080);

	SceneCapture->TextureTarget = RenderTarget;

	SetActorEnableCollision(false);
}

void AIceSimCameraSensor::BeginPlay() {

}

void AIceSimCameraSensor::Start(FString& Name) {
	if (IPixelStreamingModule::IsAvailable() && !Streamer.IsValid()) {
		Streamer = IPixelStreamingModule::Get().CreateStreamer(Name);
		VideoInput = FPixelStreamingVideoInputRenderTarget::Create(RenderTarget);
		Streamer->SetSignallingServerURL(FIceModule::Get().GetSignallingURL());
		Streamer->SetVideoInput(VideoInput);
		Streamer->StartStreaming();
	} else {
		UE_LOG(LogTemp, Error, TEXT("Failed to start streaming, either module not loaded or camera is already streaming"));
	}
}

void AIceSimCameraSensor::Stop() {
	if (Streamer.IsValid()) {
		Streamer->StopStreaming();
		IPixelStreamingModule::Get().DeleteStreamer(Streamer);
		Streamer.Reset();

		GEngine->ForceGarbageCollection();

		if (VideoInput.GetSharedReferenceCount() > 1) {
			UE_LOG(LogTemp, Error, TEXT("MORE THAN 1 REFERENCE"));
		}

		VideoInput.Reset();
	}
}

void AIceSimCameraSensor::EndPlay(const EEndPlayReason::Type EndPlayReason) {
	Stop();
}

I’ve done some more work on this, it is the VideoInput is never being garbage collected and therefore never removing it the delegates, is this an actual bug with the plugin not removing references to it? The Log there is fired meaning even after ending the stream there is somewhere using it.

5

Hi there!

Thank you so much for your patience, we’re having a look now!
Can I please confirm which version of UE you’re using?

6

Hiya!

I am using version 5.3.2

7

Thought I would also add that I’ve tried using the blueprint components and objects as well but the same result occurs in C++. I did get it working when using a blueprint but not with dynamic creation