[HTTP] Sending continuous FHttpServerResponse for MJPEG data

Nikkitta · March 9, 2024, 2:00pm

Hello,
I’m currently delving into the utilization of built-in HTTP classes within Unreal Engine, such as FHttpServerModule, IHttpRouter, FHttpServerResponse.

I’ve developed an engine plugin tasked with capturing generated frames at a certain frequency, for example, 25 frames per second. These frames are stored as const TArray64<uint8> &ImgData. Additionally, the plugin operates an HTTP server. My objective is to stream this frame data as MJPEG to any visitor accessing http://127.0.0.1:8000/stream.

Below, I’ve included snippets of the relevant code.

The problem.
I’m encountering difficulties understanding how to manage client connections to the HTTP server and how to continuously send responses to all connected clients. Despite extensive searches and examination of Unreal Engine sources, I haven’t found a satisfactory solution. The documentation for such modules is quite sparse. As far as I can tell, when I send a response using the following code:

TUniquePtr<FHttpServerResponse> Response = FHttpServerResponse::Create(ResponseContent, ContentType);
OnComplete(MoveTemp(Response));

the response is sent, but the connection is closed immediately. However, I need to maintain continuous responses until the client disconnects. When I try to execute the above code in a while(true) loop app crashes on second pass.

Any guidance or advice would be greatly appreciated!

Here are some excerpts from my HTTP server code. I acknowledge that there are issues present:

void AHttpServer::BeginPlay()
{
    StartServer();
    Super::BeginPlay();
}

void AHttpServer::StartServer()
{
    if (ServerPort <= 0)
    {
        UE_LOG(LogTemp, Error, TEXT("Could not start HttpServer, port number must be greater than zero!"));
        return;
    }

    FHttpServerModule &httpServerModule = FHttpServerModule::Get();
    TSharedPtr<IHttpRouter> httpRouter = httpServerModule.GetHttpRouter(ServerPort);

    if (httpRouter.IsValid())
    {
        httpRouter->BindRoute(FHttpPath(HttpPathSTREAM), EHttpServerRequestVerbs::VERB_GET,
                              [this](const FHttpServerRequest &Request, const FHttpResultCallback &OnComplete)
                              { return RequestSTREAM(Request, OnComplete); });

        httpServerModule.StartAllListeners();

        _isServerStarted = true;
        UE_LOG(LogTemp, Log, TEXT("Web server started on port = %d"), ServerPort);
    }
    else
    {
        _isServerStarted = false;
        UE_LOG(LogTemp, Error, TEXT("Could not start web server on port = %d"), ServerPort);
    }
}

void AHttpServer::UpdateJpegData(const TArray64<uint8> &NewJpegData)
{
    JpegData = NewJpegData;
}

bool AHttpServer::RequestSTREAM(const FHttpServerRequest &Request, const FHttpResultCallback &OnComplete)
{
    // Define the content type as MJPEG
    FString ContentType = "multipart/x-mixed-replace; boundary=--frame";

    // Start the response with the appropriate headers
    FString ResponseContent = FString::Printf(TEXT("HTTP/1.1 200 OK\r\nContent-Type: %s\r\n\r\n"), *ContentType);

    // Send each JPEG frame in a loop
    while (true)
    {
        // Add the boundary delimiter
        ResponseContent += "--frame\r\n";

        // Add the content type for each frame
        ResponseContent += "Content-Type: image/jpeg\r\n\r\n";

        // Append the JPEG data
        ResponseContent.Append(reinterpret_cast<const char *>(JpegData.GetData()), JpegData.Num());

        // Add a newline after each frame
        ResponseContent += "\r\n";

        // Send the response asynchronously
        TUniquePtr<FHttpServerResponse> Response = FHttpServerResponse::Create(ResponseContent, ContentType);
        // Here I get the crash on second pass of the loop
        OnComplete(MoveTemp(Response));

        // Delay between frames (adjust as needed)
        FPlatformProcess::Sleep(1.0f / 25.0f);
    }

    TUniquePtr<FHttpServerResponse> response = FHttpServerResponse::Create(TEXT("HttpServer STREAM"), TEXT("text/html"));
    OnComplete(MoveTemp(response));
    return true;
}

In my main class of the plugin in BeginPlay() I create server:
HttpServer = (AHttpServer *)GetWorld()->SpawnActor(AHttpServer::StaticClass());

And here is the Tick function where I update frame in HttpServer class:

void AStreamManager::Tick(float DeltaTime)
{
    Super::Tick(DeltaTime);

    if (!RenderRequestQueue.IsEmpty())
    {
        // Peek the next RenderRequest from queue
        FRenderRequestStreamStruct *nextRenderRequest = nullptr;
        RenderRequestQueue.Peek(nextRenderRequest);

        if (nextRenderRequest)
        { // nullptr check
            if (nextRenderRequest->RenderFence.IsFenceComplete())
            {
                // Check if rendering is done, indicated by RenderFence
                // Load the image wrapper module
                IImageWrapperModule &ImageWrapperModule = FModuleManager::LoadModuleChecked<IImageWrapperModule>(FName("ImageWrapper"));

                // Prepare data to be JPEG
                static TSharedPtr<IImageWrapper> imageWrapper = ImageWrapperModule.CreateImageWrapper(EImageFormat::JPEG); // EImageFormat::JPEG
                imageWrapper->SetRaw(nextRenderRequest->Image.GetData(), nextRenderRequest->Image.GetAllocatedSize(), FrameWidth, FrameHeight, ERGBFormat::BGRA, 8);
                const TArray64<uint8> &ImgData = imageWrapper->GetCompressed(0);

                HttpServer->UpdateJpegData(ImgData);

                ImgCounter += 1;

                // Delete the first element from RenderQueue
                RenderRequestQueue.Pop();
                delete nextRenderRequest;
            }
        }
    }
}

Rev0verDrive · March 12, 2024, 5:05am

HTTP is discrete. Blast, then close. What you want is UDP. More so you want a fully opened stable connection to the socket and IP/Port.

Take a look at the following. What you want is a Plugin that will broadcast/Stream your output over UDP.

Nikkitta · March 13, 2024, 11:08am

Thanks for the info!

As I understand it, you’re suggesting the following approach:

Wait for a client connection to the HTTP address http://127.0.0.1:8000/stream (from browser or some app that wants this MJPEG stream).
Extract client info from this HTTP request.
Open a new UDP connection to this client.
Start sending MJPEG data via this UDP connection.

But will the browser understand that received UDP MJPEG data should be shown on the same http page? I doubt it. So a client needs some additional logic to parse that data from UDP channel.

What I aim to implement is multipart/x-mixed-replace for HTTP. I’ve successfully used it in a Python script, which sends single_frame.jpeg to each client connected to the HTTP address. I will share the Python code below. Now I want to recreate this logic using C++ Unreal Engine’s HTTP classes.

from http.server import HTTPServer, SimpleHTTPRequestHandler
from socketserver import ThreadingMixIn

class ThreadingSimpleServer(ThreadingMixIn, HTTPServer):
    daemon_threads = True
    allow_reuse_address = True
    
def run(server_class=HTTPServer, handler_class=SimpleHTTPRequestHandler):
    server_address = ('', 8000)
    handler_class = StreamingHandler
    httpd = ThreadingSimpleServer(server_address, handler_class)
    httpd.serve_forever()

class StreamingHandler(SimpleHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/stream':
            # response
            self.send_response(200)
            # header
            self.send_header('Age', 0)
            self.send_header('Cache-Control', 'no-cache, private')
            self.send_header('Pragma', 'no-cache')
            self.send_header('Content-Type', 'multipart/x-mixed-replace; boundary=--frame')
            self.end_headers()

            srcaddr, srcport = self.request.getpeername()
            print("Connection from {}:{}".format(srcaddr, srcport))

            try:
                while True:
                    file = open("single_frame.jpeg", "rb")
                    f = file.read()
                    file.close()
                    b = bytearray(f)
                    image_bytes = f
                    frame = b
                    self.wfile.write("--frame\r\n".encode())
                    self.send_header('Content-Type', 'image/jpeg')
                    self.send_header('Content-Length', len(frame))
                    self.end_headers()
                    self.wfile.write(frame)
                    self.end_headers()

            except Exception as e:
                print(str(e))
        else:
            super().do_GET()

Nikkitta · March 28, 2024, 11:30am

Hello everyone,

I’m still seeking suggestions on leveraging built-in HTTP services for a particular functionality I’m trying to implement.

Recently, I’ve been experimenting with adapting the C library available at GitHub - nola-a/mjpeg2http. After rewriting the main libmjpeg2http to C++ and creating a separate class, I managed to successfully build the project. However, I encountered an issue where clients are unable to connect. Additionally, I realized that this approach won’t be viable on Windows due to the library’s dependency on Unix-specific functions.

I would greatly appreciate any insights or alternative approaches that could help me resolve this challenge.

Thank you for your time and assistance!

Best regards,
Nikita