Unreal tries to build my third party library

Hi all!
I try to add a third-party library to my Unreal project, I am fairly new to Unreal so there could be some settings I am missing but the behavior is weird. I am using unreal 5.1

The library that I am trying to add is Spout https://github.com/leadedge/Spout2. I know there is a plugin for that but it has bad performance for my use case so I want to implement it myself to see if I can optimize anything.

I have added the third-party library as a module and I got it to work after a while, however when I turn off my computer and started it the next day it does not longer work. This happened twice when I turned off the computer for the day and continued the next day. The error I get is:
spout
and if pressed yes:
spout_error

It feels like unreal tries to build my module instead of including its lib and dll file. My build.cs file is:

public class Spout : ModuleRules
{

    public Spout(ReadOnlyTargetRules Target) : base(Target)
    {
        Type = ModuleType.External;

        // Add any include paths for the plugin
        PublicIncludePaths.Add(Path.Combine(ModuleDirectory, "ThirdParty"));

        // Add any import libraries or static libraries
        PublicAdditionalLibraries.Add(Path.Combine(ModuleDirectory, "ThirdParty", "SpoutLibrary.lib"));

        RuntimeDependencies.Add(Path.Combine(ModuleDirectory, "Binaries/Win64/SpoutLibrary.dll"));

    }
}

I have also added my module to my uproject file as this:

{
"Name": "Spout",
"Type": "Runtime",
"LoadingPhase": "Default"
}

My file structure in the Source folder is:
My_Module(Game folder)
Private
Public
My_Module.Build.cs
(All other files)
Spout
Private
SpoutModule.cpp
ThirdParty
SpoutLibrary.dll
SpoutLibrary.lib
SpoutLibrary.h
Spout.Build.cs

It is very weird that it works and then the next day it does not without any changes to the code. I have also tried setting up a new project and made it work, committed that state to git, and then made changes so it did not work anymore. When I reset it back to the working commit it does not work anymore. It feels like there is some state in Unreal that ignore my Type = ModuleType.External; flag.

Any suggestions?