So I have a large string of replay data that is generated at the end of each level, and is then sent to a server to be stored. Ideally I would like this string to be compressed as to take up as little space as possible, and allow faster upload/download to and from the server.
I was recently browsing the source code and stumbled across the “lz4.h” file in “Source/Runtime/Core/Public/Compression” and noticed it appeared to contain some basic functions for compression and decompression.
I have a very limited knowledge of c++, and my project consists of 99% blueprints, with only a few custom blueprint nodes written in c++. Because of this I would love it if someone would be able to point me in the right direction to turn these c++ functions into blueprints for my project.
My plan is to add these two functions into my existing c++ Blueprint function library class if possible.
The two functions are:
LZ4LIB_API int LZ4_compress_default(const char* src, char* dst, int srcSize, int dstCapacity);
LZ4LIB_API int LZ4_decompress_safe (const char* src, char* dst, int compressedSize, int dstCapacity);
Note that the compressed buffer can contain zeroes (string terminators) so it is probably a very bad idea to store the result back in a FString.
Unfortunately blueprints do not support things like uint8* or TArray<uint8> so it’s gonna be difficult to send such a binary buffer over the network via standard RPCs.
I can think of two possible approaches :
Wrap results in a custom USTRUCT, store the buffer natively (it won’t be visible to BP, but you’ll still be able to pass the struct around), and write custom serializer for networking.
Convert the result buffer into a TArray<int32>, by merging bytes 4 by 4, so you can actually use it in blueprints.
Thank you so much for taking the time to respond with this!
This looks a lot more daunting than I was first hoping ill be honest.
From my limited knowledge of c++, it appears you are using the input string to calculate all of the required variables for the function, such as the dst, input size and bound, which i was originally planning on exposing as inputs to the blueprint and filling them in using other already existing nodes. This solution is so much more elegant though, so i appreciate you doing this.
I will put this code to the test once I get back to my computer. I am however a little worried by the comments you added towards the end. Unfortunately with my limited knowledge, I’m not sure I understand either of the two solutions you provide after the code to avoid storing as a string. It appears the output of the function returns a string, so am I correct in assuming that when you say it would be a bad idea to store the result as a string, you are referring to simply taking the output and throwing it into a variable?
Previously with uncompressed data, I have just been using http requests to send the replay data to a server in string form, and it would be handy to be able to do the same only with compressed data instead. Is this completely out of the question?
Also as a side question, you use the include #include "TraceLog/Private/Trace/LZ4/lz4.c.inl". Can I ask why this is used instead of something like #include "Compression/lz4.h"
char* doesn’t necessarily mean string. It’s also an old way of declaring byte/int8. In this case, a compressed “string” is more like an array of bytes rather than a string.
A string is also an array of bytes, but generally the useable/printable character codes are between 10 and 127, and a string is terminated by a null (zero) byte.
In c/c++, the standard way of finding a string length is to walk the char* array byte by byte until you find the zero terminator. So even though here you DO have the number of bytes in a separate variable, if you convert it back to a string you’ll most likely lose a big chunk of the data as the string implementation will only store the byte array up until the first zero byte.
To use "Compression/lz4.h" you need to explicitly enable LZ4 with a global definition and rebuild engine from source, which is annoying for most people. The file i’m including brings the whole lib into your module so you don’t get linkage problems.
I thought you were talking about sending replay data from client to server within UE4. The problem remains similar for sending a HTTP request, but resolution will be a bit different. However I can’t really help you with that part without knowing what kind of HTTP API you are using and what parameters it accepts.
Ok so from what I think I can understand is, I shouldn’t store it as a string because if I try to do anything with that string going forward, if there was a zero terminator contained within, anything I try to do with it will be stopped early because of this and I will effectively lose anything after it.
Is this correct? and if so, under what circumstance would a zero termination be able to appear in the middle of the my compressed data, if any?
Also, assuming that is the reason for not storing it as a string, would I run into issues if I then stored the string into a json object to send via a http request. My server is a simple lamp stack and i’m using the Varest plugin to make the http requests.
As far as the actual function goes, it appears to be working, although without a way to also decompress it I cant be too sure. Would the decompression function be somewhat similar only using the actual safe decompress function instead?
Edit: I just stumbled onto this plugin on the marketplace and it appears to be somewhat similar but uses a byte array instead of a string input. Would it be possible to output into a byte array in this version also if it makes it safer to use?
About your finding : looks like the editor actually supports byte array, so I was wrong on that part. This makes things a lot simpler on that front.
Regarding json object - it does not seem like a good idea as JSON format is plain text, and converting a byte array to plain text is gonna take much more space. Base64 could work, but it’s still like 30% bigger than pure binary, so not sure if it would be worth it.
VaRest does provide base64 encoding so you can test it for yourself :
Sending true binary over http is a bit painful as you need multipart form data instead of json, and it’s also a pain to handle on the server side. I’m not sure if there’s any plugin that makes it easier.
If your server supports multipart, you can write your own request handler with a bit of effort. It would look like something like this.
Once i have figured out how to convert the function to output a byte array I think that will probably be my limit without asking far too much of you or anyone else willing to help since I have no idea what multipart etc is, and likely if I had the budget I would have probably just bought that plugin anyway which would force me to do the base64 encode so that is how I will do it now. It seems like the simple solution and although maybe not the most effective way, it will probably suffice for what I need.
As far as rewriting this to change the input, I assume the function will now be int32 UMyLib::LZ4_Compress(const FString& Input, TArray<uint8>& Result).
As for changing the actual output in the function, I’m a little stuck because I’m not entirely sure what these lines do:
dst[written] = '\0';
Result = dst;
I know that i need to set the result variable to the dst variable from the compression function, but I’m not sure what dst[written] = '\0'; is doing.
I was adding a null terminator “manually” to make it behave like a string, but since there might be null terminators in the middle of the array it’s kinda moot, as discussed above.
Well it looks like the functions are already being included by the Core module, so adding the full .inl file in your own module seems to be causing conflict when building monolithic.
Try replacing your include with the other one #include "Compression/lz4.h"
Seems to have built correctly now, although still need to test the functions etc. Is it possible to remove those errors or will I just have to live with them?
Edit: it does indeed seem to be functioning correctly.
<3
