[Plugin] ZipUtility (7zip)

Hey UE developers! A certain vr project of mine needed the ability to peek into archives and extract them efficiently and I couldn’t find any good zip solution for UE4 so I decided to port 7zip, update it for VS2015 and make it easy to use. I hope this will be useful to some of you :). If you can improve it, please make pull requests!

An event-driven, multi-threaded, C++ & blueprint accessible 7zip archiver and file manipulation plugin for Unreal Engine 4. Built on 7zip-cpp modernization of the SevenZip++.

Supports the following compression algorithms: Zip, 7-Zip, GZip, BZip2, RAR(decompress only), TAR, ISO, CAB, LZMA, LZMA86.

Plugin works in Windows only.

Download
Stable
https://img.shields.io/github/release/getnamo/ziputility-ue4.svg

Unstable
Master Branch

Documentation and Resources
Github Repository

Quick Install & Setup

  1. Download
  2. Create new or choose project.
  3. Browse to your project folder (typically found at Documents/Unreal Project/{Your Project Root})
  4. Copy Plugins folder into your Project root.
  5. Restart the Editor and open your project again. Plugin is now ready to use.

Basic Usage - Blueprint
After the quick setup, you will have new functions available to you in blueprint

if you wish to receive callback events add ZipUtilityInterface to your blueprint class and pass a reference to self into your function calls

which exposes these events

so to e.g. Unzip a file you simply do

Note that the event callbacks are optional and are called asynchronously on your game thread allowing you to easily show a progress indicator. The plugin can handle Zipping, Unzipping, Listing contents, Making Folders and Moving/Renaming Files.

For more documentation, read the github repo readme, the main source of documentation.

Changelist

0.3.1
-Fixes for packaging

0.3.0
-Compile fixes for 4.15
-Added scaffold for status return on ZipUtility actions
-misc internal changes from 0.2.2 to 0.2.6

0.2.2
-Thanks to #4](Allow concurrent usage of zip library for list, zip and unzip operations by error454 · Pull Request #4 · getnamo/ZipUtility-ue4 · GitHub) (@hyperdr1ve](error454 (Zachary Burke) · GitHub)) the plugin now supports multiple simultaneous operations working independently on their own threads with proper callbacks.
-Added windows listing folder content function with FileListInterface expected to receive the content data

0.2.0
-Windows API functions split into WindowsFileUtility module
-Added a folder watcher interface (IFolderWatchInterface) and bp library function which will allow you to watch a folder for file changes and respond to such events e.g. directory changed or file changed
-ZipUtility trimmed to only zipping/archive functions
-Thanks to #1](Master by Deams51 · Pull Request #1 · getnamo/ZipUtility-ue4 · GitHub) (@deams51](Deams51 (Mickaël) · GitHub)) now supports extracting a single file
-Unreal engine 4.14 compile fixes

0.1.1
-Added delete file/folder functions
-Added UnzipTo
-Merged UnzipWithFormat with Unzip. Compression set to unknown will auto-detect compression.
-All callbacks now return with the archive path, allowing you to distinguish between simultaneous actions.
-Fixed callback consistency, unzip and zip will emit all progress updates and list callback will return with OnDone when listing has completed.
-Update to 7z-cpp bind to v0.2.0
-Added macro definitions for C++ integration

0.1.0
-First public commit for UE4.10

VR comic reader? :stuck_out_tongue:

Awesome work man :smiley:

This is really, neat! I have been thinking about dealing with zipped files and unreal but decided to not bother (as it never got to the point of needing it). Awesome utility, now ill have to check it out!

Let me know if you think anything is missing, feel free to modify/make pull requests.

Cheers! Not quite comics, although that sounds pretty cool. It’s more to do with automatic download and checking if they’re VR experiences and moving them into place/etc. If you’re interested in the use case, keep an eye out in the Nexus thread.

Are there any license concerns here? I thought rar and/or 7zip has license issues

I’d love to see Nexus support Hydra and other motion controls in addition to Leap Motion.

That is in the works, multiplayer and easy portal downloads first though :slight_smile:

The plugin modifications and 7zip-cpp are MIT, SevenZip++ is public domain.

7-Zip is LGPL + UnRar restriction which simply means you can’t compress RAR or make a RAR archiver (this plugin won’t allow you to compress RAR anyway, just decompress), decompressing/unarchiving is allowed and you just need to retain that specific notice in your license

  • The unRAR sources cannot be used to re-create the RAR compression algorithm,
    which is proprietary. Distribution of modified unRAR sources in separate form
    or as a part of other software is permitted, provided that it is clearly
    stated in the documentation and source comments that the code may
    not be used to develop a RAR (WinRAR) compatible archiver.*

You can comment one line of code to remove the Unrar restriction using 7za.dll instead of 7z.dll by changing line 138 and 139 in ZipFileFunctionLibrary, this will however limit your archive support to .7z only due to how 7za.dll is compiled (you can in theory re-enable everything but RAR, but you will need to recompile a compatible 7zip dll yourself using the LZMA sdk which is public domain), for my own purposes the unrar restriction is acceptable and I prefer the wider archive support so I default to 7z.dll.

Regarding LGPL, because this plugin is using the com-like port interface to the 7zip .dll it doesn’t cause the rest of the code (e.g. modifications and any unrelated code) to fall under LGPL. From the wiki on LGPL: a standalone executable that dynamically links to a library through a .so, .dll, or similar medium is generally accepted as not being a derivative work as defined by the LGPL. It would fall under the definition of a “work that uses the Library”.

Paragraph 5 of the LGPL version 2.1 states: A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being compiled or linked with it, is called a “work that uses the Library”. Such a work, in isolation, is not a derivative work of the Library, and therefore falls outside the scope of this License.

While not relevant, that dll barrier is actually another static library (7zip-cpp) away from the plugin in release mode and another dynamic link barrier in editor mode (this plugin .dll).

TL;DR:
It may be not trivial to parse but you are ok with using this plugin commercially with the license file included, that said I’m not a lawyer and you should make your own judgement, refer to the plugin license file for details.

Update to 0.1.1
-Added delete file/folder functions
-Added UnzipTo
-Merged UnzipWithFormat with Unzip. Compression set to unknown will auto-detect compression.
-All callbacks now return with the archive path, allowing you to distinguish between simultaneous actions.
-Fixed callback consistency, unzip and zip will emit all progress updates and list callback will return with OnDone when listing has completed.
-Update to 7z-cpp bind to v0.2.0
-Added macro definitions for C++ integration

This update brings fixes from 7z-cpp and improves the reliability of callbacks. You can now also call simultaneous actions and distinguish between results and progress updates via the archive path string parameter attached to each callback.

New file manipulation functions (delete file/folder) allow you to clean up temporary files e.g. an archive after you have unzipped. UnzipTo function allows you to unzip to a different location than where the archive is present.

Note that DeleteFolderRecursively can be dangerous, but it shouldn’t allow you to delete outside your game directory (basic sandboxing).

New Callbacks Example

New UnzipTo Function Example

Also changed git structure to better follow distinction between source and releases, testing the waters to see if this is more easily maintainable than what I was doing earlier.

Hi getnamo! I really like your Plugin and have been using it c++ side. I’m now trying to package my game and am getting this error



UE4-ZipUtility.lib(Module.ZipUtility.cpp.obj) : error LNK2001: unresolved external symbol "class ATL::CAtlBaseModule ATL::_AtlBaseModule" (?_AtlBaseModule@ATL@@3VCAtlBaseModule@1@A)
7zpp_u.lib(stdafx.obj) : error LNK2001: unresolved external symbol "class ATL::CAtlBaseModule ATL::_AtlBaseModule" (?_AtlBaseModule@ATL@@3VCAtlBaseModule@1@A)


Any clues as to what is going on? I’ve tried all changing all kinds of plugin/Engine settings but can’t get it to work. I see that CAtlBaseModule is a Windows thing, but not sure what to do with that information.

Thanks!

Seems to be that we’re missing links to atls.lib (unresolved external symbol ATL::AtlWinModuleInit | Microsoft Learn) or some form of basic windows lib/dependency. The plugin uses some windows specific functions to move files around (convenience functions found at https://github.com/getnamo/ZipUtility-ue4/blob/master/Source/ZipUtility/Private/ZipFileFunctionLibrary.cpp#L398), if you’re not using them you could disable them or consider maybe finding a good pull request to make it less windows specific?

Otherwise the lib dependency needs to be resolved.

Haven’t tried to package this plugin before, so you’re first to try this out :slight_smile:

Great. Looks like we did the same research on this one. Thanks for pointing out where the windows functions are being called. I’m changing my strategy and am not using the plugin anymore (for a host of other reasons not related to the plugin itself), but it looks like a solvable problem.

  • I get a crash if I hit play in editor, do some unzipping work, stop, then hit play again. Seems like the thread isn’t cleaned up always when ending the in-editor play.

Thanks for this btw, it works great for my comic book reader.

Got any logs for when these crashes happen?


2>A:\Work\Projects\NY_Loft - PC\Plugins\ZipUtility\Source\ZipUtility\Private\ZipFileFunctionLibrary.cpp(4): fatal error C1083: Cannot open include file: 'ListCallback.h': No such file or directory

A few files appear to be missing for me.

Along with


#include "ListCallback.h"
#include "ProgressCallback.h"
#include "7zpp.h"

Is this @ the latest master branch commit? Might be that the pre-requisites are desynced. Let me know which version is broken and I’ll have a look.

I am using 4.13.1 and the latest release from this forum post.

Update to 0.2.0
-Windows API functions split into WindowsFileUtility module
-Added a folder watcher interface (IFolderWatchInterface) and bp library function which will allow you to watch a folder for file changes and respond to such events e.g. directory changed or file changed
-ZipUtility trimmed to only zipping/archive functions
-Thanks to #1](Master by Deams51 · Pull Request #1 · getnamo/ZipUtility-ue4 · GitHub) @deams51](Deams51 (Mickaël) · GitHub) now supports extracting a single file
-Unreal engine 4.14 compile fixes

This update cleans some of my utility functions into a separate Windows API module and adds a new folder watcher that will emit events when something changes in the folder, e.g. file saved or directory created.

Grab latest at Releases · getnamo/ZipUtility-ue4 · GitHub

Do you have any plans in making it cross-platform? Using gzip on unix system for example, there are plenty of libs that could be used.

Hey thanks a bunch for this project. I got a basic implementation up in C++ in quite short order. There’s one shortcoming of this plugin and I’d like to ask for some advice on how to structure an improvement.

My use-case is basically generating thumbnails of zipped image archives:

  • Scanning a folder of dozens of archives
  • Getting file contents of archive and sorting by name
  • Unzipping the first file to display as cover art

I’ve noticed that when calling


UZipFileFunctionLibrary::UnzipFileNamedTo

that


InternalCallback

is reset for each call. The result seems to be that if I fire off an unzip operation before an in-progress one completes, I don’t get either of the callbacks and it seems there are some race conditions that cause the file extractions to fail in interesting ways. So while new zip events might fire off on a new thread, I don’t have the ability to queue up more than one at a time.

I’ve only dipped into the project code a bit but am getting more familiar with it. If you have a good idea of how to fix this but don’t have time I’d love to discuss what you think the best approach is to supporting this and perhaps I can send a pull request over.

No current plans to personally add other platform support. This is a fully liberal community plugin though, so by all means consider making pull requests when you get your platform working :slight_smile:

Looking online, there appears to be a project that has ported 7zip to Linux, p7zip p7zip download | SourceForge.net. If you manage to make it compile against GitHub - getnamo/7zip-cpp: Fork of SevenZip++ for modern builds. which is the underlying C++ library used in this plugin, then you could expose the build as a static lib (.a) to this plugin and the rest should work as expected (minus Windows API functions, but those have now been extracted from the 7zip core).

Optionally if you wish to use something like GZip, boost has convenient filters to utilize zlib (e.g. https://gist.github.com/yfnick/6ba33efa7ba12e93b148), which you can expose to the engine as a plugin with the correct libraries (something like Can't link zlib(Boost) to the project - Rendering - Epic Developer Community Forums).

Finally the engine source itself contains zlib as a third party source: https://github.com/EpicGames/UnrealEngine/tree/master/Engine/Source/ThirdParty/zlib, but I’ve never had any success loading engine third party libraries not exposed as modules in projects, YMMV.

Looking at the windows utility, I don’t have a function added yet that lists the content of a folder, I should probably add a function for that (made an issue add Windows API folder listing · Issue #3 · getnamo/ZipUtility-Unreal · GitHub).

To look through an archive on the other hand, you should use the list commands (https://github.com/getnamo/ZipUtility-ue4/blob/master/Source/ZipUtility/Public/ZipFileFunctionLibrary.h#L66), which will give you file names without extracting the contents.

Regarding simultaneous actions, that’s sadly a current shortcoming with the way I originally structured the callback interface for this (didn’t originally consider multiple simultaneous unzips/zips, just that they would happen off-thread to not impact game performance). Ideally each unzip operation should wrap it’s callback object into a handler of sort that would contain all the current progress data as well as a reference to it’s thread/etc which would allow for multiple callbacks and a way to control or early terminate certain operations. Probably holding an array of callback handlers as a static variable on https://github.com/getnamo/ZipUtility-ue4/blob/master/Source/ZipUtility/Public/ZipFileFunctionLibrary.h would suffice and then simply removing the handlers after the operations have finished (the lambda threads already auto-delete on completion).

The core functionality of the plugin is all contained within https://github.com/getnamo/ZipUtility-ue4/blob/master/Source/ZipUtility/Private/ZipFileFunctionLibrary.cpp which wraps the underlying C++ library implementation (GitHub - getnamo/7zip-cpp: Fork of SevenZip++ for modern builds.) with UE4 methods.

Having a pull request that would fix this shortcoming would be awesome :).