Announcement

Collapse
No announcement yet.

[Plugin] ZipUtility (7zip)

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    #16
    Originally posted by getnamo View Post
    Is this @ the latest master branch commit? Might be that the pre-requisites are desynced. Let me know which version is broken and I'll have a look.
    I am using 4.13.1 and the latest release from this forum post.

    Comment


      #17
      Update to 0.2.0
      -Windows API functions split into WindowsFileUtility module
      -Added a folder watcher interface (IFolderWatchInterface) and bp library function which will allow you to watch a folder for file changes and respond to such events e.g. directory changed or file changed
      -ZipUtility trimmed to only zipping/archive functions
      -Thanks to #1 @deams51 now supports extracting a single file
      -Unreal engine 4.14 compile fixes

      This update cleans some of my utility functions into a separate Windows API module and adds a new folder watcher that will emit events when something changes in the folder, e.g. file saved or directory created.

      Grab latest at https://github.com/getnamo/ZipUtility-ue4/releases
      Last edited by getnamo; 12-04-2016, 08:34 PM.
      Plugins: TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense - CIM

      Comment


        #18
        Do you have any plans in making it cross-platform? Using gzip on unix system for example, there are plenty of libs that could be used.
        Sr. Engine Programmer @ www.playspace.com - moritzwundke.com
        Remember: be polite and respect other peoples opinions - Join the Unofficial Unreal Discord Channel - Found a bug? Then use the Bug Report Form to get it fixed ^^

        Comment


          #19
          Hey thanks a bunch for this project. I got a basic implementation up in C++ in quite short order. There's one shortcoming of this plugin and I'd like to ask for some advice on how to structure an improvement.

          My use-case is basically generating thumbnails of zipped image archives:
          • Scanning a folder of dozens of archives
          • Getting file contents of archive and sorting by name
          • Unzipping the first file to display as cover art


          I've noticed that when calling
          Code:
          UZipFileFunctionLibrary::UnzipFileNamedTo
          that
          Code:
          InternalCallback
          is reset for each call. The result seems to be that if I fire off an unzip operation before an in-progress one completes, I don't get either of the callbacks and it seems there are some race conditions that cause the file extractions to fail in interesting ways. So while new zip events might fire off on a new thread, I don't have the ability to queue up more than one at a time.

          I've only dipped into the project code a bit but am getting more familiar with it. If you have a good idea of how to fix this but don't have time I'd love to discuss what you think the best approach is to supporting this and perhaps I can send a pull request over.
          Dev Blog http://www.error454.com
          Arconia: Twinstick Shooter Adventure

          Comment


            #20
            Originally posted by Moss View Post
            Do you have any plans in making it cross-platform? Using gzip on unix system for example, there are plenty of libs that could be used.
            No current plans to personally add other platform support. This is a fully liberal community plugin though, so by all means consider making pull requests when you get your platform working

            Looking online, there appears to be a project that has ported 7zip to Linux, p7zip https://sourceforge.net/projects/p7zip/. If you manage to make it compile against https://github.com/getnamo/7zip-cpp which is the underlying C++ library used in this plugin, then you could expose the build as a static lib (.a) to this plugin and the rest should work as expected (minus Windows API functions, but those have now been extracted from the 7zip core).

            Optionally if you wish to use something like GZip, boost has convenient filters to utilize zlib (e.g. https://gist.github.com/yfnick/6ba33efa7ba12e93b148), which you can expose to the engine as a plugin with the correct libraries (something like https://answers.unrealengine.com/que...e-project.html).

            Finally the engine source itself contains zlib as a third party source: https://github.com/EpicGames/UnrealE...hirdParty/zlib, but I've never had any success loading engine third party libraries not exposed as modules in projects, YMMV.

            Originally posted by hyperdr1ve View Post
            Hey thanks a bunch for this project. I got a basic implementation up in C++ in quite short order. There's one shortcoming of this plugin and I'd like to ask for some advice on how to structure an improvement.

            My use-case is basically generating thumbnails of zipped image archives:
            • Scanning a folder of dozens of archives
            • Getting file contents of archive and sorting by name
            • Unzipping the first file to display as cover art


            I've noticed that when calling
            Code:
            UZipFileFunctionLibrary::UnzipFileNamedTo
            that
            Code:
            InternalCallback
            is reset for each call. The result seems to be that if I fire off an unzip operation before an in-progress one completes, I don't get either of the callbacks and it seems there are some race conditions that cause the file extractions to fail in interesting ways. So while new zip events might fire off on a new thread, I don't have the ability to queue up more than one at a time.

            I've only dipped into the project code a bit but am getting more familiar with it. If you have a good idea of how to fix this but don't have time I'd love to discuss what you think the best approach is to supporting this and perhaps I can send a pull request over.
            Looking at the windows utility, I don't have a function added yet that lists the content of a folder, I should probably add a function for that (made an issue https://github.com/getnamo/ZipUtility-ue4/issues/3).

            To look through an archive on the other hand, you should use the list commands (https://github.com/getnamo/ZipUtilit...nLibrary.h#L66), which will give you file names without extracting the contents.

            Regarding simultaneous actions, that's sadly a current shortcoming with the way I originally structured the callback interface for this (didn't originally consider multiple simultaneous unzips/zips, just that they would happen off-thread to not impact game performance). Ideally each unzip operation should wrap it's callback object into a handler of sort that would contain all the current progress data as well as a reference to it's thread/etc which would allow for multiple callbacks and a way to control or early terminate certain operations. Probably holding an array of callback handlers as a static variable on https://github.com/getnamo/ZipUtilit...ctionLibrary.h would suffice and then simply removing the handlers after the operations have finished (the lambda threads already auto-delete on completion).

            The core functionality of the plugin is all contained within https://github.com/getnamo/ZipUtilit...ionLibrary.cpp which wraps the underlying C++ library implementation (https://github.com/getnamo/7zip-cpp) with UE4 methods.

            Having a pull request that would fix this shortcoming would be awesome .
            Plugins: TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense - CIM

            Comment


              #21
              Update to 0.2.2
              -Thanks to #4 @hyperdr1ve the plugin now supports multiple simultaneous operations working independently on their own threads with proper callbacks.
              -Added windows listing folder content function with FileListInterface expected to receive the content data

              Example of listing contents of a folder:



              Note that self should respond to a FileListInterface.

              releases found at the usual place https://github.com/getnamo/ZipUtility-ue4/releases
              Last edited by getnamo; 04-27-2017, 10:50 AM.
              Plugins: TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense - CIM

              Comment


                #22
                I'll be taking a deeper look at the memory management of the new async architecture this week. In the meantime I wanted to ask about 7z library.

                I noticed that some archives of mine that were created on Linux contain illegal file names for windows. When I unzip these, 7z seems to automatically replace the invalid characters with underscores. The problem is that I use the archive listing functions to generate the archive contents map, so my map contains the illegal characters and I can't find the files after extraction.

                I didn't see a 7z call that could convert an invalid file name to a valid one. Do you know anything about where this happens? I may need to write some transfer functions if they're not accessible.
                Dev Blog http://www.error454.com
                Arconia: Twinstick Shooter Adventure

                Comment


                  #23
                  Originally posted by hyperdr1ve View Post
                  I'll be taking a deeper look at the memory management of the new async architecture this week. In the meantime I wanted to ask about 7z library.

                  I noticed that some archives of mine that were created on Linux contain illegal file names for windows. When I unzip these, 7z seems to automatically replace the invalid characters with underscores. The problem is that I use the archive listing functions to generate the archive contents map, so my map contains the illegal characters and I can't find the files after extraction.

                  I didn't see a 7z call that could convert an invalid file name to a valid one. Do you know anything about where this happens? I may need to write some transfer functions if they're not accessible.
                  Great, making sure we gc appropriately and don't have memory leaks would be helpful.

                  Not sure about the name conversion. The underlying library is found here: https://github.com/getnamo/7zip-cpp which links to the raw 7z sources here: https://github.com/keithjjones/7z/tr...736ffa1b3a180e

                  You should have plenty of access to modify and add any missing underlying functionality. Note that we use the COM-port approach to interacting with the 7z dll and that there is a UE4 specific branch that has slightly different includes for the lib to hide windows data types.
                  Plugins: TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense - CIM

                  Comment


                    #24
                    Update to 0.3.0
                    -Annoying windows header fixes for 4.15

                    grab the latest at the usual
                    https://github.com/getnamo/ZipUtility-ue4/releases
                    Plugins: TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense - CIM

                    Comment


                      #25
                      Dear, can you add a sample of C ++?Click image for larger version

Name:	QQ图片20170425164152.png
Views:	1
Size:	833 Bytes
ID:	1126859
                      I have a question,How to get a OnProgress Event in C++;
                      Last edited by StaticMao; 04-26-2017, 06:03 AM.

                      Comment


                        #26
                        Originally posted by StaticMao View Post
                        Dear, can you add a sample of C ++?[ATTACH=CONFIG]138415[/ATTACH]
                        I have a question,How to get a OnProgress Event in C++;
                        Simple C++:

                        Let's say you have a class called UMyClass. You then add the IZipUtilityInterface to it via multiple inheritance e.g.

                        Code:
                        class UMyClass : public UObject, public IZipUtilityInterface
                        {
                           GENERATED_BODY()
                            ...
                        };
                        Because the events are BlueprintNativeEvent you add the C++ implementation of the events like so

                        Code:
                        class UMyClass : public UObject, public IZipUtilityInterface
                        {
                            GENERATED_BODY()
                            ...
                        
                            //event overrides
                            virtual void OnProgress_Implementation(const FString& archive, float percentage, int32 bytes) override;
                            virtual void OnDone_Implementation(const FString& archive, EZipUtilityCompletionState CompletionState) override;
                            virtual void OnStartProcess_Implementation(const FString& archive, int32 bytes) override;
                            virtual void OnFileDone_Implementation(const FString& archive, const FString& file) override;
                            virtual void OnFileFound_Implementation(const FString& archive, const FString& file, int32 size) override;
                        };
                        and in your c++ file you add the implementation

                        Code:
                        void UMyClass::OnProgress_Implementation(const FString& archive, float percentage, int32 bytes)
                        {
                            //your code here
                        }
                        To call a ziputility function you first get a valid pointer to your class (I leave that up to you) e.g.

                        Code:
                        UMyClass* MyZipClass = NewObject<UMyClass>(); //or you may already have a valid pointer from allocating elsewhere
                        then you pass the pointer to your class with the IZipUtilityInterface as your second parameter (and if you use them, any other optional parameters such as compression format)

                        Code:
                        UZipFileFunctionLibrary::Unzip(FString("C:/path/to/your/zip.7z"), MyZipClass);
                        and then you should receive your callbacks!

                        Optional Method using Lambda Expressions:
                        There is a lambda method I've added a while back https://github.com/getnamo/ZipUtilit...mbdaDelegate.h used here: https://github.com/getnamo/ZipUtilit...brary.cpp#L457

                        The signature is

                        Code:
                        static bool UnzipWithLambda(	const FString& ArchivePath,
                        							TFunction<void()> OnDoneCallback,
                        							TFunction<void(float)> OnProgressCallback = nullptr,
                        							TEnumAsByte<ZipUtilityCompressionFormat> format = COMPRESSION_FORMAT_UNKNOWN);
                        example using this

                        Code:
                        UZipFileFunctionLibrary::UnzipWithLambda(FString("C:/path/to/your/zip.7z"),
                            []()
                            {
                                 //Called when done
                            },
                            [](float Percent)
                            {
                                 //called when progress updates with % done
                            });
                        or

                        Code:
                        UZipFileFunctionLibrary::UnzipWithLambda(FString("C:/path/to/your/zip.7z"), nullptr, [](float Percent)
                            {
                                 //called when progress updates with % done
                            });
                        if you're not interested in the done callback.

                        Let me know if that helps

                        Edit:
                        I've added these instructions on the github readme:
                        https://github.com/getnamo/ZipUtility-ue4#c
                        Last edited by getnamo; 04-28-2017, 09:10 PM. Reason: expanded answer and added documentation
                        Plugins: TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense - CIM

                        Comment


                          #27
                          very clarity!thanks big god。

                          Comment


                            #28
                            Originally posted by StaticMao View Post
                            very clarity!thanks big god。
                            [MENTION=35357]StaticMao[/MENTION] 不客气
                            Plugins: TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense - CIM

                            Comment


                              #29
                              what!你是中国人啊

                              Comment


                                #30
                                Originally posted by StaticMao View Post
                                what!你是中国人啊
                                我不是,但由于技术,世界很小
                                Plugins: TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo - RealSense - CIM

                                Comment

                                Working...
                                X