Error extracting package .zip using SharpCompress

calebgregory · 16 March 2016 14:32

Hello! I’m deploying a package that’s been compressed using the node octopack module. It is building fine without errors, but when I go to deploy it on Octopus, I get

Extracting package to: C:\Octopus\Applications\Test\dimple\1.0.156
System.NotSupportedException: Unknown header: 56
   at SharpCompress.Common.Zip.ZipHeaderFactory.ReadHeader(UInt32 headerBytes, BinaryReader reader)
   at SharpCompress.Common.Zip.StreamingZipHeaderFactory.<ReadStreamHeader>d__1.MoveNext()
   at SharpCompress.Reader.Zip.ZipReader.<GetEntries>d__6.MoveNext()
   at SharpCompress.Reader.AbstractReader`2.NextEntryForCurrentStream()
   at SharpCompress.Reader.AbstractReader`2.MoveToNextEntry()
   at Calamari.Integration.Packages.ZipPackageExtractor.Extract(String packageFile, String directory, Boolean suppressNestedScriptWarning) in Y:\work\14ffc968155e4956\source\Calamari\Integration\Packages\ZipPackageExtractor.cs:line 19
   at Calamari.Deployment.Conventions.ExtractPackageConvention.Install(RunningDeployment deployment) in Y:\work\14ffc968155e4956\source\Calamari\Deployment\Conventions\ExtractPackageConvention.cs:line 31
   at Calamari.Deployment.ConventionProcessor.RunInstallConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 60
   at Calamari.Deployment.ConventionProcessor.RunConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 28
Running rollback conventions...
Adding journal entry:
<Deployment Id="d49de879-6ae4-4877-924c-d66f95ca7b3e" EnvironmentId="Environments-1" ProjectId="Projects-21" PackageId="dimple" PackageVersion="1.0.156" InstalledOn="2016-03-16 14:12:08" ExtractedFrom="C:\Octopus\Files\dimple.1.0.156.zip-3d1ecfbc-1b9f-4d7b-ac8d-54533ccaf160" ExtractedTo="" RetentionPolicySet="Environments-1/Projects-21/Step-dimple/Machines-1/&lt;default&gt;" CustomInstallationDirectory="C:\inetpub\wwwroot\Dimple" WasSuccessful="False" />
System.ArgumentNullException: Value cannot be null.
Parameter name: path
   at System.IO.DirectoryInfo..ctor(String path)
   at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.EnumerateFiles(String parentDirectoryPath, String[] searchPatterns) in Y:\work\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 181
   at Calamari.Integration.Scripting.PackagedScriptRunner.FindScripts(RunningDeployment deployment) in Y:\work\14ffc968155e4956\source\Calamari\Integration\Scripting\PackagedScriptRunner.cs:line 55
   at Calamari.Integration.Scripting.PackagedScriptRunner.RunScripts(RunningDeployment deployment) in Y:\work\14ffc968155e4956\source\Calamari\Integration\Scripting\PackagedScriptRunner.cs:line 29
   at Calamari.Deployment.ConventionProcessor.RunRollbackConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 69
   at Calamari.Deployment.ConventionProcessor.RunConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 48
   at Calamari.Commands.DeployPackageCommand.Execute(String[] commandLineArguments) in Y:\work\14ffc968155e4956\source\Calamari\Commands\DeployPackageCommand.cs:line 110
   at Calamari.Program.Execute(String[] args) in Y:\work\14ffc968155e4956\source\Calamari\Program.cs:line 38
The remote script failed with exit code 100

This appears to be an issue with SharpCompress rather than Calamari. The exception is getting thrown from https://github.com/adamhathcock/sharpcompress/blob/2a630e04b279b3491fda9276b09a86b8fedbcb3d/SharpCompress/Common/Zip/ZipHeaderFactory.cs#L90. If you need the file for debugging, I’d be happy to send it your way.

Thanks,

Caleb

henrik · 17 March 2016 07:49

Hi Caleb,

Thanks for getting in touch and I’m sorry you are having this issue.

Yes please, if you could send through this package to us that would be very helpful as it will help us investigate why that exception gets thrown.

Could you also tell us a bit more about the machine you used to build the package on?

Thank you and warm regards,
Henrik

calebgregory · 17 March 2016 15:12

Hey Henrik,

The machine I am building the package on is a Windows Server 2012 R2 on AWS Elastic Cloud. The instance size is t2.medium, the architecture is AMD64.

The package is too large to send via email. What is a better way to get it to you?

As for the package, for what it’s worth, I’m able to unzip it on a Mac, but not on a Windows machine. It was made using https://github.com/OctopusDeploy/octopackjs. Here is the publish task in my ‘gulpfile.js’ -

gulp.task('publish', function() {
  return gulp.src(['**/*', '!www/**/*', '!./gulpfile.js'])
    .pipe( octo.pack('zip', { version: config.TEAM_CITY_BUILD_NUMBER }) )
    .pipe( octo.push({ host: config.OCTOPUS_SERVER, apiKey: config.OCTOPUS_API_KEY }) );
});

Thank you Henrik,

Caleb Gregory | Web Application Developer
[cid:987ADB44-384D-4086-85C7-B305E98A9F8B]
1620 Westgate Circle, Suite 100 | Brentwood, TN 37027
P: 615.777.6400
workinstitute.com

CONFIDENTIALITY NOTICE: This email is confidential and solely intended for the recipient(s) named above. If you have received this email and any of its attachments in error, do not read, forward, or disclose any of the information contained within. Please, immediately notify the sender and properly destroy any and all information pertaining to this email correspondence.

henrik · 17 March 2016 22:34

Hi Caleb,

You can upload your package here. Please let me know if you have any issues accessing the URL or uploading your file.

Thank you and warm regards,
Henrik

calebgregory · 17 March 2016 22:51

Hey Henrik,

Uploading now. The package name is dimple.1.0.156.zip.

Best,

Caleb

calebgregory · 18 March 2016 14:52

Hey Henrik,

Just wanted to ask for verification that you got the file. I walked away from the computer while it was uploading and when I came back, it appeared to have uploaded but I didn’t see any indication that it had.

Thank you!

Caleb

henrik · 20 March 2016 23:11

Hi Caleb

Yes, the file uploaded correctly, I’m in the process of troubleshooting the issue as I have replicated in my dev environment.

Hopefully, I’ll have an answer to you today, or tomorrow.

Thank you and warm regards,
Henrik

calebgregory · 25 March 2016 15:30

Hey Henrik,

I’m looking for an update. Any progress?

Caleb

henrik · 29 March 2016 00:37

Hi Caleb,

My sincere apologies for taking so long to get back to you with an update!

I’ve been able to reproduce the error when running octo.pack from gulp, it seems that one of the installed modules under node_modules is causing this issue, as when I remove everything except the ones needed by gulp and octo-pack the resulting zip archive works as expected.

I found another user that has the same error https://github.com/altryne/extensionizr/issues/14, and it was the inject-css npm package that was causing it. Not terribly useful to find the root cause of your issue as you’re not using said npm package.

I’ll keep investigating a bit more to see if I can figure out which folder it is that is causing the error.

Thank you and warm regards,
Henrik

henrik · 29 March 2016 02:45

Hi Caleb,

After a bit more investigating, the issue could be related to the number of files in the archive.

I tested this by excluding the node_modules folder entirely and also by excluding devDevepencies by running npm install --production, both of these tests resulted in a zip file that could be opened on Windows.

I don’t really have a good solution for you at this point in time, as neither of the above tests would work for your current workflow.

Thank you and best regards,
Henrik

branden_devries · 24 August 2016 00:48

Any update on this issue? I’m seeing the same thing. I also think it may be the number of files. My zip has 1714 files, is 66.1 MB compressed, 368.4 MB uncompressed.

17:28:42 Verbose | Octopus Server version: 3.3.15+Branch.master.Sha.477470fd43c6d16974b64f8741787ab248cc3ff9

17:28:42 Info | Deploying package ‘D:\Octopus\Files\med.3.13.0.12.zip-65225afc-3583-4a00-8dcf-5e0e8b404ea6’ to machine ‘https://sd-kp-autinf01.cfnp.local:10933/’

17:28:42 Verbose | Octopus Deploy: Calamari version 3.3.13+Branch.master.Sha.821b4ffaf94e26f64cd9c14be2f3969587822fad

17:28:42 Info | Deploying package: D:\Octopus\Files\med.3.13.0.12.zip-65225afc-3583-4a00-8dcf-5e0e8b404ea6

17:28:43 Verbose | Extracting package to: D:\Octopus\Applications\SD-Auto\med\3.13.0.12

17:28:52 Error | System.NotSupportedException: Unknown header: 1263534083

17:28:52 Error | at SharpCompress.Common.Zip.ZipHeaderFactory.ReadHeader(UInt32 headerBytes, BinaryReader reader)

17:28:52 Error | at SharpCompress.Common.Zip.StreamingZipHeaderFactory.d__1.MoveNext()

17:28:52 Error | at SharpCompress.Reader.Zip.ZipReader.d__6.MoveNext()

17:28:52 Error | at SharpCompress.Reader.AbstractReader`2.NextEntryForCurrentStream()

17:28:52 Error | at SharpCompress.Reader.AbstractReader`2.MoveToNextEntry()

17:28:52 Error | at Calamari.Integration.Packages.ZipPackageExtractor.Extract(String packageFile, String directory, Boolean suppressNestedScriptWarning) in Y:\work\14ffc968155e4956\source\Calamari\Integration\Packages\ZipPackageExtractor.cs:line 19

17:28:52 Error | at Calamari.Deployment.Conventions.ExtractPackageConvention.Install(RunningDeployment deployment) in Y:\work\14ffc968155e4956\source\Calamari\Deployment\Conventions\ExtractPackageConvention.cs:line 33

17:28:52 Error | at Calamari.Deployment.ConventionProcessor.RunInstallConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 60

17:28:52 Error | at Calamari.Deployment.ConventionProcessor.RunConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 28

17:28:52 Error | Running rollback conventions…

17:28:52 Verbose | Adding journal entry:

17:28:52 Verbose |

17:28:52 Error | System.ArgumentNullException: Value cannot be null.

17:28:52 Error | Parameter name: path

17:28:52 Error | at System.IO.DirectoryInfo…ctor(String path)

17:28:52 Error | at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.EnumerateFiles(String parentDirectoryPath, String[] searchPatterns) in Y:\work\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 180

17:28:52 Error | at Calamari.Integration.Scripting.PackagedScriptRunner.FindScripts(RunningDeployment deployment) in Y:\work\14ffc968155e4956\source\Calamari\Integration\Scripting\PackagedScriptRunner.cs:line 55

17:28:52 Error | at Calamari.Integration.Scripting.PackagedScriptRunner.RunScripts(RunningDeployment deployment) in Y:\work\14ffc968155e4956\source\Calamari\Integration\Scripting\PackagedScriptRunner.cs:line 31

17:28:52 Error | at Calamari.Deployment.ConventionProcessor.RunRollbackConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 71

17:28:52 Error | at Calamari.Deployment.ConventionProcessor.RunConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 48

17:28:52 Error | at Calamari.Commands.DeployPackageCommand.Execute(String[] commandLineArguments) in Y:\work\14ffc968155e4956\source\Calamari\Commands\DeployPackageCommand.cs:line 111

17:28:52 Error | at Calamari.Program.Execute(String[] args) in Y:\work\14ffc968155e4956\source\Calamari\Program.cs:line 38

17:28:52 Fatal | The remote script failed with exit code 100

henrik · 30 August 2016 09:49

Hi Branden,

I’m sorry to say that we don’t have an update on this issue, looking at the SharpCompress repository on GitHub it doesn’t look like any work has been to resolve this issue.

I still have the original package so what I’ll do over the next few days is to do some testing and see if I can get to the bottom of this issue.

Thank you and best regards,
Henrik

CID-DevOps · 9 November 2016 12:14

Hi at all,

any updates about this topic?
We’ve got the same error during a deployment.

Posted by branden.devries on Aug 24, 2016 @ 02:48 AM

I also think it may be the number of files.

So I think that error is maybe not relate to the count of files in the packages.
This is our package data:

Serverversion: 3.4.11
Packagetype: Nuget
Size (compressed): ~ 10GB
Size (uncompressed): ~ 15GB
Filecount: 20

And here the deployment error:

Extracting package to: C:\Octopus\Applications\INT-TEST\CID.KnowledgeBaseMongo\8.0.0.636-v20161103
12:28:26   Error    |       System.NotSupportedException: Unknown header: 25771494
12:28:26   Error    |       at SharpCompress.Common.Zip.ZipHeaderFactory.ReadHeader(UInt32 headerBytes, BinaryReader reader)
12:28:26   Error    |       at SharpCompress.Common.Zip.StreamingZipHeaderFactory.<ReadStreamHeader>d__1.MoveNext()
12:28:26   Error    |       at SharpCompress.Reader.Zip.ZipReader.<GetEntries>d__6.MoveNext()
12:28:26   Error    |       at SharpCompress.Reader.AbstractReader`2.MoveToNextEntry()
12:28:26   Error    |       at Calamari.Integration.Packages.NuGet.NupkgExtractor.Extract(String packageFile, String directory, Boolean suppressNestedScriptWarning) in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Integration\Packages\NuGet\NupkgExtractor.cs:line 63
12:28:26   Error    |       at Calamari.Deployment.Conventions.ExtractPackageConvention.Install(RunningDeployment deployment) in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Deployment\Conventions\ExtractPackageConvention.cs:line 31
12:28:26   Error    |       at Calamari.Deployment.ConventionProcessor.RunInstallConventions() in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 60
12:28:26   Error    |       at Calamari.Deployment.ConventionProcessor.RunConventions() in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 28
12:28:26   Error    |       Running rollback conventions...

Vanessa_Love · 17 November 2016 23:50

Hi,

In 3.5.2 we changed from ZipReader to ZipArchive in an attempt to help with this issue. If anyone has or can upgrade we would love feedback on if this has helped.

Vanessa

narayana_rengaswamy · 23 November 2016 06:14

I had the same error earlier, and it has been fixed with 3.5.2
Thanks!

Vanessa_Love · 25 November 2016 01:20

Hi @Narayana,

Thanks for the update and confirmation. Anyone else please let me know if this still occurs after upgrading past 3.5.2.

Vanessa

narayana_rengaswamy · 25 November 2016 14:48

I think I spoke too soon. I’m still getting the error

05:11:15   Verbose  |       Octopus Server version: 3.5.2+Branch.master.Sha.33e995aab37d118b4e9b33289d0c955b12bd997f
05:11:15   Info     |       Deploying package 'C:\Octopus\Files\WebApp.2016.12.0.16032.nupkg-a017377c-e738-4c32-b701-a3341080f664' to machine 'https://is-gnv-hpapp02:10933/'
05:11:17   Verbose  |       Octopus Deploy: Calamari version 0.0.0
05:11:17   Info     |       Deploying package:    C:\Octopus\Files\WebApp.2016.12.0.16032.nupkg-a017377c-e738-4c32-b701-a3341080f664
05:11:19   Verbose  |       Extracting package to: C:\Octopus\Applications\SAIA\QA\WebApp\2016.12.0.16032
05:11:19   Error    |       SharpCompress.Compressor.Deflate.ZlibException: Bad state (invalid block type)
05:11:19   Error    |       at SharpCompress.Compressor.Deflate.InflateManager.Inflate(FlushType flush)
05:11:19   Error    |       at SharpCompress.Compressor.Deflate.ZlibBaseStream.Read(Byte[] buffer, Int32 offset, Int32 count)
05:11:19   Error    |       at SharpCompress.Common.EntryStream.Read(Byte[] buffer, Int32 offset, Int32 count)
05:11:19   Error    |       at SharpCompress.Common.EntryStream.SkipEntry()
05:11:19   Error    |       at SharpCompress.Common.EntryStream.Dispose(Boolean disposing)
05:11:19   Error    |       at System.IO.Stream.Close()
05:11:19   Error    |       at SharpCompress.Reader.AbstractReader`2.Write(Stream writeStream)
05:11:19   Error    |       at SharpCompress.Reader.AbstractReader`2.WriteEntryTo(Stream writableStream)
05:11:19   Error    |       at SharpCompress.Reader.IReaderExtensions.WriteEntryToFile(IReader reader, String destinationFileName, ExtractOptions options)
05:11:19   Error    |       at Calamari.Integration.Packages.NuGet.NupkgExtractor.Extract(String packageFile, String directory, Boolean suppressNestedScriptWarning) in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Integration\Packages\NuGet\NupkgExtractor.cs:line 57
05:11:19   Error    |       at Calamari.Deployment.Conventions.ExtractPackageConvention.Install(RunningDeployment deployment) in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Deployment\Conventions\ExtractPackageConvention.cs:line 31
05:11:19   Error    |       at Calamari.Deployment.ConventionProcessor.RunInstallConventions() in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 60
05:11:19   Error    |       at Calamari.Deployment.ConventionProcessor.RunConventions() in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 28
05:11:19   Error    |       Running rollback conventions...
05:11:19   Verbose  |       Adding journal entry:
05:11:19   Verbose  |       <Deployment Id="58f96260-41f6-405d-b878-4b06da938f87" EnvironmentId="Environments-2" TenantId="Tenants-21" ProjectId="Projects-1" PackageId="WebApp" PackageVersion="2016.12.0.16032" InstalledOn="2016-11-25 13:11:19" ExtractedFrom="C:\Octopus\Files\WebApp.2016.12.0.16032.nupkg-a017377c-e738-4c32-b701-a3341080f664" ExtractedTo="" RetentionPolicySet="Tenants-21/Environments-2/Projects-1/Step-Deploy DSS/Machines-82/&lt;default&gt;" CustomInstallationDirectory="" WasSuccessful="False" />
05:11:19   Error    |       System.ArgumentNullException: Value cannot be null.
05:11:19   Error    |       Parameter name: path
05:11:19   Error    |       at System.IO.DirectoryInfo..ctor(String path)
05:11:19   Error    |       at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.EnumerateFiles(String parentDirectoryPath, String[] searchPatterns) in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 183
05:11:19   Error    |       at Calamari.Integration.Scripting.PackagedScriptRunner.FindScripts(RunningDeployment deployment) in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Integration\Scripting\PackagedScriptRunner.cs:line 55
05:11:19   Error    |       at Calamari.Integration.Scripting.PackagedScriptRunner.RunScripts(RunningDeployment deployment) in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Integration\Scripting\PackagedScriptRunner.cs:line 29
05:11:19   Error    |       at Calamari.Deployment.ConventionProcessor.RunRollbackConventions() in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 69
05:11:19   Error    |       at Calamari.Deployment.ConventionProcessor.RunConventions() in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 48
05:11:19   Error    |       at Calamari.Commands.DeployPackageCommand.Execute(String[] commandLineArguments) in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Commands\DeployPackageCommand.cs:line 115
05:11:19   Error    |       at Calamari.Program.Execute(String[] args) in Z:\BuildAgent\work\14ffc968155e4956\source\Calamari\Program.cs:line 40

narayana_rengaswamy · 25 November 2016 16:02

I’ve updated to latest OctoPack (3.4.6) and latest Octopus, and changed download option to individually on the tentacles. I’m not sure if this is a coincidence, but one of this seems to fix the issue for me, but I don’t know if it will recur.