Hi,
I seem to be getting the following errors while using the latest beta version of octopus to deploy zip files. Seems to me like there is some problems encountered while extracting zip files.
System.NotSupportedException: Memory stream is not expandable.
20:48:40Error
at System.IO.MemoryStream.set_Capacity(Int32 value)
20:48:40Error
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
20:48:40Error
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
20:48:40Error
at SharpCompress.IO.RewindableStream.Read(Byte[] buffer, Int32 offset, Int32 count)
20:48:40Error
at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
20:48:40Error
at System.IO.BinaryReader.ReadUInt32()
20:48:40Error
at SharpCompress.Common.Zip.StreamingZipHeaderFactory.d__1.MoveNext()
20:48:40Error
at SharpCompress.Reader.Zip.ZipReader.d__6.MoveNext()
20:48:40Error
at SharpCompress.Reader.AbstractReader2.NextEntryForCurrentStream() 20:48:40Error at SharpCompress.Reader.AbstractReader2.MoveToNextEntry()
20:48:40Error
at Calamari.Integration.Packages.ZipPackageExtractor.Extract(String packageFile, String directory, Boolean suppressNestedScriptWarning) in Y:\work\14ffc968155e4956\source\Calamari\Integration\Packages\ZipPackageExtractor.cs:line 19
20:48:40Error
at Calamari.Deployment.Conventions.ExtractPackageConvention.Install(RunningDeployment deployment) in Y:\work\14ffc968155e4956\source\Calamari\Deployment\Conventions\ExtractPackageConvention.cs:line 33
20:48:40Error
at Calamari.Deployment.ConventionProcessor.RunInstallConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 60
20:48:40Error
at Calamari.Deployment.ConventionProcessor.RunConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 28
20:48:40Error
Running rollback conventions…
20:48:40Error
System.ArgumentNullException: Value cannot be null.
20:48:40Error
Parameter name: path
20:48:40Error
at System.IO.DirectoryInfo…ctor(String path)
20:48:40Error
at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.EnumerateFiles(String parentDirectoryPath, String[] searchPatterns) in Y:\work\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 181
20:48:40Error
at Calamari.Integration.Scripting.PackagedScriptRunner.FindScripts(RunningDeployment deployment) in Y:\work\14ffc968155e4956\source\Calamari\Integration\Scripting\PackagedScriptRunner.cs:line 55
20:48:40Error
at Calamari.Integration.Scripting.PackagedScriptRunner.RunScripts(RunningDeployment deployment) in Y:\work\14ffc968155e4956\source\Calamari\Integration\Scripting\PackagedScriptRunner.cs:line 31
20:48:40Error
at Calamari.Deployment.ConventionProcessor.RunRollbackConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 71
20:48:40Error
at Calamari.Deployment.ConventionProcessor.RunConventions() in Y:\work\14ffc968155e4956\source\Calamari\Deployment\ConventionProcessor.cs:line 48
20:48:40Error
at Calamari.Commands.DeployPackageCommand.Execute(String[] commandLineArguments) in Y:\work\14ffc968155e4956\source\Calamari\Commands\DeployPackageCommand.cs:line 113
20:48:40Error
at Calamari.Program.Execute(String[] args) in Y:\work\14ffc968155e4956\source\Calamari\Program.cs:line 38
20:48:40Fatal
The remote script failed with exit code 100
I solved this by rolling out a custom version of calamari. While doing this i also came across another issue with custom calamari. Seems like the tentacle is unable to accept a custom calamari properly anymore. It loads in the custom calamari and deletes it for some reason. I got it to deploy my custom version of calamari by replacing the one in C:\Program Files\Octopus Deploy\Octopus.
Just to add on to this, i replaced the original sharpcompress with ionic.zip. It seems like zip files generated from the teamcity compress plugin doesn’t work too well with sharpcompress.
If anyone is interested in the fixed version it’s here
Hi,
Thanks for trying out our new 3.3 build.
It indeed seems as though you must be hitting upon a bug within the sharpcompress library that we use for tar\zip\gzip extraction. We have tested across many zip packages with various compression methods but so far been unable to replicate the failure.
Would it be possible to get you to upload the package that was causing this failure to occur? We have a secure location at https://file.ac/iitU4hihCFg/ that you can upload this package to which no-one else but us can access. Alternatively you can attach and email them through to us at support@octopus.com. Feel free to create another sample zip package if you wish with some dummy contents, that exhibits the same behaviour.
If we can get to the bottom of this problem hopefully we can easily fix it to prevent it becoming an issue for other users. In the meantime I will further investigate the other issue you encountered regarding using a custom Calamari build.
Thanks again for getting in touch, and bringing these problems to our attention.
Cheers,
Rob
Hi again,
I have managed to replicate the failing problem you had with a custom Calamari build refusing to be used by the Tentacle. I have created a GitHub ticket for you to track its progress as it gets resolved.
Thanks again for these issues! We should have most of these kinks ironed out of the beta before full release.
Cheers,
Rob
Fanatastic! how about the compressed file issue? I think the compression library bundled with octopus might not be supporting a full range of zip files. Will you guys put in another zip library or improve this somehow?
Ah sorry please disregard my previous post on the compression library. I missed out the first reply. Will try to send you a sample of the affected zip file. Mine is generated using the compress plugin in teamcity (our packages are prepared by teamcity)
Hi,
Thanks for the extra information, I have just downloaded and tested with the compress plugin that you mentioned. It appears to be able to extract just fine without any problems. Perhaps once I get the offending package I will be able to reproduce the issue and investigate what sort of fix needs to be put in to SharpCompress.
Thanks again for sending this through.
Cheers,
Rob
I actually stripped off all the files within the zip file and it is still crashing. Most likely it has something to do with the way the headers are constructed.
I tested the file with a very small piece of code which runs sharpcompress v0.11.4
static void Main(string[] args)
{
var file = @"file.zip";
using (var inStream = new FileStream(file, FileMode.Open, FileAccess.Read))
using (var reader = ZipReader.Open(inStream))
{
while (reader.MoveToNextEntry())
{
}
}
}
Hi Jin,
Thanks for providing the zip file. Unfortunately Octopus Deploy is still extracting it correctly during deployments, despite being empty. I have also attempted to run the script you provided with sharpcompress 0.11.4 and again, with the file you provided it runs just fine. I did have some odd behaviour when trying to open the zip via finder on a Mac as described here. Opening in windows or with 7zip seemed to show the empty folder so the download itself seems ok, but perhaps archived in an unexpected format.
I cant rule out a problem with this external library but would love to reproduce it first before trying to replace or fix.
Perhaps I might be able to discover something If we schedule a support call. Perhaps looking at the packing and deployment directly we might discover the missing piece that allows us to hit the same bug.
Thanks again for reporting this issue, please let us know if you have any questions about setting up this call.
Thanks,
Rob
Sorry I didn’t get back to you sooner. It seems like the problem went away after updating to beta 2 so it could be already fixed. Although I found it strange that the library did not unzip even a blank file on my development machine
Hi Jin,
Great to hear that you no longer have this problem. Please don’t hesitate to contact us again if you see this problem recur. As we are using a 3rd party compression library we also need to make sure we keep up to date with all its updates so that fixes get passed straight on to our users.
Thanks again,
Robert
We got the same problem here. The package cannot be fully extracted and stops with the expandable memory error message. Our package is quite big as of now (140MB compressed to 85 MB) and our biggest files are around 6 MB.
Update #1: Even with a reduced size package (85MB compressed to 32 MB), it fails.
Update #2: We also use TeamCity to generate the zip file.
We JUST started experiencing this (we also use TeamCity builds to generate our ZIP package) and even after upgrading to the latest OctoDeploy (v3.5.1) and Calamari (v3.5.3) across our agents, we are still experiencing the error: System.NotSupportedException: Memory stream is not expandable
We are getting the same error with our .zip files inside our packages.
If we change the file extention to something else than .zip, it fixes the issue.
So instead of JDK/java8.zip we have JDK/java8.hest this works.
Hi Brett,
Its possible that this may be a problem with the third party library sharpcompress which we use for package extraction.
So that we can try and debug through this problem, would it be possible to get you to send the package (or any with the same probem) through to us. You can securely upload to via this address: https://file.ac/Lsy2rThn9LI/ which will only be accessable to ourselves.
Once we are able to reliably reproduce the issues, we may be able to patch the problem.
Thanks in advance,
Robert
Hi Jim,
The fact that it has been working fine until recently tells me that there must be something either in your build environment or bundled source files that has changed. Without the zip package it is hard for us to diagnose what the problem may be since these sorts of issues are typically problems with the way in which the zip specifications are followed (either by the compressor or extractor).
Based on your error message my guess is that it may have been fixed as part of this previous GitHub issue that was released in 3.5.2. Please upgrade to the latest Octopus version and let us know if you are still experiencing this problem.