FileFormatException when deploying via SSH

Hi

I was happilly deploying to my CentOS VM then sometime after updating mono (v4.4.2) and calamari (v3.3.5) deployments no longer work. Below is the long deploy log. Can you please assist?

Thanks

20:15:23
Verbose
Octopus Server version: 3.3.5+Branch.master.Sha.fd25ae4d94c0f38b428602d314795be340e7c2ff
20:15:23
Info
Deploying package '$HOME/.octopus/OctopusServer/Files/MyDeployment.1.2.41.nupkg-50ec7d65-ce80-4f74-9042-547445936a3a' to machine 'ssh://myserver.australiaeast.cloudapp.azure.com/'
20:15:23
Verbose
Requesting upload...
20:15:23
Verbose
Establishing SSH connection...
20:15:24
Verbose
SSH connection established
20:15:24
Verbose
Beginning streaming transfer of variables.json to $HOME\.octopus\OctopusServer\Work\20160805101523-334
20:15:24
Verbose
Establishing SFTP connection...
20:15:25
Verbose
SSFTP connection established
20:15:26
Verbose
Stream transfer complete
20:15:26
Verbose
Requesting upload...
20:15:26
Verbose
Beginning streaming transfer of calamari-deploy-package.sh to $HOME\.octopus\OctopusServer\Work\20160805101523-334
20:15:26
Verbose
Stream transfer complete
20:15:27
Verbose
Octopus Deploy: Calamari version 3.3.5+Branch.master.Sha.755501eda21d63861022f1a2d87c98d5e7f9f23b
20:15:28
Info
Deploying package:    /home/octopus/.octopus/OctopusServer/Files/MyDeployment.1.2.41.nupkg-50ec7d65-ce80-4f74-9042-547445936a3a
20:15:29
Error
System.IO.FileFormatException: The specified archive is invalid.
20:15:29
Error
  at System.IO.Packaging.Package.OpenCore (System.IO.Stream stream, FileMode packageMode, FileAccess packageAccess, Boolean ownsStream) <0x405f4550 + 0x001d7> in <filename unknown>:0 
20:15:29
Error
  at System.IO.Packaging.Package.Open (System.IO.Stream stream, FileMode packageMode, FileAccess packageAccess, Boolean ownsStream) <0x405f4510 + 0x00023> in <filename unknown>:0 
20:15:29
Error
  at System.IO.Packaging.Package.Open (System.String path, FileMode packageMode, FileAccess packageAccess, FileShare packageShare) <0x405f03c0 + 0x000e7> in <filename unknown>:0 
20:15:29
Error
  at Calamari.Integration.Packages.OpenPackagingConventionExtractor.GetMetadata (System.String packageFile) <0x405f0320 + 0x0002f> in <filename unknown>:0 
20:15:29
Error
  at Calamari.Integration.Packages.GenericPackageExtractor.GetMetadata (System.String packageFile) <0x405ecf80 + 0x00029> in <filename unknown>:0 
20:15:29
Error
  at Calamari.Deployment.Conventions.ExtractPackageConvention.Install (Calamari.Deployment.RunningDeployment deployment) <0x405ece10 + 0x0005f> in <filename unknown>:0 
20:15:29
Verbose
Adding journal entry:
20:15:29
Verbose
<Deployment Id="c190ec55-e37e-4ccb-859a-7bf7cf9f6d7b" EnvironmentId="Environments-1" ProjectId="Projects-121" PackageId="MyDeployment" PackageVersion="1.2.41" InstalledOn="2016-08-05 10:15:29" ExtractedFrom="/home/octopus/.octopus/OctopusServer/Files/MyDeployment.1.2.41.nupkg-50ec7d65-ce80-4f74-9042-547445936a3a" ExtractedTo="" RetentionPolicySet="Environments-1/Projects-121/Step-Deploy/Machines-61/&lt;default&gt;" CustomInstallationDirectory="" WasSuccessful="False" />
20:15:29
Error
  at Calamari.Deployment.ConventionProcessor.RunInstallConventions () <0x405ca320 + 0x0006d> in <filename unknown>:0 
20:15:29
Error
  at Calamari.Deployment.ConventionProcessor.RunConventions () <0x405ca1c0 + 0x0001f> in <filename unknown>:0 
20:15:29
Error
Running rollback conventions...
20:15:29
Error
System.ArgumentNullException: Value cannot be null.
20:15:29
Error
Parameter name: path
20:15:29
Error
  at System.IO.FileSystemInfo.CheckPath (System.String path) <0x7fd17c1400d0 + 0x000e5> in <filename unknown>:0 
20:15:29
Error
  at System.IO.DirectoryInfo..ctor (System.String path, Boolean simpleOriginalPath) <0x7fd17c136820 + 0x00027> in <filename unknown>:0 
20:15:29
Error
  at System.IO.DirectoryInfo..ctor (System.String path) <0x7fd17c136800 + 0x00013> in <filename unknown>:0 
20:15:29
Error
  at (wrapper remoting-invoke-with-check) System.IO.DirectoryInfo:.ctor (string)
20:15:29
Error
  at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.EnumerateFiles (System.String parentDirectoryPath, System.String[] searchPatterns) <0x405f7640 + 0x00066> in <filename unknown>:0 
20:15:29
Error
  at Calamari.Integration.Scripting.PackagedScriptRunner.FindScripts (Calamari.Deployment.RunningDeployment deployment) <0x405f6580 + 0x0014b> in <filename unknown>:0 
20:15:29
Error
  at Calamari.Integration.Scripting.PackagedScriptRunner.RunScripts (Calamari.Deployment.RunningDeployment deployment) <0x405f6350 + 0x00033> in <filename unknown>:0 
20:15:29
Error
  at Calamari.Deployment.Conventions.RollbackScriptConvention.Rollback (Calamari.Deployment.RunningDeployment deployment) <0x405f6320 + 0x00013> in <filename unknown>:0 
20:15:29
Error
  at Calamari.Deployment.ConventionProcessor.RunRollbackConventions () <0x405f61a0 + 0x0006d> in <filename unknown>:0 
20:15:29
Error
  at Calamari.Deployment.ConventionProcessor.RunConventions () <0x405ca1c0 + 0x00117> in <filename unknown>:0 
20:15:29
Error
  at Calamari.Commands.DeployPackageCommand.Execute (System.String[] commandLineArguments) <0x405a8620 + 0x01027> in <filename unknown>:0 
20:15:29
Verbose
Disposing SFTP connection...
20:15:29
Verbose
Disposing SSH connection...
20:15:30
Fatal
The remote script failed with exit code 100

Hi Steve,
Looking at some of the release notes for mono 4.4.2 it appears as though there have been a lot of changes to the System.IO base libraries used for archiving. Some further investigation will be needed to pinpoint exactly where their bug lies but at the moment it seems that this release might be a bit unstable. Is it possible for you to roll back to an earlier version of mono in the meantime?
I will let you know when any further information comes to light.
Cheers
Rob

Thanks Rob. Rolling back Mono to v4.4.1 didn’t work. But then I rolled back to v4.2.4 and it worked. HTH. I look forward to hearing from you about the fix for 4.4.x.

Hi Steve,
I’m glad to hear you have your deployments working again.
I have updated our testing matrix of Calamari to include a fresh AWS build of Centos 7.2 with Mono 4.4.2 and running tests against the latest version of Calamari v3.3.32 seems to be extracting packages fine so its entirely possible that there is something different with the package itself.
Would it be possible for you to upload your problem package to the following location: https://file.ac/dIqaBpEoTrQ/ ? This upload directory is only accessable to ourselves however it is understandable if you are unable to do so.
Im also wondering if there is something different that has occurred as part of your update process that doesn’t break if its a clean install. Is this something you would also be able to test (even if its just to a local\temporary VM)
Thanks for the extra info,
Robert

Thanks again Rob.

Ok, maybe changes I’ve made are conflicting. I have been experimenting with Docker and thought maybe the Dockerfiles’ lack of a file extension might be the cause but even without the Dockerfile’s there it fails. I’ve uploaded the package to the url you linked too, hopefully that helps (filename: LogsApiMono.1.2.61.nupkg-5f8dbef8-4c11-40cb-885e-497f9c431ed7). Please delete the package once you’ve finished with it.

When you say a clean install. Do you mean a clean install of the package, mono, or of the OS?

Let me know how you go. Cheers
Steve

Hi Rob. Have you had a chance to try the package I uploaded for yourself? I’m keen to know if you have the same issue with my package so that I can use the latest version of mono.

Thanks
Steve

I am currently out of the office until Monday the 29th of August.

I will review your message on my return. If you need immediate assistance, please contact Phil Brightman (phillip.brightman@havenpower.com).

Thanks
Andy

Andy Dix
Lead DevOps Engineer

Direct: +44 (0)1473 632566 | Mob: n/a
Email: andy.dix@havenpower.commailto:andy.dix@havenpower.com | Website: www.havenpower.comhttp://www.havenpower.com/

This email has been scanned for email related threats and delivered safely by Mimecast.
For more information please visit http://www.mimecast.com

Steve.
Apologies for the delay.
As you can see in the screenshot I have successfully performed a deployment with your provided package to an AWS instance running CentOS Linux release 7.2.1511 and Mono JIT compiler version 4.4.2 (Stable 4.4.2.11/f72fe45 Fri Jul 29 09:49:37 UTC 2016).

Note that this was a fresh clean install of Centos with the 4.4.2 version of Mono being the first installed. Perhaps there is a problem happening with the Mono upgrade path so will have to try with an install of 4.2.4 then upgrade to mono 4.4.2 and see if there are any strange issues.

The only other thing I could suggest trying is perhaps updating the server version. I believe we have updated SharpCompress which is used to decompress files in Calamari since the 3.3.5 version you are currently running so if you can update to a newer version of the Server (or manually build and update to test a newer Calamari as outlined in the ReadMe.txt) perhaps that may fix the error which appears to be happening during the unpacking phase of the deployment.

Let me know if the upgrade helps,
Cheers,
Rob

Something with that nuget package. I’ve just learnt of the ‘Push Package’ step in TeamCity which means I don’t have to nuget up my app, I can zip it. And a zipped copy of the app transfers fine… I can’t see any need to go back to nuget packages so I won’t worry about this issue anymore.
Thanks for all your help Rob.