Linux deployment all frozen up

Legacy Uncategorised
1

Simple linux deployment that is supposed to unpack the nupkg to /var/dir just takes forever and doesn’t finish.
This is what’s showing on the task log:

Task ID: ServerTasks-3246
Task status: Executing
Task queued: Thursday, November 5, 2015 6:56 PM
Task started: Thursday, November 5, 2015 6:56 PM
Task duration: 14 seconds
Server version: 3.2.0+Branch.master.Sha.391e7e2a161abf18405930022a65470fbfeb195d

                | == Running: Deploy SL RocketChat release 1.0.0.9 to Web.Development ==

18:56:53 Verbose | Guided failure is not enabled for this task
|
| == Running: Acquire packages ==
18:56:53 Info | Acquiring packages
18:56:53 Info | Making a list of packages to download
18:56:53 Verbose | Checking package cache for package SangLucci.RocketChat 1.0.0.9
18:56:55 Verbose | Package X version 1.0.0.9 was found in cache. No need to download. Using file: C:\Octopus\OctopusServer\PackageCache\feeds-teamcity\X.1.0.0.9_4711B6167A447A44B52522451628C9B7.nupkg
18:56:55 Verbose | SHA1 hash of package X is: 03dc4301bb8fcaf308cce59584567475183f32cc
|
| Running: X.cloudapp.net
|
| Running: Upload package X 1.0.0.9
18:56:55 Verbose | Requesting upload…
18:56:55 Verbose | Establishing SSH connection…
18:56:55 Verbose | SSH connection established
18:56:56 Info | Beginning streaming transfer of calamari-find-package.sh to .octopus\OctopusServer\Work\20151105185653-4
18:56:56 Verbose | Establishing SFTP connection…
18:56:57 Verbose | SSFTP connection established
18:56:57 Info | Stream transfer complete
18:56:58 Info | This server does not have version 3.1.2 of the Calamari package. It will be pushed automatically.
18:57:04 Verbose | Requesting upload…
18:57:04 Info | Beginning streaming transfer of calamari.3.1.2.tar.gz to .octopus\OctopusServer\Work\20151105185653-4
18:57:05 Info | Stream transfer complete
18:57:05 Verbose | Requesting upload…
18:57:06 Info | Beginning streaming transfer of extract-calamari-package.sh to .octopus\OctopusServer\Work\20151105185653-4
18:57:06 Info | Stream transfer complete
|
| == Pending: Step 1: Deploy package ==

2

It got a bit further after a long time but only up to the unpacking step and froze again - took about 10 minutes which is still unreasonable as both are Azure servers in the same region - SSH transfer between them can’t be that slow (pkg is 22MB)

3

Hi Georgios,

Thanks for getting in touch and I’m sorry to hear you are having these issues!

To help me troubleshoot the slowness you are experiencing, could you please tell me a bit more about your current setup you is in Azure?

  • VM sizes, OS (both OD server and Linux server) installed
  • Mono version
  • Could you also provide us with your NuGet package that you are trying to deploy? If you can I will provide you with a link to a secure upload as the package is a bit big for sending through the forum.

Thank you and kind regards,
Henrik

4

It’s a Standard A1 instance with Ubuntu 15.10, Mono 4.0.4. OD is latest, upgraded last night in case there was a fix I was missing.

Happy to secure upload package.

Will try Ubuntu 14.04 because I have to get going with the deployment but I’ve kept the other VM up if you want to try anything out.

5

Similar results on 14.04 - seems it gets stuck on this command:

mono /home/octopus/.octopus/OctopusServer/Calamari/3.1.2/Calamari.exe find-package -packageId X -packageVersion 1.0.0.11

CPU peaks - stopped it at 3 minutes. Even on such a small VM this is surely unexpected?

6

I downgraded to Mono 4.0.1 to match your test environment - I had to compile from source because there doesn’t seem to be a snapshot for that version.

Now getting an exception:

11:20:36 Error | System.NullReferenceException: Object reference not set to an instance of an object
11:20:37 Error | at System.IO.Packaging.Package.LoadRelationships () [0x00000] in :0
11:20:37 Error | at System.IO.Packaging.Package.get_Relationships () [0x00000] in :0
11:20:37 Error | at System.IO.Packaging.Package.GetRelationshipsByType (System.String relationshipType) [0x00000] in :0
11:20:37 Error | at Calamari.Integration.Packages.LightweightPackageExtractor.ProcessManifest (System.IO.Packaging.Package package) [0x00000] in :0
11:20:37 Error | at Calamari.Integration.Packages.LightweightPackageExtractor.GetMetadata (System.String packageFile) [0x00000] in :0
11:20:37 Error | at Calamari.Deployment.Conventions.ExtractPackageConvention.Install (Calamari.Deployment.RunningDeployment deployment) [0x00000] in :0
11:20:37 Error | at Calamari.Deployment.ConventionProcessor.RunInstallConventions () [0x00000] in :0
11:20:37 Error | at Calamari.Deployment.ConventionProcessor.RunConventions () [0x00000] in :0
11:20:37 Error | Running rollback conventions…
11:20:37 Error | System.ArgumentNullException: Argument cannot be null.
11:20:37 Error | Parameter name: path
11:20:37 Error | at System.IO.FileSystemInfo.CheckPath (System.String path) [0x00000] in :0
11:20:37 Error | at System.IO.DirectoryInfo…ctor (System.String path, Boolean simpleOriginalPath) [0x00000] in :0
11:20:37 Error | at System.IO.DirectoryInfo…ctor (System.String path) [0x00000] in :0
11:20:37 Error | at (wrapper remoting-invoke-with-check) System.IO.DirectoryInfo:.ctor (string)
11:20:37 Error | at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.EnumerateFiles (System.String parentDirectoryPath, System.String[] searchPatterns) [0x00000] in :0
11:20:37 Error | at Calamari.Deployment.Conventions.PackagedScriptConvention.FindScripts (Calamari.Deployment.RunningDeployment deployment) [0x00000] in :0
11:20:37 Error | at Calamari.Deployment.Conventions.PackagedScriptConvention.RunScripts (Calamari.Deployment.RunningDeployment deployment) [0x00000] in :0
11:20:37 Error | at Calamari.Deployment.Conventions.RollbackScriptConvention.Rollback (Calamari.Deployment.RunningDeployment deployment) [0x00000] in :0
11:20:37 Error | at Calamari.Deployment.ConventionProcessor.RunRollbackConventions () [0x00000] in :0
11:20:37 Error | at Calamari.Deployment.ConventionProcessor.RunConventions () [0x00000] in :0
11:20:37 Error | at Calamari.Commands.DeployPackageCommand.Execute (System.String[] commandLineArguments) [0x00000] in :0
11:20:37 Verbose | Disposing SFTP connection…
11:20:37 Verbose | Disposing SSH connection…
11:20:37 Error | The remote script failed with exit code 100

7

Similar results with Mono 4.0.4.1 compiled from source… sigh…

8

Same exception with 3.0.12 from deb

9

Hm, I wonder if it has anything to do with this pull request: https://github.com/mono/nuget/pull/5

I had to custom compile Octo.exe with this patch to successfully package the meteor app.

Doubtful though - it’s just a different way of loading the file stream. I went through the relevant usages and FileStream.Name which is the only place affected by this patch, is not used anywhere.

I can’t think of anything else, I hope you have some ideas… :slight_smile:

10

It seems that the Mono 4.0.1 NRE is related to this issue: https://bugzilla.xamarin.com/show_bug.cgi?id=26205 - it’s supposed to be resolved in 4.0.3.17 which is why it doesn’t appear in 4.0.4 but there’s obviously another issue there that makes mono take forever.

11

Hi Georgios,

Thank you for providing this extra information!

I have created an area for you to upload your package here so that I can try and replicate this issue, let me know if you have any issues accessing or uploading your files there.

Thank you and kind regards,
Henrik

12

Hi again Georgios,

I’ve had a chat to the team and there is a bug in mono that causes this issue and there is a workaround for it, the process would be

  1. tar all your app files
  2. pack the tar file into a nuget package
  3. use the deploy nuget package step to extract the tar file from the package
  4. untar the tar file using a post-deploy/custom script

We have had a few other customers that have run into this same issue, and they have used this method and it’s worked for them

Hope that helps!

Thank you and kind regards,
Henrik

13

Hi Henrik,

I did indeed do what you suggested and got around this issue.

It would however be good to see this resolved.

I have created a pull request at the Mono.Nuget.Core project to cope with the PathTooLongException here: https://github.com/mono/nuget/pull/5

This doesn’t solve the Mono side of things but I’ll leave that to you.

Georgios