Intermittent problem that can be resolved after re-trying the release

We getting the following error quite often now we’re trying to deploy a more complex application with many steps. Currently I’m able to retry the release excluding the steps that have completed. This is working but we’d really like to get to the bottom of the problem.

@@@
2013-04-08 14:01:43 ERROR System.NullReferenceException: Object reference not set to an instance of an object.
at Octopus.Tentacle.Deployment.DeploymentController.LogDiagnosticInformation(PackageMetadata package, VariableDictionary variables, IActivityLog log) in c:\w\e6923628be6eaf72\source\Octopus.Tentacle\Deployment\DeploymentController.cs:line 59
at Octopus.Tentacle.Deployment.DeploymentController.Execute(StoredPackage package, VariableDictionary variables, IActivityLog log) in c:\w\e6923628be6eaf72\source\Octopus.Tentacle\Deployment\DeploymentController.cs:line 30
at Octopus.Tentacle.Services.Jobs.JobQueue.RunDeploymentsOnBackgroundThread(Object ignored) in c:\w\e6923628be6eaf72\source\Octopus.Tentacle\Services\Jobs\JobQueue.cs:line 81
@@@

Hi Graeme,

Is anything else shown in the deployment log before that error?

Paul

We suspect the problem is that we’re trying to deploy a single package to one host/tentacle multiple times.

Each step deploys a website with a different set of configuration options specified via the octopus variables. In live these will be on separate machines but in testing they are all on the same box. To achieve this, so far, we’ve customized the Destination folder for each step that is reusing the main package with the following
@@@
C:\Octopus\Applications#{Octopus.Environment.Name}#{Octopus.Step.Package.NuGetPackageId}#{IisSiteName}#{Octopus.Step.Package.NuGetPackageVersion}
@@@
This seems to work but occasionally we get the aforementioned error, could this be a concurrency problem?

We’ve take a look using process monitor to see what’s going on and it does appear that the deployments seem to overlap a bit chronologically anyway. At least the writing to log files seems to overlap with the extraction of the next nupkg.

Any suggestions would be welcome.

Cheers,
Graeme.

The full error is…

@@@
Step 7: STEPNAMEREMOVED

2013-04-09 07:32:11 DEBUG Finding Test_B machines in the following role(s): ROLEREMOVED
2013-04-09 07:32:11 DEBUG Found machines:
2013-04-09 07:32:11 DEBUG - HOSTNAMEREMOVED (http://HOSTNAMEREMOVED:10933/)
2013-04-09 07:32:11 INFO Execute step STEPNAMEREMOVED

Execute step STEPNAMEREMOVED against machine HOSTNAMEREMOVED

2013-04-09 07:32:11 INFO Begin deployment
2013-04-09 07:32:11 DEBUG Deploying package PACKAGENAMEREMOVED 4.4.4846.37075 to tentacle http://HOSTNAMEREMOVED:10933/
2013-04-09 07:32:14 ERROR Running job on the tentacle failed:
2013-04-09 07:32:14 INFO Tentacle output follows:

2013-04-09 07:33:05 ERROR System.NullReferenceException: Object reference not set to an instance of an object.
at Octopus.Tentacle.Deployment.DeploymentController.LogDiagnosticInformation(PackageMetadata package, VariableDictionary variables, IActivityLog log) in c:\w\e6923628be6eaf72\source\Octopus.Tentacle\Deployment\DeploymentController.cs:line 59
at Octopus.Tentacle.Deployment.DeploymentController.Execute(StoredPackage package, VariableDictionary variables, IActivityLog log) in c:\w\e6923628be6eaf72\source\Octopus.Tentacle\Deployment\DeploymentController.cs:line 30
at Octopus.Tentacle.Services.Jobs.JobQueue.RunDeploymentsOnBackgroundThread(Object ignored) in c:\w\e6923628be6eaf72\source\Octopus.Tentacle\Services\Jobs\JobQueue.cs:line 81
@@@

Hi Graeme,

Do you have multiple package steps (on the Steps tab in Octopus) or a
single step that in this case is running multiple times on the same machine
at once? If so, this could be a concurrency thing. Steps in Octopus are
executed sequentially, but within a step, the deployment to each machine
happens in parallel.

As a workaround, you could set a special variable, "OctopusMaxParallelism"
to “1” and scope it to your testing environment. This will ensure that
packages aren’t deployed in parallel.

Regards,

Paul Stovell
Octopus Deploy
W: octopusdeploy.com | T: @octopusdeploy http://twitter.com/octopusdeploy

Hi Graeme,

Actually, I just realized: the Tentacle service uses a job queuing system
to ensure a single machine is never doing more than one thing at once. So
configuring the parallelism won’t make a difference - an agent can never be
deploying more than one package at a time.

The only explanation for this stack trace is that a package wasn’t found on
the Tentacle that was expected to have been uploaded already. Would it be
possible for you to send me the complete deployment log?

Regards,

Paul Stovell
Octopus Deploy
W: octopusdeploy.com | T: @octopusdeploy http://twitter.com/octopusdeploy

Hi Paul,
We have multiple steps, that use the same package. each has variables such as SiteName and AppPool which are then subsequently used to generate a unique physical path for each web/application to coexist, even though they’ve been deployed using a single package.

Currently we only have one machine that has multiple roles (in our testing environment) therefore Octopus deploys the sites to the same machine. We can send you screen prints or is there some way to export the config?

Many thanks for your prompt replies.

Kind Regards
Graeme Stow.

Hi Graeme,

When looking at the deployment results page, there’s a link named "Raw"
where you can view the full deployment log as a text file. You can upload
that file here (mark the conversation as private) or mail it to me directly
(paul at octopusdeploy.com).

Paul

Regards,

Paul Stovell
Octopus Deploy
W: octopusdeploy.com | T: @octopusdeploy http://twitter.com/octopusdeploy

Hi Paul,
I’ve emailed you the log. Let me know if you need anything further.

Many thanks,
Graeme.

Hi Graeme,

Thanks for sending through the deployment log.

Looking through the logs, it seems that you have multiple steps deploying
the same package. This normally wouldn’t be a problem, except you also have
a retention policy that causes the packages to be deleted from the server
during deployment. The end result is:

  1. A package gets uploaded (only once)
  2. It gets installed (step 1)
  3. It gets installed again (step 3), and a retention policy is triggered
    causing the package file to be deleted
  4. It is about to get installed again (step 7), but since the package file
    has been deleted, there’s no package to install and so this error happens

If you check the last line of the output for each step, you can see this in
the line:

Removing old package file: C:\Octopus\Applications\.Tentacle\.....file

name…nupkg

This is really a bug in Octopus/Tentacle, and we need to find a better way
to handle this scenario. But for now, you could try increasing your
retention policy settings (or keeping everything) so that a higher number
of Tentacle installations are kept. This should hopefully serve as a
workaround until we can find a better solution.

Regards,

Paul Stovell
Octopus Deploy
W: octopusdeploy.com | T: @octopusdeploy http://twitter.com/octopusdeploy

Hi Paul,
That does make sense, sort of! We changed the retention period yesterday as we were quickly running out of disk space.

We look a bit silly now.

Many thanks for resolving that for us.
Cheers,
Graeme.