Deployment fails with 'Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it has already been disposed'

daniel · 27 February 2017 17:57

Hi,

I am experiencing an issue where a deployment fails with the error message: ‘Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it has already been disposed’

I can see this error was reported before but in a slightly different context and the ticket has been closed due to inactivity:
http://help.octopusdeploy.com/discussions/problems/39623

Log from the failing step:

                    |   == Failed: Step 2: ******** ==
14:05:39   Fatal    |     The step failed: Activity Configure Service User on ******** failed with error 'Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it has already been disposed.
                    |     Server exception:
                    |     System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it has already been disposed.
                    |     at Autofac.Core.Lifetime.LifetimeScope.CheckNotDisposed()
                    |     at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable`1 parameters)
                    |     at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
                    |     at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
                    |     at Autofac.ResolutionExtensions.ResolveNamed[TService](IComponentContext context, String serviceName, IEnumerable`1 parameters)
                    |     at Octopus.Shared.Communications.AutofacServiceFactory.CreateService(String serviceName) in Y:\Work\refs\tags\3.4.9\source\Octopus.Shared\Communications\AutofacServiceFactory.cs:line 18
                    |     at Halibut.ServiceModel.ServiceInvoker.Invoke(RequestMessage requestMessage) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\ServiceModel\ServiceInvoker.cs:line 21
                    |     at Halibut.Transport.Protocol.MessageExchangeProtocol.InvokeAndWrapAnyExceptions(RequestMessage request, Func`2 incomingRequestProcessor) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 153'.
14:05:39   Verbose  |     ******** completed
                    |   
                    |     Failed: ********
14:05:39   Fatal    |       Configure Service User on ********
                    |     
                    |       Success: Configure Service User
14:05:30   Verbose  |         Octopus Server version: 3.7.1+Branch.master.Sha.0fc1b4c55d31801a654cd781099a16317d8af685
14:05:30   Verbose  |         Environment Information:
                    |         OperatingSystem: Microsoft Windows NT 6.2.9200.0
                    |         OsBitVersion: x64
                    |         Is64BitProcess: True
                    |         CurrentUser: ********
                    |         MachineName: ********
                    |         ProcessorCount: 4
                    |         CurrentDirectory: C:\Windows\system32
                    |         TempDirectory: C:\Users\********\AppData\Local\Temp\
                    |         HostProcessName: Octopus.Server
14:05:30   Info     |         Deploying package 'C:\Octopus\Files\********.2.0.510.nupkg-1a20e5d5-6988-4546-a3ec-29157b5d9aef' to machine 'https://********:*****/'
14:05:30   Info     |         Cannot start this task yet. There is already another task running that cannot be run in conjunction with any other task. Please wait...
14:05:36   Verbose  |         Octopus Deploy: Calamari version 0.0.0
14:05:36   Verbose  |         Environment Information:
14:05:36   Verbose  |         OperatingSystem: Microsoft Windows NT 6.2.9200.0
14:05:36   Verbose  |         OsBitVersion: x64
14:05:36   Verbose  |         Is64BitProcess: True
14:05:36   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
14:05:36   Verbose  |         MachineName: ********
14:05:36   Verbose  |         ProcessorCount: 4
14:05:36   Verbose  |         CurrentDirectory: C:\Octopus\Work\********
14:05:36   Verbose  |         TempDirectory: C:\Windows\TEMP\
14:05:36   Verbose  |         HostProcessName: Calamari
14:05:36   Info     |         Deploying package:    C:\Octopus\Files\********.2.0.510.nupkg-1a20e5d5-6988-4546-a3ec-29157b5d9aef
14:05:37   Verbose  |         Extracting package to: C:\Octopus\Applications\********\********\2.0.510
14:05:38   Verbose  |         Extracted 6 files
14:05:38   Info     |         Copying package contents to '********'
14:05:38   Info     |         Copied 6 files
14:05:38   Verbose  |         Executing '********\Octopus.Action.CustomScripts.PostDeploy.ps1'

Please advise.

Many thanks,
Daniel

Shane_Gill · 28 February 2017 01:41

Hi Daniel,

Thanks for getting in touch.

That error is generally caused by the Tentacle shutting down while attempting to run something on it. The error message has been improved since Tentacle version 3.7.0.

Are you seeing this consistently but the Tentacle appears online? Do health checks to that Tentacle pass?

Cheers,
Shane

dvargha · 2 March 2017 14:53

Hi Shane,

Sounds about right. In the tentacle log I see two requests coming in. The first one is likely instructing the tentacle to restart, and the second fails with the above error:

2017-02-27 14:03:00.2313      7  INFO  Agent listening on: [::]:*****
2017-02-27 14:03:00.2313      7  INFO  The Windows Service has started
2017-02-27 14:04:25.2010      3  INFO  listen://[::]:*****/              3  Accepted TCP client: [::ffff:************]:*****
2017-02-27 14:04:25.2322     12  INFO  listen://[::]:*****/             12  Performing TLS server handshake
2017-02-27 14:04:29.6817     12  INFO  listen://[::]:*****/             12  Secure connection established, client is not yet authenticated, client connected with Tls12
2017-02-27 14:04:29.6973     12  INFO  listen://[::]:*****/             12  Client at [::ffff:************]:***** authenticated as ************************
2017-02-27 14:05:21.9990      3  INFO  listen://[::]:*****/              3  Accepted TCP client: [::ffff:************]:60813
2017-02-27 14:05:21.9990     27  INFO  listen://[::]:*****/             27  Performing TLS server handshake
2017-02-27 14:05:21.9990     28  INFO  listen://[::]:*****/             28  Accepted TCP client: [::ffff:************]:60818
2017-02-27 14:05:21.9990     20  INFO  listen://[::]:*****/             20  Performing TLS server handshake
2017-02-27 14:05:24.0286     27  INFO  listen://[::]:*****/             27  Secure connection established, client is not yet authenticated, client connected with Tls12
2017-02-27 14:05:24.0286     20  INFO  listen://[::]:*****/             20  Secure connection established, client is not yet authenticated, client connected with Tls12
2017-02-27 14:05:24.0286     20  INFO  listen://[::]:*****/             20  Client at [::ffff:************]:60818 authenticated as ************************
2017-02-27 14:05:24.0286     27  INFO  listen://[::]:*****/             27  Client at [::ffff:************]:60813 authenticated as ************************
2017-02-27 14:05:38.9385     10  INFO  Stopping the Windows Service
2017-02-27 14:05:39.0166     10  INFO  listen://[::]:*****/             10  Listener stopped
2017-02-27 14:05:39.0166     10  INFO  The Windows Service has stopped
2017-02-27 14:05:49.9452      7  INFO  Agent will trust Octopus servers with the thumbprint: ************************
2017-02-27 14:05:50.3044      7  INFO  listen://[::]:*****/              7  Listener started
2017-02-27 14:05:50.3044      7  INFO  Agent listening on: [::]:*****
2017-02-27 14:05:50.3044      7  INFO  The Windows Service has started
2017-02-27 14:05:50.3668      5  INFO  listen://[::]:*****/              5  Accepted TCP client: [::ffff:************]:60846
2017-02-27 14:05:50.3668      5  INFO  listen://[::]:*****/              5  Performing TLS server handshake

I suspect that the issue is related to the tentacle auto-update feature. The target machine is a newly provisioned box in an elastic environment and it is running an old version of the tentacle. The deployment is triggered by an auto-deployment trigger, and the machine policy has the ‘Tentacle updates’ option set to ‘Automatically update Tentacle’.

I cannot just ignore the error as it was suggested in another thread because after the failed deployment Octopus stops auto-deploying the project because it considers the last deployment as failed.

Regards,
Daniel

Shane_Gill · 6 March 2017 01:35

Hi Daniel,

Could you use the option in “Project > Process > Change machine connectivity settings” and select “Skip machines if they are or become unavailable”?

I think this will allow the deployment to finish successfully if a machine goes down because of a Tentacle update and then deploy to that machine when the update is finished.

Cheers,
Shane

dvargha · 6 March 2017 15:02

Hi Shane,

Thanks for the suggestion, I’ll give it a try.

BTW: I already have a ‘Health Check’ step as the first step in my process as recommended by the , and it is configured to ‘Skip deployment targets that are unavailable’

Regards,
Daniel

Shane_Gill · 6 March 2017 22:27

Hi Daniel,

I’ve created a GitHub issue to prevent deployments from running while a Tentacle upgrade is in progress: https://github.com/OctopusDeploy/Issues/issues/3254

Cheers,
Shane