Windows hosted unable to run docker

jussi.mattila · 25 May 2022 05:18

We are currently unable to deploy on hosted node octopus-i009324-6b7c644645-8kv6j.

This is part of the error message:

Unable to connect to docker service
05:13:39   Error    |         NotSpecified: You will need docker installed and running to pull docker images
05:13:39   Error    |         At C:\Octopus\Tentacle\Work\20220525051333-27842-14\Bootstrap.Octopus.DockerLogin.ps1:915 char:2
05:13:39   Error    |         +     . '.\Octopus.DockerLogin.ps1'
05:13:39   Error    |         +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
05:13:39   Error    |         at <ScriptBlock>, C:\Octopus\Tentacle\Work\20220525051333-27842-14\Octopus.DockerLogin.ps1: line 29
05:13:39   Error    |         at <ScriptBlock>, C:\Octopus\Tentacle\Work\20220525051333-27842-14\Bootstrap.Octopus.DockerLogin.ps1: line 915
05:13:39   Error    |         at <ScriptBlock>, <No file>: line 1
05:13:39   Error    |         at <ScriptBlock>, <No file>: line 1
05:13:39   Error    |         Failed to download package octopusdeploy/worker-tools v3.0.0-windows.ltsc2019 from feed: 'https://index.docker.io'
05:13:39   Error    |         Unable to log in Docker registry
05:13:39   Verbose  |         Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Tentacle\Work\20220525051333-27842-14 exited with code 1
05:13:39   Verbose  |         Exit code: 1

Is there something that I can do to get the deployments working again?

finnian.dempsey · 25 May 2022 05:37

Hi @jussi.mattila,

Thanks for reaching out, sorry to hear your having trouble with Docker on our Windows Dynamic Worker!

I’ll check in with the devs about how this instance managed to slip through our tests and not have docker installed, however to get you unblocked in the meantime I can delete the current worker lease so that another one will be used for your next deployment.

Please let me know if you are happy for me to do this and I’ll get it done asap!

Best Regards,

jussi.mattila · 25 May 2022 05:49

Thanks for your quick response. I managed to find a workaround Docker not installed on dynamic Windows 2019 worker? - #8 by adam.hollow, restarting the Docker service using a runbook helped.

Deployment is currently underway, no need to do anything or delete the worker lease anymore.

finnian.dempsey · 25 May 2022 05:58

Hi @jussi.mattila,

That is great to hear, I won’t touch your current worker!

Feel free to reach out again if you experience any other issues!

Best Regards,

jussi.mattila · 25 May 2022 06:01

Hi again. It looks like the workaround didn’t fully fix the situation. Our deployment is hanging. It just stopped doing anything after pulling the Docker image. Maybe you should delete this specific worker lease after all and allow us to try with a new one.

finnian.dempsey · 25 May 2022 06:06

Hi @jussi.mattila,

That is strange indeed, I will get your worker rotated asap!

Best Regards,

jussi.mattila · 25 May 2022 06:07

Thanks, please let me know when I should retry the deployment!

finnian.dempsey · 25 May 2022 06:15

Hi @jussi.mattila,

I’ve gone ahead and rotated your worker, a new one should be leased when you retry your deployment.

Please let us know how you get on or if there are still any issues!

Best Regards,

jussi.mattila · 25 May 2022 06:21

Unfortunately, that didn’t help. Looks like the server node is still the same, don’t know if that was supposed to change?

Server node:    octopus-i009324-6b7c644645-8kv6j

                    | == Failed: Deploy release 2.1.0-main7090 to Preview for West EU ==
06:17:01   Verbose  |   Guided failure is not enabled for this task
06:18:36   Verbose  |   Releasing workers that required packages.
06:18:36   Verbose  |   Successfully called release for dynamic pool lease Leases-5424747 : 22-05-25-0400-pifge
06:18:36   Fatal    |   The deployment failed because one or more steps failed. Please see the deployment log for details.
                    | 
                    |   == Failed: Acquire packages ==
06:17:02   Info     |     Acquiring packages
06:17:02   Info     |     Making a list of packages to acquire
06:17:02   Info     |     Leasing Windows2019 dynamic worker...
06:17:02   Info     |     Worker not ready, trying again in 10 seconds
06:17:12   Info     |     Worker not ready, trying again in 5 seconds
06:17:17   Info     |     Worker not ready, trying again in 5 seconds
06:17:22   Info     |     Worker not ready, trying again in 5 seconds
06:17:27   Info     |     Worker not ready, trying again in 5 seconds
06:17:32   Info     |     Worker not ready, trying again in 5 seconds
06:17:37   Info     |     Obtained worker lease successfully.
06:17:37   Verbose  |     Leased worker 22-05-25-0400-pifge from pool Default Worker Pool (lease Leases-5424747).
06:17:37   Verbose  |     Package Projects-25 v2.1.0-main7090 is required by action 'Deploy Infrastructure'
06:17:37   Verbose  |     Package Projects-1 v2.1.0-main7090 is required by action 'Deploy Suite'
06:17:37   Verbose  |     Package octopusdeploy/worker-tools v3.0.0-windows.ltsc2019 is required by action 'Deploy Suite'
06:17:37   Verbose  |     Package Projects-41 v2.1.0-main7090 is required by action 'Deploy Gateway'
06:17:37   Verbose  |     Package octopusdeploy/worker-tools v3.0.0-windows.ltsc2019 is required by action 'Deploy Gateway'
06:17:37   Verbose  |     Package Projects-2 v2.1.0-main7090 is required by action 'Deploy MRI'
06:17:37   Verbose  |     Package octopusdeploy/worker-tools v3.0.0-windows.ltsc2019 is required by action 'Deploy MRI'
06:17:37   Verbose  |     Package Projects-61 v1.0.0-main7090 is required by action 'Deploy PET'
06:17:37   Verbose  |     Package Projects-22 v2.1.0-main7090 is required by action 'Deploy DSI'
06:17:37   Verbose  |     Package octopusdeploy/worker-tools v3.0.0-windows.ltsc2019 is required by action 'Deploy DSI'
06:17:37   Verbose  |     No packages are required on the Octopus Server
06:17:37   Verbose  |     Delta compression is enabled for package transfers from the Octopus Server to deployment targets
06:17:37   Verbose  |     Machine DynamicWorker 22-05-25-0400-pifge still needs packages octopusdeploy/worker-tools v3.0.0-windows.ltsc2019, octopusdeploy/worker-tools v3.0.0-windows.ltsc2019, octopusdeploy/worker-tools v3.0.0-windows.ltsc2019 and octopusdeploy/worker-tools v3.0.0-windows.ltsc2019 for actions ('Deploy Suite', 'Deploy Gateway', 'Deploy MRI' and 'Deploy DSI')
06:17:37   Verbose  |     Package Projects-25 v2.1.0-main7090 does not need to be acquired
06:17:37   Verbose  |     Package Projects-1 v2.1.0-main7090 does not need to be acquired
06:17:37   Verbose  |     Package Projects-2 v2.1.0-main7090 does not need to be acquired
06:17:37   Verbose  |     Package Projects-41 v2.1.0-main7090 does not need to be acquired
06:17:37   Verbose  |     Package Projects-61 v1.0.0-main7090 does not need to be acquired
06:17:37   Verbose  |     Package Projects-22 v2.1.0-main7090 does not need to be acquired
06:18:36   Fatal    |     The step failed: Activity failed with error 'The remote script failed with exit code 1'.
06:18:36   Verbose  |     Acquire Packages completed
                    |   
                    |     Failed: DynamicWorker 22-05-25-0400-pifge
06:18:36   Fatal    |       The remote script failed with exit code 1
06:18:36   Verbose  |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.SuccessArbitrator.ThrowIfNotSuccessful(IActionHandlerResult result) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/SuccessArbitrator.cs:line 22
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation actionHandler, ITaskLog taskLog, VariableCollection variables) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 61
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageAcquisitionExecutionHandlers.Invoke(ActionHandlerInvocation invocation, ITaskLog taskLog) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageAcquisitionExecutionHandlers.cs:line 116
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageAcquisitionExecutionHandlers.DownloadPackage(String packageId, IVersion version, IFeed feed, ITaskLog taskLog, PackageCachePolicy packageCachePolicy) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageAcquisitionExecutionHandlers.cs:line 71
                    |       at Octopus.Server.Orchestration.Targets.Common.PackageStagingMediator.DownloadPackageOnAgent(PackageIdentity packageIdentity, IDeploymentPackageDownloader deploymentPackageDownloader, PackageAcquisitionExecutionHandlers execs, ITaskLog taskLog) in ./source/Octopus.Server/Orchestration/Targets/Common/PackageStagingMediator.cs:line 66
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireMachinePackageTask.StagePackage(PackageAcquisitionExecutionHandlers packageAcquisitionHandlers, ITaskLog stepTaskLog, ITaskLog deploymentTargetTaskLog) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/AcquireMachinePackageTask.cs:line 152
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireMachinePackageTask.Acquire(AcquiredPackageMap acquiredPackageMap, ITaskContext taskContext, ITaskLog stepTaskLog, ITaskLog deploymentTargetTaskLog) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/AcquireMachinePackageTask.cs:line 104
                    |       at Octopus.Server.Infrastructure.EitherAsyncOrSync.Execute(CancellationToken cancellationToken) in ./source/Octopus.Server/Infrastructure/EitherAsyncOrSync.cs:line 38
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(EitherAsyncOrSync callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Guidance.cs:line 148
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(EitherAsyncOrSync callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Maybe`1 callbackOnExclude, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Guidance.cs:line 78
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.AcquirePackagesStepController.<>c__DisplayClass5_3`3.<<Execute>b__4>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/AcquirePackagesStepController.cs:line 85
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/TransientErrorDetectionExecutor.cs:line 49
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.AcquirePackagesStepController.<>c__DisplayClass5_2`3.<<Execute>b__3>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/AcquirePackagesStepController.cs:line 89
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass5_0`3.<<Execute>b__0>d.MoveNext() in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 126
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Core/Infrastructure/UnitsOfWork/UnitOfWorkExtensionMethods.cs:line 75
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Core/Infrastructure/UnitsOfWork/UnitOfWorkExtensionMethods.cs:line 75
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3](Func`5 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 129
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.AcquirePackagesStepController.<>c__DisplayClass5_1`3.<<Execute>b__2>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/AcquirePackagesStepController.cs:line 92
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Server.Orchestration.ServerTasks.ParallelWorkOrderAsync`1.<>c__DisplayClass8_1.<<Execute>b__4>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/ParallelWorkOrderAsync.cs:line 62
                    |       --- End of stack trace from previous location ---
                    |       at Octopus.Server.Infrastructure.EitherAsyncOrSync.Execute(CancellationToken cancellationToken) in ./source/Octopus.Server/Infrastructure/EitherAsyncOrSync.cs:line 38
                    |       at Octopus.Server.Orchestration.ServerTasks.OctoThreadClosureAsync`1.Execute(CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/OctoThreadClosureAsync.cs:line 63
                    |     
                    |       Failed: Download package octopusdeploy/worker-tools v3.0.0-windows.ltsc2019 directly from Docker Registry v1 'Docker Hub' at https://index.docker.io

finnian.dempsey · 25 May 2022 06:35

Hi @jussi.mattila,

Something strange is definitely going on here…Sorry for the hassle this is causing!

I’ve asked the engineers to investigate further and I’ll keep you posted as soon as we have any updates. I’ve requested that they prioritise getting you a healthy worker to get you unblocked.

Just confirming that the Server Node will stay the same, however the worker used will now be different:

Leased worker 22-05-25-0324-0gxbg - Old Worker
Leased worker 22-05-25-0400-pifge - New Worker

Best Regards,

dane.falvo · 25 May 2022 07:45

Hi @jussi.mattila,

The issue was identified and rectified. Everything should be working as expected now.
Please try it again and let me know if you are still out of luck.

I’m sorry for the issues this has caused.

Regards,

jussi.mattila · 25 May 2022 11:23

Thanks for your quick responses to our issue! We have not retried deployments yet, there’s a trade show starting today where we are demoing the system and don’t want to disrupt that now. We were hoping to push some last minute changes but we are fine without them. I assume that things will work the next time we deploy a release. If not, I’ll add another comment in this thread, early next week.

clare.martin · 25 May 2022 11:31

Hey @jussi.mattila,

There have been a few customers that have had this issue this morning and upon redeploying they have managed to lease a worker. Our cloud team have confirmed they have fixed this issue and are now monitoring the workers to make sure we are in the all-clear.

I hope that helps alleviate any concerns you may have had with this issue, again we are sorry this has stopped you from pushing some changes out but it’s good to know you are still able to demo the system at the trade show without the last minute changes.

Please do let us know if you are still struggling to lease a worker when you go to deploy again and we will do our best to help!

Kind Regards,

Clare Martin

system · 25 June 2022 11:32

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.