Deploy steps hangs when using Container

Octopus Cloud
1

Have a step we have built in one of our projects that makes use of the octopusdeploy/worker-tools:ubuntu.22.04 container image. The issue is that after the step has completed it just hangs around and never ends, and will eventually time out. So i’m unsure if i am doing something wrong or if this is just one of those things. Is there some step or command i supposed to use to indicate final success of the deployment process?

This is the tail end of the log from the deployment step after it has finished.

2:25:59   Verbose  |       Process /bin/bash in /home/Octopus/Work/20230823222133-117686-22 exited with code 0
22:35:59   Verbose  |       Updating manifest with output variables
22:35:59   Verbose  |       Updating manifest with action evaluated variables
22:35:59   Verbose  |       An error occurred when sending a request to 'https://X.X.X.X:YYYY/', after the request began: Unable to read data from the transport connection: Connection timed out.
                    |       Halibut.HalibutClientException
                    |       at Octopus.Server.Orchestration.Targets.Tentacles.Observability.HalibutTentacleRpcTimerProxy.Invoke(MethodInfo targetMethod, Object[] args) in HalibutTentacleRPCTimerProxy.cs:line 42
                    |       at generatedProxy_3.CompleteScript(CompleteScriptCommand )
                    |       at Octopus.Server.Orchestration.Targets.Tentacles.ScriptObserver.Finish(ScriptTicket ticket, Int64 next, ScriptOutputFilter output) in ScriptObserver.cs:line 94
                    |       at Octopus.Server.Orchestration.Targets.Tentacles.ScriptObserver.ObserveThenFinish(ScriptTicket ticket, ScriptOutputFilter output, Func`3 collectArtifacts, CancellationToken cancellationToken) in ScriptObserver.cs:line 52
                    |       at Octopus.Server.Orchestration.Targets.Tentacles.ScriptObserver.ObserveUntilComplete(ScriptTicket ticket, ITaskLog taskLog, Func`3 collectArtifacts, CancellationToken cancellationToken) in ScriptObserver.cs:line 38
                    |       at Octopus.Server.Orchestration.Targets.Tentacles.TentacleRemoteEndpointFacade.ExecuteCommand(StartScriptCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in TentacleRemoteEndpointFacade.cs:line 57
                    |       at Octopus.Server.Orchestration.Targets.Common.RemoteEndpointFacadeCancellationTokenDecorator.ExecuteCommand(StartScriptCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in RemoteEndpointFacadeCancellationTokenDecorator.cs:line 36
                    |       at Octopus.Server.Orchestration.Targets.Tentacles.Observability.ErrorLoggingRemoteEndpointFacadeDecorator.ExecuteCommand(StartScriptCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in ErrorLoggingRemoteEndpointFacadeDecorator.cs:line 75
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ExecutionTargets.TentacleExecutionTarget.Execute(ScriptCollection bootstrapperScripts, IReadOnlyList`1 bootstrapperArguments, IReadOnlyList`1 files, Nullable`1 forceIsolationLevel, Boolean raw, ITaskLog taskLog, String isolationMutexName, CancellationToken cancellationToken, Nullable`1 isolationMutexTimeout) in TentacleExecutionTarget.cs:line 70
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.RunOnTarget(ScriptCollection finalBootstrapperScripts, String platform, IBootstrapperScriptGenerator bootstrapperScriptGenerator, IReadOnlyList`1 bootstrapperArguments, IReadOnlyList`1 allFiles, Nullable`1 isolation, Nullable`1 isolationMutexTimeout, String isolationMutexName, CalamariFlavour flavour, IReadOnlyList`1 deploymentTools, ExecutionTargetDetails executionTargetDetails, CalamariPlatformConstraint calamariPlatformConstraint, ITaskLog taskLog, CancellationToken cancellationToken) in ImmediateExecutor.cs:line 296
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.ExecuteCalamari(CalamariFlavour calamariFlavour, String calamariCommand, IReadOnlyList`1 calamariArguments, IReadOnlyList`1 files, IReadOnlyList`1 deploymentTools, VariableCollection extraVariables, TargetManifest targetManifest, CalamariPlatformConstraint calamariPlatformConstraint, Nullable`1 isolationMutexTimeout, String isolationMutexName, ITaskLog taskLog, CancellationToken cancellationToken) in ImmediateExecutor.cs:line 228
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.CommandBuilders.CalamariCommandBuilder.Execute(ITaskLog taskLog, CancellationToken cancellationToken) in CalamariCommandBuilder.cs:line 163
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Script.ScriptActionHandler.ExecuteWithDefaultScriptHandler(IActionHandlerContext context, ITaskLog taskLog, CancellationToken cancellationToken) in ScriptActionHandler.cs:line 66
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Script.ScriptActionHandler.Execute(IActionHandlerContext context, ITaskLog taskLog, CancellationToken cancellationToken) in ScriptActionHandler.cs:line 45
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.ExecuteAction(ActionCommand command, DeploymentTarget inContextOf, TargetManifest targetManifest, ITaskLog taskLog, IActionHandler handler, IActionHandlerContext handlerContext, Nullable`1 executionTimeout, CancellationToken ct) in NewActionDispatcher.cs:line 359
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.<>c__DisplayClass20_0.<Execute in NewActionDispatcher.cs:line 318
                    |       at Octopus.Server.Orchestration.ServerTasks.ActionRetry.Execute(Func`2 callback, ITaskLog taskLog, TargetManifest targetManifest, IActionAndTargetScopedVariables variablesSnapshot, PlannedAction action, CancellationToken cancellationToken) in ActionRetry.cs:line 53
                    |       at Octopus.Server.Orchestration.ServerTasks.ActionRetry.Execute(Func`2 callback, ITaskLog taskLog, TargetManifest targetManifest, IActionAndTargetScopedVariables variablesSnapshot, PlannedAction action, CancellationToken cancellationToken) in ActionRetry.cs:line 38
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.<>c__DisplayClass20_0.<Execute in NewActionDispatcher.cs:line 324
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken) in Guidance.cs:line 148
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Action callbackOnExclude, CancellationToken cancellationToken) in Guidance.cs:line 78
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.Execute(ActionCommand command, DeploymentTarget inContextOf, IExecutor executor, TargetManifest targetManifest, Maybe`1 stagedPackagePath, IEnumerable`1 packageAcquisitionInformation, IActionAndTargetScopedVariables variablesSnapshot, ITaskLog taskLog, IActionHandler handler, Action guidanceExcludeCallback, CancellationToken cancellationToken) in NewActionDispatcher.cs:line 330
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.ExecuteOnWorker(ActionCommand command, DeploymentTarget inContextOf, ITaskLog taskLog, IActionHandler handler, CancellationToken cancellationToken) in NewActionDispatcher.cs:line 229
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.ExecuteOnWorker(ActionCommand command, DeploymentTarget inContextOf, ITaskLog taskLog, IActionHandler handler, CancellationToken cancellationToken) in NewActionDispatcher.cs:line 229
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.NewActionDispatcher.Dispatch(ActionCommand command, DeploymentTarget deploymentTarget, ITaskLog taskLogForTarget, ITaskLog taskLogRoot, IActionHandlerResolver actionHandlerResolver, Action guidanceExcludeCallback, CancellationToken cancellationToken) in NewActionDispatcher.cs:line 127
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.<>c__DisplayClass8_2.<TryExecuteActionAndInitLoggingContext in ProcessStepController.cs:line 297
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken) in Guidance.cs:line 148
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Action callbackOnExclude, CancellationToken cancellationToken) in Guidance.cs:line 78
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.<>c__DisplayClass8_1.<TryExecuteActionAndInitLoggingContext in ProcessStepController.cs:line 303
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in TransientErrorDetectionExecutor.cs:line 50
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in TransientErrorDetectionExecutor.cs:line 50
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.<>c__DisplayClass8_0.<TryExecuteActionAndInitLoggingContext in ProcessStepController.cs:line 308
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass7_0`4.<Execute in UnitOfWorkExecutor.cs:line 124
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 58
                    |       at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 58
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3,T4](Func`6 action, CancellationToken cancellationToken, String name) in UnitOfWorkExecutor.cs:line 127
                    |       at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3,T4](Func`6 action, CancellationToken cancellationToken, String name) in UnitOfWorkExecutor.cs:line 127
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.ProcessStepController.TryExecuteActionAndInitLoggingContext(ExecutionPlan plan, ExecutionPlanner planner, PlannedStep step, DeploymentTarget targetContext, PlannedAction action, ITaskLog taskLogForTarget, ITaskLog taskLogRoot, CancellationToken cancellationToken) in ProcessStepController.cs:line 329
                    |       --Inner Exception--
                    |       Unable to read data from the transport connection: Connection timed out.
                    |       System.IO.IOException
                    |       at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
                    |       at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
                    |       at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
                    |       at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
                    |       at Halibut.Transport.RewindableBufferStream.Read(Byte[] buffer, Int32 offset, Int32 count)
                    |       at Halibut.Transport.Observability.ByteCountingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
                    |       at System.IO.Compression.DeflateStream.ReadCore(Span`1 buffer)
                    |       at Halibut.Transport.Observability.ByteCountingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
                    |       at System.IO.BinaryReader.InternalRead(Int32 numBytes)
                    |       at System.IO.BinaryReader.ReadInt32()
                    |       at Newtonsoft.Json.Bson.BsonDataReader.ReadNormal()
                    |       at Newtonsoft.Json.Bson.BsonDataReader.Read()
                    |       at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
                    |       at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
                    |       at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
                    |       at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
                    |       at Halibut.Transport.Protocol.MessageSerializer.DeserializeMessage[T](JsonReader reader)
                    |       at Halibut.Transport.Protocol.MessageSerializer.ReadCompressedMessageRewindable[T](Stream stream, IRewindableBuffer rewindable)
                    |       at Halibut.Transport.Protocol.MessageExchangeStream.Receive[T]()
                    |       at Halibut.HalibutRuntime.<>c__DisplayClass53_0.<SendOutgoingHttpsRequest>b__0(MessageExchangeProtocol protocol)
                    |       at Halibut.Transport.SecureListeningClient.ExecuteTransaction(ExchangeAction protocolHandler, CancellationToken cancellationToken)
                    |       --Inner Exception--
                    |       Connection timed out
                    |       System.Net.Sockets.SocketException
                    |       at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
22:35:59   Fatal    |       The action Build and Deploy  on a Worker failed
                    |     
                    |   == Success: Release packages ==
22:35:59   Info     |     There are no packages to be released.
22:35:59   Verbose  |     Release Packages completed
2

Hi @paul2,

Thanks for reaching out, I’d be happy to help with this issue with the workertools container timing out!

There shouldn’t be any configuration needed for the container to finish so I have a feeling there might be a specific workertools image that’s not working right, could you please send through the full task logs to our secure upload portal?

That should provide all the info we need, however a potential workaround might be to use a specific image tag like octopusdeploy/worker-tools:5.2.0-ubuntu.22.04. I’m also curious if you’re able to reproduce this consistently or does it just happen intermittently?

Looking forward to hearing how you get on, feel free to reach out with any questions at all!

Best Regards,

3

Finn, I’ve uploaded the log and also the script steps for the failing step (step 2)

Also as for reproducability, has happened consistent, but we also haven’t used this a lot due to the failures. Its a new deploy and so i have a lot of failed attempts before we got to this stage.

1 Like
4

Hi @paul2,

Cheers for that, unfortunately there isn’t anything obvious sticking out but I’ll keep digging into them!

Is it just this one project or does it ever happen to other projects too? I wonder if you make another simple script step e.g. echo 'test' and run it on the same target/worker using that container does it terminate ok?

I’m thinking there could potentially be a child process being kept open somehow, as the main script process had an exit code 0 but I’ll keep you posted if I have any further updates!

Best Regards,

6

hey Finn, so this is only one project as it is the only one where we are running a specific container image (unique deployment process mandated it).

I will make a dummy step and see what happens.

7

Looks linked to the nvm install command. No idea why, there is no process left behind… the shell still exits cleanly.

8

Hi @paul2,

Cheers for confirming that, glad to hear you managed to narrow it down!

Typically docker images are built with any required programs preinstalled, e.g. vercel on alpine linux: https://github.com/dockette/vercel/blob/master/Dockerfile

And we recommend only really using the workertools image as a way to get started and to create your own custom images instead (tldr: workertools have more tooling installed than is needed for a lot of use cases): https://octopus.com/docs/projects/steps/execution-containers-for-workers#worker-tools-images

You can check out what’s installed on the Workertool images here: https://github.com/OctopusDeploy/WorkerTools/blob/master/ubuntu.22.04/Dockerfile

I definitely recommend checking out this blog about configuring your own Docker image from workertools: https://octopus.com/blog/extending-octopus-execution-container

However I noticed it looks like you’re trying to use Octopus as a Build Server, which it really isn’t what it’s designed for: https://octopus.com/blog/how-octopus-complements-build-server

Bonus blog from our CEO covering this when Octopus was first created: Octopus vs. Build Servers - Octopus Deploy

Hope that helps but feel free to reach out if you have any questions at all!

Best Regards,

9

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.