Octopus Timeout Creating Release w/API

[John_Keippel]

We’ve started to see this occasionally across multiple recent releases. TeamCity will build, package, and publish a nuget package just fine to Octopus, it will resolve all the details in order to create the release, and then timeout. You can see this in the attached screenshot. The exception after the 10 minute timeout is below. We have 1839 packages within the Octopus library currently and the default retention policy is to keep packages for 30 days. We keep packages used in releases in prod forever and the previous 5 packages used in lower environments. Are we running into an performance issue here? Should we maybe not keep 30 days of packages in the library? Any other info or logs that might be helpfup? I checked C:\Octopus\Logs and there doesnt appear to be anything around the time of the timeout.

Footnote: it appears to be stuck again on another build while I’m typing this. Going to restart the Octopus service.

System.TimeoutException: Timeout getting response, client timeout is set to 00:10:00.
[10:19:42][Octopus Deploy]    at Octopus.Client.OctopusAsyncClient.<DispatchRequest>d__47`1.MoveNext()
[10:19:42][Octopus Deploy] --- End of stack trace from previous location where exception was thrown ---
[10:19:42][Octopus Deploy]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[10:19:42][Octopus Deploy]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[10:19:42][Octopus Deploy]    at Octopus.Client.OctopusAsyncClient.<Create>d__35`1.MoveNext()
[10:19:42][Octopus Deploy] --- End of stack trace from previous location where exception was thrown ---
[10:19:42][Octopus Deploy]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[10:19:42][Octopus Deploy]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[10:19:42][Octopus Deploy]    at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
[10:19:42][Octopus Deploy]    at Octopus.Cli.Commands.CreateReleaseCommand.<Execute>d__31.MoveNext()
[10:19:42][Octopus Deploy] --- End of stack trace from previous location where exception was thrown ---
[10:19:42][Octopus Deploy]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[10:19:42][Octopus Deploy]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[10:19:42][Octopus Deploy]    at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
[10:19:42][Octopus Deploy]    at Octopus.Cli.Commands.ApiCommand.<Execute>d__31.MoveNext()
[10:19:42][Octopus Deploy] --- End of stack trace from previous location where exception was thrown ---
[10:19:42][Octopus Deploy]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[10:19:42][Octopus Deploy]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[10:19:42][Octopus Deploy]    at Octopus.Cli.Program.Run(String[] args)
[10:19:42][Octopus Deploy] Exit code: -3
[10:19:42][Octopus Deploy] Octo.exe exit code: -3

[John_Keippel]

Attempted to stop/start the Octopus service from the Octopus Manager and it wasn’t happy with that.

Waiting for service to become Stopped. Current status: StopPending
Error: The service could not be stopped
Error: ===============================================================================
Error: Exceeded max retries
Error: System.Exception
Error:    at Octopus.Shared.Startup.ConfigureServiceHelper.Retry(Func`1 func, Int32 maxRetries)
Error:    at Octopus.Shared.Startup.ConfigureServiceHelper.ConfigureService()
Error:    at Octopus.Shared.Startup.AbstractCommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions)
Error:    at Octopus.Shared.Startup.ConsoleHost.Run(Action`1 start, Action shutdown)
Error:    at Octopus.Shared.Startup.OctopusProgram.Run()
Error: -------------------------------------------------------------------------------
Error: Terminating process with exit code 100
Error: Full error details are available in the log files at:
Error: C:\Octopus\Logs
Error: C:\Users\admjk\AppData\Local\Octopus\Logs
Error: If you need help, please send these log files to https://octopus.com/support
Error: -------------------------------------------------------------------------------
Error: 
Error: The previous command returned a non-zero exit code of: 100
Error: The command that failed was: "C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe" service --instance "OctopusServer" --stop

[henrik (Henrik Andersson)]

Hi John,

Thanks for getting in touch and I’m sorry to hear that you are having these issues.

I don’t think the amount of packages in the built-in feed plays in to the release creation as we don’t look for the packages at the point where it’s getting stuck. I’ll keep investigating to try and figure out what it could be that is causing the delay. In the meantime, you could add the --timeout parameter to your octo.exe call to increase the default 10 minute timeout to see if this at least allows the release to be created.

What’s worrying is that when you tried to restart the Octopus service while it was stuck creating a release it failed, have you been able to restart the service since?

The next time it gets stuck creating a release could you take a process dump of the Octopus service and send that to us, this might help us pinpoint what is causing the delay in release creation. I’ve created a secure file share here that you can use to upload the process dump.

Thank you and best regards,
Henrik