Long deployments "failing" from TeamCity's perspective

Hey Paul,

I’ve recently converted my original TeamCity-initiated deployment from raw powershell/octo.exe to your new plugin. The plugin works as advertised and this is a much cleaner approach. Thanks for putting that together.

Sometimes my deployments take longer than 10 minutes (usually a result of a stubbornly slow service start). This is normal behavior and Octopus always reports success. However, I’ve noticed that when a deployment crosses the 10 minute threshold, TeamCity gives up and assumes failure. When I converted to your new plugin and saw the “Wait for deployment to complete” checkbox, I was hopeful that it would wait beyond the 10 minutes that it sometimes takes. Alas, I still get the same “failure” result in TeamCity even though Octopus shows a successful deployment.

Is it possible to extend the apparent timeout that exists under the covers?

Hi David,

In the ‘additional command line arguments’ for your create/deploy release step, add this:

--deploymenttimeout=00:20:00

This will make the deployment wait 20 minutes (the default is 10).

Paul

Perfect. Thanks!

Hey Paul,

Now that I’ve got the custom build of OctoPack working (hopefully you’ll accept my pull request), I was able to revisit this issue and try your fix. Unfortunately, it didn’t work. Prior to applying the parameter you suggested, TeamCity would time out after 10 minutes. After applying the parameter (see the log below), TeamCity has curiously decided to timeout after 15 minutes despite specifying a 20 minute timeout.

Do you have any ideas?

Thanks for your help.

David

Log snippet showing the timeout occur and the (sanitized) Octo command that was executed:

[08:55:29][Octopus Deploy] Deployment has not yet finished. Time taken: 13m:22s
[08:55:39][Octopus Deploy] Deployment has not yet finished. Time taken: 14m:32s
[08:55:49][Octopus Deploy] Deployment has not yet finished. Time taken: 14m:42s
[08:55:59][Octopus Deploy] Deployment has not yet finished. Time taken: 14m:52s
[08:56:09][Octopus Deploy] Deployment has not yet finished. Time taken: 14m:2s
[08:56:19][Octopus Deploy] Deployment has not yet finished. Time taken: 14m:12s
[08:56:29][Octopus Deploy] Deployment has not yet finished. Time taken: 14m:22s
[08:56:39][Octopus Deploy] Deployment has not yet finished. Time taken: 15m:32s
[08:56:49][Octopus Deploy] Deployment has not yet finished. Time taken: 15m:42s
[08:56:59][Octopus Deploy] Deployment has not yet finished. Time taken: 15m:52s
[08:57:01][Octopus Deploy] The build BUILD - Dev #1.6.0.387 {builId=2191} has been running for more than 15 minutes. Terminating…
[08:57:10][Octopus Deploy] Deployment has not yet finished. Time taken: 15m:3s
[08:57:10][Octopus Deploy] Dumping threads before termination:
[08:57:10][Octopus Deploy] PID: 1832, PPID: 2992, Command line: “C:\TeamCity\buildAgent\temp\buildTmp\octo-temp\octo.exe” create-release --server=“http://octopus” --apikey=“SECRET” --project=“PROJECT” --enableservicemessages --version=“1.6.0.387” --deployto=“Development” --waitfordeployment --deploymenttimeout=00:20:00

Hi David,

This seems to be down to TeamCity now. In Administration->General Settings, do you have a value specified in “Default build execution timeout”? Or do you have a build failure condition specified on your build?

Paul

It is set to 0 based upon the label next to the box: “0 indicates no execution timeout”.

I didn’t think to review the Build Failure Conditions. Though I don’t have any custom ones, you did discover the culprit. “it runs longer than” is checked with a value of 15 minutes. This does not appear to be a default setting as I created a new build configuration to see what gets checked by default. Clearly human error on our part.

Thanks for helping us resolve the issue, Paul. Might be important to note in your module’s documentation that this setting will override (trump) your command-line timeout setting.

David