We are using Octo.exe to deploy from our TFS build. When Octo is looking for the latest packages, it times out. Here is the log:
Octopus Command Line Tool, version 1.1.17.62
Finding project: Prepaid
Handshaking with Octopus server: http://(server name):8080/api
Handshake successful. Octopus version: 1.5.1.1652; API version: 2.0.0
Finding environments…
Finding steps for project…
Resolving NuGet package versions…
Finding latest NuGet package for step: (package name here)
The operation has timed outSystem.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at OctopusTools.Client.OctopusSession.Get[TResource](String path, QueryString queryString)
at OctopusTools.Client.OctopusSession.Get[TResource](String path)
at OctopusTools.Client.OctopusSession.List[TResource](String path)
at ProjectExtensions.GetLatestPackageForStep(IOctopusSession session, Step step)
at OctopusTools.Commands.CreateReleaseCommand.Execute()at OctopusTools.Infrastructure.CommandProcessor.Process(String[] args)
This timeout is apparently not related to the “–deploymenttimeout” command line argument, because I have that set to one hour. When creating a release manually in Octopus, sometimes it takes minutes to find the latest NuGet package versions. It would be nice if Octo.exe had a configurable timeout for this.
You may need to look into using a faster NuGet server - I recommend NuGet.Lucene. We’ll be bundling this into the next release of Octopus Deploy.
As a workaround, if you know the version number of the package you want to include in the release, you can tell Octo.exe that explicitly and it will avoid querying NuGet. You can read more about how to do this here:
Is there any updates to this, we also see random timeouts with octopus on the same host and using a fileshare nuget. It doesn’t happen all the time, just randomly (and typically when we are keen to get a build out!).
Is there a way we can increase the timeout for Octo.exe as I would like to stick with the fileshare nuget repository.
Another related issue with octo.exe is that octo.exe might timeout, but the deployment would still succeed. Had this happen alot with long-running uploads to Azure.
I am also seeing octo.exe timeouts fairly frequently, from a TC agent on the octo server as well as a TC agent on a separate server. Are there any workarounds until the next release?
Hi,
Earlier, intermittently we are running into below exception, but now the exception is happening always. Is there a suggestion to overcome this issue?
[22:27:31]E: bt33 (2m:02s)
[22:27:31] : TeamCity server version is 8.1.3 (build 30101)
[22:27:31] : Skip checking for changes - there are no VCS roots defined
[22:27:31] : Agent time zone: America/Chicago
[22:27:31] : Agent is running under JRE: 1.7.0_51-b13
[22:27:31] : Publishing internal artifacts (3s)
[22:27:35] : [Publishing internal artifacts] Sending using ArtifactsCachePublisher
[22:27:35] : [Publishing internal artifacts] Sending using WebPublisher
[22:27:31] : Clearing temporary directory: E:\TeamCity\buildAgent\temp\buildTmp
[22:27:31] : Checkout directory: E:\TeamCity\buildAgent\work\9ef29b99cdbc22e7
[22:27:31] : Build preparation done
[22:27:31]E: Step 1/3: Orchard.Web: Create release and deploy (OctopusDeploy: Create release) (2m:01s)
[22:27:31] : [Step 1/3] ##teamcity[buildStatisticValue key=‘buildStageDuration:firstStepPreparation’ value=‘202.0’]
[22:27:31] : [Step 1/3] ##teamcity[buildStatisticValue key=‘buildStageDuration:buildStepRUNNER_49’ value=‘0.0’]
[22:27:31] : [Step 1/3] Octopus Deploy (2m:01s)
[22:27:31] : [Octopus Deploy] Running command: octo.exe create-release --server http://prodnetbld01:82 --apikey SECRET --project Client.Web --enableservicemessages --deployto QA --waitfordeployment
[22:27:31] : [Octopus Deploy] Creating Octopus Deploy release
[22:27:32] : [Octopus Deploy] Octopus Deploy Command Line Tool, version 2.4.9.22
[22:27:32] : [Octopus Deploy]
[22:27:32] : [Octopus Deploy] Handshaking with Octopus server: http://xxxx
[22:27:32] : [Octopus Deploy] Handshake successful. Octopus version: 2.3.3.1369; API version: 3.0.0
[22:27:32] : [Octopus Deploy] Finding project: Client.Web
[22:27:32] : [Octopus Deploy] Finding deployment process for project: Orchard.Web
[22:27:33] : [Octopus Deploy] Finding release template…
[22:27:33] : [Octopus Deploy] Resolving NuGet package versions…
[22:27:33] : [Octopus Deploy] Finding latest NuGet package for step: Deploy
[22:29:33] : [Octopus Deploy] System.Net.WebException: The operation has timed out
[22:29:33] : [Octopus Deploy] at System.Net.HttpWebRequest.GetResponse()
[22:29:33] : [Octopus Deploy] at Octopus.Client.OctopusClient.DispatchRequest[TResponseResource](OctopusRequest request, Boolean readResponse)
[22:29:33] : [Octopus Deploy] at Octopus.Client.OctopusClient.Get[TResource](String path, Object pathParameters)
[22:29:33] : [Octopus Deploy] at OctopusTools.Commands.CreateReleaseCommand.Execute()
[22:29:33] : [Octopus Deploy] at OctopusTools.Commands.ApiCommand.Execute(String[] commandLineArguments)
[22:29:33] : [Octopus Deploy] at OctopusTools.Program.Main(String[] args)
[22:29:33] : [Octopus Deploy] Exit code: -3
[22:29:33] : [Octopus Deploy] Octo.exe exit code: -3
[22:29:33] : [Step 1/3] Unable to create or deploy release. Please check the build log for details on the error.
[22:29:33] : [Step 1/3] ##teamcity[buildStatisticValue key=‘buildStageDuration:buildStepRUNNER_49’ value=‘121513.0’]
Can you please try to browse to your TeamCity NuGet feed using the TC interface.
What we are seeing is a timeout browsing that feed, and our plugin browses it in much the same way as you would.
Are you able to search your own feeds?
Hi Vanessa,
Here is how our build and deployment systems are setup:
TeamCity builds the code and publishes output to a folder
TeamCity then runs another step to package content of folder as nuget
Package
TeamCity calls Octo.exe to deploy a Release (e.g., QA)
Octo.exe try to get the latest NuGet package created in step 2 and
deploy to server (e.g., QA environment).
The step is failing when Octo.exe try to get latest feed. But if I try the
same using OctopusDeploy UI, it works and* I do see that it takes a while
to get the latest build# *and show it on UI during deployment.
Is there a setting to increase timeout for scanning for latest nuget
package? What is default timeout?
Sorry for the delay in getting back to you here. Octo.exe deploy release has a 10 minute time out but your build log shows a timeout after 2 minutes.
Are you able to run the following command octo.exe create-release --server http://prodnetbld01:82 --apikey SECRET --project Client.Web --enableservicemessages --deployto QA --waitfordeployment from command line on the team city server just to be sure that it does not timeout after 2 minutes and provide any logs that it gives.
Thanks
Hi Vanessa,
I am able to create release using Octopus UI. The issue I am seeing is
intermittent. After our last communications, it started working most of the
time, intermittently failed to create a release from teamcity. Not sure if
there is any option to capture debug logs around this process.
TeamCity does have build logs - so if you think the error is different then it would be worth a shot at looking at them.
But unless we can pin point the issue, like running the command from the TC server outside of TeamCity and seeing the same behavior I am not sure what we can track down here.
I’m seeing a similar intermittent timeout on TeamCity (v9) via Octo.exe (v2.5.3.33) to OctoDeploy server (v2.6.0.778) with no obvious error as Rajesh:
[19:38:19][Octopus Deploy] Waiting for 1 deployment(s) to complete....
[19:39:14][Octopus Deploy] Success: Acquire packages
[19:39:25][Octopus Deploy] Success: Step 1: My Web
[19:39:25] [Success: Step 1: My Web] Executing step: My Web
[19:39:25] [Success: Step 1: My Web] Success: MyWeb.Cloudapp.net
[19:39:25] [Success: MyWeb.Cloudapp.net] Success: Deploy My Web
[19:39:25] [Success: MyWeb.Cloudapp.net] Success: Clean up config files
[19:39:25] [Success: MyWeb.Cloudapp.net] Success: Set Credentials
[19:39:25] [Success: Set Credentials] Running "Set Credentials" on "My Web"
[19:39:25] [Success: Set Credentials] Success: Tentacle script execution
[19:41:44][Octopus Deploy] System.Net.WebException: The operation has timed out
[19:41:44][Octopus Deploy] at System.Net.HttpWebRequest.GetResponse()
The deployment succeeds, and the website has the release that was configured for deployment is present on the server (e.g. 1.3.49 was deployed, and that’s what is on the server), however Octo.exe throws this TimeoutException to TeamCity, and this fails the build. Why is there this 2m19s lag after success of the last step and Octo.exe failing with a timeout?
Thanks for getting in touch. After the final log line is printed, we do make one extra request to get the task status and check the deployment log again. It looks like this request is timing out. The timeout is set to 2 minutes - it may be that your Octopus deploy server is busy at this time.
What might also make sense, because the timeout could indicate more than just a transient problem, is to retry a few times, and write warnings on timeout fail. This could give troubleshooters valuable information to help root cause issues.