Problems with OctoDeploy in TeamCity

voicebooth · 4 February 2019 16:46

I have installed multiple instances of TeamCity (latest build 2018.2.2 (build 61245)) on different Debian 9 instances, and used the OctoDeploy plugin on it to push NuGet packages to my cloud instance of Octopus Deploy.

Roughly every other build I try to push to Octopus I get the following error in the log of the build project:

[16:36:56][Octopus Deploy] Running command:   dotnet Octo.dll push --server https://xxxxxxx.octopus.app --apikey SECRET --package /opt/teamcity-agent-2/work/xxxxxxxxx/xxxxxxxxxxxxx.nupkg
[16:36:56][Octopus Deploy] Pushing packages to Octopus server
[16:36:57][Octopus Deploy] Octopus Deploy Command Line Tool, version 4.42.0
[16:36:57][Octopus Deploy] 
[16:37:57][Octopus Deploy] 
[16:37:57][Octopus Deploy] System.Exception: Unable to connect to the Octopus Deploy server. See the inner exception for details. ---> System.Net.Http.HttpRequestException: Resource temporarily unavailable ---> System.Net.Sockets.SocketException: Resource temporarily unavailable
[16:37:57][Octopus Deploy]    at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
[16:37:57][Octopus Deploy]    --- End of inner exception stack trace ---
[16:37:57][Octopus Deploy]    at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
[16:37:57][Octopus Deploy]    at System.Threading.Tasks.ValueTask`1.get_Result()
[16:37:57][Octopus Deploy]    at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[16:37:57][Octopus Deploy]    at System.Threading.Tasks.ValueTask`1.get_Result()
[16:37:57][Octopus Deploy]    at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
[16:37:57][Octopus Deploy]    at System.Threading.Tasks.ValueTask`1.get_Result()
[16:37:57][Octopus Deploy]    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
[16:37:57][Octopus Deploy]    at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
[16:37:57][Octopus Deploy]    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[16:37:57][Octopus Deploy]    at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
[16:37:57][Octopus Deploy]    at Octopus.Client.OctopusAsyncClient.DispatchRequest[TResponseResource](OctopusRequest request, Boolean readResponse) in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Client\OctopusAsyncClient.cs:line 553
[16:37:57][Octopus Deploy]    at Octopus.Client.OctopusAsyncClient.Get[TResource](String path, Object pathParameters)
[16:37:57][Octopus Deploy]    at Octopus.Client.OctopusAsyncClient.EstablishSession() in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Client\OctopusAsyncClient.cs:line 481
[16:37:57][Octopus Deploy]    --- End of inner exception stack trace ---
[16:37:57][Octopus Deploy]    at Octopus.Client.OctopusAsyncClient.EstablishSession() in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Client\OctopusAsyncClient.cs:line 494
[16:37:57][Octopus Deploy]    at Octopus.Client.OctopusAsyncClient.Create(OctopusServerEndpoint serverEndpoint, OctopusClientOptions options, Boolean addHandler) in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Client\OctopusAsyncClient.cs:line 134
[16:37:57][Octopus Deploy]    at Octopus.Client.OctopusAsyncClient.Create(OctopusServerEndpoint serverEndpoint, OctopusClientOptions options) in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Client\OctopusAsyncClient.cs:line 116
[16:37:57][Octopus Deploy]    at Octopus.Cli.Commands.ApiCommand.Execute(String[] commandLineArguments) in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Cli\Commands\ApiCommand.cs:line 140
[16:37:57][Octopus Deploy]    at Octopus.Cli.CliProgram.Run(String[] args) in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Cli\CliProgram.cs:line 52
[16:37:57][Octopus Deploy] Exit code: -3
[16:37:57][Octopus Deploy] Octo.exe exit code: 253

if I re-run the build straight after, most of the time the build finishes successfully and the NuGet packages gets pushed through.

I don’t understand what the issue here is?

Regards,
Alvaro

benpearce · 5 February 2019 01:58

Hi Alvaro,

Thanks for getting in touch.
The error message you are receiving seems to be network related. The frequency that you are receiving errors is concerning though.

How long does the connection take to fail? Is it immediate or is it taking some time before it fails?
Are you able to connect to the Octopus server using Octo.exe from a different machine that isn’t one of your build servers?
Are there any proxies between your build server and the public internet, or any firewalls that may be blocking or throttling traffic?
Can you also try running a trace route from your build server to the Octopus server and let me know the results.

Regards
Ben

voicebooth · 6 February 2019 08:52

Hi Ben,

The connection takes 30 seconds to fail, which is the default Socket timeout length it seems.

The servers in question are all Linux, so rather than using Octo.exe, it’s using the dotnet core Octo.dll file. As mentioned before, every other attempt works, but obviously for every 1 or 2 attempts, 1 of them will fail.

I’ve tried running the execution directly from bash, but I get the same random problems:

debian@build:/opt/teamcity-agent-1/temp/buildTmp/octo-temp/3.0/Core$ dotnet Octo.dll push --server https://xxxxxxx.octopus.app --apikey SECRET --package /opt/teamcity-agent-1/work/d1df6864f98d2599/xxxxxxx.nupkg --package /opt/teamcity-agent-1/work/d1df6864f98d2599/xxxxxxxxxxx.nupkg --package /opt/teamcity-agent-1/work/d1df6864f98d2599/xxxxxxxxx.nupkg
Octopus Deploy Command Line Tool, version 4.42.0



System.Exception: Unable to connect to the Octopus Deploy server. See the inner exception for details. ---> System.Net.Http.HttpRequestException: Resource temporarily unavailable ---> System.Net.Sockets.SocketException: Resource temporarily unavailable
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Octopus.Client.OctopusAsyncClient.DispatchRequest[TResponseResource](OctopusRequest request, Boolean readResponse) in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Client\OctopusAsyncClient.cs:line 553
   at Octopus.Client.OctopusAsyncClient.Get[TResource](String path, Object pathParameters)
   at Octopus.Client.OctopusAsyncClient.EstablishSession() in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Client\OctopusAsyncClient.cs:line 481
   --- End of inner exception stack trace ---
   at Octopus.Client.OctopusAsyncClient.EstablishSession() in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Client\OctopusAsyncClient.cs:line 494
   at Octopus.Client.OctopusAsyncClient.Create(OctopusServerEndpoint serverEndpoint, OctopusClientOptions options, Boolean addHandler) in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Client\OctopusAsyncClient.cs:line 134
   at Octopus.Client.OctopusAsyncClient.Create(OctopusServerEndpoint serverEndpoint, OctopusClientOptions options) in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Client\OctopusAsyncClient.cs:line 116
   at Octopus.Cli.Commands.ApiCommand.Execute(String[] commandLineArguments) in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Cli\Commands\ApiCommand.cs:line 140
   at Octopus.Cli.CliProgram.Run(String[] args) in Z:\buildAgent\workDir\d8818cc5a2905d1\source\Octopus.Cli\CliProgram.cs:line 52
Exit code: -3

The build server is on a dedicated server with no proxies or firewalls in the way.

As I’ve been trying to get a traceroute done for you, I’ve spotted some DNS resolving issues, which may related to the openstack DNS itself, this may be the cause of the issue so I’ll go and change some nameservers first to see if this fixes the issue.

system · 8 March 2019 23:44

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.