Tomcat Manager deploy fails with read timeout

steve.small.slalom · 25 April 2018 13:45

Problem

I’m working on deploying WAR files to Tomcat / TomEE with Octopus Deploy using the built-in step for Tomcat. I had done test deployments with a sandbox environment that worked great, but now when targeting more realistic infrastructure I’m running into issues. It seems like Octopus is not receiving a response back from the Tomcat Manager that it is deploying the WAR to, even though Tomcat is successfully deploying. This seems to cause Octopus to retry 5 times until it eventually fails the deployment. When I look at Tomcat after this, the app seems to be deployed successfully. The ultimate exception that Octopus is throwing is a java.net.SocketTimeoutException: Read timed out, which suggests to me that Octopus isn’t receiving the response it expects from the Tomcat Manager in time, and so it retries until it eventually fails.

Has anyone encountered this in the past, or have any tips about how I can affect the timeout value that Octopus/Calamari is using when communicating with the Tomcat Manager? Any ideas would be appreciated, thanks in advance.

Technical Specifications

Polling tentacle, Windows 2008
Deployment to Tomcat / TomEE+ (Tomcat 7/8 family) via Tomcat Manager

Log Output

06:18:33   Verbose  |       Executing feature-class 'Calamari.Deployment.Features.Java.TomcatFeature'
06:18:33   Verbose  |       Setting environment variable: OctopusEnvironment_Octopus_Tentacle_CurrentDeployment_PackageFilePath = 'C:\Octopus\Applications\Staging\com.REDACTED%3Aweb-services\5.7.0-SNAPSHOT_5\com.REDACTED%3Aweb-services@M5.7.0-SNAPSHOT@62DE25AF6040A6458BF87B2EEA677FF7.war'
06:18:33   Verbose  |       Setting environment variable: OctopusEnvironment_Tomcat_Deploy_Name = 'ed5'
06:18:33   Verbose  |       Setting environment variable: OctopusEnvironment_Tomcat_Deploy_Controller = 'http://localhost/manager'
06:18:33   Verbose  |       Setting environment variable: OctopusEnvironment_Tomcat_Deploy_User = '********'
06:18:33   Verbose  |       Setting environment variable: OctopusEnvironment_Tomcat_Deploy_Password = '********'
06:18:33   Verbose  |       Setting environment variable: OctopusEnvironment_Tomcat_Deploy_Enabled = 'True'
06:18:33   Verbose  |       Setting environment variable: OctopusEnvironment_Tomcat_Deploy_Version = ''
06:18:33   Verbose  |       Invoking java.exe to perform Tomcat integration
06:27:50   Verbose  |       TomcatOptions(controller=http://localhost/manager, user=********, password=******, deploy=true, application=C:\Octopus\Applications\Staging\com.REDACTED%3Aweb-services\5.7.0-SNAPSHOT_5\com.REDACTED%3Aweb-services@M5.7.0-SNAPSHOT@62DE25AF6040A6458BF87B2EEA677FF7.war, name=ed5, tag=, version=, state=true, alreadyDumped=true)
06:27:50   Verbose  |       Attempt 1 to deploy C:\Octopus\Applications\Staging\com.REDACTED%3Aweb-services\5.7.0-SNAPSHOT_5\com.REDACTED%3Aweb-services@M5.7.0-SNAPSHOT@62DE25AF6040A6458BF87B2EEA677FF7.war to http://localhost/manager/text/deploy?update=true&path=/ed5&
06:27:50   Verbose  |       Making request to http://localhost/manager/text/deploy?update=true&path=/ed5&
06:27:50   Verbose  |       Attempt 2 to deploy C:\Octopus\Applications\Staging\com.REDACTED%3Aweb-services\5.7.0-SNAPSHOT_5\com.REDACTED%3Aweb-services@M5.7.0-SNAPSHOT@62DE25AF6040A6458BF87B2EEA677FF7.war to http://localhost/manager/text/deploy?update=true&path=/ed5&
06:27:50   Verbose  |       Making request to http://localhost/manager/text/deploy?update=true&path=/ed5&
06:27:50   Verbose  |       Attempt 3 to deploy C:\Octopus\Applications\Staging\com.REDACTED%3Aweb-services\5.7.0-SNAPSHOT_5\com.REDACTED%3Aweb-services@M5.7.0-SNAPSHOT@62DE25AF6040A6458BF87B2EEA677FF7.war to http://localhost/manager/text/deploy?update=true&path=/ed5&
06:27:50   Verbose  |       Making request to http://localhost/manager/text/deploy?update=true&path=/ed5&
06:27:50   Verbose  |       Attempt 4 to deploy C:\Octopus\Applications\Staging\com.REDACTED%3Aweb-services\5.7.0-SNAPSHOT_5\com.REDACTED%3Aweb-services@M5.7.0-SNAPSHOT@62DE25AF6040A6458BF87B2EEA677FF7.war to http://localhost/manager/text/deploy?update=true&path=/ed5&
06:27:50   Verbose  |       Making request to http://localhost/manager/text/deploy?update=true&path=/ed5&
06:27:50   Verbose  |       Attempt 5 to deploy C:\Octopus\Applications\Staging\com.REDACTED%3Aweb-services\5.7.0-SNAPSHOT_5\com.REDACTED%3Aweb-services@M5.7.0-SNAPSHOT@62DE25AF6040A6458BF87B2EEA677FF7.war to http://localhost/manager/text/deploy?update=true&path=/ed5&
06:27:50   Verbose  |       Making request to http://localhost/manager/text/deploy?update=true&path=/ed5&
06:27:50   Verbose  |       TOMCAT-DEPLOY-ERROR-0005: An exception was thrown during the deployment. http://g.octopushq.com/JavaAppDeploy#tomcat-deploy-error-0005
06:27:50   Verbose  |       java.net.SocketTimeoutException: Read timed out
06:27:50   Verbose  |       at java.net.SocketInputStream.socketRead0(Native Method)
06:27:50   Verbose  |       at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
06:27:50   Verbose  |       at java.net.SocketInputStream.read(SocketInputStream.java:170)
06:27:50   Verbose  |       at java.net.SocketInputStream.read(SocketInputStream.java:141)
06:27:50   Verbose  |       at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
06:27:50   Verbose  |       at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
06:27:50   Verbose  |       at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
06:27:50   Verbose  |       at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
06:27:50   Verbose  |       at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
06:27:50   Verbose  |       at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
06:27:50   Verbose  |       at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
06:27:50   Verbose  |       at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
06:27:50   Verbose  |       at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
06:27:50   Verbose  |       at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
06:27:50   Verbose  |       at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
06:27:50   Verbose  |       at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
06:27:50   Verbose  |       at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
06:27:50   Verbose  |       at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
06:27:50   Verbose  |       at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
06:27:50   Verbose  |       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
06:27:50   Verbose  |       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
06:27:50   Verbose  |       at org.apache.http.client.fluent.Request.internalExecute(Request.java:173)
06:27:50   Verbose  |       at org.apache.http.client.fluent.Executor.execute(Executor.java:262)
06:27:50   Verbose  |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1$2.invoke(TomcatDeploy.kt:124)
06:27:50   Verbose  |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1$2.invoke(TomcatDeploy.kt:27)
06:27:50   Verbose  |       at org.funktionale.tries.Try$map$1.invoke(Try.kt:68)
06:27:50   Verbose  |       at org.funktionale.tries.Try$map$1.invoke(Try.kt:30)
06:27:50   Error    |       TOMCAT-DEPLOY-ERROR-0005: An exception was thrown during the deployment. http://g.octopushq.com/JavaAppDeploy#tomcat-deploy-error-0005
06:27:50   Error    |       java.net.SocketTimeoutException: Read timed out
06:27:50   Error    |       at java.net.SocketInputStream.socketRead0(Native Method)
06:27:50   Error    |       at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
06:27:50   Error    |       at java.net.SocketInputStream.read(SocketInputStream.java:170)
06:27:50   Error    |       at java.net.SocketInputStream.read(SocketInputStream.java:141)
06:27:50   Error    |       at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
06:27:50   Error    |       at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
06:27:50   Error    |       at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
06:27:50   Error    |       at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
06:27:50   Error    |       at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
06:27:50   Error    |       at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
06:27:50   Error    |       at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
06:27:50   Error    |       at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
06:27:50   Error    |       at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
06:27:50   Error    |       at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
06:27:50   Error    |       at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
06:27:50   Error    |       at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
06:27:50   Error    |       at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
06:27:50   Error    |       at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
06:27:50   Error    |       at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
06:27:50   Error    |       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
06:27:50   Error    |       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
06:27:50   Error    |       at org.apache.http.client.fluent.Request.internalExecute(Request.java:173)
06:27:50   Error    |       at org.apache.http.client.fluent.Executor.execute(Executor.java:262)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1$2.invoke(TomcatDeploy.kt:124)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1$2.invoke(TomcatDeploy.kt:27)
06:27:50   Error    |       at org.funktionale.tries.Try$map$1.invoke(Try.kt:68)
06:27:50   Error    |       at org.funktionale.tries.Try$map$1.invoke(Try.kt:30)
06:27:50   Error    |       at org.funktionale.tries.Try.flatMap(Try.kt:61)
06:27:50   Error    |       at org.funktionale.tries.Try.map(Try.kt:68)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1.doWithRetry(TomcatDeploy.kt:123)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1.doWithRetry(TomcatDeploy.kt:27)
06:27:50   Error    |       at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
06:27:50   Error    |       at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy.deployArtifact(TomcatDeploy.kt:111)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy.doDeployment(TomcatDeploy.kt:90)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy.main(TomcatDeploy.kt:34)
06:27:50   Verbose  |       at org.funktionale.tries.Try.flatMap(Try.kt:61)
06:27:50   Verbose  |       at org.funktionale.tries.Try.map(Try.kt:68)
06:27:50   Verbose  |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1.doWithRetry(TomcatDeploy.kt:123)
06:27:50   Verbose  |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1.doWithRetry(TomcatDeploy.kt:27)
06:27:50   Verbose  |       at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
06:27:50   Verbose  |       at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
06:27:50   Verbose  |       at com.octopus.calamari.tomcat.TomcatDeploy.deployArtifact(TomcatDeploy.kt:111)
06:27:50   Verbose  |       at com.octopus.calamari.tomcat.TomcatDeploy.doDeployment(TomcatDeploy.kt:90)
06:27:50   Verbose  |       at com.octopus.calamari.tomcat.TomcatDeploy.main(TomcatDeploy.kt:34)
06:27:50   Error    |       Calamari.Integration.Processes.CommandLineException: The following command: "C:\Program Files\Java\jdk\bin\java" -cp calamari.jar com.octopus.calamari.tomcat.TomcatDeploy
06:27:50   Error    |       With the working directory of: C:\Octopus\Calamari\JavaLib\1.0.97\contentFiles\any\any
06:27:50   Error    |       Failed with exit code: 1
06:27:50   Error    |       TOMCAT-DEPLOY-ERROR-0005: An exception was thrown during the deployment. http://g.octopushq.com/JavaAppDeploy#tomcat-deploy-error-0005
06:27:50   Error    |       java.net.SocketTimeoutException: Read timed out
06:27:50   Error    |       at java.net.SocketInputStream.socketRead0(Native Method)
06:27:50   Error    |       at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
06:27:50   Error    |       at java.net.SocketInputStream.read(SocketInputStream.java:170)
06:27:50   Error    |       at java.net.SocketInputStream.read(SocketInputStream.java:141)
06:27:50   Error    |       at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
06:27:50   Error    |       at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
06:27:50   Error    |       at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
06:27:50   Error    |       at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
06:27:50   Error    |       at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
06:27:50   Error    |       at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
06:27:50   Error    |       at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
06:27:50   Error    |       at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
06:27:50   Error    |       at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
06:27:50   Error    |       at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
06:27:50   Error    |       at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
06:27:50   Error    |       at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
06:27:50   Error    |       at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
06:27:50   Error    |       at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
06:27:50   Error    |       at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
06:27:50   Error    |       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
06:27:50   Error    |       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
06:27:50   Error    |       at org.apache.http.client.fluent.Request.internalExecute(Request.java:173)
06:27:50   Error    |       at org.apache.http.client.fluent.Executor.execute(Executor.java:262)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1$2.invoke(TomcatDeploy.kt:124)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1$2.invoke(TomcatDeploy.kt:27)
06:27:50   Error    |       at org.funktionale.tries.Try$map$1.invoke(Try.kt:68)
06:27:50   Error    |       at org.funktionale.tries.Try$map$1.invoke(Try.kt:30)
06:27:50   Error    |       at org.funktionale.tries.Try.flatMap(Try.kt:61)
06:27:50   Error    |       at org.funktionale.tries.Try.map(Try.kt:68)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1.doWithRetry(TomcatDeploy.kt:123)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1.doWithRetry(TomcatDeploy.kt:27)
06:27:50   Error    |       at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
06:27:50   Error    |       at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy.deployArtifact(TomcatDeploy.kt:111)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy.doDeployment(TomcatDeploy.kt:90)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy.main(TomcatDeploy.kt:34)
06:27:50   Error    |       at Calamari.Integration.Processes.CommandResult.VerifySuccess()
06:27:50   Error    |       at Calamari.Deployment.Features.Java.TomcatFeature.Execute(RunningDeployment deployment)
06:27:50   Error    |       at Calamari.Deployment.Conventions.FeatureConventionBase.Run(RunningDeployment deployment)
06:27:50   Error    |       at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
06:27:50   Error    |       at Calamari.Deployment.ConventionProcessor.RunConventions()
06:27:50   Error    |       Running rollback conventions...
06:27:50   Verbose  |       Adding journal entry:
06:27:50   Verbose  |       <Deployment Id="06e547ed-95a0-40d6-a248-aeedadedc8ab" EnvironmentId="Environments-1" TenantId="" ProjectId="Projects-21" PackageId="com.REDACTED-services" PackageVersion="5.7.0-SNAPSHOT" InstalledOn="2018-04-24 13:27:50" ExtractedFrom="C:\Octopus\Files\com.REDACTED%3Aweb-services@M5.7.0-SNAPSHOT@62DE25AF6040A6458BF87B2EEA677FF7.war" ExtractedTo="C:\Octopus\Applications\Staging\com.REDACTED%3Aweb-services\5.7.0-SNAPSHOT_5" RetentionPolicySet="Environments-1/Projects-21/Step-Deploy WAR/Machines-56/&lt;default&gt;" CustomInstallationDirectory="C:\Octopus\Applications\Staging\com.REDACTED%3Aweb-services\5.7.0-SNAPSHOT_5" WasSuccessful="False" />
06:27:50   Error    |       The following command: "C:\Program Files\Java\jdk\bin\java" -cp calamari.jar com.octopus.calamari.tomcat.TomcatDeploy
06:27:50   Error    |       With the working directory of: C:\Octopus\Calamari\JavaLib\1.0.97\contentFiles\any\any
06:27:50   Error    |       Failed with exit code: 1
06:27:50   Error    |       TOMCAT-DEPLOY-ERROR-0005: An exception was thrown during the deployment. http://g.octopushq.com/JavaAppDeploy#tomcat-deploy-error-0005
06:27:50   Error    |       java.net.SocketTimeoutException: Read timed out
06:27:50   Error    |       at java.net.SocketInputStream.socketRead0(Native Method)
06:27:50   Error    |       at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
06:27:50   Error    |       at java.net.SocketInputStream.read(SocketInputStream.java:170)
06:27:50   Error    |       at java.net.SocketInputStream.read(SocketInputStream.java:141)
06:27:50   Error    |       at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
06:27:50   Error    |       at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
06:27:50   Error    |       at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
06:27:50   Error    |       at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
06:27:50   Error    |       at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
06:27:50   Error    |       at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
06:27:50   Error    |       at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
06:27:50   Error    |       at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
06:27:50   Error    |       at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
06:27:50   Error    |       at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
06:27:50   Error    |       at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
06:27:50   Error    |       at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
06:27:50   Error    |       at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
06:27:50   Error    |       at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
06:27:50   Error    |       at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
06:27:50   Error    |       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
06:27:50   Error    |       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
06:27:50   Error    |       at org.apache.http.client.fluent.Request.internalExecute(Request.java:173)
06:27:50   Error    |       at org.apache.http.client.fluent.Executor.execute(Executor.java:262)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1$2.invoke(TomcatDeploy.kt:124)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1$2.invoke(TomcatDeploy.kt:27)
06:27:50   Error    |       at org.funktionale.tries.Try$map$1.invoke(Try.kt:68)
06:27:50   Error    |       at org.funktionale.tries.Try$map$1.invoke(Try.kt:30)
06:27:50   Error    |       at org.funktionale.tries.Try.flatMap(Try.kt:61)
06:27:50   Error    |       at org.funktionale.tries.Try.map(Try.kt:68)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1.doWithRetry(TomcatDeploy.kt:123)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy$deployArtifact$1.doWithRetry(TomcatDeploy.kt:27)
06:27:50   Error    |       at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
06:27:50   Error    |       at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy.deployArtifact(TomcatDeploy.kt:111)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy.doDeployment(TomcatDeploy.kt:90)
06:27:50   Error    |       at com.octopus.calamari.tomcat.TomcatDeploy.main(TomcatDeploy.kt:34)
06:27:50   Error    |       Calamari.Integration.Processes.CommandLineException
06:27:50   Error    |       at Calamari.Integration.Processes.CommandResult.VerifySuccess()
06:27:50   Error    |       at Calamari.Deployment.Features.Java.TomcatFeature.Execute(RunningDeployment deployment)
06:27:50   Error    |       at Calamari.Deployment.Conventions.FeatureConventionBase.Run(RunningDeployment deployment)
06:27:50   Error    |       at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
06:27:50   Error    |       at Calamari.Deployment.ConventionProcessor.RunConventions()
06:27:50   Error    |       at Calamari.Commands.Java.DeployJavaArchiveCommand.Execute(String[] commandLineArguments)
06:27:50   Error    |       at Calamari.Program.Execute(String[] args)
06:27:50   Verbose  |       Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20180424131810-4112-73 exited with code 100
06:27:51   Verbose  |       Updating manifest with output variables
06:27:51   Verbose  |       Updating manifest with action evaluated variables
06:27:51   Verbose  |       Planning retention policy
06:27:51   Fatal    |       The remote script failed with exit code 100
06:27:51   Verbose  |       at Octopus.Server.Orchestration.Targets.Scripting.ScriptResult.EnsureSuccessful()
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Java.JavaActionHandler.Execute(ActionCommand command, Machine machine)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.<>c__DisplayClass30_0.<ExecuteActionAndInitLoggingContext>b__0()
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.ExecuteWithTransientErrorDetection(Action action, Machine machine)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.ExecuteActionAndInitLoggingContext(PlannedStep step, Machine machine, PlannedAction action)
06:27:51   Verbose  |       The remote script failed with exit code 100
                    |       Octopus.Server.Orchestration.Targets.Tasks.ActivityFailedException
                    |       at Octopus.Server.Orchestration.Targets.Scripting.ScriptResult.EnsureSuccessful()
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Java.JavaActionHandler.Execute(ActionCommand command, Machine machine)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.<>c__DisplayClass30_0.<ExecuteActionAndInitLoggingContext>b__0()
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.ExecuteWithTransientErrorDetection(Action action, Machine machine)
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.ExecuteActionAndInitLoggingContext(PlannedStep step, Machine machine, PlannedAction action)
06:27:51   Fatal    |       The action Deploy WAR on COAUWEBSTG512 failed

Observations

The Tomcat server’s logs show a deployment about once every 10 seconds, but the logs from Octopus show all the retries happening at the same timestamp, which is curious…

I’ve already checked that the WAR file is smaller than the maximum file upload size set in the Tomcat Manager for this host (per the instructions from Octopus ). My WAR is about 48mb, which should be just below the 50mb default.

Shane_Gill · 27 April 2018 00:42

Hi Steve,

Thanks for getting in touch.

Tomcat timeouts were increased from 30 seconds to 60 seconds in version 2018.3.3: https://github.com/OctopusDeploy/Issues/issues/4370

If you are on a version prior to 2018.3.3 it would be worth upgrading to see if it resolves the issue.

The logs all get returned once the call to Java is complete, that is why they all have the same timestamp. They are timestamped when they are collected, that is why it appears that the retries happened all at once but they actually happened in the big gap between 06:18:33 and 06:27:50.

Cheers,
Shane

steve.small.slalom · 27 April 2018 13:26

Hi Shane,

Thank you very much for your response. Another user in the Slack community followed up with this information about the increased timeout in version 2018.3.3, and it turns out that I was on something like 2018.1.x, so I upgraded the instance and re-tested.

At first there was no appreciable change, but after testing deployments to this server with curl, I realized that the server was not responding to any deployment api calls for over 4 minutes. The 60 second timeout would certainly help, but the target was so unhealthy that it still wouldn’t matter. After cleaning up the box and restarting tomcat, I was able to get deployments to run successfully with Octopus.

Again, thank you for the quick response here, and for the great community engagement in the Slack that helped me arrive at this success. Much appreciated.

/Steve

system · 27 May 2018 13:26

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.