Health check fails on 1 tentacle

Using 3.0.6. Did a HealthCheck on this machine and this is what I get.

Starting health check for a limited set of deployment targets
07:54:39Info
The following deployment targets will be checked:
07:54:39Info

  • TestUTIL02 at https://10.10.3.98:10933/
    07:54:59Error
    The health check failed. One or more deployment targets were not available.
    Check deployment target: TestUTIL02
    07:54:39Info
    Running health check on TestUTIL02
    07:54:39Info
    Sending health check request to TestUTIL02 at https://10.10.3.98:10933/
    07:54:42Info
    This server does not have version 3.0.1.376 of the Calamari package. It will be pushed automatically.
    07:54:44Info
    d:\Octopus\Data\Calamari\3.0.1.376\Success.txt
    07:54:46Error
    D:\Octopus\Data\Work\20150728145443-2\Bootstrap.ps1 : Calamari extraction
    07:54:46Error
    failed
    07:54:46Error
    At line:1 char:37
    07:54:46Error
  • … ‘Stop’; . {. ‘d:\Octopus\Data\Work\20150728145443-2\Bootstrap.ps1’; …
    07:54:46Error
  •             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    

07:54:46Error
+ CategoryInfo : NotSpecified: (:slight_smile: [Write-Error], WriteErrorExcep
07:54:46Error
tion
07:54:46Error
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
07:54:46Error
n,Bootstrap.ps1
07:54:46Error
07:54:49Info
d:\Octopus\Data\Calamari\3.0.1.376\Success.txt
07:54:51Error
D:\Octopus\Data\Work\20150728145448-3\Bootstrap.ps1 : Calamari extraction
07:54:51Error
failed
07:54:51Error
At line:1 char:37
07:54:51Error

  • … ‘Stop’; . {. ‘d:\Octopus\Data\Work\20150728145448-3\Bootstrap.ps1’; …
    07:54:51Error
  •             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    

07:54:51Error
+ CategoryInfo : NotSpecified: (:slight_smile: [Write-Error], WriteErrorExcep
07:54:51Error
tion
07:54:51Error
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
07:54:51Error
n,Bootstrap.ps1
07:54:51Error
07:54:53Info
d:\Octopus\Data\Calamari\3.0.1.376\Success.txt
07:54:55Error
D:\Octopus\Data\Work\20150728145452-4\Bootstrap.ps1 : Calamari extraction
07:54:55Error
failed
07:54:55Error
At line:1 char:37
07:54:55Error

  • … ‘Stop’; . {. ‘d:\Octopus\Data\Work\20150728145452-4\Bootstrap.ps1’; …
    07:54:55Error
  •             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    

07:54:55Error
+ CategoryInfo : NotSpecified: (:slight_smile: [Write-Error], WriteErrorExcep
07:54:55Error
tion
07:54:55Error
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
07:54:55Error
n,Bootstrap.ps1
07:54:55Error
07:54:57Info
d:\Octopus\Data\Calamari\3.0.1.376\Success.txt
07:54:59Error
D:\Octopus\Data\Work\20150728145456-5\Bootstrap.ps1 : Calamari extraction
07:54:59Error
failed
07:54:59Error
At line:1 char:37
07:54:59Error

  • … ‘Stop’; . {. ‘d:\Octopus\Data\Work\20150728145456-5\Bootstrap.ps1’; …
    07:54:59Error
  •             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    

07:54:59Error
+ CategoryInfo : NotSpecified: (:slight_smile: [Write-Error], WriteErrorExcep
07:54:59Error
tion
07:54:59Error
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
07:54:59Error
n,Bootstrap.ps1
07:54:59Error
07:54:59Error
Tentacle upgrade failed
07:54:59Fatal
The remote script failed with exit code 1

Here is the output file from the Work directory

Output.log (1003 Bytes)

When I manually copied the Calamari files from another tentacle, I was able to get past that step in the HealthCheck process. But now I get the error below. One question is why Octopus keeps trying to put things at the root drive level, rather than in the “D:\Octopus\Work” directory. Notice below it tries to find files in D:\ps_scripts.

Check TestUTIL02 health
09:00:44Info
Starting health check for a limited set of deployment targets
09:00:44Info
The following deployment targets will be checked:
09:00:44Info

  • TestUTIL02 at https://10.10.3.98:10933/
    09:00:46Error
    The health check failed. One or more deployment targets were not available.
    Check deployment target: TestUTIL02
    09:00:44Info
    Running health check on TestUTIL02
    09:00:44Info
    Sending health check request to TestUTIL02 at https://10.10.3.98:10933/
    09:00:46Error
    Could not find variables file: D:\ps_scripts\Variables.json
    09:00:46Fatal
    The remote script failed with exit code 1
    Health results:
    09:00:46Info
  • OFFLINE: TestUTIL02 at https://10.10.3.98:10933/
    09:00:46Fatal
    One or more Agents were not online. Please see the output log for details.

Hi Chris,

Could you please try hitting the “ReInstall” button from the Tentacle manager? (see attached screenshot)

Let me know how it goes after that

Thanks

Did that then re-ran the health check. Got the following

Starting health check for a limited set of deployment targets
09:29:16Info
The following deployment targets will be checked:
09:29:16Info

  • TestUTIL02 at https://10.10.3.98:10933/
    09:29:21Info
    Health check completed successfully.
    Check deployment target: TestUTIL02
    09:29:16Info
    Running health check on TestUTIL02
    09:29:16Info
    Sending health check request to TestUTIL02 at https://10.10.3.98:10933/
    09:29:18Error
    Set-Location : Cannot find path ‘D:\ps_scripts’ because it does not exist.
    09:29:18Error
    At
    09:29:18Error
    C:\Windows\System32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1:2
    09:29:18Error
    char:1
    09:29:18Error
  • Set-Location D:\ps_scripts
    09:29:18Error

09:29:18Error
+ CategoryInfo : ObjectNotFound: (D:\ps_scripts:String) [Set-Loc
09:29:18Error
ation], ItemNotFoundException
09:29:18Error
+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.SetLo
09:29:18Error
cationCommand
09:29:18Error
09:29:20Error
Set-Location : Cannot find path ‘D:\ps_scripts’ because it does not exist.
09:29:20Error
At
09:29:20Error
C:\Windows\System32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1:2
09:29:20Error
char:1
09:29:20Error

  • Set-Location D:\ps_scripts
    09:29:20Error

09:29:20Error
+ CategoryInfo : ObjectNotFound: (D:\ps_scripts:String) [Set-Loc
09:29:20Error
ation], ItemNotFoundException
09:29:20Error
+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.SetLo
09:29:20Error
cationCommand
09:29:20Error
09:29:21Info
Drive C: has 48 GB available
09:29:21Info
Drive D: has 60 GB available
09:29:21Info
Running Calamari version 3.0.1.376
09:29:21Info
Running Tentacle version 3.0.6.2140
09:29:21Info
Deployment Target health check successful.
09:29:21Info
Tentacle health check done
Health results:
09:29:21Info

Hi Chris,

The Set-Location call seems to be happening from your Powershell profile at C:\Windows\System32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1. This file gets executed at the beginning of each Powershell session, reason why it shows up during our health check.

Please check the contents of that file, specially around the call for Set-Location (which can also be found with the alias CD)

Thanks,

Dalmiro

That file:

C:\Windows\System32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1

Does not exist on the tentacle machine.

So I thought I’d just create the missing directory for Octopus. Then I get this:

Starting health check for deployment targets in: LogViewerPROD
13:31:43
Info
The following deployment targets will be checked:
13:31:43
Info

  • EBIUTIL02 at https://10.10.3.98:10933/
    13:31:46
    Error
    The health check failed. One or more deployment targets were not available.
    Check deployment target: EBIUTIL02https://ebideploy.exactbid.com/app
    13:31:43
    Info
    Running health check on EBIUTIL02
    13:31:43
    Info
    Sending health check request to EBIUTIL02 at https://10.10.3.98:10933/
    13:31:45
    Error
    Could not find variables file: D:\ps_scripts\Variables.json
    13:31:45
    Fatal
    The remote script failed with exit code 1

From: Chris Becker
Sent: Tuesday, July 28, 2015 1:29 PM
To: 'Dalmiro Grañas’
Subject: RE: Health check fails on 1 tentacle [Problems #36512]

That file:

C:\Windows\System32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1

Does not exist on the tentacle machine.

I also get this in the Tentacle log:

2015-07-28 08:54:36.0380 12 INFO listen://0.0.0.0:10933/ 12 Accepted TCP client: 10.10.3.157:59029
2015-07-28 08:54:36.0380 31 INFO listen://0.0.0.0:10933/ 31 Performing SSL (TLS 1.0) server handshake
2015-07-28 08:54:36.0380 31 INFO listen://0.0.0.0:10933/ 31 Secure connection established, client is not yet authenticated
2015-07-28 08:54:36.0693 31 INFO listen://0.0.0.0:10933/ 31 Client at 10.10.3.157:59029 authenticated as C06CCA6A4C92BE004B5802D21D05318814462C09
2015-07-28 09:10:46.5242 31 INFO listen://0.0.0.0:10933/ 31 Unhandled error when handling request from client: 10.10.3.157:59029
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadLine()
at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 91
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 65
at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func2 incomingRequestProcessor) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 110 at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsServer(Func2 incomingRequestProcessor, Func`2 pendingRequests) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 90
at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 122

From: Chris Becker
Sent: Tuesday, July 28, 2015 1:33 PM
To: 'Dalmiro Grañas’
Subject: RE: Health check fails on 1 tentacle [Problems #36512]

So I thought I’d just create the missing directory for Octopus. Then I get this:

Starting health check for deployment targets in: LogViewerPROD
13:31:43
Info
The following deployment targets will be checked:
13:31:43
Info

  • EBIUTIL02 at https://10.10.3.98:10933/
    13:31:46
    Error
    The health check failed. One or more deployment targets were not available.
    Check deployment target: EBIUTIL02https://ebideploy.exactbid.com/app
    13:31:43
    Info
    Running health check on EBIUTIL02
    13:31:43
    Info
    Sending health check request to EBIUTIL02 at https://10.10.3.98:10933/
    13:31:45
    Error
    Could not find variables file: D:\ps_scripts\Variables.json
    13:31:45
    Fatal
    The remote script failed with exit code 1

From: Chris Becker
Sent: Tuesday, July 28, 2015 1:29 PM
To: 'Dalmiro Grañas’
Subject: RE: Health check fails on 1 tentacle [Problems #36512]

That file:

C:\Windows\System32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1

Does not exist on the tentacle machine.

Hi Chris,

This path “D:\ps_scripts” isn’t something Octopus/Tentacle sets, so we’re pretty stumped about where it’s coming from. But this error:

+ Set-Location D:\ps_scripts\ 09:29:18Error
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 09:29:18Error
+ CategoryInfo : ObjectNotFound: (D:\ps_scripts:String) [Set-Loc 09:29:18Error
ation], ItemNotFoundException 09:29:18Error
+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.SetLo 09:29:18Error
cationCommand 09:29:18Error
09:29:20Error
Set-Location : Cannot find path 'D:\ps_scripts\' because it does not exist.
09:29:20Error
At
09:29:20Error
C:\Windows\System32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1:2
09:29:20Error
char:1

Really suggests that there’s a custom PowerShell profile trying to set that path. There are a few places where this profile can be set:

http://blogs.technet.com/b/heyscriptingguy/archive/2013/01/04/understanding-and-using-powershell-profiles.aspx

If you use the Script Console in Octopus (go to Tasks -> Script Console in the web UI) can you run this script on the Tentacle?

Write-Host "$profile"

Paul

When I run that script on my tentacle, I get the exact same error.

So I did find C:\Windows\System32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1 on my machine (I think I reported it did not exist prior, and I am not sure what happened.)

And that Set-Location is in there. So that is where it is coming from.

However, if I create that folder, I then get the following error:

Could not find variables file: D:\ps_scripts\Variables.json

That seems to me that Octopus is writing variables.json to some other directory, but assuming that its current directory is somewhere else and not taking into account that the Microsoft.PowerShell_profile.ps1 can and does change the current directory with the Set-Location call.

Hi Chris,

You’re gonna have to remove the Set-Location call from the Profile script for this to work.

Thanks,

Dalmiro

I get that… but what if that is what other people need on that machine

Can you respond to this question:

It seems to me that Octopus is writing variables.json to some other directory (the Octopus\Work<JobID> directory), but assuming that its current directory is that somewhere else and not taking into account that the Microsoft.PowerShell_profile.ps1 can and does change the current directory with the Set-Location call.
Is there a fix that Octo can put in to make sure it is always in the Work<JobID> subfolder?

This tentacle box is shared among many applications and IT users, so that Set-Location is apparently something that is helpful to our IT department. I will work with them to maybe move the Set-Location call out of the global profile and into their own user PS profile but also might be indicating some assumptions that the Octopus scripts are making that might be able to be cleaned up?

Hi Chris,

I’ll bring up that question to the team to see if there’s something we can do to make sure the Tentacle always uses the correct path.

I do recommend you though to work with your IT department to avoid having that line of code into the global profile.

Thanks,

Dalmiro

I had this same exact problem and I absolutely have to have Set-Location to a specific path in my global profile for our business to run.

I wanted to drop a note that a work around is that you only have to remove the Set-Location from your PS global profile for a few minutes while you install the Tentacle. Once you have the tentacle installed you can add the Set-Location back to the global profile.

You custom PS scripts will run fine and the Tentacle will be properly installed.

Brad.

Guys we need to try and get a solution to this problem. All of the servers that we deploy to have a custom $profile for each service account. I’m using a specific service account for my Octopus Tentacle service so I can set a custom profile and Path for my Octopus Tentacle service but I need to know what path to use?

I cannot leave it blank because it will still default to an invalid path.

We solved this problem!

We moved from using a single PowerShell Profile for all users (C:\Windows\System32\WindowsPowerShell\v1.0\Profile.ps1) to separate profiles per user\service account (C:\Users\UserAccount\Documents\WindowsPowerShell\Profile.ps1).

In theory this should work because specific profile paths are now set at the user level and not globally. The problem was that the Tentacle was still failing the health\connectivity check.

We found that we still had a global profile at C:\Windows\System32\WindowsPowerShell\v1.0\Profile.ps1 with only two commented out lines in it, effectively empty. After renaming this file to Profile.ps1.bak (or deleting it) the health check was then successful. So the problem was you cannot have a profile at C:\Windows\System32\WindowsPowerShell\v1.0 at all, even if it is blank!

If you have the requirement that you have to have applications running with default PowerShell profiles run those apps with a different user account than the user running your Octopus Deploy tentacle. Then create your PowerShell custom profiles at the user level - C:\Users\UserAccount\Documents\WindowsPowerShell\Profile.ps1.