SSH connection with Failed: Check deployment target

richardedward_poole · 22 April 2016 09:24

Hi,

We are having an issue with setting up a SSH tentacle. We are getting the following output from our health check:

Task ID: ServerTasks-1439
Task status: Failed
Task queued: Friday, April 22, 2016 5:08 AM
Task started: Friday, April 22, 2016 5:08 AM
Task duration: 2 seconds
Server version: 3.3.9+Branch.master.Sha.35de9da28e9759d98a404c505b882114f10e2e44

                | == Failed: Check NAME health ==

05:08:12 Info | Starting health check for a limited set of deployment targets
05:08:12 Info | 1 machines will have their health checks taken.
05:08:14 Fatal | The health check failed. One or more deployment targets were not available.
|
| == Failed: Check deployment target: NAME ==
05:08:12 Verbose | Requesting upload…
05:08:12 Verbose | Establishing SSH connection…
05:08:12 Verbose | SSH connection established
05:08:12 Verbose | Beginning streaming transfer of health-check.sh to $HOME.octopus\OctopusServer\Work\20160422090812-33
05:08:12 Verbose | Establishing SFTP connection…
05:08:12 Verbose | SSFTP connection established
05:08:13 Verbose | Stream transfer complete
05:08:13 Error | Required command ‘mono’ is not available.
05:08:13 Verbose | Disposing SFTP connection…
05:08:13 Verbose | Disposing SSH connection…
05:08:13 Fatal | The remote script failed with exit code 127
05:08:13 Verbose | The remote script failed with exit code 127
| Octopus.Shared.Tasks.ActivityFailedException: The remote script failed with exit code 127
| at Octopus.Worker.Scripting.ScriptResult.EnsureSuccessful() in Y:\work\refs\tags\3.3.9\source\Octopus.Worker\Scripting\ScriptResult.cs:line 71
| at Octopus.Worker.Ssh.SshWorker.CheckHealth() in Y:\work\refs\tags\3.3.9\source\Octopus.Worker\Ssh\SshWorker.cs:line 65
| at Octopus.Server.Orchestration.Health.MachineTaskController.PerformTask(Machine machine) in Y:\work\refs\tags\3.3.9\source\Octopus.Server\Orchestration\Health\MachineTaskController.cs:line 86
| Octopus.Server version 3.3.9 (3.3.9+Branch.master.Sha.35de9da28e9759d98a404c505b882114f10e2e44)
|
| == Failed: Summary ==
05:08:14 Info | OFFLINE:
05:08:14 Info | - NAME at ssh://NAME/, error: The remote script failed with exit code 127
05:08:14 Fatal | One or more deployment targets were not online. Please see the output Log for details.

So we have checked that MONO is available and on the path.
On the server with the same user use to connect we ran:

$ which mono
/opt/mono/bin/mono

$ mono --version
Mono JIT compiler version 4.4.0 (Stable 4.4.0.122/a3fabf1 Thu Apr 21 10:52:11 EDT 2016)
Copyright © 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
LLVM: supported, not enabled.
GC: sgen

We would like to get hold of a example health-check.sh to run interactively to debug but I guess this is created dynamical and then removed.

We used the following script on the server, found from discussion ticket:
http://help.octopusdeploy.com/discussions/problems/39756-install-mono-on-centos-66

yum -y install bison gettext glib2 fontconfig libpng libpng-devel libX11 libX11-devel glib2-devel libexif glibc-devel urw-fonts java unzip gcc gcc-c++ automake autoconf libtool make bzip2 wget
cd /usr/local/src
wget http://download.mono-project.com/sources/mono/mono-4.4.0.122.tar.bz2
tar jxf mono-4.4.0.122.tar.bz2
cd mono-4.4.0
./configure --prefix=/opt/mono
make
make install

We had to remove to “freetype” dependency as it was not resolving correctly.

Any help would be appreciated.

Thanks Rich

Shane_Gill · 25 April 2016 23:57

Hi Rich,

Thanks for getting in touch.

The health check script for SSH targets:

#!/bin/bash

# Ensure required commands are available
declare -a arr=("df" "ls" "grep" "base64" "mkdir" "tar" "mono")
deps=0
for cmd in "${arr[@]}"; 
do 
	$(hash "$cmd" 2>/dev/null || { echo  >&2 "I require $cmd but it's not available."; exit 1; });
	if [[ $? != 0 ]]; then deps=1; fi
done;
if [[ $deps != 0 ]]; then exit 127; fi

# Print standard diagnostics
echo "##octopus[stdout-verbose]"
echo "Host Name: $HOSTNAME"
echo "Running As: $DOMAINNAME\\$USER"	
df -TH
echo "Bash version: $BASH_VERSION"

# Print and return Mono version
property=$(echo MonoVersion | base64)
value=$(mono -V | grep version |  grep -o '\([0-9]\+\.\+\)\+\([0-9]\+\)' | base64)
echo "##octopus[stdout-verbose]"
echo "Mono version: " $(mono -V | grep version)
echo "##octopus[setVariable name='"$property"' value='"$value"']"

# Ensure Calamari up to date
ls ~/"{{CalamariLocation}}/success" &>/dev/null && echo "Calamari version: {{CalamariVersion}}" && hasLatestCalamari=true || (echo "##octopus[stdout-warning]" && echo "Not running latest version of Calamari");
property=$(echo -n HasLatestCalamariVersion | base64)
if test "$hasLatestCalamari" = ""; then
	hasLatestCalamari=false
fi
value=$(echo -n $hasLatestCalamari | base64)
echo "##octopus[setVariable name='"$property"' value='"$value"']"

Is mono working for the user that has been configured in Octopus?

Cheers,
Shane

richardedward_poole · 28 April 2016 09:32

Hi, I rolled up the above into a script to check interactivity with the same user and got the following which looks ok. (Did not add the Calamari check)

$ ./test.sh
##octopus[stdout-verbose]
Host Name: ####
Running As: $USER
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg00-root
ext4 26G 6.5G 18G 27% /
tmpfs tmpfs 2.1G 0 2.1G 0% /dev/shm
/dev/sda1 ext4 529M 100M 403M 20% /boot
/dev/mapper/vg00-diskdump
ext4 11G 158M 9.9G 2% /diskdump
/dev/mapper/vg00-opt_CA
ext4 2.2G 415M 1.6G 21% /opt/CA
/dev/mapper/vg00-opt_OV
ext4 1.1G 243M 761M 25% /opt/OV
/dev/mapper/vg00-opt_bmc
ext4 2.2G 71M 2.0G 4% /opt/bmc
/dev/mapper/vg00-opt_bsa_bladelogic
ext4 2.2G 419M 1.6G 21% /opt/bsa/bladelogic
/dev/mapper/vg00-opt_perf
ext4 1.1G 80M 924M 8% /opt/perf
/dev/mapper/vg00-opt_tivoli_cit
ext4 1.1G 59M 945M 6% /opt/tivoli/cit
/dev/mapper/vg00-users
ext4 3.2G 97M 3.0G 4% /users
/dev/mapper/vg00-usr_bltemp
ext4 2.2G 473M 1.6G 24% /usr/bltemp
/dev/mapper/vg00-var_opt_OV
ext4 1.1G 43M 961M 5% /var/opt/OV
/dev/mapper/vg00-var_opt_perf
ext4 1.1G 90M 914M 9% /var/opt/perf
/dev/mapper/vgapp-data
ext4 53G 189M 50G 1% /Data
/dev/mapper/vgapp-opt_tripwire
ext4 2.2G 242M 1.8G 13% /opt/tripwire
Bash version: 4.1.2(1)-release
##octopus[stdout-verbose]
Mono version: Mono JIT compiler version 4.4.0 (Stable 4.4.0.122/a3fabf1 Thu Apr 21 10:52:11 EDT 2016)
##octopus[setVariable name=‘TW9ub1ZlcnNpb24K’ value=’’]
$

Few questions that might help:

What SSH client are you using, guess is a .net lib of some kind? (been using putty to check connection and run script)
Whats the exact SSH connection settings?

I think its a either a path problem with how octopus and putty differ in ssh session connection or our internal cloud security is creating a temp user for non interactive ssh.

Thanks Rich

Shane_Gill · 28 April 2016 23:26

Hi Rich,

We are using Renci.SshNetand just creating a new SshClient(connInfo).

Rob has pointed me to a stackoverflow article that might help: http://stackoverflow.com/questions/216202/why-does-an-ssh-remote-command-get-fewer-environment-variables-then-when-run-man

He also thinks a difference in interactive vs non-interactive ssh may be the culprit. Setting up a test Renci.SshNet.ShhClient may help or we could add some diagnostics to the health check.

Cheers,
Shane

richardedward_poole · 29 April 2016 10:46

Hi Guys,

I setup a test console app and started to get some more info about the paths.
The fix in my case was the following:

Edit /etc/ssh/sshd_config with the following: PermitUserEnvironment yes
Restart ssh : # /etc/init.d/sshd restart
Add: ~/.ssh/environment and add PATH to the file.

When I get a new Linux node I’ll try and write a script to set this up and test, as there was a lot of edits before this change. So could be a mix of different guesses

Thanks Rich

nandini · 1 June 2016 16:44

HI,

I have a octopus tentacle installed on a mac machine… I am getting the same " mono command not found error" when I try ssh
Requesting upload…
09:44:00Verbose
Establishing SSH connection…
09:44:00Verbose
SSH connection established
09:44:01Verbose
Beginning streaming transfer of health-check.sh to .octopus\OctopusServer\Work\20160601164400-39
09:44:01Verbose
Establishing SFTP connection…
09:44:02Verbose
SSFTP connection established
09:44:02Verbose
Stream transfer complete
09:44:03Error
Required command ‘mono’ is not available.
09:44:03Verbose
Disposing SFTP connection…
09:44:03Verbose
Disposing SSH connection…
09:44:03Fatal
The remote script failed with exit code 127
09:44:03Verbose
The remote script failed with exit code 127
Octopus.Shared.Tasks.ActivityFailedException: The remote script failed with exit code 127
at Octopus.Worker.Ssh.SshWorker.CheckHealth() in Y:\work\refs\tags\3.2.23\source\Octopus.Worker\Ssh\SshWorker.cs:line 60
at Octopus.Server.Orchestration.Health.MachineTaskController.PerformTask(Machine machine) in Y:\work\refs\tags\3.2.23\source\Octopus.Server\Orchestration\Health\MachineTaskController.cs:line 86
Octopus.Server version 3.2.23 (3.2.23+Branch.master.Sha.fd3dc908dc6c7fa39be0312f65a4449ddbe95c1c)

Could anybody help?

nandini · 1 June 2016 17:21

I tried changing sshd.config and set the path too … But no luck

Vanessa_Love · 2 June 2016 04:48

Hi Nandini,

Can you confirm that you have mono installed? That is where the error is pointing at/suggesting. Our SSH deployments are mono-dependant.

Let me know!

Vanessa

fhenin · 5 December 2016 14:43

Any update on this? I’m having the same issue currently trying to setup up a Mac as a build server

Vanessa_Love · 8 December 2016 04:54

Hi Freddy,

Thanks for getting in touch! Can you confirm that you have mono installed? If you do see the section in the following documentation (Bash startup files): http://docs.octopusdeploy.com/display/OD/SSH+Targets
Can you also provide the exact deployment log (or health check log) with the errors that you are seeing? http://docs.octopusdeploy.com/display/OD/Get+the+raw+output+from+a+task
When you say setup a Mac as a build server, how does Octopus come into play ?

Vanessa