Octopus Tentacle/Agent Inactivity Timeout?

Jon_Govednik · 7 July 2015 19:09

I’m designing a production Octopus environment where the Octopus server lives in the Dev domain with the Source Control and Build servers, however the Production domain has more than a few levels of security behind it and our internal security team has deemed that the Octopus agents cannot have permanent network access to the Octopus server.

The compromise is that we will script/automate the enable/disable of the firewall rules to allow the Octopus server to talk to the agents in the production tier; however I’m concerned that installing an agent then taking it “offline” (by disabling the network access between the Octopus server and the Agents) will result in the eventual timeout/expiration of the Agent boxes.

There may be 180+ days between deployments for some of our (“legacy”) applications, so any kind of timeout could be problematic if it needs to “phone home” on a regular basis to stay alive - or if the Octopus server needs to “ping” the agents to keep them listed as viable deployment endpoints.

If the agents are installed and configured as “Listening Tentacles”, is there a maximum amount of time they can be offline before Octopus will not allow a deployment to be sent to those agents?

Dalmiro · 8 July 2015 16:36

Hi Jon,

Thanks for reaching out. There’s is a time limit where the Octopus Server will stop trying to reach the Listening Tentacles. I don’t have the specific time at hand, but i can get it for you if you need it.

What you can do to avoid this, is disable the machines when you are not deploying to them to stop any unnecessary communication (see attached screenshot). The disable/enable process could be automated and bundled along the firewall script. I can give you a hand with this process if you want.

Thanks,

Dalmiro

Jon_Govednik · 12 August 2015 21:42

Thanks Dalmiro, I imagine the call to disable a machine would be to post to the REST API :

/api/machines/{id}
{IsDisabled=true}

And to get the machines in a given environment:
/api/environments/{id}/machines

Do you happen to have a powershell script that would query for each server in an environment & disable it? If so, i’ll gladly turn this into a step template (disable all machines in an environment) - which may have limited audience, but i’ll have to do it anyway & might as well share it.

Damian_Brady · 13 August 2015 07:05

Hi Jon,

Thanks for the reply. I’ll jump in here simply because I’ve done something very similar before

The call to disable a machine is actually an update. You’ll need to do a PUT request with the machine details as a JSON body, changing the IsDisabled property to true.

To get all machines in an environment, you’re correct - GET /api/environments/{id}/machines will do it.

The best strategy would be:

Get all machines using api/environments/{id}/machines
Loop through the Items collection in the resulting object
Update each machine to set the IsDisabled property to true
For each machine, make a PUT request to the Links.Self property - including the updated JSON in the body.

Also be careful that the machines URL returns paged data. You might need to follow the links at the end of the JSON body to get subsequent pages.

I don’t have a Powershell script, however I did create a Gist (for 2.6) which does a similar thing using JavaScript and DOM manipulation.

Good luck, and let me know how you go!

Damo

Jon_Govednik · 14 August 2015 17:35

Trying to do an update against a single machine using invoke-webRequest -Method POST seems to be failing with a “Method not allowed”.
I’m able to browse to this url using my browser (http://octopusserver/api/machines/Machines-40), and confirm I can get there with my API key via powershell (apikey for my user is system administrator - testing with a restricted API key for that project returned the same “method not allowed” error).
Get works fine, but POST is failing as not allowed.

Powershell code:

Function Set-OcotpusTentacleDisabled{
	Param(
		[switch]$SetEnabled,
		[string]$MachineID, #"Machines-40"
		[string]$APIKey, #API-XXXXXXXXXXXXXXXXXXXXXXXXX
		[string]$OctopusApiURI #http://octopus/api
	)
	try{
		$machineJSON=Invoke-WebRequest -URI "$OctopusApiURI/machines/$MachineID" -Header @{ "X-Octopus-ApiKey" = $apiKey } -Method GET;
	}catch{
		throw "Set-OcotpusTentacleDisabled: Failed to find Tentacle ID: $MachineID :: $_ "
	}
	$machine=$machineJSON.content|ConvertFrom-JSON;
	#$machine=$($Machines.items|?{$_.id -eq $MachineID})
	if($SetEnabled -eq $true){
		write-host "Set-OcotpusTentacleDisabled: Set to ENABLED";
		if($machine.IsDisabled -ne $False){$machine.IsDisabled=$False;}
	}else{
		write-host "Set-OcotpusTentacleDisabled: Set to DISABLED";
		if($machine.IsDisabled -ne $True){$machine.IsDisabled=$True;}
	}
	$bodyJson=($machine|ConvertTo-JSON);
	write-host "Set-OcotpusTentacleDisabled: Posting changes to the MachineID to Octopus...";
	try{
		Invoke-WebRequest -URI "$OctopusApiURI/machines/$($machine.id)" -Header @{ "X-Octopus-ApiKey" = $apiKey } -Method POST -Body $bodyJson;
	}catch{
		throw "Set-OcotpusTentacleDisabled: Failed to set the MachineID ($MachineID):: $_"
	}
}
Set-OcotpusTentacleDisabled -MachineID "Machines-40" -APIKey 'API-XXXXXXXXXXXXXXXXXXXXXXXXXXX' -OctopusApiURI 'http://octopusserver/api'

Error from command line execution:

Set-OcotpusTentacleDisabled: Failed to set the MachineID (Machines-40):: The remote server returned an error: (405) Method Not Allowed.
At line:27 char:1
+ throw "Set-OcotpusTentacleDisabled: Failed to set the MachineID ($MachineID):: $ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Set-OcotpusTent...od Not Allowed.:String) [], RuntimeException
    + FullyQualifiedErrorId : Set-OcotpusTentacleDisabled: Failed to set the MachineID (Machines-40):: The remote server returned an error: (405) Method Not Allowed.

Damian_Brady · 17 August 2015 00:20

Hi Jon,

A POST would mean creating a new machine. You’ll need to make it a PUT - i.e. modifying an existing machine.

The failure will be because you’re trying to create a machine with the same Id as an existing one.

I hope that helps!

Damo

Jon_Govednik · 19 August 2015 15:53

Damo,
Perfect, thanks! just that simple change from POST to PUT now properly disables/enables the machine. Cycling through the ITEM list for Environment allows retrieval of all servers within an environment, and then enable/disabling them.
Here’s the fully functional code + examples for sharing.

Function Get-OcotpusEnvironmentMachines{
	Param(
		[string]$EnvironmentName, #"App1 DEV"
		[string]$APIKey, #API-XXXXXXXXXXXXXXXXXXXXXXXXX
		[string]$OctopusBaseURI #http://octopus
	)
	try{
		$environmentsJSON=Invoke-WebRequest -URI "$OctopusBaseURI/api/environments" -Header @{ "X-Octopus-ApiKey" = $apiKey } -Method GET;
	}catch{
		throw "Get-OcotpusEnvironmentMachines: Failed to retrieve list of environments :: $_ "
	}
	$environments=$environmentsJSON.content|ConvertFrom-JSON;
	$environment=$($environments.items|?{$_.name.contains($EnvironmentName)})
	write-host "EnvironmentMachinesURI: $($environment.links.machines)"
	#Get array of environment machines 
	try{
		$environmentMachinesJSON=Invoke-WebRequest -URI "$OctopusBaseURI$($environment.links.machines)" -Header @{ "X-Octopus-ApiKey" = $apiKey } -Method GET;
	}catch{
		throw "Get-OcotpusEnvironmentMachines: Failed to retrieve list of environment machines :: $_ "
	}
	$environmentMachines=$environmentMachinesJSON.content|ConvertFrom-JSON;
	if(($environmentMachines.Items -eq $null) -or ($environmentMachines.Items.count -lt 1)){
		return $false;
	}else{
		return ($environmentMachines.Items | %{$_ | select ID,Name})
	}
}

Function Set-OcotpusTentacleDisabled{
	Param(
		[switch]$SetEnabled,
		[string][Parameter(Mandatory=$true)]$MachineID, #"Machines-40"
		[string]$APIKey, #API-XXXXXXXXXXXXXXXXXXXXXXXXX
		[string]$OctopusBaseURI, #http://octopus
		[switch]$DisableOctopusServer
	)
	try{
		$machineJSON=Invoke-WebRequest -URI "$OctopusBaseURI/api/machines/$MachineID/" -Header @{ "X-Octopus-ApiKey" = $apiKey } -Method GET;
	}catch{
		throw "Set-OcotpusTentacleDisabled: Failed to find Tentacle ID: $MachineID :: $_ "
	}
	$machine=$machineJSON.content|ConvertFrom-JSON;
	#$machine=$($Machines.items|?{$_.id -eq $MachineID})
	
	#Do not disable the Octopus Server's deployment agent unless the override flag is in place
	if(($Machine.Name -eq "OctopusServer") -and ($DisableOctopusServer -eq $false)){return $null;}
	
	if($SetEnabled -eq $true){
		write-host "Set-OcotpusTentacleDisabled: MachineID($MachineID) Set to ENABLED";
		if($machine.IsDisabled -ne $False){$machine.IsDisabled=$False;}
		else{return $true}
	}else{
		write-host "Set-OcotpusTentacleDisabled: MachineID($MachineID) Set to DISABLED";
		if(($machine.IsDisabled -ne $null) -and ($machine.IsDisabled -ne $True)){$machine.IsDisabled=$True;}
		else{return $true;}
	}
	$bodyJson=($machine|ConvertTo-JSON);
	write-host "Set-OcotpusTentacleDisabled: Posting changes to the MachineID to Octopus...";
	try{
		Invoke-WebRequest -URI "$OctopusBaseURI/api/machines/$($machine.id)" -Header @{ "X-Octopus-ApiKey" = $apiKey } -Method PUT -Body $bodyJson;
	}catch{
		throw "Set-OcotpusTentacleDisabled: Failed to set the MachineID ($MachineID):: $_"
	}
}
#Get List of Machines
$EMachines=Get-OcotpusEnvironmentMachines -EnvironmentName "App1 Dev" -APIKey 'API-XXXXXXXXXXXXXXXXXXXXXXXXXXX' -OctopusBaseURI 'http://octopusserver'
#Disable the Machines
$EMachines|%{Set-OcotpusTentacleDisabled -MachineID ($_.id) -APIKey 'API-XXXXXXXXXXXXXXXXXXXXXXXXXXX' -OctopusBaseURI 'http://octopusserver'}
#Enable the Machines
$EMachines|%{Set-OcotpusTentacleDisabled -MachineID ($_.id) -APIKey 'API-XXXXXXXXXXXXXXXXXXXXXXXXXXX' -OctopusBaseURI 'http://octopusserver' -SetEnabled}

Damian_Brady · 20 August 2015 00:15

Hi Jon,

That’s awesome, many thanks for sharing that!

I’ve also put it in a Gist which references this thread.

Kind Regards,
Damo