Runbook with Load Balancer Config

I’ve been trying to think of a way to reboot a set of servers. The issue is that each server may be in more than one load balancer and more than one backend (haproxy term). I have rolling deployments working great already with the load balancer, the issue is when I am affecting the whole server. I feel that I might not be doing something the octopus deploy way and thought to reach out for some advice before going too far.

Scenario
2 Servers

  • Server 1
  • Server 2
    3 Web apps
  • app.example[.]com
  • api.example[.]com
  • help.example[.]com

I would like to have a runbook that goes like “Reboot Server(s)”

Step 1
Remove Server 1 from the load balancer
This would drop the server from app.example[.]com, api.example[.]com, help.example[.]com
Reboot
Add Server 1 to app.example[.]com, api.example[.]com, help.example[.]com

The issue is I don’t know how to map in variables, or in some place in Octopus Deploy that Server 1 is mapped to 3 websites. I see that I can add multi-line variables, should I just use multiple lines and some | to say “server1|app.example[.]com\nserver1|api.example[.]com”? Is that the best approach or is there another?

Hi @joshua.bartley,

Welcome to the Octopus community and thank you for your question.

Apologies for the delay, this is due to public holidays here in the UK.

I am glad to hear you are looking to use Runbooks to perform operational tasks such as server reboots.

Just to be upfront, I have not used HAProxy before, but I assume you are leveraging (or planning to use) their available REST APIs to take nodes (servers) out of service while you reboot them?

From a Runbook perspective, your logic makes sense, personally I tend to follow this logic (where possible):

  1. Drain node(s) active sessions.
  2. Wait till sessions reach 0.
  3. Disable node(s).
  4. Perform desired task(s) to node(s) that are out of service.
  5. Health checks/smoke tests.
  6. Put nodes back in service.

I am not 100% sure what you are trying to achieve with your variable mappings, if you are comfortable sharing, I would like to understand this further. Are you able to tell me what you need to achieve with the variables, so I have a clearer idea and maybe I can advise further.

I am wondering at this point, if you haven’t seen this feature within Octopus then maybe Scoping Variables is what you may be after?

Doing this will allow you to be quite specific on how and where a variable can be used as part of your process.

I look forward to hearing from you.

All the best
Doug

Sure. I am using the HaProxy dataplane API. Everything is grouped into HaProxy backends, like a virtual server in apache terms, and those backends have a list of servers. To remove a server from a load balancer, I have to send a request using the backend name, and the server name. There is not a single api call for “Remove server SERVERXYZ” as I have to issue something like

“Remove server SERVERXYZ from backend app.example[.].com”
“Remove server SERVERXYZ from backend api.example[.].com”

I don’t believe scopes will work as a production server may host multiple sites. I need a single variable to list all backends a server may be a part of. Using the example above.

Name: SERVERXYS
Value: app.example[.]com
Scope: Production

I also need another value or I won’t properly drain the active sessions.

Name: SERVERXYZ
Value: api.example[.]com
Scope: Production

The variables during execution feel one to one, and I need something that is one to many in this case.

Hi @joshua.bartley,

Thank you for getting back to me with this.

I have a couple of thoughts here:

My first thought is mapping the variables 1:1 as you have mentioned and using PowerShell (or your preferred scripting language) to loop through an array.

For example, storing your variables within your Octopus project:

Using the “Run A Script” template to run a PowerShell script to loop through an array:

$server = $OctopusParameters["Project.ServerName.Web01"]

$appList = $OctopusParameters["Project.AppName.API"],
		   $OctopusParameters["Project.AppName.APP"],
           $OctopusParameters["Project.AppName.Help"]

foreach ($app in $appList){

		# Add in your call to the HAProxy dataplane API
        Write-Host "Removing $server from backend $app"
    }

Resulting with this:

You may even wish to use a Prompted Variable to pass in your server names (this depends on how your servers have been setup, so I am just making assumptions here).

Another idea may be to look at Variable Substitution - Iterating over sets of values to see if you could achieve what you are after.

For example you could look at using:

Endpoint[SERVERXYS].Address = app.example.com
Endpoint[SERVERXYZ].Address = app.example.com

I hope this helps

Thanks,
Doug

Thanks! I think this got me on the right path. Variable Substitution would be exactly what I needed except that I don’t see a way to do that with an inline powershell script. I could be wrong about that.

I think I’m going this approach

$variables = $OctopusParameters.getenumerator() | where-object {$_.key -like ($OctopusParameters["Octopus.Machine.Name"]+ "*")}

foreach($variable in $variables)
{
    # Perform the desired action here e.g.
    Write-Host $variable
}

Not 100% on the syntax of the variables, but this gets me closer to the idea where I can update these variables and have a form of a load balancer to backend to server catalog so a reboot doesn’t have to wait for a health check to drop a server out.

Thanks!