Controlling load balancers for deployments

Advice
1

Hello,

We are still setting up our Octopus deployments and the product does meet most of our deployment needs.

One area we are struggling with is dealing with the load balancer (lb).

As a test, we have written PowerShell scripts that can enable/disable a server from the lb pool using Octopus.

The lb control is a project in itself which can deal with taking the servers in and out for each environment, however the issue is that we have to create a release time we want to update the LB pool. For example:

  1. Dev has 2 IIS servers
  2. both servers sit behind lb
  3. run project “update lb pool” to remove IIS server 1 from lb pool
  4. we release to IIS server - as part of the deployment, we have a trigger step in here for teamcity to carry out acceptance tests on the required server
  5. we run project “update lb pool” to add IIS server 1 to lb pool.

Now for server 2, we go through the same process - we can either “create release” or use the “deploy again” option, in this scenario, would our release notes get lost?

We are also unable to see or present anywhere to say which server has been taken out, and each time a server is updated in the lb we are unable to see the previous machine which was worked on, as in taken out and put back in

as we go further into the higher environments, we have more machines, and so I can imagine the deployments becoming very “messy” or not very transparent.

Do you guys have any ideas on how we can make this deployment more simple, or straightforward

Thanking in advance

4

Anything?..Anyone?

:sos:

5

Hey @supafly270,

I’m just taking a look at this and I’ll come back to you very shortly with an update.

Thanks

6

Hey @supafly270,

Do you need to be able to control which server is taken out of the load balancer first or can it be either server if you have the ability to pass that information over to team city?

Thanks

7

Hi Adam,

It can be any server. The operator doing the deployment will choose the server

Thank you

8

I’ve got an idea of how we can make this deployment straightforward using rolling deployments.

The last question I probably have is what happens after the machine has been through acceptance testing? Is there a process where if it fails you take action and roll back the update then add it back to the loadbalancer?

Thanks

9

The deployment is a rolling update, currently the server lb operation is in a separate project and the update to IIS is a separate project

if the acceptance testing fails or there are issues then we roll back and add the machine back in

10

Hey @supafly270,

I’m just writing up and testing out a recommendation and I’ll come back to you very soon.

Thanks

11

Morning @adam.close

Hope you had a good weekend

Thank you - looking forward to it

1 Like
12

Hey @supafly270,

So I think the easiest way to set up your deployment process is using a mixture of rolling and canary deployments using an additional unique canary environment.

I’m inspired by a KB article here that I think suits your deployment right now. I can mark out all the steps you need to add to the deployment process in a further reply if you think this solution would work.

Create a new environment and add one machine from production into the canary environment (so the target exists in both canary and production). You deploy a new release to the canary environment which removes one machine from the load balancer, deploys the package, and runs the tests. If the tests fail it fails the deployment process and you re-deploy the last known good release to the canary environment. If the tests pass you can automatically kick off the deployment to go the production environment and deploy to the rest of the machines.

There’s some script logic that needs working out for the LB but does that sound like something that could work for you?

Thanks