Defining some deployment steps that run once pr. deployment, and some pr. tenant?

Legacy Uncategorised
1

Is it possible to configure a multi-tennanted deployment process, so that certain steps only run once during the process, while others run pr. tenant?

In our scneario, all “tenants” reside on the same app server and use the same IIS sites. Each tenant has an independant IIS binding, and a seperate database for data, but other than that, they share environments and sites.

We basically just need to:

  • Deploy our sites - once pr. deployment
  • Flip some IIS bindings and potentally run a migration script - once pr. tenant

Is this possible somehow?

Currently I’ve created a tag set with a “System” and “Tenant” tag, assigned all tenants to the “Tennant” tag, and created a single fake “System” tenant with the “System” tag. This allows me to achieve the behavior using tenant tag set filters - but this also means I have to remember to deploy to the fake “System” tenant everytime. Hoping for a less ugly solution.

Thanks in advance :slight_smile:

2

Hi Carsten,

Thanks for getting in touch! It looks like you’ve got a nice multi-tenant system running at high density, using the request to route to the right database. I’ve been in this exact situation before with several projects myself, some using the same SaaS model as yourself, others where we deployed separate instances.

By the way, Paul did a nice podcast on architecting multi-tenant applications on .NET Rocks recently: https://www.dotnetrocks.com/?show=1332_

The multi-tenant features were optimized towards the situation where you deploy an entirely separate stack (top to bottom) per customer/tenant, which is why the documentation/guides etc don’t spend too much time on other models yet.

Personally, I would model your situation as two projects:

  1. Application project - this project deploys the common software components for hosting your multi-tenant application. Its job is to apply upgraded DLLs and content etc, but doesn’t do any tenant-specific configuration. This could be modelled as an untenanted project (see this discussion for an existing problem and the workaround).
  2. Tenant configuration project - this project is responsible for configuring the IIS bindings, and other tenant-specific configuration values like database connection string etc. It doesn’t deploy any software components. This would be modelled as a multi-tenant only project. You would deploy this project for a tenant only when you are provisioning them for the first time, or changing their configuration.

This means you can upgrade the hosting application platform as one deployment, and then roll through all of your tenants in another series of deployments as configuration changes are necessary.

The remaining question is: when should the database upgrades be performed? I think this is where a potential problem lies.

When you upgrade the application, you may have made breaking changes to the database schema, and you need to upgrade all of your tenants immediately. I feel like this would be the case regardless of how you model in Octopus. All of your tenants are coupled to the same application hosting environment anyhow. Perhaps you could tie the database schema upgrades into the startup of your application - but then you have to deal with race conditions and the like.

Floating crazy ideas time! If there was a way to have an Octopus Step that executed a script in a “foreach Tenant in Tenants with this Tag” - would that have been more straightforward for you? One problem here would be how to handle failures in one tenant when you’ve already upgraded the application…

I’m really interested in your thoughts since this is the first time we’ve seen one of our customers using the multi-tenant features for this kind of multi-tenanted architecture.

Hope that helps!
Mike

3

Hi Michael

Thanks a lot for such a comprehensive answer!

You’re absolutely right - we’re deploying a SaaS product to a high-density environment. I’m trying to support a model where we have multiple “partners” of our system, each with their own database and endpoint. Each partner can be considered an individual customer, and should be able to run independent versions of the app from each other, but ultimately we still control all the partners / tenants.

I’ve thought about a dual-project solution, but was hoping that I could abstact thoose invidual steps away. I consider them implementation details of the deployment process, and ideally I don’t want any implicit protocols for whomever initates this process. If I cannot find a pretty single-project approach though, I’ll probably end up using two projects as you suggest because it seems prettier on the surface.

I not excactly sure that a “Foreach Tenant in Tennants with this Tag” would help me much in this particular situation - I guess this would still require me to have a single special tenant to represent the “Application” when deploying the fileset of the app, which is what I was hoping I could avoid altogether.

If I could define a deployment step that only runs once pr. environment, regardless of how many tenants were being deployed to, that would be very nifty. Actually most of the heavy lifting during deployment should only happen once pr. version pr. environment. The only thing required pr. tenant is flipping an IIS binding, and sometimes run some migration scripts if we upped the major version part - But that doesn’t solve the problem completely; If we’re upgrading a handful of tenants/partners from version 1.0 to 1.1, then the IIS sites for version 1.1 should only be deployed once. If we then later decide that the remaining tenants should now be updated to 1.1, that operation should not require deploying the site again, because the fileset required is already present on the server.

I’ve thought about that having some advanced conditionals on the deployment step could help - If I could tell the ‘Deploy Website’ step only to do so if a given site was missing (or potentially some other criteria), then I wouldn’t care if that step runs a 100 times, because it would be idempotent this way.

I hope I am making some sense? :slight_smile:

4

Hi Carsten,

Thanks for getting back to me! Sounds to me like you’ve got options to work with and see what suits best.

I agree with what you are saying, theoretically, you could use a suggestion like this reasonably popular UserVoice suggestion to test whether the application files need to be deployed or not: https://octopusdeploy.uservoice.com/forums/170787-general/suggestions/6594872-allow-the-run-condition-of-a-step-to-be-based-on-a

Practically though, it really depends on what you’re trying to avoid by only deploying the application files once.

When you update the bindings of the IIS Web Site, you will be causing a rotation of the Application Pool… We use a package cache and delta compression (in most cases) so you avoid streaming the package multiple times… I guess the main thing you’d save on is the actual unpacking and configuration of the site itself.

One way of achieving this kind of shape right now in Octopus would be something like Set-OctopusVariable('Octopus.Action.SkipRemainingConventions', 'True') as described here: http://docs.octopusdeploy.com/display/OD/Package+deployment+feature+ordering

You could add the detection to a PreDeploy.* script and then set this variable to skip the actual web site deployment.

Another idea worth considering, to avoid the first tenant upgrading all the others would be to build your website name based on the Release Version, let Octopus deploy the package as per normal, and then remove the Tenant binding from the old Web Site, and add it to the new Web Site.

A better way to do this might actually be to configure ARR or another reverse proxy in front of IIS. This way you could:

  1. Ensure the new version of the Web Application is deployed into IIS and bound to localhost and a TCP port
  2. Change the ARR rules to start redirecting traffic for that tenant to the new application version

Now as you start to deploy to new Tenants, that deployment could use what Nick suggested to skip the web site deployment, avoiding unnecessary rotation, and simply redirect traffic for the tenant to the new location.

Either way this is a fun problem to solve, and I hope you find a palatable solution for your scenario.

Keep me posted because I’ll add it to our list of patterns in our documentation.

Hope that helps!
Mike

5

Hi Mike

Thanks a lot for your help!

The need for us to avoid multiple file deploys is simply the time factor - Theoretically we might want to roll our a new relase one partner at a time, so I just want to avoid the uploading unpacking and app pool recycling each time. We have 50+ partners, so it would be very tedious waiting each time, even if we could cut out the upload part.

Just to reiterate on the deploy process I had in mind, take the following scenario: “Want to update app from 1.0 - > 1.1 for 10/50 tenants”:

  • Step 1: Deploy “App.1.1” site, if it does not exist (using no actual bindings) - Once pr. deploy
  • Step 2: Run code pr. tenant that notes the target version of each tenant - Once pr. tenant
  • Step 3: Move bindings from App.1.0 to App.1.1 in a single operation, based on what previous step noted - Once pr. deploy
  • Step 4: Determine if any remaning tenants are running on App.1.0, and delete it if not - Once pr deploy

This way, the fileset is only deployed once pr. version, and the app pool is only recycled once pr. deploy. Basically it is only step 2 that needs to be run pr. tenant, and that’s probably the most lightweight step and it basically only notes that “Tennant X want to go to version Y”

Regarding your suggestions, I think the Set-OctopusVariable apporach looks interesting, and could be of help - While perhaps not the most elegant way, I could work in my scenario, and any “uglyness” woudl be something I have to deal with, and not the deployer.

I’ll give it a look, and keep you posted on whatever ends up being our solution :slight_smile:

Thanks again!