Our projects typically follow a DEV->QA->UAT->PROD Blue -> PROD Green lifecycle. We periodically run into situations where teams will deploy a project’s release through to UAT, but then get side tracked and context switch to work on another project. This sometimes results in teams forgetting about that release and it just sits there in UAT as inventory. By the time they remember (2 weeks later) they’ve all but forgotten what that release was about!
Although this situation is rare, it does happen often enough that we need some safeguard in place. We would like to build some sort of query to find all projects where the latest release in their respective default channel has sat in a pre-production deployment stage for some period of time (say > 48hrs). Is this something that we can build via the API? Would SQL be better suited for this? Is there already such a tool that does this sort of detection already?
While we don’t have a product or script “out of the box” that will accomplish this task I do believe that what you are requesting can be done via the Octopus API. I would definately recommend the API over attempting to access the database directly as the API is tested against every release, however the database we assume is not end user accessable and so may change at any point in any release.
The other advantage of the API is it has multiple options available, namely using the REST API directly (for example we have pre-written powershell examples), leveraging Octopus.Client or using a tool like OctoPosh.
In (very) rough terms you will need to accomplish the following via the API:
Determine the latest release version for a Project
Grab all deployments for this release
Check if environment is production
If not production confirm time of deployment
Alert if time of deployment is > 48 hours
As I mentioned, that is a very rough outline of what will need to occur.
As a side project I am working on coming up with a solution for this issue via the API, however it will take time for this to be developed.
I’d be very interested to know if you come up with your own solution!
I was able to come up with a good enough solution for our needs using the Octopus API. Based on our specific deployment model, I first query our load balancer to see the current state of our VIP for a project (we use a variant of blue/green deployments). From there we traverse the release progressions to find the last prod deployment to see if the deployment state matches what blue/green pool the load balancer is pointing to. If not, then we inform the team via Slack.
In the case that an app is not in the load balancer, we just short cut and check the most recent release for the project. If it’s in a non terminating state (that is, if it’s in QA or UAT), we send a friendly reminder that the team’s Slack channel that the app needs to be promoted.