We upgraded our Octopus deploy server to 2019.3.1 LTS version 3 days back. While doing so, we upgraded our tentacles as well from v3.12.7 to v4.0.1. As soon as the tentacles were upgraded, 350+ deployments were triggered by System to various environments for different projects . we drained the Octopus deploy server and canceled the deployments using a custom script.
On further analysis, we found that deployments were triggered for release versions where project lifecycle have at least 1 environment currently set for auto-deployment and the release version didn’t went to the environment. Reason for no deployment could be either:
Lifecycle was different at the time, and environment was not set for auto-deploy
Lifecycle didn’t had the environment present at the time.
For some deployments, release’s last deployment date was months old. There were multiple release versions (some belonging to same project and some to different) queued up for an environment .
Also, if there are multiple phases in the lifecycle for the release where environment are set to auto-deploy and release didn’t went to those auto-deploy environment, deployment triggered only for lowermost phase out of those phases and in that phase deployed to all auto-deploy environments where releases didn’t happened in the past.
Can someone please check into this, so that we don’t run into such situation again.
Details of Octopus Server:
Windows 2012, AWS hosted
Octopus deploy v2019.3.1 LTS (recently upgraded from 3.17.14)
Details of Tentacles:
v4.0.1 (recently upgraded from 3.12.7)
I’m really sorry to hear that auto-deploy kicked off after you performed a Tentacle upgrade.
Auto deploy can be configured by project trigger or auto-promotion through a lifecycle. Do you know if you are using project triggers, lifecycle auto-promote, or both?
There is a section in the Octopus portal Configuration > Diagnostics > Auto deploy logs. Could you download and send those logs to support@octopus.com so we can have a look?
I look forward to getting to the bottom of this issue and ensuring it doesn’t happen again.
Thanks Shane for reaching out. Deployments happened in some projects which didn’t have any trigger and in some they have. The only thing common was, auto-deployment set in lifecycle.
Right now auto-deploy logs are present in the console for today only. Is there any way, I can get the logs for past days?
Older auto-deploy logs should be on the Octopus server machine in the TaskLogs folder. By default the folder is C:\Octopus\TaskLogs. They start with scheduledtasks_processautodeployments.
It’s going to be very difficult diagnosing what happened without the logs.
I have two theories, depending on which type of auto-deploy happened: Tentacle health changed or the auto-deploy counter was reset.
You mentioned the auto-deploys were initiated after a Tentacle upgrade. My immediate thought was that the Tentacle health changed during the Tentacle upgrade which would initiate auto-deploys for any projects that had triggers set to fire for Tentacle health changes. Tentacle upgrade blocks health checks from running for this reason, but perhaps somehow a health check and Tentacle upgrade started at the same time. You could discount this theory by checking Configuration > Audit selecting “Show Advanced Filters” and filtering by the “Machine” document type. There should be a lot of entries for machines becoming healthy right after you upgraded the Tentacles.
If the auto-deploy counter was reset then auto-deploy would kick off for every lifecycle and project that you have configured for auto-deploy. From your description it sounds like only lifecycle-related auto-deploys happened. This one I think is impossible to figure out without logs.
I’m not sure how to proceed without those logs. Maybe the task logs from one of the auto-deploys that you cancelled has some information about which machines were involved in the deployment and why it was triggered? If you have those handy could you send one to support@octopus.com?