Eventual Consistency Bug? - removing a step/package from a Project

Hi,

Version: Octopus Deploy 2.6.3.886

Given an Octopus Project
And I have added a new Deployment step to the project
And the Nuget package for that step is not available
And I realised I screwed something up so I removed the deployment step from the project

When I went to deploy the project again
Then it failed the deployment because it was still looking for the missing nuget package for the step that I had removed

Deleting and recreating the indexes in RavenDb fixed this.

Cheers,

Dan

Hi Dan,

Thanks for reaching out. Each time you create a release, Octopus creates a snapshot of the variables & steps at that time and then starts a deployment based on them. If you make a change such as deleting a step from your deployment process, but you re-deploy the same release you created before (where the step still existed), Octopus will use the same snapshot of that deployment and therefore execute the now deleted step.

Everytime you make a change on your Variables or Deployment Process, you need to create a new release for the changes to take effect on the next deployment. By touching the RavenDB you sort of forced the update in a way we wouldn’t recommend you to do again.

Regards,

Dalmiro.

Yep, we had the issue on new releases created subsequently, hence having to brute force the update – the UI was showing that the release only had two steps but it was still trying to fetch the package that was removed.

Cheers,

Dan

Hi Daniel,

Could you please send us a deployment log and a screenshot of how your steps look like?

Also, what is the name of the package that Octopus is still trying to fetch?

Thanks

Dalmiro

Hi Dalmiro,

Our Project currently consists of two steps, deploying the packages WordWatch.Server and WordWatch.Monitor. The current deployment process is described in the attached deploymentProcess.json file.

[cid:image003.jpg@01D08719.2DB36C90]

I’ve attached deployment logs and also the json documents from the API describing 5 releases, let me walk you through the scenario to explain:

  •      4.0.11-build01494
    

o This is the previous good deploy, deploying WordWatch.Server and WordWatch.Monitor

o The ProjectDeploymentProcessSnapshot is deploymentprocess-projects-65-snapshot-7

  •      4.0.11-build01495
    

o We added a new deployment step, “Deploy Ingester” with a nuget package id of “WordWatch.Ingester”.

o This nuget package does not exist with the given version number and so the deployment will fail because it cannot find the given Nuget package.

o The ProjectDeploymentProcessSnapshot is deploymentprocess-projects-65-snapshot-8

  •      4.0.11-build01496
    

o We removed the “Deploy Ingester” step from the project and attempted to deploy it

o Octopus still attempted to perform the “Deploy Ingester” step

o The deployment failed because it could not find the “WordWatch.Ingester” nuget package

o The ProjectDeploymentProcessSnapshot is deploymentprocess-projects-65-snapshot-8

  •      4.0.11-build01497
    

o We did not make any changes to the project in Octopus

o We retried the build and it failed because it was trying to run the “Deploy Ingester” step

o It seems that Octopus is intent on running the “Deploy Ingester” step even though we have removed it and the Process UI seems to show that it has been removed.

o The ProjectDeploymentProcessSnapshot is deploymentprocess-projects-65-snapshot-8

  •      4.0.11-build01499
    

o We did not make any changes to the project in Octopus

o I shut down Octopus

o I deleted and recreated the RavenDb indexes as described here: https://octopusdeploy.com/blog/resetting-raven-indexes (out of date but still relevant – replace port 10930 with 10931 for Octopus v2)

o I restarted Octopus

o The ProjectDeploymentProcessSnapshot is deploymentprocess-projects-65-snapshot-9

o The deploy worked, it did not try to run the “Deploy Ingester” step

o I’m not sure if recreating the indexes or restarting Octopus fixed the issue, in hindsight I should have tried restarting Octopus first

I hope that is enough information to go on, I haven’t tried to recreate the issue in isolation yet so it could be one of those weird heisenbugs.

Regards,

Dan

Octopus_Deploy_Logs.zip (18 KB)

Hi Dan,

Thanks for the information. I cant see anything that would have caused the mentioned behavior. If it happens again, please send the screenshot of the deployment process at that moment (like you did on image003.jpg) and then send the deployment log of just that deployment please. And even though it seems like restarting the server fixed the issue, its definitely not a recommended workaround. Try sending us the info if it happens again please.

Thanks

Dalmiro