It seems that the rolling deployment window size is overridden by the value in Octopus.Acquire.MaxParallelism. My expectation was that it would be the other way round and that Octopus.Acquire.MaxParallelism would only be enabled for tasks that was setup to start in parallel with the previous task. We are currently running OD 3.4.1.
I experienced it in my production environment yesterday, but have constructed a simplified project to reproduce it. The attached log files are from the example project.
Here I have two steps: One is a regular script running with a deployment window of one. The step will echo, sleep for 5 seconds and echo again.
In the second process the behaviour is identical, but this time the action is a nested with the parant defining the deployment window.
In both cases the steps are run serially when Octopus.Acquire.MaxParallelism is unset. Setting it to 10 causes the steps to run in parallel disregarding the deployment window.
In my production environment the steps involved starts by closing traffic to a host. Deploying to it, health check it and open for traffic on success before proceding to the next step. When tasks are run in parallel it causes all traffic to my website to be closed which was not the intention.
I don’t think this is the behaviour is what I read in the documentation long ago. My expectation would be for the window size to take precedence and Octopus.Acquire.MaxParallelism to be involved when different steps are setup to start simultanously.
deploy-with-no-parallelism.txt (14 KB)
deploy-with-parallelism-10.txt (17 KB)
Hi Morten,
Thanks for getting in touch! I had a look at your logs, and based on the steps you are running this should not occur. Acquire should only be used on package acquisition and not effect the steps.
I will have it investigated and let you know the outcome.
Thanks for reporting this!
Vanessa
Hi Vanessa
Thank you! – Both “action” and “acquire” was set. I have not checked which of the values were causing the problem.
Kind regards
Morten
"Id": "7a8de22b-be93-1478-78d8-40a791803581",
"Name": "Octopus.Acquire.MaxParallelism",
"Value": "10"
},
{
"Id": "ca98faa6-1427-2a9a-b024-0a854eed02e6",
"Name": "Octopus.Action.MaxParallelism",
"Value": "10"
},
Fra: Vanessa Love tender2+dee931c11d@tenderapp.com
Sendt: 12. september 2016 14:39:28
Til: Morten Fæster
Emne: Re: parallelism and Octopus.Acquire.MaxParallelism [Problems #47438]
Hi Morten,
Thanks for getting in touch.
We were not able to reproduce this issue using the Octopus.Acquire.MaxParallelism variable. This variable controls the number of package acquisitions that will be allowed to run concurrently, so it shouldn’t have any effect on steps/actions concurrency.
However we were able to see the behaviour you described when we added the Octopus.Action.MaxParallelism variable that you mentioned in your reply. If you are also setting this variable, we would expect it to override the window size, as this variable controls the maximum number of machines on which the action will concurrently execute.
Can you please confirm if this is the case? If you remove the action variable, does the deployment process then behave as you would expect?
Cheers
Mark
Hi Mark
Removing the variable makes the deployment behave as defined in window size.
But I would still claim that the behavior is a bit unfortunate:
The window size should define the maximum number of tentacles running a task in parallel whereas - to my understanding - the Action.MaxParallelism defines the number of yaks running in parallel on a specific tentacle. What I saw was that the variable was overriding the the deployment window whereas the task on the individual tentacle continued to run serially.
As it is a long time since I reported this issue it is possible that you have fixed it - possibly unknowingly (?) - already. I did however receive a notification from someone named Vanessa shortly after reporting the error who recognizing there was a problem and promising that you would look into it.
I now understand that you describe the behavior as “by design”. But how would you then achieve the situation where tasks can run in parallel on a tentacle but the same task cannot be running concurrently on more than deployWindow hosts?
The ability to run in parallel on the same host may speed up installation time considerably, so the feature has justification.
Regards
Morten
Get Outlook for iOShttps://aka.ms/o0ukef
Hi Morten,
If we’re understanding the scenario correctly, would Running multiple processes on a Tentacle Simultaneously be what you’re looking for?
OctopusBypassDeploymentMutex would allow the Tentacle to run tasks in parallel, and then the action window could restrict the same task running concurrently.
You would also need to make sure you have your steps setup to run in parallel, as described in this documentation (assuming your steps aren’t dependent on each other).
There shuldn’t have been any changes in Octopus recently surrounding this behaviour. We originally thought this may have been a bug with the Octopus.Acquire.MaxParallelism variable until we saw your reply about the Octopus.Action.MaxParallelism variable, then the behaviour made sense. We hope the above documentation on running multiple processes on a Tentacle simultaneously (plus running steps in parallel) will help.
Cheers
Mark
This is fine.
Just to reiterate: I had the settings as mentioned for quite a while after reading i link from the log file.
After an upgrade the behaviour of the deploy failed causing downtine on my production in the order of minutes. The change was undocumented.
I am glad to read the link you provide which seems to elaborate of a behaviour introduced around three months ago.
But the RTFM-attitude annoys me. This was a real problem as well as a behaviour change in OD.
Thank you for your help.
Kind regards
Morten
Fra: Mark Siedle tender2+dee931c11d@tenderapp.com
Sendt: 15. november 2016 06:28:43
Til: Morten Fæster
Emne: Re: parallelism and Octopus.Acquire.MaxParallelism [Problems #47438]