We have a large number of users trying to deploy releases to internal machines. It would be useful to have a max number of concurrent deployments variables to limit the server from getting overwhelmed. We are easily overwhelming our test deployment server which is 8 cores and 8G Ram 3-4 times a day. When we do this you see that the tentacles seem to hang and when you try and shutdown the service it will hang, forcing you to terminate the process and clean the DB
Hi Jeff,
Thanks for getting in touch! Just a few questions so I can understand this scenario better so we can come up with the best possible solution.
Are these deployments that are running at the same time running on the same or different tentacles?
Could you explain the workflow/timing and to which servers this process is running on?
Vanessa
Vanessa,
We have one project that runs and deploys to n environments all individual tentacles. We allow pushing the product out to multiple machines for testing purposes. This occurs all on the same sever that has a single Octopus instance on it. The server is 8 vCPU, 8G Ram and during the time of the deployments will usually be running 100% CPU and Octopus server is about 2 G RAM.
The workflow is as follows:
Developer checks in code,
Build happens
New release is created
Deployment is started automatically to 2 environments
n Developers start pushes to individual environments shortly after
each deployment it to separate environments and tentacles usually 1 machine to en environment. Average deployment ~15 mins
deployment queue shows 6-10 deployments started/queued
Checking newest request will show no log actions even though showing started
Of Note:
Originally this server had 2 vCPU and logging would lag by 6 -20 minutes on tasks. We could see actions and deployments complete and the logs would not be updated until significantly later. When I boosted the machine to 8 vCPU the logging only lags by ~2 minutes. We log a log of data to the console when we are doing our database actions. Could the overall logging be slowing the system down as it tries to aggregate the log files and put them back into sequence?
Jeffrey Feinberg
Vanessa,
Just as a follow on here is a screen shot of what is occurring. Once we get into a state like this we end up having to take the server down forcibly, the service will not stop on its own you need to kill it. Then we run a DB cleanup and a reset-activity and clean up the tentacles as well.
-Jeff
Jeffrey Feinberg
Director | Release Engineering
Eze Software Group / Eze OMS
From: Feinberg, Jeffrey
Sent: Monday, November 03, 2014 7:05 AM
To: ‘Vanessa Love’
Subject: RE: Max concurrent deployment variable [Problems #26354]
Vanessa,
We have one project that runs and deploys to n environments all individual tentacles. We allow pushing the product out to multiple machines for testing purposes. This occurs all on the same sever that has a single Octopus instance on it. The server is 8 vCPU, 8G Ram and during the time of the deployments will usually be running 100% CPU and Octopus server is about 2 G RAM.
The workflow is as follows:
Developer checks in code,
Build happens
New release is created
Deployment is started automatically to 2 environments
n Developers start pushes to individual environments shortly after
each deployment it to separate environments and tentacles usually 1 machine to en environment. Average deployment ~15 mins
deployment queue shows 6-10 deployments started/queued
Checking newest request will show no log actions even though showing started
Of Note:
Originally this server had 2 vCPU and logging would lag by 6 -20 minutes on tasks. We could see actions and deployments complete and the logs would not be updated until significantly later. When I boosted the machine to 8 vCPU the logging only lags by ~2 minutes. We log a log of data to the console when we are doing our database actions. Could the overall logging be slowing the system down as it tries to aggregate the log files and put them back into sequence?
Jeffrey Feinberg
Hi Jeff,
I’ve scheduled a chat about this specific issue. I am going to see if I can push this one and see what can be done, and when.
So to be 100% clear a solution would be a singular configuration number, that only allows the defined number of deployments on the Octopus server (for all environments) to be completed at one time?
Vanessa
Correct it would represent the maximum number of deployments at a single time
Do you have a guide to trouble shoot problem with the server? I am having to restart the service 2 to 3 times a day to clear up hung tasks or tentacles that time out in the middle of the job
Typos courtesy of my mobile device
Hi Jeff,
There is nothing I can think of to ease the pain - so I will ask about a workaround - see if as a group we can think of something.
Vanessa
Can we schedule. A webex for tomorrow your time to review logs etc. to see if there’s is an underlying issue? I have been reviewing decrypted messages and actors to see what is occurring. I can send them to you if you like ahead of time to see anything and we can focus on the findings
Jeff
Typos courtesy of my mobile device
Hi Jeff,
We do not have anything available on Wednesdays it’s a group meetup day. That is the scheduled meeting I was referring to, I am going to corner the team and talk about this issue.
Do send through the logs again though that might help us come up with some kind of workaround.
Vanessa
Vanessa,
Here are the logs. If we can schedule a call for Thursday it would be helpful. I have a lot riding on this tool as it is a central part of my release pipeline as we approach continuous delivery for our cloud platform.
-Jeff
Jeffrey Feinberg
Logs.zip (197 KB)
Hi Jeff,
Pick a time, I will have a call with you, talk about what we discussed today and get some more details from you.
https://octopusdeploy.acuityscheduling.com/schedule.php
Vanessa
Vanessa,
Thank you, I have scheduled a call for this evening my time/your tomorrow. Were you able to see anything in the logs that attributes to the locking in the Database?
-Jeff
Jeffrey Feinberg
Director | Release Engineering