Hi
We are running several tasks on different machines and we have noticed that same type tasks are queuing.
I opened a request with this same problem with Health Check (but the reason for the health check might be related with the start up of the machine) but these are tasks run on machines that are up and the tentacle responds ok to health check.
The biggest problem we have are ad hock tasks that keep piling and only executes one at a time.
Check attached image please, on the bottom you have Execute remote PowerShell wich is a Ad hock task
Hi Carlos,
Thanks for getting in touch.
I have read your thread regarding health checks and think I understand the issue you are experiencing. You are queuing health checks and ad hock scripts to your machines. You may queue many of these but Octopus will only execute them one at a time. That becomes a problem because the queue gets longer and longer waiting for the first task to finish. If a health check goes wrong or takes a very long time it is holding up the rest of your machines because they must wait for that health check to finish before their own is finally executed.
As you have discovered, Octopus will only run one task of each type that is created via the API. We could be smarter about running tasks in parallel and allow running a task per Tentacle. That would take some time for use to implement after it made it to the top of our backlog (which is quite full!) so you would need to wait for a solution. Maybe we can figure out a good way to achieve the result you are seeking using the features Octopus currently has available.
A few ideas come to mind, please let me know if these would suit your situation. We are putting the finishing touches on some features that would allow you to automatically deploy to your machines. You may be able to:
- Tell your projects to automatically deploy when a machine is enabled
- Before you restore a machine to the previous state, make an API call to Octopus to set the machine to disabled
- After the machine reboots, make an API call to Octopus to set the machine to enabled
Now whenever your machines are set to enabled Octopus will automatically deploy whatever you require on to them. You could turn your “Remote powershell” script into a project and it would be deployed to your machines. Rather than queuing a bunch of ad hock tasks it would deploy to every waiting machine at once.
You can also choose to do a health check at the start of the deployment so only your healthy machines are deployed to and you wouldn’t need to create health checks via the API.
Would you be interested in trying this method? I think it would make your life easier and you could implement it now instead of waiting for us to potentially change the way tasks are executed. I’m here to help you get set up if you need assistance.
Cheers,
Shane
This is a big problem for us… We are deploying each hour around almost 80 deployments and a lot of tasks for several servers (automated tests and live deployments). With this many deployments fully automated by PowerShell/Bash we cannot use interface, and only the API.
I don’t know why there are two ways of work, one queue for requests via interface and API but it should work the other way, there will be more requests on API than interface and it should deal better with queuing…
This is a serious problem for us and i would like to request, if possible, to escalate this problem to fix.
Thank you for your answers and have a nice day
Carlos
Hi Carlos,
Running AdHoc scripts was designed as a troubleshooting mechanism and intended to be an exception, not the rule. If we start asking what kind of parallelism you would like applied to AdHoc scripts it would get complicated, making it difficult for you to debug.
If you moved your AdHoc scripts to a project and deployed the project for each machine you would get parallelism out of the box. Alternatively you could investigate any of the suggestions we have provided and it would smooth your process to the point where you would not need to maintain your own custom API scripts or worry about parallelism - Octopus would manage it for you.
Hope this helps.
Cheers,
Shane
