All our tentacles are polling tentacles using web sockets with a default timeout of two minutes.
I have a project with 11 packages between 50 and 400MB. During the acquire packages step in the deployment I keep running into timeout exceptions:
Server exception: System.TimeoutException: A request was sent to a polling endpoint, but the polling endpoint did not collect the request within the allowed time (00:02:00), so the request timed out.'
Raising the “Polling Request Queue Timeout” to 6 min works, however I’m guessing I’ll run into the same problem when deploying the project to targets with slow internet connections. The second downside is that increasing the timeout has negative effects on the heath checks, as the duration increases if targets are offline (an there are always a couple).
How can I get around the issue? Could I raise the timeout just for the acquiring packages step? I would have guessed the upload to the targets would be async and wouldn’t timeout as long as the upload is running.
Unfortunately, this is a limitation of the polling mechanism. As the server doesn’t have any direct connection to the target it needs to set a limit on the time a request will remain alive before it is cancelled.
The only option is to raise the global timeout as you have already done.
this puts me into an awkward position with 700 polling tentacles with some poor internet connections. Wouldn’t an async transfer be an option?
would it work to change the timeouts during execution?
step 1: change timeout of machine policy
step 2 acquire package
step 3 change timeout back
step 4… deploy
So, the timeout wouldn’t normally trigger just because the package takes a while to transfer.
I think the issue here may be that the package transfer is using all of the available bandwidth preventing the tentacle machine from being able to complete the poll request.
If running with the timeout in the machine policy set to higher isn’t feasible, then it would definitely be worth trying a script to amend the policy during the deployment.
Another option, though this will depend on the location of the tentacles, may be to deploy the package to a shared local location and script the the tentacles to transfer from there using the local network.
I’ll try an play with the bandwith and see what happen.
I seem to have multiple problems with polling tentacles.
I was quickly trying to evaluate something in our custormer servers (ca. 450) and decided this would be the perfect use case for adhoc task as it was going to be an one off job.
It started off nicely but soon hit the first offline target and it timed out after 3min. no worries as it was happily carrying on until it hit another 2 offline targets. After the third target timed out the whole task ended with an error.
Success = 68
Failed = 3
Pending / Not executed = 375