Attempting to deploy to 100’s or 1000’s of remote nodes at a time and seeing the errors below from Octopus Deploy. This issue is now a show stopper for our company’s use of OD.
Scenario is during a Deploy a Package step doing the Acquire operation one (or more) nodes connection is lost OR even when one or more nodes are unavailable in an environment, which is a possible normal occurrence when deploying to 100’s of remote locations across North America and around the world. So see the typical message in the log:
“after the request began: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.”
Then during the actual install attempting to run Pre-Deploy and Deploy scripts a node that failed connectivity during the Acquire package operation reports the following exception in the logs:
No package for the action ‘51ec4296-5e21-41fd-b70c-19c395fbfded’ and machine ‘Machines-2708’ was acquired.
System.Collections.Generic.KeyNotFoundException
at Octopus.Core.Model.Projects.AcquiredPackageMap.GetPackageForMachine(String actionId, String machineId) …
After that every node just fails without any error and just see this in the logs:
Failed:
This error make Octopus Deploy unusable for our needs to deploy to 100’s to 1000’s of remote nodes.
Hi David,
Thanks for reaching out. One potential solution you could try is to run a health check against your deployment targets to exclude any machine that may be offline. There is a “Health Check” deployment step can be configured to simply check the connectivity of the targets for a deployment and exclude any that may be unavailable at the time of the health check. There is some documentation around handling transient targets here.
I hope that helps, please let me know if you have any further questions.
Kinds Regards,
Tom W
Unfortunate that does not help as I had a situation where connection was lost during the acquire package and the same behavior was observed and thus the run was a complete failure. There is something that needs to be fixed in Octopus Deploy or some setting. The expected behavior is to just fail the one node and the rest of the run complete.
