We are facing an intermittent issue while package acquire from external feed (Proget) to windows tentacle.
Package acquire fails for some packages(different package each time) on couple of tentacles (different tentacles each time), though the same package is successfully downloaded at other tentacles in the deployment. Packages are being downloaded directly on the tentacles. We are downloading around 27 packages during the deployment on each tentacle. Below are few of our observations:
- Facing this issue in deployments having 8 or more deployment targets.
- Most of the times restarting tentacle service on the target resolves the issue.
- The affected tentacle doesn’t fail for all package, usually 1 or 2 packages. If multiple tentacles are affected in same deployment, they are usually failing at different packages.
- In couple of failed deployment cases, when we faced timeout issues on our forwarder feed endpoint, we found package downloaded with smaller size than the actual package with proper naming (not with downloading appended), and the subsequent retries keep on failing till we remove the package from tentacle. We have tried with option “Re-download packages from feed”, but even that doesn’t resolve the issue. Only way is to remove the package from tentacle. Though the same package downloaded on other tentacle fine.
- Frequency for these issues have increased since our last Octo server upgrade to 2019.6.7 LTS in Oct 2019.
Octo Server Details:
Version: 2019.6.7 LTS
AWS Instance type: c5.2xlarge
OS: windows 2012
Tentacle details:
Version: 3.12.8
CPU: 6 cores
Memory: 12 GB
OS: windows 2012
Octo DB server details:
AWS Instance type: c5.2xlarge
SQL server: 2016
OS: Windows 2012
I have also sent an email (16th Jan) at support@octopus.com with attached raw task logs for both cases, where package was of smaller size and where we have to re-start the tentacle. Please let us know if any further information is required.
Please help us in identifying the root cause for it as it is impacting the production release reliability and creating drag in the process.
Thanks and Regards,
Devan