Is this typical for it to take 16 minutes to check for previous packages on a tentacle? Obviously the environment could be to blame, but is there anything I could troubleshoot to determine the root cause of this slowness? Previous deployments perform this step in 3 mins or less.
05:38:53 Verbose | Octopus Deploy: Calamari version 3.3.29+Branch.master.Sha.2b…
05:38:53 Verbose | Package OnlineServer.2016 version 2017.3.7.1 hash 3be… has not been uploaded.
05:38:53 Verbose | Finding earlier packages that have been uploaded to this Tentacle.
05:54:50 Verbose | Found 5 earlier versions of OnlineServer.2016 on this Tentacle
Thanks for getting in touch! It is not typical. Between those two messages it looks for packages starting with the package id in <octopushome>\Files (on the tentacle machine). It then extracts the metadata from them and calculates a hash. This should not take that long considering there are only 5 files found.
I’m wondering whether the machine is hitting some sort of disk read limit (eg IOPS in Azure). Is the machine hosted in AWS/Azure? Try checking the machine diagnostics for high disk/CPU activity.
We have run into this before, but have not been able to get to the bottom of what exactly is running slow. I have built a version of Calamari 3.3.29 that has extra logging around this area, are you able to try that version out and send me the results? You can download the two packages from here and follow the instructions here to configure the server to use that version. Once you have the results, the same link has instructions on how to revert to the built in version.
Could you check that Files directory to see how many files there are, and how large the “OnlineServer.2016” packages are? You can clear out the packages in that directory, which should resolve the problem, but it may return in the future.
Thanks for responding Robert. Because we are on octopus version 3.3.24, I think we are going to upgrade the server version and tentacles to the current version before running this version of calamari.
The version in the link is based on the version of Calamari you are currently running (3.3.29, based on the log snippet you gave me). If you upgrade, I will need to supply you with another build of Calamari. Note that the Calamari and Octopus versions don’t normally match.
We were having the same problem, our deployments were taking 20 minutes. I was able to resolve this by deleting very old nuget packges in <%octopushome%>\Files
I deleted 2 years worth of files and now our deployment is down to 30 seconds.