When we first installed Octopus (can’t remember initial version, 3.0.*), our deployment times were ~20 seconds. It was very fast and was a good selling point when we decided to move past trial and purchase a license for Octopus. Shortly thereafter (maybe 3.1.0?), our deployment times slowed to ~3-5 minutes. Our projects have not changed significantly enough to merit such a slowdown, certainly not in application size. Now we’re on 3.2.1 (as of this morning) and the deployment times have not improved since. Looking at “Verbose | All” logs, I see the substep that’s taking ~5 minutes, under the Upload Package step (forgive formatting):
Octopus Deploy: Calamari version 3.1.2+Branch.master.Sha.08f592c501569f343a331d542df3878d35959e69
08:00:24Verbose
Octopus Deploy: Calamari version 3.1.2+Branch.master.Sha.08f592c501569f343a331d542df3878d35959e69
08:00:24Verbose
Package webapp version 0.1.5788.14407 hash 2faea18b27635c0ccd841db8e2d76fbb392127a5 has not been uploaded.
08:00:24Verbose
Finding earlier packages that have been uploaded to this Tentacle. <----- (THIS IS THE STEP)
08:05:20Verbose
No earlier packages for webapp has been uploaded
08:05:20Info
Uploading package webapp 0.1.5788.14407
08:05:20Verbose
Requesting upload…
08:05:20Info
Beginning streaming transfer of webapp.0.1.5788.14407.nupkg-41db6a84-186f-453d-a15e-283861f9ecb6
08:05:21
Sometimes our deployments are fast and go back to ~30 seconds, but this 5 minute deploy is becoming the new norm. Can I get some help with this? Not sure how to diagnose where the issue lies (whether it’s infrastructure on our end or an issue with Octopus).
That step is part of the new delta compression functionality in the package acquisition step, we have had another customer report that they also had issues with slowness on this step. We have introduced a system variable Octopus.Acquire.DeltaCompressionEnabled to disable delta compression (just add that variable with a value of false to the project(s) you are having issues with) and just upload the full package everytime (packages that have not yet been uploaded). You can use this as a workaround for your slow deployments.
Let me start by saying the workaround was effective. Had some fun getting it in place though:
Tried to add it as a variable set, and when trying to add the variable set to my projects on project Save ended up getting a 400 on
POST /api/projects/Projects-1
Response Body:
{
"ErrorMessage": "There was a problem with your request.",
"Errors": [
"A channel ID must be provided to enable automatic release creation."
]
}
I was able to add it just as a variable and Save with success. I ran into that same error when trying to set deployment to always deploy all packages. Workaround was to disable automatic release creation, change the deployment setting, then re-enable automatic release creation. (I know this is unrelated to this issue from this forum but you guys might want to check on that Project Save call).
All that being said, our deployment times have been reduced from 5 minutes average to 9 seconds and 11 seconds. Thank you for your help!
Great to hear your deployments are back down to their normal durations again!
I have raised #2128 to fix the error about channel Id and automatic release creation you ran into. Can I just confirm if this project was created prior to your upgrade from 3.0.x to 3.2 (from the ProjectId it certainly looks like this is the case)?
I just noticed the same problem, starting maybe a month or so ago? THe upload Package step has a 5 minute window where nothing is logged. This only happens when octo is deploying to calamari on the same server octo is hosted on.
| == Success: Acquire packages ==
11:55:10 Info | Acquiring packages
11:55:10 Info | Making a list of packages to download
11:55:10 Verbose | Checking package cache for package m2 version 2016.602.407
11:55:12 Verbose | Package m2 version 2016.602.407 was found in cache. No need to download. Using file: C:\Octopus\OctopusServer\PackageCache\feeds-tc\m2.2016.602.407_DD5DD9783140DF4ABBF4EBC8731406B1.nupkg
11:55:12 Verbose | SHA1 hash of package m2 is: 61f385dc777bf3c2bc052c40f5b02e0f8c42391e
11:55:12 Verbose | Checking package cache for package m3 version 2016.602.407
11:55:12 Verbose | Package m3 version 2016.602.407 was found in cache. No need to download. Using file: C:\Octopus\OctopusServer\PackageCache\feeds-tc\m3.2016.602.407_4DB3BEE5419F6B4D85DD5A20B06931DA.nupkg
11:55:12 Verbose | SHA1 hash of package m3 is: f87c80cf22f4587360fb523186b5e2434dbadd54
12:02:47 Info | All packages have been acquired
12:02:47 Verbose | Acquire Packages completed
|
| Success: Boxy
|
| Success: Upload package m2 version 2016.602.407
11:55:12 Verbose | Octopus Deploy: Calamari version 3.3.13+Branch.master.Sha.821b4ffaf94e26f64cd9c14be2f3969587822fad
11:55:13 Verbose | Name Value
11:55:13 Verbose | ---- -----
11:55:13 Verbose | PSVersion 5.0.10514.6
11:55:13 Verbose | WSManStackVersion 3.0
11:55:13 Verbose | SerializationVersion 1.1.0.1
11:55:13 Verbose | CLRVersion 4.0.30319.42000
11:55:13 Verbose | BuildVersion 10.0.10514.6
11:55:13 Verbose | PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
11:55:13 Verbose | PSRemotingProtocolVersion 2.3
11:55:13 Verbose | Octopus Deploy: Calamari version 3.3.13+Branch.master.Sha.821b4ffaf94e26f64cd9c14be2f3969587822fad
11:55:13 Verbose | Package m2 version 2016.602.407 hash 61f385dc777bf3c2bc052c40f5b02e0f8c42391e has not been uploaded.
11:55:13 Verbose | Finding earlier packages that have been uploaded to this Tentacle.
12:02:23 Verbose | Found 5 earlier versions of m2 on this Tentacle
12:02:23 Verbose | - 2016.602.405: C:\Octopus\Files\m2.2016.602.405.nupkg-7f8d771a-a624-47be-b771-c4a140284306
12:02:23 Verbose | - 2016.602.404: C:\Octopus\Files\m2.2016.602.404.nupkg-1a8018d6-718b-4994-b8bc-c0c8c0f9844b
12:02:23 Verbose | - 2016.602.402: C:\Octopus\Files\m2.2016.602.402.nupkg-c391e72c-07c4-47ef-8e73-682270b5cc46
12:02:23 Verbose | - 2016.602.399: C:\Octopus\Files\m2.2016.602.399.nupkg-57ecae8e-9cce-41da-a75d-e499562c62da
12:02:23 Verbose | - 2016.602.398: C:\Octopus\Files\m2.2016.602.398.nupkg-e31020a9-ca7a-408d-b7a0-eafba7b35eda
12:02:30 Info | Found matching version 2016.602.405: C:\Octopus\OctopusServer\PackageCache\feeds-tc\m2.2016.602.405_26F2A483A6B86F41861F08F9E0A5EC07.nupkg
12:02:46 Info | Package m2 version 2016.602.407 successfully uploaded to this server
|
| Success: Building delta for m2 version 2016.602.407
12:02:33 Info | Using package C:\Octopus\OctopusServer\PackageCache\feeds-tc\m2.2016.602.405_26F2A483A6B86F41861F08F9E0A5EC07.nupkg with hash 1d44da545abbe09eeba2375e5f4b560b3564a551 for creating delta
12:02:33 Verbose | Signature file m2.2016.602.405_26F2A483A6B86F41861F08F9E0A5EC07.nupkg.octosig already exists, using C:\Octopus\OctopusServer\PackageCache\m2.2016.602.405_26F2A483A6B86F41861F08F9E0A5EC07.nupkg.octosig
12:02:33 Verbose | Delta file m2.2016.602.405_to_2016.602.407.octodelta already exists, using file C:\Octopus\OctopusServer\PackageCache\m2.2016.602.405_to_2016.602.407.octodelta
12:02:33 Info | Original package was 103 MB, delta file is 36 MB
|
| Success: Uploading and applying delta m2.2016.602.405_to_2016.602.407.octodelta
12:02:37 Verbose | Octopus Deploy: Calamari version 3.3.13+Branch.master.Sha.821b4ffaf94e26f64cd9c14be2f3969587822fad
12:02:40 Info | Applying delta to C:\Octopus\Files\m2.2016.602.405.nupkg-7f8d771a-a624-47be-b771-c4a140284306 with hash 1d44da545abbe09eeba2375e5f4b560b3564a551 and storing as C:\Octopus\Files\m2.2016.602.407.nupkg-36f2316a-1fe5-4510-9b1d-7578e6f522bd
12:02:40 Info | Applying delta: 0%
12:02:41 Info | Applying delta: 10%
12:02:41 Info | Applying delta: 20%
12:02:42 Info | Applying delta: 40%
12:02:42 Info | Applying delta: 50%
12:02:42 Info | Applying delta: 90%
|
| Success: Boxy
|
| Success: Upload package m3 version 2016.602.407
11:55:12 Verbose | Octopus Deploy: Calamari version 3.3.13+Branch.master.Sha.821b4ffaf94e26f64cd9c14be2f3969587822fad
11:55:13 Verbose | Name Value
11:55:13 Verbose | ---- -----
11:55:13 Verbose | PSVersion 5.0.10514.6
11:55:13 Verbose | WSManStackVersion 3.0
11:55:13 Verbose | SerializationVersion 1.1.0.1
11:55:13 Verbose | CLRVersion 4.0.30319.42000
11:55:13 Verbose | BuildVersion 10.0.10514.6
11:55:13 Verbose | PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
11:55:13 Verbose | PSRemotingProtocolVersion 2.3
11:55:13 Verbose | Octopus Deploy: Calamari version 3.3.13+Branch.master.Sha.821b4ffaf94e26f64cd9c14be2f3969587822fad
11:55:13 Verbose | Package m3 version 2016.602.407 hash f87c80cf22f4587360fb523186b5e2434dbadd54 has not been uploaded.
11:55:13 Verbose | Finding earlier packages that have been uploaded to this Tentacle.
11:56:53 Verbose | Found 5 earlier versions of m3 on this Tentacle
11:56:53 Verbose | - 2016.602.405: C:\Octopus\Files\m3.2016.602.405.nupkg-2188bad2-161d-498e-860d-a6fac8d58ae6
11:56:53 Verbose | - 2016.602.404: C:\Octopus\Files\m3.2016.602.404.nupkg-74f71608-16e8-4949-87c8-118bed8c5808
11:56:53 Verbose | - 2016.602.402: C:\Octopus\Files\m3.2016.602.402.nupkg-f1bb3745-132d-41de-8ffe-d6d64c456785
11:56:53 Verbose | - 2016.602.399: C:\Octopus\Files\m3.2016.602.399.nupkg-3c30c85c-4c46-433e-aaf7-d8b5a5fd0f8c
11:56:53 Verbose | - 2016.602.398: C:\Octopus\Files\m3.2016.602.398.nupkg-4ce3772e-55a2-49fd-922d-cd1b51f1fe63
11:56:53 Info | Found matching version 2016.602.405: C:\Octopus\OctopusServer\PackageCache\feeds-tc\m3.2016.602.405_95FBD18D3785FB4894F7EF1A08992A38.nupkg
11:56:55 Info | Package m3 version 2016.602.407 successfully uploaded to this server
|
| Success: Building delta for m3 version 2016.602.407
11:56:53 Info | Using package C:\Octopus\OctopusServer\PackageCache\feeds-tc\m3.2016.602.405_95FBD18D3785FB4894F7EF1A08992A38.nupkg with hash 7f4e2aa4d25b1c1af4646b171e4f9a2cbed5b13c for creating delta
11:56:53 Verbose | Signature file m3.2016.602.405_95FBD18D3785FB4894F7EF1A08992A38.nupkg.octosig already exists, using C:\Octopus\OctopusServer\PackageCache\m3.2016.602.405_95FBD18D3785FB4894F7EF1A08992A38.nupkg.octosig
11:56:53 Verbose | Delta file m3.2016.602.405_to_2016.602.407.octodelta already exists, using file C:\Octopus\OctopusServer\PackageCache\m3.2016.602.405_to_2016.602.407.octodelta
11:56:53 Info | Original package was 29 MB, delta file is 13 MB
|
| Success: Uploading and applying delta m3.2016.602.405_to_2016.602.407.octodelta
11:56:55 Verbose | Octopus Deploy: Calamari version 3.3.13+Branch.master.Sha.821b4ffaf94e26f64cd9c14be2f3969587822fad
11:56:55 Info | Applying delta to C:\Octopus\Files\m3.2016.602.405.nupkg-2188bad2-161d-498e-860d-a6fac8d58ae6 with hash 7f4e2aa4d25b1c1af4646b171e4f9a2cbed5b13c and storing as C:\Octopus\Files\m3.2016.602.407.nupkg-73be3c4b-30f6-4c63-a1cb-1497357ea089
11:56:55 Info | Applying delta: 0%
11:56:55 Info | Applying delta: 10%
11:56:55 Info | Applying delta: 50%
11:56:55 Info | Applying delta: 100%
Thanks for getting in touch. If you see the following lines in your logs:
11:55:13 Verbose | Finding earlier packages that have been uploaded to this Tentacle.
12:02:23 Verbose | Found 5 earlier versions of m2 on this Tentacle
There is about a 5 minute delay where Octopus needs to check the package cache to find files to compare for the delta compression. Are you using any form of retention policy? Could you clear out some of the older files in your Tentacle cache?
This will significantly improve the time that this operation needs to run for.
Is there any sort of policy to automatically clear out the cache in C:\Octopus\OctopusServer\PackageCache ? Or should I script that out to keep octo performing well?
Are there plans to speed up the algorithm that does that step? It seems silly that the cache is slowing things down!
I have retention policies set to 7 release/7 days for all of our lifecycles. Our build process auto deploys master to a the test and staging environments, probably 3 times a day. It pulls the packages from teamcity nuget. There are currently 86 releases in the list, the project uses 2 nuget packages.
The PackageCache (C:\Octopus\OctopusServer\PackageCache) folder has 149 .octodelta files per packge (so 300 total) and 256 .octosig files. The feeds-tc subfolder has 256 nupkg files, 128 of each package.
C:\Octopus\Files has 267 copies of the nupkgs (133 and 134)
The packagecache folder numbers vs the release count doesn’t add up to me.
c:\Octopus\Applications\Staging$nugetname\ has 7 extracted packages each.
The retention policy was previously (like a month ago) set to 21 days/21 packages.
Re Deploying a release to the same environment goes quickly.
It is only slow on staging, which is also the octopus server
Our default for package cache is 20 days. But this is configurable. Octopus.Sever.exe has a configure command that allows you to change the value of this --cachePackages=VALUE
You can set it to a value that is better for your workflow as you have a better idea than we do! Do not forget to first stop the service then run this command.
Here is the doco: http://docs.octopus.com/display/OD/Server+configuration+and+File+storage