We have an old legacy site with hundreds of thousands of old content files. When we deploy our .net code to this site it seems to take several minutes between the extraction step and updating .config files, as seen in this log snippet:
2012-06-06 22:20:18 INFO Extracting package contents to 'E:\Projects\WWW'
2012-06-06 22:24:08 INFO Updating any .config files
Since we’re deploying in 10 steps to roll out to a few servers at a time, this results in a 40 minute+ deployment time.
I’m not sure what’s going on under the covers (other than recursively searching for *.config), but it would be helpful to us if operations like this happened in the Tentacle/Applications staging area before files are copied when OctopusPackageDirectoryPath is specified.
Thanks,
Chris
Hi Chris,
Thanks for the post. I can see how a 40 minute deployment would be painful!
I just checked, and those 4 minutes aren’t spent updating config files, but actually in extracting the package (the “Updating any .config files” log entry is written before scanning).
This is good news, because I have we have a solution in mind:
https://trello.com/card/pre-upload-packages/4e907de70880ba000079b75c/98
So there’ll be a “stage deployment” button that:
- Uploads the packages to all tentacles
- Extracts them, and possibly copies to OctopusPackageDirectory (need to think that through)
Later, the deployment will realize that the packages have been pre-staged, and so it will just configure them (update IIS, run powershell scripts, etc.).
What’s nice about this is that you can stage the deployment a few hours ahead of time, to keep the downtime really minimal.
Let me know if you think that will solve your problem and I’ll increase the priority.
Paul
Interesting. So it sounds like the disks on my web servers are overworked and it’s taking longer than it should to unzip the packages. The zip files aren’t any bigger than our other sites (which don’t have gigantic legacy docroots), so this sounds like a load issue to me. Thanks for the hints.
I took a process snapshot of Tentacle to take a peek during deployment today and my analysis seems to confirm that the slowness is during the XmlConfigTransforms convention, though the timestamps in the log don’t match up.
Log shows:
2012-06-12 12:08:35 INFO Extracting package contents to 'E:\Projects\WWW'
2012-06-12 12:11:46 INFO Updating any .config files
Snapshot taken at Tue Jun 12 12:11:30.000 2012 shows a thread with this call stack:
Child SP IP Call Site
000000001a79e6d0 00000000779b166a [NDirectMethodFrameStandalone: 000000001a79e6d0] Microsoft.Win32.Win32Native.FindNextFile(Microsoft.Win32.SafeHandles.SafeFindHandle, WIN32_FIND_DATA)
000000001a79e440 000007fef8682706 DomainNeutralILStubClass.IL_STUB_PInvoke(Microsoft.Win32.SafeHandles.SafeFindHandle, WIN32_FIND_DATA)
000000001a79e7b0 000007fef8e4608e System.IO.FileSystemEnumerableIterator`1[[System.__Canon, mscorlib]].AddSearchableDirsToStack(SearchData)
000000001a79e850 000007fef924fee8 System.IO.FileSystemEnumerableIterator`1[[System.__Canon, mscorlib]].MoveNext()
000000001a79e8b0 000007fef6bc692b System.Linq.Enumerable+d__14`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()
000000001a79e910 000007ff0090c1a8 Octopus.Tentacle.Conventions.XmlConfigTransforms.AfterDeployment(Octopus.Tentacle.Conventions.ConventionContext)
000000001a79e980 000007ff008f8341 Octopus.Tentacle.Jobs.Deployment.DeployPackageJobExecutor.Execute(Octopus.Tentacle.Jobs.Deployment.DeployPackageJob)
I suspect that the thing that is slow is the scanner which builds a list of **\*.config files, and then the log says after the list is built which files are being scanned/updated.
We’re working on moving the legacy content into virtual directories to side-step this issue, but it would also be nice if we could disable config file updates (we don’t use them here) and also if the convention applied to files in the staging directory (Octopus/Tentacles/Applications) before the files are copied when $OctopusPackageDirectoryPath is set.
Ahh, thanks for the post Chris. I see the issue.
The order of the conventions is:
public static readonly int PreDeployScript = 0;
public static readonly int CopyPackage = 500;
public static readonly int ConfigTransforms = 1000;
public static readonly int ConfigVariables = 1001;
The message “updating any .config files” is written by the ConfigVariables task. But the stack trace you have above is the ConfigTransforms task. So you were right with your initial suggestion and I was wrong.
You are right that there is no reason not to run the config tasks prior to copying, so I’ll change that order. I’ll also try and find some ways to speed up those tasks. I’ll include the fixes in the next release (which isn’t far away).
Thanks for the info,
Paul
We took a look at performance in Octopus 1.0 and verified that we are no longer experiencing this problem.
Cheers,
Chris