4 logs attached. Combination of hung/success and dev/staging servers.
Some facts:
Octopus Deploy server 3.2.8
Deployed app size: 240MB and 3300 files
Successful deployment takes 4-6 minutes, where timestamp checking makes up about 2-3 of those minutes
Azure web app S1 Standard tier app service plan with two web apps hosted on it, total load is very low
Upload speed of big binary to web app FTP is stable at 1.5-1.7 MiB/s, meaning it should theoretically complete a full upload of web app in about 3.5 minutes, and practice have shown a clean upload (no timestamp checking) to take about 4-6 minutes too.
Frequency varies but we almost always have to retry once or twice for each deployment, sometimes it fails several times in a row
Clean upload (delete deployment slot first) always succeeds, or at least the 4-5 times we tried it, which indicates that timestamp checking is one of the culprits here.
Deploy from Visual Studio seems to work well, at least the 2 times we have tried it so far. Might need to test more here to be certain.
A hung deployment’s last log output is always timestamp checking.
I also frequently see hangs that look exactly like the files shared earlier. Our app will deploy and then at some point just stop updating the log files, these can sit there for an hour or more and never complete or have new log messages.
We had another issue today with a hang where it looked like it got past the deploy step but then never proceeded to the next step. I have attached that log.
Details on our setup:
Octopus Deploy - 3.2.17
Server and tentacle (same machine) running on an Azure VM (North Central US)
Uploading to a Azure Web App (also North Central US)
FYI. Getting rid of parallel processes fixed it. We decided to bump up our Octopus VM as deployments were maxing out our CPU. Will update as we do more testing.
We are still seeing what appears to be a file check for every file. But at least the deployment didn’t fail on the Octopus Server.
Thanks for getting back in touch. I’ll try and summarise what we’ve done so far to address this issue. I can see you’ve been participating in a few threads about this issue, so I apologise if I’m repeating anything you already know.
The core issue
The examples we have seen so far look look like threading dead-locks in WebDeploy. Microsoft.Web.Deployment has reportedly had issues with locking when using Checksum for file comparisons.
The solution
You can now use Timestamp instead of Checksum and see if this fixes the issue for your deployments. Other customers have reported success. Requires Octopus 3.3.3 or greater.
You will need to modify existing steps to use Timestamp (we do not automatically switch them over)
Any new Azure Web App steps you create will use Timestamp by default.
A note about timestamps
To get the most benefit of using timestamps for WebDeploy sync, start using zip packages instead of NuGet packages. We cannot maintain consistent timestamps when extracting NuGet packages - see this GitHub Issue for more information.
Plans for the future
We will continue to roll out further improvements to Azure Web App support, though we don’t have any fixed delivery timeframes for these:
Thank you for the detailed response, I will try the Timestamp solution and report back. Also look forward to the bugfixes you mention, could you please update this ticket when that happens? I’m sure others will find that info useful too.
I stand corrected. I went through and logged the past few weeks worth of deployments. It appears to have cut the time down by at least 1/2 in most cases even more.
One note, one of the options you recommend, is to try doing a deploy to a new slot and then switching. A few questions that blocked us when considering this, but perhaps you have the answer.
We use Stackify.com, others i’m sure use NewRelic for Server monitoring because currently the Azure monitoring is just not powerful enough. To do so however, you have to install extensions and you have to do this via the portal there is no way that we could find to automate and configure these installations. Have you seen some other way?
Application Settings: if we were to do this, then we would need to keep all of our Application settings/ Configs in Octopus or manage somewhere else and pulled in during deploy. What are you suggesting around this?
APMs like Stackify and NewRelic can be installed in different ways. One way NewRelic is installed as a NuGet package, and included when you build, package, and eventually deploy your application.
Stackify offers a Web App Extension (which I think you’re using) and you would need to add that Web App Extension to any new Site or Slot as part of the deployment, and uses some AppSettings for your API key etc. In Octopus I would do this as follows:
Azure Script Step: Create Web App (if required) adding the Web App Extension
Azure Script Step: Use PowerShell to set your appSettings for Stackify for the Web App
Azure Web App Step: As per normal
Sadly I haven’t been able to find a PowerShell CmdLet for adding Web App Extensions, and that’s something you may find before I do, or raise an issue here
The benefit of this is that you can replace Slots (or even Web Apps) any time without needing special handling - at least once you can add WebApp Extensions via PowerShell…