We encountered an issue when deploying multiple website to a production server. The deployment succeeded in test and stage, but when promoting to production, we receive the error:
Attempt 4 of 5 failed: Filename: \?\C:\Windows\system32\inetsrv\config\applicationHost.config
Line number: 1
Error: Configuration file is not well-formed XML
Waiting for 2 seconds before retrying…
Retrying…
get-webitemstate : Filename: \?\C:\Windows\system32\inetsrv\config\applicationHost.config
Line number: 1
Error: Configuration file is not well-formed XML
At C:\xxxxx\Octopus.Features.IISWebSite_BeforePostDeploy.ps1:165 char:3
This error has occurred on multiple web sites on the server, in different parts of the deployment of the site. It has failed setting windows authentication, and updating bindings. Once the error has occurred, the applicationHost.config file has contained a powershell script, and what appears to be the binary of a DLL.
I have compared the applicationHost.config file from our stage and prod server, and found no significant difference. They both appear to be valid XML, and the only differences are expected because of the environment/machine difference.
Are there any known causes of this issue? The deployment steps are Octopus Deploy build-in steps, any recommendations for debugging them?
Just a note for anyone that has had this happen to them, the config file can be recovered by copying it from c:\inetpub\history.
Thanks for getting in touch, this is a strange one. We’ve seen it at least once before but have not been able to reproduce it. Our suspicion is multiple threads causing the issue, in version 3.11.18 of Octopus we put in some safe guards to make IIS deployments more reliable. If you can you can try and update to a version after that to see if it solves the problem: https://octopus.com/downloads
Could we get some more details from you to help us work out what’s going on. What version of IIS and on which version of Windows is it running on, and which version of Octopus are you currently running.
We are using the OctopusBypassDeploymentMutex flag. However, the IIS steps that were failing were failing when no other steps were in parallel with it.
We are in the process of scheduling the upgrade of Octopus Deploy.
This is definitely happens in our environment sporadically. Looking at the applicationhost.config file after the failed deploy looks like it contains garbage either from disk or RAM, almost like a memory overflow.
This has been happening to us for over a year now, so I’m not sure if it’s something that will be fixed with a new release unless it was specifically addressed.
This causes some major issues when release our websites and we would really like to find a solution to this.
Thanks for getting in touch, sorry to hear you have also been experiencing this issue too. Would you be able to help us narrow down the cause, by providing some information about your set up and installed version.
What is the current version of Octopus you are running that still shows the corruption issue? Does it always happen on the same machine that’s being deployed to? What versions of Windows Server/IIS is/are the affected machines?
Are you also making use of the OctopusBypassDeploymentMutex flag for your deployments?
[Get-WebItemState], COMException