Error "Cannot write configuration file" while setting bindings

Legacy Uncategorised
1

We are randomly getting the following error:

New-ItemProperty : Filename: \?\C:\Windows\system32\inetsrv\config\applicationHost.config
Error: Cannot write configuration file
At C:\Octopus\Applications.Tentacle\Octopus.Tentacle\2.0.13.1100\Scripts\Octopus.Features.IISWebSite_BeforePostDeploy.ps1:117 char:5

  • New-ItemProperty $sitePath -name bindings -value ($wsbindings[$i])

  • ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    • CategoryInfo : NotSpecified: (:slight_smile: [New-ItemProperty], FileLoadException
    • FullyQualifiedErrorId : System.IO.FileLoadException,Microsoft.PowerShell.Commands.NewItemPropertyCommand
      Fatal
      Script ‘C:\Octopus\Applications.Tentacle\Octopus.Tentacle\2.0.13.1100\Scripts\Octopus.Features.IISWebSite_BeforePostDeploy.ps1’ returned non-zero exit code: 1. Deployment terminated.

Redeploying the project always results in success.

It appears most often when we are deploying multiple sites in succession (via automated build), but we have seen it happen with a single deployment as well. Is there anyway you could catch this exception, pause and retry?

Researching the actual error lead me to finding that the system.applicationHost/configHistory potentially locks the configuration, which might be our root cause. I’ve internally changed the values to not backup the configuration as often, but that’s not ideal and I’m not sure that it’s actually prevented the error from happening at this point. (If it has, I’ll let you all know.)

2

in v1.6 we came up with our own pre-deploy script to create websites and setup some wcf and appfabric autostart stuff. this error is a very common problem we ran into also. it is more related to the fact that IIS likes to hold on to applicationhostconfig as it recycles or if any other changes are made to IIS. for us it occurs more often when we are deploying more than one wcf service at the same time. Im sure with the built in processes in v2.x that do some of the IIS setup, those types of errors will still occur. not sure if octopus support has other suggestions. we just rerun our deployments as you have. I’d love to find a better solution too, but I think its the nature of IIS and not really an Octopus issue.

3

What should we do here? Should the IIS configuration steps do a try/catch retry with a small sleep a couple of times? Or would that be a bad idea?

Paul

4

I don’t know that sleep will help because it all depends on what else is happening on the server or it could be worse with multiple servers. ive had to restart a deployment multiple times due to one server out of 4 failing randomly with this. I don’t know that there is a real solution because this is being caused by IIS itself, but at the very least as long as this shows up as an error in the deploy log, we will know about it and be able to take appropriate action. not sure if Michael’s idea of a pause/retry type thing would help because ultimately you are having to retry it manually in either case. I suppose it might go faster though if you were able to hit that error, and immediately be able to retry that one server right away.

5

I actually like the idea that it should at least pause and retry a couple of times before failing out. In our case, we take the server out of rotation from our server farm prior to installation. When a failure occurs, the machine is left out of rotation because it could be in an invalid state. Granted, we do just reinstall at that point, but then have to manually bring the server back into rotation. It’s just an added step in our case. I don’t mind being a guinea pig and actually writing the code for you all – just to see if it makes a difference on our end. I’ll give a try and see how it goes.

6

Everything seems to be working with the retries in the modified version of my “Octopus.Features.IISWebSite_BeforePostDeploy.ps1” file. I’ve attached it for your review and testing. I’ve verified that it does actually work, meaning it will retry when a failure occurs.

Basically it will retry every piece that fails up to a specified maxFailures with a specified sleep time in between retries. If you do a compare against this version and version 2.1.3.1223, you should see what I’ve modified (and made configurable.)

Errors are logged using Write-Host, although it might be nice to spit out the actual exception type so the code could be a bit more narrow in scope about which exceptions are retried. Right now it just catches [System.Exception]. This is probably ok, since it really only retries 5 times for a maximum of 10 seconds - using the defaults. I don’t think 10 seconds with a few retries will hurt anything if the failure is something other than trying to write configuration changes - like conflicting bindings for example.

Let me know when if you all use it and what version it makes it to.

Thanks,
Michael McDaniel
http://mcdaniel.ws

Octopus.Features.IISWebSite_BeforePostDeploy.ps1 (12 KB)

7

Thanks very much for this Michael! I really appreciate it.

I made some changes to reuse parts of the script - you can view it here:

Also, I made the default sleep time random; the idea being that if two deployments on the same machine hit a lock, then they both retry at a fixed interval, then they’ll keep locking. Making it random seems like a simple way increase the chance there won’t be contention next time.

Paul

8

I’ve been running your version for a couple of weeks now without having any issues. Just wanted to say thanks and that I hope it goes out soon! Also, your function for doing the retry was pretty slick! nice work!

9

Thanks for the update Michael, it will be in 2.4!