Octopus webserver crashes when trying transform xml

david_barnes · 14 August 2015 19:28

There are 3 errors in my event Logs

================================================
2015-08-14 19:09:31.3964 14 FATAL Unhandled AppDomain exception occurred: Name cannot begin with the ‘’ character, hexadecimal value 0x5C. Line 1, position 4.
System.Xml.XmlException: Name cannot begin with the ‘’ character, hexadecimal value 0x5C. Line 1, position 4.
at System.Xml.XmlTextReaderImpl.Throw(String res, String[] args)
at System.Xml.XmlTextReaderImpl.ParseQName(Boolean isQName, Int32 startOffset, Int32& colonPos)
at System.Xml.XmlTextReaderImpl.ParseElement()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlReader.MoveToContent()
at System.Xml.Linq.XElement.Load(XmlReader reader, LoadOptions options)
at System.Xml.Linq.XElement.Parse(String text, LoadOptions options)
at Octopus.Worker.Scripting.ServiceMessageParser.ProcessMessage(ProcessOutputSource source, String message) in y:\work\refs\heads\master\source\Octopus.Worker\Scripting\ServiceMessageParser.cs:line 101
at Octopus.Worker.Scripting.ServiceMessageParser.Append(ProcessOutputSource source, String line) in y:\work\refs\heads\master\source\Octopus.Worker\Scripting\ServiceMessageParser.cs:line 73
at Octopus.Worker.Scripting.ScriptOutputFilter.Write(ProcessOutputSource source, String text) in y:\work\refs\heads\master\source\Octopus.Worker\Scripting\ScriptOutputFilter.cs:line 67
at System.Diagnostics.AsyncStreamReader.FlushMessageQueue()
at System.Diagnostics.AsyncStreamReader.GetLinesFromStringBuilder()
at System.Diagnostics.AsyncStreamReader.ReadBuffer(IAsyncResult ar)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.IO.Stream.ReadWriteTask.System.Threading.Tasks.ITaskCompletionAction.Invoke(Task completingTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task.Finish(Boolean bUserDelegateExecuted)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
at System.Threading.ThreadPoolWorkQueue.Dispatch()

=================

Application: Octopus.Server.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Xml.XmlException
Stack:
at System.Xml.XmlTextReaderImpl.Throw(System.String, System.String[])
at System.Xml.XmlTextReaderImpl.ParseQName(Boolean, Int32, Int32 ByRef)
at System.Xml.XmlTextReaderImpl.ParseElement()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlReader.MoveToContent()
at System.Xml.Linq.XElement.Load(System.Xml.XmlReader, System.Xml.Linq.LoadOptions)
at System.Xml.Linq.XElement.Parse(System.String, System.Xml.Linq.LoadOptions)
at Octopus.Worker.Scripting.ServiceMessageParser.ProcessMessage(Octopus.Shared.Contracts.ProcessOutputSource, System.String)
at Octopus.Worker.Scripting.ServiceMessageParser.Append(Octopus.Shared.Contracts.ProcessOutputSource, System.String)
at Octopus.Worker.Scripting.ScriptOutputFilter.Write(Octopus.Shared.Contracts.ProcessOutputSource, System.String)
at System.Diagnostics.AsyncStreamReader.FlushMessageQueue()
at System.Diagnostics.AsyncStreamReader.GetLinesFromStringBuilder()
at System.Diagnostics.AsyncStreamReader.ReadBuffer(System.IAsyncResult)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.IO.Stream+ReadWriteTask.System.Threading.Tasks.ITaskCompletionAction.Invoke(System.Threading.Tasks.Task)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task.Finish(Boolean)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
at System.Threading.Tasks.Task.ExecuteEntry(Boolean)
at System.Threading.ThreadPoolWorkQueue.Dispatch()

=================
Faulting application name: Octopus.Server.exe, version: 3.0.6.2140, time stamp: 0x55b5b74b
Faulting module name: KERNELBASE.dll, version: 6.3.9600.17415, time stamp: 0x54505737
Exception code: 0xe0434352
Fault offset: 0x0000000000008b9c
Faulting process id: 0x750
Faulting application start time: 0x01d0d6c46e005662
Faulting application path: C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: fd952570-42b7-11e5-80be-000d3a308b2c
Faulting package full name:
Faulting package-relative application ID:

we are running v3.0.6.2140

I’ve noticed this happens when I have remove a variable from my web.config but in my web.release.config there is still a transform referencing that variable. Let me know if you need any more information to help debug this issue.

Damian_Maclennan · 17 August 2015 01:05

Hi David,

Thanks for getting in touch. Sorry you are getting this error.

Is there any chance you can attach those config files so we can have a look ? (we can make the thread private).
It looks like a bug we can fix pretty quickly.

Damian

Aaron · 19 August 2015 08:52

I am getting this exact same error when transforming my configs. I have just upgraded to Octopus 3.0 and have not change the Process steps at all from when I was running v2.6 and now I am getting these errors and having to edit the configs manually after deployment. I dont believe there is anything actually wrong with my config syntax as it has been working for months on Octopus 2.6 without any hassles.

Damian_Maclennan · 25 August 2015 23:57

Hi Aaron,

Thanks for chiming in.

Are you able to post a full deployment log ?

Damian

aaron_morse · 26 August 2015 07:23

Hi Damian,

I managed to get passed this issue by removing the first line of the config

<?xml version="1.0" encoding="utf-8"?>

Not sure why this made a difference as it was present in my configs before with Octopus 2.6 which was working fine.
Please see the full deployment log below.

Running: MyApp
15:19:35 Info | 15:19:35 Info | 15:19:35 Info | 15:19:36 Verbose | 15:20:12 Verbose | 15:20:12 Info | 15:20:12 Info | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Info | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Info | 15:20:12 Info | 15:20:12 Info | 15:20:12 Info | 15:20:12 Verbose | 15:20:12 Verbose | 15:20:12 Info | 15:20:12 Info | 15:20:12 Info | 15:20:12 Info | 15:20:12 Info | 15:20:12 Info | 15:20:12 Info | 15:20:12 Info | 15:20:12 Info | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Verbose | 15:20:12 Verbose |
15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | 15:20:12 Error | |
| Cancelled: 15:20:12 Verbose | | Deploying package ‘C:\Software\Octopus\Files\MyAppWeb.3.10.0.288.nupkg-c1085499-1cc7-435a-b69b-2247676c1781’ to machine 'xxx’
Deploying package: C:\Software\Octopus\Files\MyAppWeb.3.10.0.288.nupkg-c1085499-1cc7-435a-b69b-2247676c1781
Using variables from: C:\Software\Octopus\Work\20150818131935-9\Variables.json
Extracting package to: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1
Extracted 3939 files
Performing variable substitution on 'C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Web.Production.config’
Transforming ‘C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Web.config’ using ‘C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Web.Release.config’.
Executing RemoveAttributes (transform line 18, 18)
on /configuration/system.web/compilation
Applying to ‘compilation’ element (source line 39, 6)
Removed ‘debug’ attribute
Removed 1 attributes
Done executing RemoveAttributes
Transforming ‘C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Web.config’ using ‘C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Web.Production.config’.
Executing SetAttributes (transform line 8, 80)
on /configuration/appSettings/add[@key=‘AuthServiceUrl’]
Applying to ‘add’ element (source line 26, 6)
Set ‘key’ attribute
Set ‘value’ attribute
Set 2 attributes
Done executing SetAttributes
Executing SetAttributes (transform line 9, 77)
on /configuration/appSettings/add[@key=‘TPGServiceUrl’]
Applying to ‘add’ element (source line 25, 6)
Set ‘key’ attribute
Set ‘value’ attribute
Set 2 attributes
Done executing SetAttributes
Executing SetAttributes (transform line 23, 260)
on /configuration/connectionStrings/add[@name=‘DataStaging’]
Applying to ‘add’ element (source line 33, 6)
Set ‘name’ attribute
Set ‘connectionString’ attribute
Set 2 attributes
Done executing SetAttributes
Executing SetAttributes (transform line 24, 261)
on /configuration/connectionStrings/add[@name=‘AUMyApp’]
Applying to ‘add’ element (source line 34, 6)
Set ‘name’ attribute
Set ‘connectionString’ attribute
Set 2 attributes
Done executing SetAttributes
Executing SetAttributes (transform line 25, 266)
on /configuration/connectionStrings/add[@name=‘MyAppContext’]
Applying to ‘add’ element (source line 35, 6)
Set ‘name’ attribute
Set ‘connectionString’ attribute
Set 2 attributes
Done executing SetAttributes
Executing SetAttributes (transform line 28, 255)
on /configuration/system.web/sessionState[@mode=‘SQLServer’]
Applying to ‘sessionState’ element (source line 38, 6)
Set ‘mode’ attribute
Set ‘sqlConnectionString’ attribute
Set ‘allowCustomSqlDatabase’ attribute
Set ‘cookieless’ attribute
Set ‘timeout’ attribute
Set 5 attributes
Done executing SetAttributes
Looking for appSettings and connectionStrings in any .config files
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\NLog.config
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Web.config
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Web.Debug.config
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Web.Demo.config
File ‘C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Web.Production.config’ was interpreted as an XML configuration transform; variable substitution won’t be attempted.
File ‘C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Web.Release.config’ was interpreted as an XML configuration transform; variable substitution won’t be attempted.
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Web.Test.config
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\Areas\AU\Views\Web.config
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\bin\AUMyApp.Web.dll.config
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\bin\NLog.config
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\bin\Web.Debug.config
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\bin\Web.Demo.config
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\bin\Web.Production.config
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\bin\Web.Release.config
No matching setting or connection string names were found in: C:\Software\Octopus\Applications\Production\MyAppWeb\3.10.0.288_1\bin\Web.Test.config
System.Xml.XmlException: Data at the root level is invalid. Line 1, position 1.
at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
at Calamari.Integration.ConfigurationVariables.ConfigurationVariablesReplacer.ModifyConfigurationFile(String configurationFilePath, VariableDictionary variables) in y:\work\b1fd300731d5f2fc\source\Calamari\Integration\ConfigurationVariables\ConfigurationVariablesReplacer.cs:line 20
at Calamari.Deployment.Conventions.ConfigurationVariablesConvention.Install(RunningDeployment deployment) in y:\work\b1fd300731d5f2fc\source\Calamari\Deployment\Conventions\ConfigurationVariablesConvention.cs:line 37
at Calamari.Deployment.ConventionProcessor.RunInstallConventions() in y:\work\b1fd300731d5f2fc\source\Calamari\Deployment\ConventionProcessor.cs:line 60
at Calamari.Deployment.ConventionProcessor.RunConventions() in y:\work\b1fd300731d5f2fc\source\Calamari\Deployment\ConventionProcessor.cs:line 28
Running rollback conventions…
Adding journal entry:
System.Xml.XmlException: Data at the root level is invalid. Line 1, position 1.
at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
at Calamari.Integration.ConfigurationVariables.ConfigurationVariablesReplacer.ModifyConfigurationFile(String configurationFilePath, VariableDictionary variables) in y:\work\b1fd300731d5f2fc\source\Calamari\Integration\ConfigurationVariables\ConfigurationVariablesReplacer.cs:line 20
at Calamari.Deployment.Conventions.ConfigurationVariablesConvention.Install(RunningDeployment deployment) in y:\work\b1fd300731d5f2fc\source\Calamari\Deployment\Conventions\ConfigurationVariablesConvention.cs:line 37
at Calamari.Deployment.ConventionProcessor.RunInstallConventions() in y:\work\b1fd300731d5f2fc\source\Calamari\Deployment\ConventionProcessor.cs:line 60
at Calamari.Deployment.ConventionProcessor.RunConventions() in y:\work\b1fd300731d5f2fc\source\Calamari\Deployment\ConventionProcessor.cs:line 50
at Calamari.Commands.DeployPackageCommand.Execute(String[] commandLineArguments) in y:\work\b1fd300731d5f2fc\source\Calamari\Commands\DeployPackageCommand.cs:line 107
at Calamari.Program.Main(String[] args) in y:\work\b1fd300731d5f2fc\source\Calamari\Program.cs:line 21
The remote script failed with exit code 100
Apply retention policy on Tentacles
Step “Apply Tentacle Retention Policy” runs only when all previous steps succeeded; skipping

Regards,
Aaron

Damian_Maclennan · 28 August 2015 01:35

Hi Aaron,

Thanks for all the info!
Looks like we have an XML encoding issue in there somewhere in this version.

We probably should have a verbose log entry for the file we’re about to try to open, was the file that broke it the only one with that declaration line in there ? It seems like there’s a bug, and would love to be able to narrow down on it’s cause.

Thanks again

Damian

aaron_morse · 28 August 2015 13:07

Hi Damian,

That is the strange thing. I have a few projects and it only seemed to be an issue in the one project however all the projects have that line in the top of the config transforms.

Regards,
Aaron

Damian_Maclennan · 31 August 2015 01:06

Hi Aaron,

That is strange. I wonder if it’s a text encoding issue, and removing that line fixes it because you’ve re-saved the file ?
What are you using for build / source control ? Could we have some line ending issue ?

Damian

aaron_morse · 31 August 2015 07:03

I am using BitBucket and Team City. I will try add the line back and resave and see if it is perhaps fixed just by resaving and I will feedback.

Regards,
Aaron