Octopus Server stopped with fatal exception

Legacy Uncategorised
1

Just got a bit suspicious behavior of Octopus Server (ver. 3.4.9). It just crashed with the following message in the log

2016-11-09 05:00:09.8240    135 FATAL  Unhandled AppDomain exception occurred: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: chunkLength
System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: chunkLength
   at System.Text.StringBuilder.ToString()
   at Octopus.Worker.Scripting.ServiceMessageParser.Flush(Action`2 to) in Y:\Work\refs\tags\3.4.9\source\Octopus.Worker\Scripting\ServiceMessageParser.cs:line 107
   at Octopus.Worker.Scripting.ServiceMessageParser.Finish() in Y:\Work\refs\tags\3.4.9\source\Octopus.Worker\Scripting\ServiceMessageParser.cs:line 93
   at Octopus.Worker.Scripting.ServiceMessageParser.Append(ProcessOutputSource source, String line) in Y:\Work\refs\tags\3.4.9\source\Octopus.Worker\Scripting\ServiceMessageParser.cs:line 28
   at Octopus.Worker.Scripting.LocalScriptLogWriter.WriteOutput(ProcessOutputSource source, String message) in Y:\Work\refs\tags\3.4.9\source\Octopus.Worker\Scripting\LocalScriptLogWriter.cs:line 18
   at System.Diagnostics.Process.OutputReadNotifyUser(String data)
   at System.Diagnostics.AsyncStreamReader.FlushMessageQueue()
   at System.Diagnostics.AsyncStreamReader.GetLinesFromStringBuilder()
   at System.Diagnostics.AsyncStreamReader.ReadBuffer(IAsyncResult ar)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.IO.Stream.ReadWriteTask.System.Threading.Tasks.ITaskCompletionAction.Invoke(Task completingTask)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.Finish(Boolean bUserDelegateExecuted)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

I’m not sure what caused this exception, i’ll check if it is temporary or not. But i’m not sure that unhandled exception in thread pool task should crash the server itself.

2

Hi,

Thanks for getting in touch and sharing this. We haven’t really seen anything like this so I’ll need to dig in a bit.

  • Can you send me your full server log file? Often, we find something interesting in the lead up to the crash/error.
  • When did this occur? i.e. In the middle of a deployment? Or browsing projects etc.

Looking forward to your reply.

Thanks

Rob

3

Hi Rob,

I’ve attached full log file before the crash.

I’m not sure what exactly server did when it crashed, hope you’ll get it from the log.

Roman

OctopusServer.1.txt (1 MB)

4

Hi Roman,

Thanks for the reply and attaching your log file. I didn’t really see anything bad in your log file. The exception is due to concurrent access to a StringBuilder used to buffer log output. It’s odd as we haven’t seen this previously. That said, I did notice that your log file was primarily filled with statements related to executing retention policies and they seemed to be taking a lot of time. I’m suspicious that these tasks might not be cleaning up correctly but I’m not 100% sure at this point.

If this crash happens again, can I ask you to turn on crash dump reporting so we can get more low level information on what happened? The following page has more information on this. http://docs.octopusdeploy.com/display/OD/Capture+a+crash+dump

Thanks

Rob