Random Server Restarts

Legacy Uncategorised
1

Once every few days, our Octopus server restarts itself canceling any operations that are occuring in the system and generating errors when applications are being deployed both through automated and non-automated processes. Each time, the windows application logs has the error below. Could you all assist in trying to understand how to remedy this error and resolve it completely? Right now we have a process in place that restarts it; however, that tends to take five or ten minutes while our processing is down.

Level Date and Time Source Event ID Task Category

Error 8/31/2015 4:29:53 PM Application Error 1000 (100) "Faulting application name: Octopus.Server.exe, version: 2.6.4.951, time stamp: 0x5507bc85
Faulting module name: KERNELBASE.dll, version: 6.3.9600.17415, time stamp: 0x54505737
Exception code: 0xe0434352
Fault offset: 0x0000000000008b9c
Faulting process id: 0x700
Faulting application start time: 0x01d0e15139b7704b
Faulting application path: D:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 6a747464-5027-11e5-80dd-005056b93b13
Faulting package full name:
Faulting package-relative application ID: "

Error 8/31/2015 4:29:52 PM .NET Runtime 1026 None Application: Octopus.Server.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Reflection.TargetInvocationException
Stack:
at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])
at System.Delegate.DynamicInvokeImpl(System.Object[])
at System.ExecutionContextLightup.Run(System.ExecutionContextLightup, System.Action1<System.Object>, System.Object) at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run() at Microsoft.Runtime.CompilerServices.TaskAwaiter.RunNoException(System.Action) at Microsoft.Runtime.CompilerServices.TaskAwaiter+<>c__DisplayClassa.<OnCompletedInternal>b__4(System.Threading.Tasks.Task) at System.Threading.Tasks.Task.Execute() at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) at System.Threading.Tasks.Task.ExecuteEntry(Boolean) at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean) at System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean) at System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean) at System.Threading.Tasks.Task.FinishContinuations() at System.Threading.Tasks.Task1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.__Canon)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetResult(System.__Canon) at Raven.Client.LinqExtensions+<FirstOrDefaultAsync>d__2e1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
at System.Runtime.CompilerServices.TaskAwaiter.RunNoException(System.Action)
at System.Runtime.CompilerServices.TaskAwaiter+<>c__DisplayClassa.b__4(System.Threading.Tasks.Task)
at System.Threading.Tasks.Task.Execute()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
at System.Threading.Tasks.Task.ExecuteEntry(Boolean)
at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)
at System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean)
at System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task.Finish(Boolean)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
at System.Threading.Tasks.Task.ExecuteEntry(Boolean)
at System.Threading.ThreadPoolWorkQueue.Dispatch()

2

We have the exact same symptoms(same stack trace) with octopus server 2.6.5.1010
I will try the the following workaround helps:

3

We will probably go for upgrading to 3.0 to resolve this.

http://help.octopusdeploy.com/discussions/problems/34953-octopus-deploy-2651010-crashes-on-unhandled-exception

4

It appears an upgrade definitely would help. Did the workaround prove to be useful for you all? I may try that first on my side since we don’t really have the resources at this point to perform an upgrade.

5

Hi,

I’m very sorry to hear you guys are experiencing this issue. Unfortunately, if the workaround mentioned by Jonas doesn’t work, then the only thing left to do would be to upgrade to 3.0.

Thanks,

Dalmiro

6

I never tried the workaround, we will go for upgrade at a later stage since this is not a major issue at the moment.

7

Dalmiro,

A quick question hopefully based on the documentation at http://docs.octopusdeploy.com/display/OD2/Switch+from+single-threaded+workstation+GC+to+concurrent+server+GC, if gcConcurrent doesn’t exist in the runtime tag then is that appropriate or should it be there with a value = false?

8

Hi,

It should not be there.

Thanks,

dalmiro