Tentacle.exe terminated csrss.exe, after upgrade from Octopus Server 3.3.6 to 3.8.2

Hello,

We have recently upgraded our Octopus Server from v3.3.6 to v3.8.2 across our test / production environments.

Everything has been working well, except for one server on production with tentacle installed, where somehow the service have managed to terminate csrss.exe process, causing the server to crash with a BSOD / and a subsequent forced restart.

The server’s operating system is Windows Server 2008r2.

After the restart the tentacle is working fine again, and so far have not encountered any further issues.

However, I am wondering if anyone had come across this issue before? It’s a bit worrying that an upgrade could potentially cause the server to crash and restart.

It would be good if there is a way we can prevent this in the future.

Below is the debugging details, which came from the memory dump by Windows when it crashed:

Page 3dc2ae not present in the dump file. Type “.hh dbgerr004” for details

DUMP_CLASS: 1

DUMP_QUALIFIER: 401

BUILD_VERSION_STRING: 7601.23564.amd64fre.win7sp1_ldr.160930-0600

SYSTEM_MANUFACTURER: VMware, Inc.

VIRTUAL_MACHINE: VMware

SYSTEM_PRODUCT_NAME: VMware Virtual Platform

SYSTEM_VERSION: None

BIOS_VENDOR: Phoenix Technologies LTD

BIOS_VERSION: 6.00

BIOS_DATE: 09/17/2015

BASEBOARD_MANUFACTURER: Intel Corporation

BASEBOARD_PRODUCT: 440BX Desktop Reference Platform

BASEBOARD_VERSION: None

DUMP_TYPE: 1

BUGCHECK_P1: 3

BUGCHECK_P2: fffffa800e93e3b0

BUGCHECK_P3: fffffa800e93e690

BUGCHECK_P4: fffff800019dba10

PROCESS_NAME: csrss.exe

CRITICAL_PROCESS: csrss.exe

IMAGE_NAME: csrss.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 0

MODULE_NAME: csrss

FAULTING_MODULE: 0000000000000000

EXCEPTION_CODE: (Win32) 0xefb6b50 (251358032) -

ERROR_CODE: (NTSTATUS) 0xefb6b50 -

CPU_COUNT: 8

CPU_MHZ: 7d0

CPU_VENDOR: GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 1a

CPU_STEPPING: 4

CPU_MICROCODE: 6,1a,4,0 (F,M,S,R) SIG: 710’00000000 (cache) 710’00000000 (init)

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

BUGCHECK_STR: 0xF4

CURRENT_IRQL: 0

ANALYSIS_SESSION_HOST: DEV-2008R2-4

ANALYSIS_SESSION_TIME: 02-16-2017 13:18:09.0570

ANALYSIS_VERSION: 10.0.10586.567 amd64fre

STACK_TEXT:
fffff88008587b18 fffff80001a65732 : 00000000000000f4 0000000000000003 fffffa800e93e3b0 fffffa800e93e690 : nt!KeBugCheckEx
fffff88008587b20 fffff80001a22f7b : 0000000000000001 fffffa800efb6b50 fffffa800e93e3b0 fffffa80108e6001 : nt!PspCatchCriticalBreak+0x92
fffff88008587b60 fffff8000198c234 : 0000000000000001 0000000000000ae8 fffffa800e93e3b0 fffffa8000000008 : nt! ?? ::NNGAKEGL::string'+0x273e6 fffff88008587bb0 fffff800016cf6d3 : 0000000000000ae8 fffffa800efb6b50 fffffa800e93e3b0 000000001c758bd0 : nt!NtTerminateProcess+0x284 fffff88008587c20 000000007743bdfa : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!KiSystemServiceCopyEnd+0x13 000000001bf5dc68 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 00000000`00000000 : 0x7743bdfa

STACK_COMMAND: kb

THREAD_SHA1_HASH_MOD_FUNC: e9460336222f4471d8ae88a3d24ad7df3aff8ef1

THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 00d8b79e89f094b2d29f42fa22cba954c8d4497d

THREAD_SHA1_HASH_MOD: f08ac56120cad14894587db086f77ce277bfae84

FOLLOWUP_NAME: MachineOwner

IMAGE_VERSION:

FAILURE_BUCKET_ID: X64_0xF4_csrss.exe_BUGCHECK_CRITICAL_PROCESS_TERMINATED_BY_Tentacle.exe_efb6b50

BUCKET_ID: X64_0xF4_csrss.exe_BUGCHECK_CRITICAL_PROCESS_TERMINATED_BY_Tentacle.exe_efb6b50

PRIMARY_PROBLEM_CLASS: X64_0xF4_csrss.exe_BUGCHECK_CRITICAL_PROCESS_TERMINATED_BY_Tentacle.exe_efb6b50

TARGET_TIME: 2017-02-15T21:22:07.000Z

OSBUILD: 7601

OSSERVICEPACK: 1000

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK: 272

PRODUCT_TYPE: 3

OSPLATFORM_TYPE: x64

OSNAME: Windows 7

OSEDITION: Windows 7 Server (Service Pack 1) TerminalServer SingleUserTS

OS_LOCALE:

USER_LCID: 0

OSBUILD_TIMESTAMP: 2016-10-01 03:57:30

BUILDDATESTAMP_STR: 160930-0600

BUILDLAB_STR: win7sp1_ldr

BUILDOSVER_STR: 6.1.7601.23564.amd64fre.win7sp1_ldr.160930-0600

ANALYSIS_SESSION_ELAPSED_TIME: 88b

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:x64_0xf4_csrss.exe_bugcheck_critical_process_terminated_by_tentacle.exe_efb6b50

FAILURE_ID_HASH: {66fbdb89-ff5f-4652-eb12-b95d291c72d9}

Followup: MachineOwner

Cheers,
Gavin

Hi Gavin,

Thanks for getting in touch. That’s a nasty crash, I checked with some of the team we haven’t seen something like happen especially something that results in a BSOD.

Keep an eye on it, and let us know if it happens again. If it’s not a lot of work; check to see if those VMs are storing the dump file from the crash, if it becomes a common occurrence a few of those dump outputs may help in narrowing down the issue.

Regards,
Nick

Hi Nick,

Thanks for the reply.

I will keep monitoring, but so far everything has been fine since the server crash.

I think the tentacle service may have been upgrading to the latest version when the crash have occurred (although we have around 11 tentacles running, and they have all upgraded without issues except for the single server that crashed).

Kind Regards,

Gavin


Thanks Gavin,
We’ll be here to help.

Regards,
Nick