We’ve been noticing that the octopus windows services sometimes do not restart during a reboot. Most of the time happens with the tentacles, but last night it happened with the octopus server. When I log into the server, I can manually start the services without any issue.
I don’t see any messages in the Application or System event logs. I see that it uses log4net, is there a way to get some logging to help troubleshoot?
+1 For this. This happens occasionally on our site as well with the tentacle service
Hi,
The services use Log4net but the events are written to the event log. It might be that your machine has too many services running on startup and isn’t able to start them up quickly enough, or that a virus scanner or other software is slowing down the startup process. The services already request extended startup time from Windows and they do nothing in startup apart from creating a thread, so they should start quickly.
Can you try changing the startup type on the services from Automatic to Automatic (Delayed)?
Paul
I will try the Automatic Delayed idea. Thanks!
Seems sporadic. If I manually reboot the VM hosting the tentacle service it restarted. However, restart after a Windows update last night did not happen.
… tried both “Automatic” and “Automatic (Delayed)”. Oddly, it worked with “Automatic” when shutting down manually, but not with “Automatic (Delayed)”. Current setting is “Automatic”.
+1 We also have this problem. Octopus server and tentacles do not always start after reboot of server even though they are set to automatic.
Hi,
Thanks for reaching out. What version of the Octopus Server and Tentacle are you running? Do you see any logs on the event viewer right after the server reboot that may indicate a possible (failed) attempt to start the service?
Regards,
Dalmiro
Hi,
I had to wait until it happened again, and it did now.
Octopus server version: 3.2.5
Tentacle version : 3.2.5
Calimari: 3.1.16
2 errors in event log:
A timeout was reached (30000 milliseconds) while waiting for the OctopusDeploy service to connect. (Eventid: 7009, Source: Service Control Manager)
The OctopusDeploy service failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion. (Eventid: 7000, Source: Service Control Manager)
Hi,
Could you please send us the full text of each event log entry and your Octopus server logs from the time of the reboot? On a default install, the logs should be on C:\Octopus\Logs.
Please send them to [support @ Octopus .com]
Thanks,
Dalmiro
I get a : Please send them to [email blocked]
So I have attached the files to this reply.
OctopusServer.1.txt (28 KB)
EventLog.txt (2 KB)
HI,
Sorry about that. I’ve formated the email address on my previous reply so you can see it.
The entries on OctopusServer.1.txt are for 00:42 - 00:44, while the event log entries are from 01:36. Can you check if there’s an Octopus server log with entries around this last time?
Regards,
Dalmiro
We have this problem off and on too. With 3.4.7 and 3.3.11. We are set to Automatic startup (non-Delayed). We’re using a Hyper-V VM.
15/10/2016
23:29:39.000
10/15/2016 11:29:39 PM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4688
EventType=0
Type=Information
ComputerName=SERVER.DOMAIN.FULL
TaskCategory=Process Creation
OpCode=Info
RecordNumber=4540450
Keywords=Audit Success
Message=A new process has been created.
Subject:
Security ID: NT AUTHORITY\SYSTEM
Account Name: SERVER$
Account Domain: DOMAIN
Logon ID: 0x3E7
Process Information:
New Process ID: 0x7f4
New Process Name: D:\Octopus Deploy\Octopus\Octopus.Server.exe
Token Elevation Type: TokenElevationTypeDefault (1)
Creator Process ID: 0x248
Process Command Line:
Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.
Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account.
Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator. An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group.
Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator.
Collapse
Hi @rjhansen - Did you reboot your Hyper-v VM using SCOM/SCCM by any chance? Cause we have an open issue for the service not starting when the VM gets rebooted using those tools: https://github.com/OctopusDeploy/Issues/issues/2375
We do not use SCOM or SCCM.
I double checked the event logs and could not find an attempt to start the Octopus Server service, nor could I find a failure. When I logged into the box the service was set to Automatic but was Stopped.
Regarding your link, the issue I’m speaking about is with our Octopus Server, not with a Tentacle.
Also, I checked my ngen logs in
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.log and
C:\Windows\Microsoft.NET\Framework\v4.0.30319\ngen.log
There are entries logged from the day in question but no log entries in close proximity to the time of the reboot.
Hi @Rjhansen,
Could you email us your Octopus server logs to Support(at)Octopus(dot)com? On a default install they should be on C:\Octopus\Logs.
Please try to send the logs that were generated around the time of the reboot.
Thanks!
Dalmiro