Delay in Octopus service to get started

Legacy Execution
1

Hello,

We have been using Octopus(3.15.7) in our build server.It works smoothly except the times when for some reason when we reboot the build server(like when I applying any patch), the octopus service takes around half an hour to start. It shows as Starting in services list(services.msc) , but when we see the logs it shows that

Login failed for user “OctopusUser”.

Then after repeated attempts by this service as seen in the logs, Octopus database is able to connect for that user, all that takes atleast half an hour, till then our deployments get impacted.

Regards,

Nishant

2

Hi Nishant,

In which logs is that appearing? Is it related to the service logon or to the SQL connection?

What account is the octopus service running under?

Are there any other services in the starting state? For example the MSSSQLServer service or something related with Active Directory authentication? Is Octopus using any network drives that may not be immediately available?

Could you send me the server logs for the time period covered by this start.

Do you have the Watchdog setup?

If you are seeing this in the Event Log, could you send me the full details? Does the Eventlog indicate that the service is continually stopping and starting, or is it just the single service start?

Robert W

3

Hi Robert,
This error is from the Octopus logs configured to be at following location D:\Octopus\Home\Logs\Octopusserver.log

This happened on July 15th , unfortunately the old logs get cleared automatically, I can’t find even in the files Octopusserver.[n].txt

Let me explain the sequence

  1.   We reboot the build server for some reason where Octopus was hosted

  2.   After that Octopus url was not working, we checked the services list using command services.msc and found that Octopus Deploy service was in stopped state

  3.   I restarted that service and the status starting showing Starting

  4.   I browsed to logs located at D:\Octopus\Home\Logs\Octopusserver.txt and found that there was repetitive error

Login failed for user “OctopusUser”.

a. During this time, we tried login to sql server manually through same credentials via SQL server management studio and we were able to login.

b. Also we checked the SQL Server (MSSQLSERVER) and SQL Server Agent (MSSQLSERVER) services those were in Started state.

  1.   We waited for this to Start, and it took half an hour for that but eventually came up

Other points

  1.   Octopus Deploy service is set up as Start automatically and runs on service account.

  2.   Octopus Deploy service uses SQL server as database and uses sql authentication to connect.

  3.   Octopus is kkw
4

Hi Nishant,

I think the end of your message got cut off.

We haven’t seen this kind of error before, so without the logs, I’m stumped as to where to start looking. The next time it happens, could you capture those logs and send them to us? Also does it only happen on server restart, or when the service restarts?

Could you check the SQL server logs to see if it logged failed login attempts?

Finally, if you can capture a process dump at the time of the problem occurring, I can analyse that to see exactly where in the code it is.

Rob

5

Hi Robert,

The cut-off message was

  1.   Octopus is not dependent upon any network drives

  2.   I don’t think we have watchdog setup

Now(recently added by me)

Actually I recalled what happened( start vs restart service)

  1.   Server rebooted, service was in stopped state.

  2.   I started service, it failed to start and throws error in less than a minute.

  3.   I restarted service, this time it takes time to start and till then shows starting in list of services. This is when multiple attempts to connect to SQL server has been logged.

Dump file is too large (in gigs )

Regards
Nishant

6

Hi Nishant,

I’ve created a secure upload site for a dump file https://file.ac/DWJCBJTkzXo/, it should also zip quite a bit.

The dump file is only useful if it is captured while the process is experiencing this issue.

Robert W

7

Hi Robert- It will take 2 hrs to upload, so I cancelled it
Is there any way to open this file in any editor and send you the information for the day when the problem occurred

Regards
Nishant

8

The dump file? It would only contain the data at the time the problem was
occurring. Have you encountered the problem again and taken a dump? A dump
when the problem is not occurring would not help.

Rob

9

I haven’t encountered the problem again,how can we proceed further

10

Hi,

The next time it occurs, could you capture the process dump along with the server and event logs. Also check the SQL Server logs for why the login failed (password, permissions, etc).

Robert W