Timeout Error During Deployments and Other Operations

Good Afternoon,

I currently run into the following error when deploying projects and when trying to add a new tentacle for a tenant. It doesn’t happen all the time during deploys, but it does happen every time when trying to install a new tentacle on a tenant’s site.

The Octopus version is v2021.2

Thank You.


Octopus Server returned an error: Error while executing SQL command in transaction ‘MachineDomainEventListener’: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The command being executed was:
DELETE FROM [dbo].[EventRelatedDocument]
WHERE ([EventId] = @eventid)
SQL Error -2 - Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Unknown error 258
Octopus.Client.Exceptions.OctopusServerException
at Octopus.Client.OctopusAsyncClient.DispatchRequest
at Octopus.Client.OctopusAsyncClient.Create
at Octopus.Client.Operations.RegisterMachineOperation.ExecuteAsync
at Octopus.Tentacle.Commands.RegisterMachineCommandBase1.RegisterMachine at Octopus.Tentacle.Commands.RegisterMachineCommandBase1.StartAsync
at Octopus.Tentacle.Commands.RegisterMachineCommandBase1.Start() at Octopus.Shared.Startup.AbstractCommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) at Octopus.Shared.Startup.ConsoleHost.Run(Action1 start, Action shutdown)
at Octopus.Shared.Startup.OctopusProgram.Run()


Hi @blakee,

Thank you for contacting Octopus Support.

Our engineers are aware of and currently investigating this issue for your Octopus Cloud instance. This has been prioritized and we are actively working on a resolution.

We appreciate your patience and we’ll update you as soon as we have additional information from the investigating team.

Best Regards,
Donny

Hi @blakee,

Apologies for the additional post. I have one quick question.

May we have your permission to access your instance’s SQL db as necessary for this investigation?

Please let me know at your earliest convenience.

Best Regards,
Donny

Yes, you have our permission to access our instance’s SQL DB as needed.

Thanks!

Hi @blakee,

Thank you for the quick reply and for granting us permission.

We’ll keep you updated as the investigation progresses.

Best Regards,
Donny

Good Afternoon,

Is there an update on this issue? Our deployment system is kind of stuck until we can fix the issues.

Thank You

Hi @blakee,

Thank you for your patience. I have followed up with the investigating team. As soon as the Support Team hears something back, we’ll let you know ASAP. The team is based in Australia and should be signing on shortly.

I wish I had more information at this moment. I’m hopeful we’ll have something for you very soon.

Best Regards,
Donny

Hi @blakee,

Thank you for your patience.

It looks like your instance was upgraded to a newer version of Octopus last night that may have resolved the issue.

Can you confirm if the instance is behaving normally now?

Let me know at your earliest convenience.

Best Regards,
Donny

Good Afternoon,

I am still getting the same error when trying to install a tentacle on a client’s server. And our octopus instance is still the same version as before: v2021.2 (Build 7650)

===============================================================================
Registering the tentacle with the server at https://auctionedgeaos.octopus.app/

Octopus Server returned an error: Error while executing SQL command in transaction ‘MachineDomainEventListener’: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The command being executed was:
DELETE FROM [dbo].[EventRelatedDocument]
WHERE ([EventId] = @eventid)
SQL Error -2 - Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Unknown error 258
Octopus.Client.Exceptions.OctopusServerException
at Octopus.Client.OctopusAsyncClient.DispatchRequest
at Octopus.Client.OctopusAsyncClient.Create
at Octopus.Client.Operations.RegisterMachineOperation.ExecuteAsync
at Octopus.Tentacle.Commands.RegisterMachineCommandBase1.RegisterMachine at Octopus.Tentacle.Commands.RegisterMachineCommandBase1.StartAsync
at Octopus.Tentacle.Commands.RegisterMachineCommandBase1.Start() at Octopus.Shared.Startup.AbstractCommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) at Octopus.Shared.Startup.ConsoleHost.Run(Action1 start, Action shutdown)
at Octopus.Shared.Startup.OctopusProgram.Run()

Hi @blakee,

Thank you for clarifying.

May I have your permission to temporarily bring your instance (auctionedgeaos.octopus.app) offline? There are slow-running queries against the db that are preventing me from pushing a fix on the back end.

The process will likely take about 30 minutes, but could possibly take longer.

Let me know if that is acceptable to do now.

Best Regards,
Donny

Yes, you can bring our instance offline for maintenance.

Hi @blakee,

Thank you for your patience.

I was unable to get my fix pushed to your instance. I’m currently waiting for our AU Cloud Engineers to come online to hopefully run this using a different tool.

I am actively working on and monitoring this issue to make sure this gets handled ASAP. I sincerely apologize for the recent issues you have experienced. I’m hopeful we can have this resolved this evening.

I’ll continue to update you once AU comes online.

Best Regards,
Donny

Thank you for keeping my updated! Just let me know if you need anything more from me.

Hi @blakee,

I just wanted to give you a brief update.

Our AU Cloud Engineers are working on pushing a solution to get this resolved for you over the next few hours. Our AU Support Team is going to keep an eye on things until I sign on with East Coast US Support in the morning.

I’ll check on everything once I sign on first thing tomorrow.

Thank you again for your patience during this ongoing issue.

Best Regards,
Donny

Thanks for the quick action! I probably won’t be on for much longer tonight since I’m on CST, so I will check back in the morning.

Thanks again!

Hi @blakee,

Thank you again for your patience.

I’m happy to report that the fix has been applied and your instance should behave normally now. Please let me know if that is not the case.

Otherwise, if there is anything else we can assist with, please let us know.

Best Regards,
Donny