All Octopus Server nodes in this cluster are draining tasks

#1

Hi
Starting early morning we have been facing issues with the octopus login and then with deploy failing followed with the below message
Server status

All Octopus Server nodes in this cluster are draining tasks.

I get the below error message when trying to access the configuration tab
“Exception occurred while executing a reader for SELECT MAX([DataVersion]) AS [Latest], MIN_ACTIVE_ROWVERSION() as [MinActive], COUNT_BIG(*) AS [Count] FROM dbo.[DeploymentEnvironment] WHERE SpaceId = @spaceId SELECT [DataVersion] FROM dbo.[DeploymentEnvironment] WHERE SpaceId = @spaceId AND [DataVersion] >= MIN_ACTIVE_ROWVERSION() ORDER BY [DataVersion] desc
SQL Error -2 - Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The wait operation timed out”

We are using Azure SQL Database and though it had a downtime early in the morning, the status of the database is now available but we still see issues with octopus.

Thanks
Sharjil

(Ziaul Islam) #3

Hi Sharjil,

Sorry to see you are having issues with your Octopus instance.

To get you back up and running I would look at this link below:

This should get you out of the state “All Octopus Server nodes in this cluster are draining tasks.”

In addition to this, you can do it from the command line using this as a guide:

Once you do that, it would fix this issue for you.

I look forward to your response once you have completed that.

Kind regards,
Ziaul

#4

Hey Ziaul,

Thanks for quick response !
And about the issue, it seemed to be because of the Azure outage of database in the region and now octopus deploy is back up again after the issue at Azure is resolved.
But however, i did go through the knowledge base article you shared before creating this issue and we have just the one leader node here which was drained out so couldn’t do anything that the article suggested.
Will be a good to know in case it happens sans Azure issue :slight_smile:

BR
Sharjil

(Ziaul Islam) #5

Hi @Sharjil

I’m glad the issue has resolved itself.

I will make a note of what you have said to me I will pass on this information to the engineers on our side to see what they think would be the best next steps to take.

Kind regards,
Ziaul