503 Error When Deleting an Environment

Octopus Server
1

We have an old Environment that can now be deleted but I am encountering a 503 error in doing so.
I have confirmed there are no deploy targets, lifecycles or variable scopes referencing this environment. Typically there is a message displayed that clearly indicates what needed clean up is blocking the deletion, but in this case it’s just an ambiguous 503 error.

Error message screenshot:
Octopus_503_error_deleting-env

During a given attempt I can see tons of noise in the server logs related to deletion of deployments associated to the environment (see snip-it below). Looking in the database there are approx. 9600 dbo.DeploymentHistory records still associated to this environment. Additionally the following query indicates that some deployment processes are still referencing the environment, however combing through each of the deployment steps, channels, triggers etc, through the UI; there is no mention of it at all. . .

SELECT TOP (1000) dp.[Id]
,[OwnerId]
,[IsFrozen]
,[Version]
,dp.[JSON]
,[RelatedDocumentIds]
–,[SpaceId]
,p.Name
FROM [dbo].[DeploymentProcess] dp
left join Project p on dp.OwnerId = p.Id
WHERE RelatedDocumentIds like ‘%Environments-1242%’

I noticed there is a stored proc called [dbo].[UpdateDeploymentHistory]. Would running this help the situation. Else what is the best way to get the environment deleted??

Snip-it from server log:

2023-01-24 12:41:31.3989 9964 384 INFO “HTTP” “GET” to “cicd.employeenavigator.net”“/api/tasks” completed with 200 in 00:00:00.1581232 (158ms) by “Ryan Philippi”
2023-01-24 12:41:31.3989 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.3989 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4145 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: All deployments to a given environment are deleted when the environment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.
2023-01-24 12:41:31.4305 9964 384 INFO Rule: The deployment task is deleted when the deployment is deleted.

3

Hi @rphilippi1,

Thanks for reaching out, and welcome to the Octopus Deploy community!

I’m sorry you are having trouble deleting this particular environment within your Octopus Deploy instance, but I’m happy to help take a closer look.

As a first step in investigating this further, would you be able to run the following SQL query against your Octopus Deploy database and send me the results? This should return any references to that particular environment across the entire database:

USE [OctopusDeploy-OctopusServer]
DECLARE @SearchStr nvarchar(100) = 'Environments-1242'
DECLARE @Results TABLE (ColumnName nvarchar(370), ColumnValue nvarchar(3630))

SET NOCOUNT ON

DECLARE @TableName nvarchar(256), @ColumnName nvarchar(128), @SearchStr2 nvarchar(110)
SET  @TableName = ''
SET @SearchStr2 = QUOTENAME('%' + @SearchStr + '%','''')

WHILE @TableName IS NOT NULL

BEGIN
    SET @ColumnName = ''
    SET @TableName = 
    (
        SELECT MIN(QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME))
        FROM     INFORMATION_SCHEMA.TABLES
        WHERE         TABLE_TYPE = 'BASE TABLE'
            AND    QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME) > @TableName
            AND    OBJECTPROPERTY(
                    OBJECT_ID(
                        QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME)
                         ), 'IsMSShipped'
                           ) = 0
    )

    WHILE (@TableName IS NOT NULL) AND (@ColumnName IS NOT NULL)

    BEGIN
        SET @ColumnName =
        (
            SELECT MIN(QUOTENAME(COLUMN_NAME))
            FROM     INFORMATION_SCHEMA.COLUMNS
            WHERE         TABLE_SCHEMA    = PARSENAME(@TableName, 2)
                AND    TABLE_NAME    = PARSENAME(@TableName, 1)
                AND    DATA_TYPE IN ('char', 'varchar', 'nchar', 'nvarchar', 'int', 'decimal')
                AND    QUOTENAME(COLUMN_NAME) > @ColumnName
        )

        IF @ColumnName IS NOT NULL

        BEGIN
            INSERT INTO @Results
            EXEC
            (
                'SELECT ''' + @TableName + '.' + @ColumnName + ''', LEFT(' + @ColumnName + ', 3630) 
                FROM ' + @TableName + ' (NOLOCK) ' +
                ' WHERE ' + @ColumnName + ' LIKE ' + @SearchStr2
            )
        END
    END    
END

SELECT ColumnName, ColumnValue FROM @Results

Could you also upload the diagnostics package for your Octopus Deploy instance? You should be able to grab this by navigating to Configuration > Diagnostics within the server UI:

image
image1004×302 34.5 KB

Here is a secure link for this information, so you don’t have to post this publically on the forum.

Looking forward to hearing back from you,

Britton

5

Thanks Britton. Can you please try sending me the secure link again. I am not seeing it in your previous reply. Cheers!

6

Hi @rphilippi1,

It looks like our support files application is currently having some issues, so instead, I’ll DM you a direct link where you can upload these things in Google Drive for now.

Look out for that message shortly, and once I have the files, I can review things on my side to determine the best next step for this.

Regards,

Britton

8

Hi @rphilippi1,

Our support files tool is back up and running again, and if you could, I would upload your files here instead, as others on my team will have access to this area if I’m not online.

Here is the secure upload link.

Sorry for the initial trouble there,

Britton

9

Files uploaded. Thanks :+1:

1 Like
10

Hi @rphilippi1,

Thank you for uploading those files for my review.

In taking a look, I wonder if you can address this issue by removing all the deployments associated with that particular environment. To get an idea if this would help, you can run the following script (after supplying the required variables), which will return all of the deployments associated with Environments-1242:

$ErrorActionPreference = "Stop";

# Define working variables
$octopusUrl = "https://your.octopus.URL"
$octopusAPIKey = "API-XXX"
$header = @{ "X-Octopus-ApiKey" = $octopusAPIKey }
$spaceId = "Spaces-XX"
$environmentIdToClear = "Environments-1242"

Write-Host "Getting environment"
$environment = Invoke-RestMethod -Method Get -Uri "$octopusUrl/api/$spaceId/environments/$environmentIdToClear" -Headers $header

$deployments = Invoke-RestMethod -Method Get -Uri "$octopusUrl/api/$spaceId/deployments?environments=$($environment.Id)" -Headers $header

Write-Host "Found $($deployments.Items.Count) deployments related to $($environmentIdToClear)"

foreach ($deployment in $deployments.Items)
{   Try
    {
        Invoke-RestMethod -Method GET -Uri "$octopusUrl/api/$spaceId/deployments/$($deployment.Id)" -Headers $header
    }
    Catch
    {
        Write-Host "Issue getting $($deployment.Id)"
    }    
}

The results you get back should line up with the deployment IDs from the previous query you ran, which I’ve compiled into a separate file that I will DM to you.

If everything lines up, I would then switch the GET to a DELETE, like so (I also changed the Catch language as well):

$ErrorActionPreference = "Stop";

# Define working variables
$octopusUrl = "https://your.octopus.URL"
$octopusAPIKey = "API-XXX"
$header = @{ "X-Octopus-ApiKey" = $octopusAPIKey }
$spaceId = "Spaces-XX"
$environmentIdToClear = "Environments-1242"

Write-Host "Getting environment"
$environment = Invoke-RestMethod -Method Get -Uri "$octopusUrl/api/$spaceId/environments/$environmentIdToClear" -Headers $header

$deployments = Invoke-RestMethod -Method Get -Uri "$octopusUrl/api/$spaceId/deployments?environments=$($environment.Id)" -Headers $header

Write-Host "Found $($deployments.Items.Count) deployments related to $($environmentIdToClear)"

foreach ($deployment in $deployments.Items)
{   Try
    {
        Invoke-RestMethod -Method Delete -Uri "$octopusUrl/api/$spaceId/deployments/$($deployment.Id)" -Headers $header
    }
    Catch
    {
        Write-Host "Unable to delete $($deployment.Id)"
    }    
}

After running this, you should hopefully be able to delete Environments-1242, but let me know how it goes.

If this doesn’t work, there may be some other ways around this, so just let me know and I’ll take a deeper look.

Best,

Britton

12

Hello @britton.riggs -

Thank you, this worked great.
Note: I did need to add the ’ &take=2500 ’ onto this line in order to get all the deployment records without having to process batches of 30:

$deployments = Invoke-RestMethod -Method Get -Uri “$octopusUrl/api/$spaceId/deployments?environments=$($environment.Id)” -Headers $header

Thanks for the assist and I’ll keep these scripts around for future use.

Cheers!

13

Please also note that now, running the initial SQL query you sent - that exposes all references to the ’ Environments-1242 ’ string, in any column, in any table - I am getting more results than before when I sent you my original output. . .

Before:
2023-02-10 09_48_08-Window

Now:
2023-02-10 09_53_25-Window

14

Hi @rphilippi1,

Thank you for the update, and I’m glad the API script got things unblocked for you!

I’m not sure what’s going on with those extra results you are seeing via SQL now, but if you want to upload the results to the following secure link I’d be happy to take a closer look.

Regards,

Britton

1 Like
15

File sent. Thank you for the support

16

Hi @rphilippi1,

Thanks for keeping in touch and sending the list of references to this environment! I’m jumping in for Britton as he’s currently offline as part of our US-based team. :slight_smile:

The unique column names are as follows:

[dbo].[EventRelatedDocument].[RelatedDocumentId]
[dbo].[DeploymentHistory].[EnvironmentId]
[dbo].[Event].[EnvironmentId]
[dbo].[Event].[RelatedDocumentIds]
[dbo].[RunbookRunHistory].[EnvironmentId]
[dbo].[VariableSet].[JSON]
[dbo].[VariableSet].[RelatedDocumentIds]
[dbo].[DeploymentProcess].[JSON]
[dbo].[DeploymentProcess].[RelatedDocumentIds]

My understanding is the related document, *history, and event entries referencing this environment should be fine and expected (retained for audit history purposes), though the variable set and deployment process entries are worth having a closer look at and try to reproduce locally. My apologies if it’s already been supplied, but could you let me know the version of Octopus you’re currently running? I don’t see any mention of this information in this thread.

One specific variable set that references this environment is mentioned multiple times in the spreadsheet. Unfortunately I can’t determine the variable set name or ID from the data provided, but there are variables within named DatabaseIp, DatabaseIpConnStrings, DatabaseIpSessionState, DocDatabaseIp`, etc. Are you familiar with the variable set containing those variables? I’m wondering if you’re able to browse to that variable set in the UI, and whether or not you see any error messages on its page. My suspicion is there may be some cascade-deletion issue not removing relevant scopes during the environment deletion.

I look forward to getting to the bottom of this one!

Best regards,

Kenny

1 Like
17

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.