Unable to delete environment - Runtime error

[mmargala]

Hello! After clicking delete on an environment we are seeing a “Object reference not set to an instance of an object” error message. I am happy to provide more info, just let me know what to collect.

This is happening on two environments I am trying to cleanup. I’ve already removed all references to the environment (Lifecycles, Process Steps, Team Permissions).

Thanks!

[dan_close]

Hi @mmargala,

Thanks for reaching out to Octopus support and I’m sorry to see you’re running into this issue attempting to delete environments. I just have a few questions to hopefully help us narrow this down.

Which version of Octopus are you currently running? Do you have any project releases or runbook snapshots that have been deployed to this environment? Those should be automatically removed along with the environment deletion, but just wanted to verify that it may be in play.

Can you run this SQL query against your database and send me the JSON column from the results? We just want to see if there is anything unexpected there.

SELECT * FROM dbo.DeploymentEnvironment
WHERE Name = 'XXXX' and SpaceId = 'Spaces-XX'

Can you also send me the API response from the browser when you edit the environment? With dev tools open in your browser you should see a call similar to the screenshot below.

Please let me know what you find or if you have any other questions.

Thanks!
Dan

[mmargala]

Hi @dan_close, thanks for the prompt response.

Thanks we are currently on v2021.3 (Build 12132), however, this error has been happening for some time. At least since 2021.3.8275. We did have project releases deployed here but like you said those normally get cleaned up when an environment gets deleted. For run books yes, we did have a recent runbook deployed here. I went a head and attempted to remove that run but still get the same error.

SQL json column:

{"Description":"Auto-provisioned environment by Gitlab and AutoMikeOps","UseGuidedFailure":false,"AllowDynamicInfrastructure":false,"ExtensionSettings":[]}

API Response is a HTTP 500 error:

[dan_close]

Hi @mmargala,

Thanks for getting back to me with this information. We suspected the API call would fail as well so I appreciate you confirming that for us. Your database query also returned what we would expect. With this type of issue that doesn’t have an obvious cause, would you be receptive to sharing a backup of your Octopus server database with us so that we can take a deeper look? We don’t need your master key and would scrub the database of sensitive data. If this is okay, you may use the following secure link to upload the database backup: Secure Upload

Thanks!
Dan

[mmargala]

Just to be clear of what type of backup is required. Is it just a backup of the SQL Server database? or of the whole Octopus Server, done through the Server Manager UI?

Thanks!
Mike

[dan_close]

@mmargala,

Apologies, I should have been more specific. We would just need a backup of the Octopus SQL Server database.

Thanks!
Dan

[mmargala]

@dan_close

I’ve uploaded the backup

[dan_close]

Thanks @mmargala!

I’ll work on this today and let you know once I have an update.

Thanks,
Dan

[dan_close]

Hi @mmargala,

Thanks for patience as we work through this issue. Luckily I was able to replicate this error using your database. Eventually I found that it was indeed being caused by one of the previous deployments that had been done to that environment. Once I deleted the associated deployments I was then able to delete the environment from the Octopus UI. I put together this script that should find any deployments associated with the environments in question and delete them from your instance.

# Octopus Info
$OctopusUrl = "https://youroctopusurl"
$APIKey = "API-12345"
$spaceId = "Spaces-X" 
$environmentId = "Environments-XXXX"
$header = @{ "X-Octopus-ApiKey" = $APIKey }

# Gather all deployments associated with the environment
$deploymentsUrl = "$OctopusUrl/api/$spaceId/deployments?environments=$($environmentId)&skip=0&take=1000"
Write-Host "Getting list of deployments: $deploymentsUrl"
$deploymentsResource = (Invoke-RestMethod $deploymentsUrl -Headers $header)
$deployments = $deploymentsResource.Items

# Delete all deployments associated with the environment
function delete($url){
    Invoke-WebRequest $url -Headers $header -Method Delete -Verbose
}

foreach ($deployment in $deployments){
    Write-Host "About to delete $($Deployment.id)"
    delete -url "$OctopusUrl$($deployment.links.self)"
}

You will just need to update the script with your Octopus URL, an API key with access to delete deployments, your Space ID, and the Environment ID for the environment you’d like to delete. As with any bulk deletion, we recommend you run this in a test instance if at all possible and be sure to have a recent database backup before running it in production. Hopefully this gets your issue resolved for you, but please let me know if you have any questions for me.

Thanks!
Dan