We were upgrading from version 2021.3.8275 to version 2022.1.2495 and received the following error:
SQL Error 2627 - Violation of PRIMARY KEY constraint ‘PK_ExtensionConfiguration_Id’. Cannot insert duplicate key in object ‘dbo.ExtensionConfiguration’. The duplicate key value is (issuetracker-azuredevops-v2).
The statement has been terminated.
You have two options from this point:
OPTION 1: Re-run the installer for the version of Octopus Server you were using just before trying this upgrade. Your data hasn’t been changed and everything will go back to how it was beforehand. Please send this log to support@octopus.com and we will work with you to get you upgraded.
OPTION 2: If you know what the problem is and how to fix it, please fix the problem and then try the upgrade again by running C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe database --upgrade
Octopus.Shared.ControlledFailureException: We encountered an error during the schema upgrade. The schema upgrade was stopped and rolled back. Don’t worry, this means we haven’t made any changes to your data, and you should be able to get back up and running quickly.
Looking in the database, the ExtensionConfiguration table has this row:
Since the message said we can go back to the previous version of the software, we have done that. We needed to restore the database first, since it has been altered, even though it said it wasn’t.
What should we do to correct this issue? do we just need to manually delete that row?
Thanks for reaching out, sorry to hear you’ve hit this Primary Key error when trying to upgrade!
I’ll check with the developers about whether deleting that entry will get you unblocked but we typically don’t like to edit the database unless absolutely required.
Would you be willing to send through the OctopusServer logs for this error? It’s possible that we might require a copy of the database too but the logs should be plenty in the meantime.
You can upload any files securely here, please feel let me know if there are any issues with it or you have any questions at all!
The log file has been uploaded. I cannot give you the original logfile, as that machine no longer exists. However, we write the log files to CloudWatch, and that is what I uploaded
A quick look into the log file does seem to indicate the only issue is the duplicate issuetracker-azuredevops-v2 entry trying to be added. I’ve specifically requested from the engineers whether deleting that DB entry is safe and if it’s likely the root cause of the issue.
I’ll reach out as soon as I have any updates or questions!
Just hopping on this ticket for Finnian as he is currently off shift as part of our Australian based team.
Sorry for taking a while to get back to you, our conversations surrounding this issue have been split between two of our engineering departments. The reason for this is one of our engineers thinks potentially what is happening here is the upgrade script is failing and not rolling back and then when you try to run the upgrade script again this is why you may be getting this duplicated PK error (which might not be the primary cause of the issue).
First, we want to get an answer on the Azure Issue Tracker DB entry and whether you are safe to remove it.
Alongside that, we want to investigate why the rollback failed as this should not have happened and we want to make sure it was an isolated incident and will not affect anyone else.
Unfortunately, there has been no update to both questions since Friday but some of our engineers were busy with other projects.
I have asked for an update on both and will get back to you when we have any updates from either team.
If there is anything else we can help you with in the meantime please reach out,
Sorry for the delay in response with this, I have a few questions from the devs which should hopefully get us closer to resolving this!
They would like to know if the contents of the dbo.ExtensionConfiguration table prior to upgrading matches the row you shared previously. If you are willing to share a backup of your database just prior to upgrading, that should provide us with all the info we need and hopefully be able to reproduce it on our end. Could please also send through a System Diagnostic Report of the instance?
They’d also just like to confirm that this was the first attempt at upgrading and there weren’t any other error messages received prior?
You should be able to upload files securely here, let me know if there are any issues with that link or you have any questions!
I am attempting to download the backup from the S3 bucket. The backup is ~ 16GB. If I am unable to download or then upload. I can restore it to another database and select the info you are looking for. Is the only table you want the contents off dbo.ExtensionConfiguration?
Just jumping in here for Finnian whilst he is off shift as part of our Australian based team.
So the engineers like a DB backup when we have issues that cannot be resolved through questions over a support ticket, they take the backup and cleanse it for any sensitive values (we don’t have the master key either in order to view those) and they import it into a test instance of the same version a customer is running.
From there they have the customer’s exact replication of the issue they are facing and they can troubleshoot it without disturbing the customer, they can also manipulate the data without affecting a live database.
The engineers usually only ask for a DB backup as a last resort because we know it can be difficult for some customers to get one out to us. It seems in this situation the engineers have asked for it so they can try and accurately reproduce this issue. Duplicate keys in tables are sometimes difficult to replicate on DBs.
In an ideal scenario, a DB backup would be the best option here for us to get a resolution out to you. But if you cant get that out to us then the engineers can probably work with the contents of the dbo.ExtensionConfiguration table.
It just won’t be an exact replication so they may not be able to reproduce the issue and it will take them longer to advise a fix. I hope that makes sense.
If you can get that DB uploaded that would be brilliant, if not then the contents of the dbo.ExtensionConfiguration table along with the system diagnostics report Finnian asked for would be something our engineers could use to help them diagnose this issue.
Let us know either way, we don’t get notified when someone uploads to our secure site so if you can reply to this ticket with when you uploaded the files (be it the DB backup or the diagnostics report and table contents) we can take a look for you and go from there.
I have repeatedly tired to upload the database backup. It has failed every time due to the size of the file. I restored the backup locally and did an upgrade and it succeeded. I have restored a copy of the database locally and re-ran the upgrade. I didn’t run into this same issue. We are going to retry the actual upgrade later this week. Will let you know how it goes.