Error starting service after restore from backup

Hi,

I’m using Octopus Deploy 2.0.5.933. After attempting a restore of a recent backup using the wizard I’m getting the following error and can no longer start the service:

Raven.Abstractions.Exceptions.OperationVetoedException: PUT vetoed by Raven.Bundles.UniqueConstraints.UniqueConstraintsPutTrigger because: Ensure unique constraint violated for fields: Name
at Raven.Database.DocumentDatabase.AssertPutOperationNotVetoed(String key, RavenJObject metadata, RavenJObject document, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 938
at Raven.Database.DocumentDatabase.<>c__DisplayClass4b.b__43(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 780
at Raven.Storage.Esent.TransactionalStorage.Batch(Action1 action) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 608 at Raven.Database.DocumentDatabase.Put(String key, Etag etag, RavenJObject document, RavenJObject metadata, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 843 at Raven.Database.Extensions.CommandExtensions.Execute(ICommandData self, DocumentDatabase database, BatchResult batchResult) in c:\Builds\RavenDB-Stable\Raven.Database\Extensions\CommandExtensions.cs:line 54 at Raven.Database.DocumentDatabase.ProcessBatch(IList1 commands) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 2086
at Raven.Database.DocumentDatabase.<>c__DisplayClass10c.b__108(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 2048
at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action1 action, EsentTransactionContext transactionContext) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 666 at Raven.Storage.Esent.TransactionalStorage.Batch(Action1 action) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 617
at Raven.Database.DocumentDatabase.Batch(IList1 commands) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 2051 at Raven.Client.Embedded.EmbeddedDatabaseCommands.Batch(IEnumerable1 commandDatas) in c:\Builds\RavenDB-Stable\Raven.Client.Embedded\EmbeddedDatabaseCommands.cs:line 709
at Raven.Client.Document.DocumentSession.SaveChanges() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\DocumentSession.cs:line 664
at Octopus.Server.Storage.Defaults.DefaultProjectGroupInstaller.Initialize(IDocumentStore store) in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\Storage\Defaults\DefaultProjectGroupInstaller.cs:line 25
at Octopus.Server.Storage.StoreInitializer.Initialize() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\Storage\StoreInitializer.cs:line 24
at Octopus.Server.OctopusServerEngine.Start() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\OctopusServerEngine.cs:line 50
at Octopus.Server.Commands.RunCommand.Start() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\Commands\RunCommand.cs:line 21
at Octopus.Shared.Startup.WindowsServiceHost.<>c__DisplayClass2.b__0() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Shared\Startup\WindowsServiceHost.cs:line 19
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Shared\Startup\WindowsServiceAdapter.cs:line 44

Hi Mike,

It sounds like the RavenDB indexes haven’t restored correctly; can you please try
working through the instructions here:

It would be good to know why a restore was required - e.g. were there data issues you’re trying to recover from?

If this is mission-critical/a production server please let me know; I’ll do my best to get you back up and running quickly.

Hi,

Thanks for the reply. It is a critical server, so I ended up abandoning the restore and rebuilding from a 1.6 backup that I had recently upgraded.

If I try to restore any one of the 2.0 backups then I always get the same error. I even tried a full uninstall, deleted all the Raven\Octopus directories prior to restore, but nothing worked.

The reason for the restore is because I ran a script against the Octopus API which deletes all releases - but this happened whilst some were still in process of being deployed. It resulted in 1 orphaned task than was continuously trying to deploy a release which no longer existed and I couldn’t delete/stop it.

To resove this I tried deleting some items in RavenDb directly, but ended up in a state whereby I could no longer start Octopus Server due to broken references. This lead me to attempting to restore a backup verison prior to my tinkering.

I would like to be able to use the backup/restore feature, but now worried I’m going to end up with the same issue and loose my work. I will try restoring to a different location when I get a chance.

Thanks,
Mike

Sorry to hear about the hassle; we’re also very keen for backup/restore to work (an understatement), I’ll do my best to pinpoint this and make sure it is fixed.

I believe the 2.0.6.xxx builds include a new command, Octopus.Server.exe repair-storage, that automates index rebuilding for situations like this. If you have the opportunity, running this after restoring from backup should get those indexes back in shape.

I’ve checked indexes after restoring in our backup/restore test suite and everything looks fine. One other thing you might try - if your database is large, there might be some lag between restoring from backup and the indexes catching up.

I’ve added code to mitigate against this (proactively creating indexes during restore); I’m not sure if there’s any workaround without it though. I’ll post here when the code makes it out in a new version.

Just a quick note, the index-recreation code is out in 2.0.7 - if my understanding of what’s going on here is correct this should allow your backups to be restored successfully; I’ve left the associated GitHub issue open until we can verify it.

Great thanks, I’ll test this out when I can.

Hi Mike - I just managed to replicate this locally and needed to make one more change to fix it. This will be in 2.0.8, which I’ll assemble in a day or two.

Regards,
Nick