Project suddenly went missing

Legacy Uncategorised
1

Help - all of a sudden one of our Projects has disappeared?!

It was successfully deployed a couple of days ago and there’s no entry in the Audit about it being deleted.

The deployment refers to an item “projects-5” - which no longer exists within the RavenDB.

Is the “projects-5” item fully gone, or would rebuilding the indexes help here?
I haven’t had great experience running the Repair RavenDB job … it has only successfully completed 50% of the time.
It mostly gets stuck on…

Checking whether any active tasks have timed out…

Error: Waited for 1,200,106ms for the query to return non stale result.

I usually have to restore the whole system from a backup…not ideal.

Would you recommend running the Repair for this incident? Or should I restore to a point in time where we know the Project was still existing…?

Many thanks!
Bronwyn,

2

Hi Bronwyn,

Thanks for getting in touch. Really sorry to hear you are having this issue. Let’s first try to repair the RavenDB from the Octopus manager and see if that helps.

Did something happen on the Octopus server since the last successful deployment to that project? Like moving the home folder, moving to another server or upgrading versions?

Let me know how it goes

Dalmiro

3

Could you also tell us which version of Octopus are you running?

Dalmiro

4

Hi Dalmiro,

Many thanks for the response.

We are running Octopus v2.6.4.951 - we upgraded to this a couple of months ago.

As I mentioned, I have had bad experiences with the Repair RavenDB process in the past so I was not confident running it would help in this instance.

Previous events:
We have not made any changes like moving the home folder or changing servers or anything, but we did have trouble with some deployments a few days ago where the following error (pasted below) started appearing. It suggests issues with the RavenDB but the Repair job consistently failed and I ended up having to restore from the previous night’s backup. We had no further issue until we noticed this Project had gone missing.

Padding is invalid and cannot be removed.
System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
at System.Security.Cryptography.CapiSymmetricAlgorithm.DepadBlock(Byte[] block, Int32 offset, Int32 count)
at System.Security.Cryptography.CapiSymmetricAlgorithm.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at Raven.Abstractions.Extensions.StreamExtensions.DisposingStream.Dispose(Boolean disposing) in c:\Builds\RavenDB-Stable\Raven.Abstractions\Extensions\StreamExtensions.cs:line 195
at System.IO.Stream.Close()
at Raven.Storage.Esent.StorageActions.DocumentStorageActions.ReadDocumentData(String key, Etag existingEtag, RavenJObject metadata) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\StorageActions\Documents.cs:line 107
at Raven.Storage.Esent.StorageActions.DocumentStorageActions.b__1(JsonDocumentMetadata metadata, Func4 createDocument) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\StorageActions\Documents.cs:line 39 at Raven.Storage.Esent.StorageActions.DocumentStorageActions.DocumentByKeyInternal[T](String key, TransactionInformation transactionInformation, Func3 createResult) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\StorageActions\Documents.cs:line 70
at Raven.Storage.Esent.StorageActions.DocumentStorageActions.DocumentByKey(String key, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\StorageActions\Documents.cs:line 36
at Raven.Database.Impl.DocumentRetriever.GetDocumentWithCaching(String key) in c:\Builds\RavenDB-Stable\Raven.Database\Impl\DocumentRetriever.cs:line 210
at Raven.Database.Impl.DocumentRetriever.RetrieveDocumentInternal(IndexQueryResult queryResult, HashSet1 loadedIds, FieldsToFetch fieldsToFetch, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable\Raven.Database\Impl\DocumentRetriever.cs:line 117 at Raven.Database.Impl.DocumentRetriever.ShouldIncludeResultInQuery(IndexQueryResult arg, IndexDefinition indexDefinition, FieldsToFetch fieldsToFetch) in c:\Builds\RavenDB-Stable\Raven.Database\Impl\DocumentRetriever.cs:line 243 at Raven.Database.Indexing.Index.IndexQueryOperation.ShouldIncludeInResults(IndexQueryResult indexQueryResult) in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\Index.cs:line 1108 at Raven.Database.Indexing.Index.IndexQueryOperation.<Query>d__52.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\Index.cs:line 939 at Raven.Database.Util.ActiveEnumerable1.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Database\Util\ActiveEnumerable.cs:line 46
at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext()
at Raven.Database.DocumentDatabase.<>c__DisplayClass91.b__88(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 1354
at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action1 action, EsentTransactionContext transactionContext) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 666 at Raven.Storage.Esent.TransactionalStorage.Batch(Action1 action) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 617
at Raven.Database.DocumentDatabase.Query(String index, IndexQuery query, Action1 headerInfo, Action1 onResult) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 1368
at Raven.Client.Embedded.EmbeddedDatabaseCommands.Query(String index, IndexQuery query, String[] includes, Boolean metadataOnly, Boolean indexEntriesOnly) in c:\Builds\RavenDB-Stable\Raven.Client.Embedded\EmbeddedDatabaseCommands.cs:line 486
at Raven.Client.Document.AbstractDocumentQuery2.ExecuteActualQuery() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 664 at Raven.Client.Document.AbstractDocumentQuery2.get_QueryResult() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 633
at Raven.Client.Linq.RavenQueryProviderProcessor1.ExecuteQuery[TProjection]() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryProviderProcessor.cs:line 1511 at Raven.Client.Linq.RavenQueryInspector1.GetEnumerator() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryInspector.cs:line 105
at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Octopus.Platform.Documents.Persistence.SessionExtensions.FetchAllUnsynchronized[TDocument](IRavenQueryable1 query) in y:\work\refs\heads\master\source\Octopus.Platform.Documents\Persistence\SessionExtensions.cs:line 113
at Octopus.Server.Orchestration.Deploy.DeploymentOrchestrator.d__e.MoveNext() in y:\work\refs\heads\master\source\Octopus.Server\Orchestration\Deploy\DeploymentOrchestrator.cs:line 112
Octopus.Server version 2.6.4.951

Resolution:
Since yesterday, I have managed to recover the missing Project. As an experiment, I manually created a “projects-5” document in the RavenDB. I had realised all the elements used by the project (variablesets-5 / deployment-process-5) were all still existing, it seemed to be just the “projects-5” document record that was missing. Once I manually created it and linked in the various elements it needed (using another project’s metadata as a guideline), the Project appeared again and deployed successfully. Does that sound like a stable solution, or may there be some underlying issues?
This was a better solution to me than trying the not-so-reliable Repair job :\ … and furthermore a full restore which required downtime.

Going forward, would you suggest the Repair DB for most issues? It seems to be the only go-to solution (before a full restore) but I dont know how to identify whether an issue will be solved this way or not, or when the DB is too corrupted for the Repair to make a difference. Are there cases where a Repair is irrelevant and its worth bypassing and going straight to a restore?

Many thanks for your help,
Bronwyn,

5

Hi Bronwyn,

I understand your lack of confidence on the DB Repair given your past experience, but that is still the recommended first step when you realize you are missing data on Raven. The restore should only be considered as the last resource.

Regarding your workaround, I can’t say i’m sure its gonna be a stable solution. Manual modifications on the database are something we almost never recommend. Its still uncertain if your database is missing more data, so i’d still recommend you to do a backup of your database now and run the repair. If something happens during the repair, let us know and we’ll give you a hand.

Luckily this will all go away in Octopus 3.0 as we’ll be switching from RavenDB to SQL.

Regards,

Dalmiro