Octopus Server Not Able to Start

Legacy Uncategorised
1

Hi,

The Octopus version we used is 2.6.5.1010. I encountered the following error when try to start the Octopus server. Please kindly help. Thanks!

2017-03-20 15:46:04.1414 1 INFO Resetting stored communication state
2017-03-20 15:46:04.3324 1 INFO Purging data from “F:\Octopus\OctopusServer\Messages”…
2017-03-20 15:46:04.3534 1 INFO Purging data from “F:\Octopus\OctopusServer\Actors”…
2017-03-20 15:46:04.3764 1 INFO Purged 0 messages (0 failed), 0 actors (0 failed)
2017-03-20 15:46:10.2246 1 INFO Waiting for service to start. Current status: StartPending
2017-03-20 15:46:11.4422 1 INFO Waiting for service to start. Current status: Running
2017-03-20 15:46:12.4642 1 INFO Service started
2017-03-20 15:46:13.8012 6 INFO Browse your Octopus server at: http://localhost:80/
2017-03-20 15:46:37.3083 6 INFO You can browse the RavenDB server at: http://localhost:10931/
2017-03-20 15:46:46.6650 6 INFO The database is up to date.
2017-03-20 15:46:53.9662 6 ERROR An exception was thrown while invoking the constructor ‘Void .ctor(Octopus.Platform.Deployment.Configuration.IHomeConfiguration)’ on type ‘BuiltInPackageRepositoryFactory’. —> Index was outside the bounds of the array. (See inner exception for details.)
Autofac.Core.DependencyResolutionException: An exception was thrown while invoking the constructor ‘Void .ctor(Octopus.Platform.Deployment.Configuration.IHomeConfiguration)’ on type ‘BuiltInPackageRepositoryFactory’. —> Index was outside the bounds of the array. (See inner exception for details.) —> System.IndexOutOfRangeException: Index was outside the bounds of the array.
at Lucene.Net.Util.BitVector.Get(Int32 bit) in d:\Lucene.Net\FullRepo\trunk\src\core\Util\BitVector.cs:line 118
at Lucene.Net.Index.AbstractAllTermDocs.SkipTo(Int32 target) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\AbstractAllTermDocs.cs:line 95
at Lucene.Net.Search.MatchAllDocsQuery.MatchAllScorer.NextDoc() in d:\Lucene.Net\FullRepo\trunk\src\core\Search\MatchAllDocsQuery.cs:line 82
at Lucene.Net.Search.Scorer.Score(Collector collector) in d:\Lucene.Net\FullRepo\trunk\src\core\Search\Scorer.cs:line 66
at Lucene.Net.Search.IndexSearcher.Search(Weight weight, Filter filter, Collector collector) in d:\Lucene.Net\FullRepo\trunk\src\core\Search\IndexSearcher.cs:line 221
at Lucene.Net.Search.IndexSearcher.Search(Weight weight, Filter filter, Int32 nDocs, Sort sort, Boolean fillFields) in d:\Lucene.Net\FullRepo\trunk\src\core\Search\IndexSearcher.cs:line 213
at Lucene.Net.Search.IndexSearcher.Search(Weight weight, Filter filter, Int32 nDocs, Sort sort) in d:\Lucene.Net\FullRepo\trunk\src\core\Search\IndexSearcher.cs:line 194
at Lucene.Net.Linq.LuceneQueryExecutorBase1.ExecuteScalar[T](QueryModel queryModel) at Remotion.Linq.Clauses.StreamedData.StreamedScalarValueInfo.ExecuteScalarQueryModel[T](QueryModel queryModel, IQueryExecutor executor) at Remotion.Linq.Clauses.StreamedData.StreamedScalarValueInfo.ExecuteQueryModel(QueryModel queryModel, IQueryExecutor executor) at Remotion.Linq.QueryProviderBase.System.Linq.IQueryProvider.Execute[TResult](Expression expression) at System.Linq.Queryable.Count[TSource](IQueryable1 source)
at NuGet.Lucene.LucenePackageRepository.UpdatePackageCount(IQueryable1 packages) at NuGet.Lucene.LuceneRepositoryConfigurator.Initialize() at Octopus.Server.BuiltInFeed.BuiltInPackageRepositoryFactory..ctor(IHomeConfiguration configuration) in y:\work\refs\heads\master\source\Octopus.Server\BuiltInFeed\BuiltInPackageRepositoryFactory.cs:line 35 at lambda_method(Closure , Object[] ) at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate() --- End of inner exception stack trace --- at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate() at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable1 parameters) at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func1 creator)
at Autofac.Core.Resolving.InstanceLookup.Execute()
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable1 parameters) at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance)
at Autofac.ResolutionExtensions.ResolveOptional[TService](IComponentContext context, IEnumerable1 parameters) at Octopus.Shared.Packages.NuGetModule.<Load>b__1(IComponentContext c) in y:\work\refs\heads\master\source\Octopus.Shared\Packages\NuGetModule.cs:line 20 at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass11.b__0(IComponentContext c, IEnumerable1 p) at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters)
at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)

2

Hi,

Thanks for getting in touch! The error is indicating that you have had an index corruption. If you open your Octopus Manager and find Repair RavenDB Storage under the Storage section.
This open will reset your Raven indexes and hopefully allow you to start the service. Depending on the amount of data this could take some time.

When you start seeing these errors they tend to start to occur more often due to the sheer number of documents in the database and our implementation.
Have you considered upgrading to Octopus 3.x?
Vanessa

3

Hi Vanessa, thank you very much for indicating the issue. I did try Repair RavenDB storage couple of time, but it still failed. Finally, I solve it out by deleting all indexes three times via RavenDB portal.

Additionally, I found I have 2903 documents in deploymentIndex. I think I can delete some of old deployment documents (1 year ago) using RavenDB studio to reduce the index size, which might also reduce the index corruption rate. Do you think it is okay to do so?

4

Hi,

The feature in the portal is supposed to do exactly as you did manually, but when you have many records it may fail as you found.

I do not suggest deleting anything directly from Raven, especially that specific index. What I would recommend is turning on Retention Policies in the UI for the server. If you do have years of deployments start with a large number and decrease it over a number of days, as our Raven implementation can seize if too many deletions happen at once.

How many documents do you have overall (in the Raven Portal the total document count is in the footer).

Index size does not necessarily reduce the corruption rate as it can depend on the content and amount in combination.

We no longer host documentation for 2.x so I cannot link to our 2.x retention policies documentation, however I can send you a PDF copy if you require. Or I can attempt to guide you if you require.

Let me know how you go.
Vanessa

5

Hi Vanessa, we have 28K+ documents in total. Please kindly send me the retention policies PDF copy. Thank you very much!

6

Hi,

The documentation can be found here: https://www.dropbox.com/s/0icf2do5eqrutye/OD2-270117-0236-260.pdf?dl=0
Unfortunately it is the complete documentation and not just the retention policies section.

Generally 60K+ is when you start to see consistent errors with the indexes. They can become corrupt at any point, and you might not find it happens that often.

Vanessa