Timeouts when upgrading all agents or checking health

Getting the following exception when I upgrade all tentacles:

2013-05-19 18:26:43 INFO Performing upgrade of all Tentacles
2013-05-19 18:27:57 ERROR One or more errors occurred. System.TimeoutException: Waited for 15,022ms for the query to return non stale result.

Server stack trace:
at Raven.Client.Document.SessionOperations.QueryOperation.IsAcceptable(QueryResult result) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\SessionOperations\QueryOperation.cs:line 290
at Raven.Client.Document.AbstractDocumentQuery2.ExecuteActualQuery() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 535 at Raven.Client.Document.AbstractDocumentQuery2.InitSync() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 516
at Raven.Client.Document.AbstractDocumentQuery2.get_QueryResult() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 502 at Raven.Client.Linq.RavenQueryProviderProcessor1.ExecuteQueryTProjection in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryProviderProcessor.cs:line 1381
at Raven.Client.Linq.RavenQueryProviderProcessor1.Execute(Expression expression) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryProviderProcessor.cs:line 1354 at Raven.Client.Linq.RavenQueryProvider1.Execute(Expression expression) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryProvider.cs:line 135
at Raven.Client.Linq.RavenQueryProvider1.System.Linq.IQueryProvider.Execute(Expression expression) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryProvider.cs:line 190 at Raven.Client.Linq.RavenQueryInspector1.GetEnumerator() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryInspector.cs:line 99
at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Octopus.Server.Tasks.Upgrades.UpgradeControllerActivity.<>c__DisplayClass14.<MarkMachinesUpToDate>b__11(IDocumentSession session) in c:\w\e6923628be6eaf72\source\Octopus.Server\Tasks\Upgrades\UpgradeControllerActivity.cs:line 64 at Octopus.Core.Persistance.UnitOfWorkFactory.InSession(Action1 callback) in c:\w\e6923628be6eaf72\source\Octopus.Core\Persistance\UnitOfWorkFactory.cs:line 33
at Octopus.Server.Tasks.Upgrades.UpgradeControllerActivity.MarkMachinesUpToDate(IEnumerable1 machines, List1 results) in c:\w\e6923628be6eaf72\source\Octopus.Server\Tasks\Upgrades\UpgradeControllerActivity.cs:line 60
at Octopus.Server.Tasks.Upgrades.UpgradeControllerActivity.d__a.MoveNext() in c:\w\e6923628be6eaf72\source\Octopus.Server\Tasks\Upgrades\UpgradeControllerActivity.cs:line 52

Exception rethrown at [0]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Octopus.Shared.Activities.ActivityRuntime.d__5.MoveNext() in c:\w\e6923628be6eaf72\source\Octopus.Shared\Activities\ActivityRuntime.cs:line 54

I’m also getting a timeout when doing an agent health check:

Details

Started: Sunday, May 19, 2013 11:37 AM
Duration: 19 minutes
Completed: Timed out
Status: Timed out
Error

The task was executing, but the Octopus Server process that was executing the task stopped updating. The server process is expected to update the task every few seconds. Please verify that the Octopus Server process is running, and check the event log for any errors.

I see this in the event log, not sure if it corresponds to the health check that times out:

2013-05-19 11:47:41,755 [8] ERROR Octopus [(null)] - An unexpected error occurred while attempting to retrieve and execute a task: Waited for 15,032ms for the query to return non stale result.
System.TimeoutException: Waited for 15,032ms for the query to return non stale result.
at Raven.Client.Document.SessionOperations.QueryOperation.IsAcceptable(QueryResult result) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\SessionOperations\QueryOperation.cs:line 290
at Raven.Client.Document.AbstractDocumentQuery2.ExecuteActualQuery() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 535 at Raven.Client.Document.AbstractDocumentQuery2.InitSync() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 516
at Raven.Client.Document.AbstractDocumentQuery2.GetEnumerator() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\AbstractDocumentQuery.cs:line 741 at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable1 source)
at Raven.Client.Linq.RavenQueryProviderProcessor1.GetQueryResult[TProjection](IDocumentQuery1 finalQuery) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryProviderProcessor.cs:line 1454
at Raven.Client.Linq.RavenQueryProviderProcessor1.ExecuteQuery[TProjection]() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryProviderProcessor.cs:line 1379 at Raven.Client.Linq.RavenQueryProviderProcessor1.Execute(Expression expression) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryProviderProcessor.cs:line 1354
at Raven.Client.Linq.RavenQueryProvider1.Execute(Expression expression) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryProvider.cs:line 135 at Raven.Client.Linq.RavenQueryProvider1.System.Linq.IQueryProvider.Execute[S](Expression expression) in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Linq\RavenQueryProvider.cs:line 178
at System.Linq.Queryable.FirstOrDefault[TSource](IQueryable1 source, Expression1 predicate)
at Octopus.Server.Tasks.TaskQueue.PollForMoreWorkOnBackgroundThread(Object state) in c:\w\e6923628be6eaf72\source\Octopus.Server\Tasks\TaskQueue.cs:line 54

I tried restarting the machine / octopus service and I just upgraded to the latest version (was on 1.3? before) and that’s when these exceptions started.

Hi Kevin,

There seems to be a problem with the RavenDB database. Can you connect to the RavenDB console (http://your-octopus:10930) and go to the Indexes tab, and reset/delete all of the indexes and then restart the Octopus Windows Service and do an IIS reset? That should recreate all the indexes, and once the indexes have been rebuilt it should resolve the issue.

Paul