The Octopus server is starting: Starting message processor

Legacy Uncategorised
1

OS: Windows 2008R2
Octopus Deploy Version: 2.5.12.666
Issue: Post restart, the Dashboard shows (see attached)
The Octopus server is starting: Starting message processor…

Tail of Octopus.txt log:
2016-02-19 12:16:18.4386 INFO Browse your Octopus server at: http://localhost:8082/
2016-02-19 12:16:41.9319 INFO You can browse the RavenDB server at: http://localhost:10931/
2016-02-19 12:16:46.4089 INFO The database is up to date.

Tail of RavenDB log:
2016-02-18 11:26:13.3966 WARN WASTE: Discarding future work item without using it, to reduce memory usage

octopus_message-processor.png
octopus_message-processor.png917×858 32.5 KB

2

Update: I ran a RavenDB repair, but still seeing the same issue. Can we bump this priority to a Severity 1? We cannot perform any deployments.

Repair logs:
Service stopped
Repairing the Octopus Server storage at D:\Octopus\RavenDB.
Recovering the ESENT database…

Extensible Storage Engine Utilities for Microsoft® Windows®
Version 6.1
Copyright © Microsoft Corporation. All Rights Reserved.

Initiating RECOVERY mode…
Logfile base name: RVN
Log files: D:\Octopus\RavenDB\logs
System files: D:\Octopus\RavenDB\system
Database Directory: D:\Octopus\RavenDB

Performing soft recovery…
Restore Status (% complete)

      0    10   20   30   40   50   60   70   80   90  100
      |----|----|----|----|----|----|----|----|----|----|
      ...................................................

Operation completed successfully in 1.108 seconds.

Defragmenting the ESENT database…

Extensible Storage Engine Utilities for Microsoft® Windows®
Version 6.1
Copyright © Microsoft Corporation. All Rights Reserved.

Initiating DEFRAGMENTATION mode…
Database: D:\Octopus\RavenDB\Data

              Defragmentation Status (% complete)

      0    10   20   30   40   50   60   70   80   90  100
      |----|----|----|----|----|----|----|----|----|----|
      ...................................................

Moving ‘TEMPDFRG6512.EDB’ to ‘D:\Octopus\RavenDB\Data’…
File Copy Status (% complete)

      0    10   20   30   40   50   60   70   80   90  100
      |----|----|----|----|----|----|----|----|----|----|
      ...................................................

Note:
It is recommended that you immediately perform a full backup
of this database. If you restore a backup made before the
defragmentation, the database will be rolled back to the state
it was in at the time of that backup.

Operation completed successfully in 43.477 seconds.

Deleting RavenDB indexes…
33 indexes deleted
Recreating RavenDB indexes…
Checking whether any active tasks have timed out…
Database repair is complete.
Waiting for service to start. Current status: StartPending
Waiting for service to start. Current status: Running
Service started

3

I managed to get Octopus server up and running.
Actions performed:
Stopped Octopus Server service / killed process
Renamed the D:\Octopus\OctopusServer\Messages to D:\Octopus\OctopusServer\Messages.old
Created a new “D:\Octopus\OctopusServer\Messages” directory
Ran the RavenDB repair (which auto-starts Octopus deploy)
[crossed fingers]
… console became available.
performing validations now.

4

Latest update:
Noted that tasks went immediately to queue with no new information in logs.
Actions taken:
Cleared out the tasks from RavenDB
Set Octopus into maintenance mode
stopped octopus server
renamed the Actors and Messages directory in "D:\Octopus\OctopusServer"
started octopus server
Noticed it went through a full retention policy cleanup
Restarted Octopus server
[crossing fingers]

5

Latest update: the same issue happened after performing the previous post’s steps, mainly the UI was stuck in “The Octopus server is starting: Starting message processor…”

Actions performed:
Stopped octopus deploy server / killed process
Renamed (for backup) and re-created the following in D:\Octopus\OctopusServer:
ActivityLogs
Actors
Messages
Started Octopus deploy server
[crossed fingers]
Deployment activities resuming :slight_smile:

6

Requesting closing this issue, as the above steps seemed to have fixed the hanging messages.

7

Hi Todd,

I am glad you are back up and running, and everything that you did seems logical.
I think some of this might have occurred due to our ‘background’ process in your version of Octopus.
Things like package indexing, indexes being built and a few other little things run on startup. They can cause this style of locking or just time to complete before you can access Octopus.

We have changed this dramatically in versions since, and even change colors on load with much better messages about what we are doing. Some of the tasks were also moved to the background to allow you in while still performing indexes.
Do you have plans to upgrade to our 3.x versions? Are there any questions or blockers that you see that I can help with?

Vanessa