Octopus Service Unavailable

[ritvij05]

Hi Team,
We have a Release Pipeline in Azure DevOps which Performs Several Task. This Task include creation of new release in octopus projects, Deploy on untenanted and tenanted.

This Pipeline works seamlessly but sometimes its failing while deploying on untenanted or tenanted with Octopus Service unavailable. Pipeline gets automatically triggered as per the schedule, but if it get failed with above error we have to go on octopus, navigate through that project and deploy the pending tenants. This seems like the api call sent to octopus is not reachable. From last 3 days the deployment is failing with above Error.

Let me know how can this issue be avoided. Also is there any retry mechanism I can use to deploy the pending tenants if such error occurs again?

Error Message:
Octopus_Error.txt (2.1 KB)

2022-05-06T11:04:56.6932070Z body {font-family:Arial; margin-left:40px; }img { border:0 none; }#content { margin-left: auto; margin-right: auto }#message h2 { font-size: 20px; font-weight: normal; color: #000000; margin: 34px 0px 0px 0px }#message p { font-size: 13px; color: #000000; margin: 7px 0px 0px0px}#errorref { font-size: 11px; color: #737373; margin-top: 41px }Service unavailable

Our services aren’t available right now

We’re working to restore all services as soon as possible. Please check back soon.

0gP90YgAAAAAN8AeI2R71RKyjTovjqvgdRE0yQUExMDkxMjA4MDExAGMxZGQyNzFmLWY4N2EtNGNiNy05N2JhLWEwMWU2MmZiYjY3YQ==

2022-05-06T11:04:56.6939398Z Error from Octopus Server (HTTP 503 ServiceUnavailable)
2022-05-06T11:04:56.6947267Z Exit code: -7
2022-05-06T11:04:56.7218687Z ##[error]"Failed to deploy release. The process ‘C:\hostedtoolcache\windows\octo\7.4.3663\x64\octo.cmd’ failed with exit code 4294967289
Error: The process ‘C:\hostedtoolcache\windows\octo\7.4.3663\x64\octo.cmd’ failed with exit code 4294967289
at ExecState._setResult (D:\a_tasks\OctopusDeployRelease_8ca1d96a-151d-44b7-bc4f-9251e2ea6971\3.1.131\node_modules\azure-pipelines-task-lib\toolrunner.js:942:25)
at ExecState.CheckComplete (D:\a_tasks\OctopusDeployRelease_8ca1d96a-151d-44b7-bc4f-9251e2ea6971\3.1.131\node_modules\azure-pipelines-task-lib\toolrunner.js:925:18)
at ChildProcess. (D:\a_tasks\OctopusDeployRelease_8ca1d96a-151d-44b7-bc4f-9251e2ea6971\3.1.131\node_modules\azure-pipelines-task-lib\toolrunner.js:838:19)
at ChildProcess.emit (events.js:198:13)
at maybeClose (internal/child_process.js:982:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)

Regards,
Ritvij Iyer

[paul.calvert]

Hi @ritvij05,

Thanks for getting in touch!

I believe one of your colleagues may have raised a similar question on Friday: Error from Octopus Server (HTTP 503 ServiceUnavailable)

As the error seems to be coming from the Octopus Server I’m wondering if there could be a load issue caused by triggering a large number of deployments at the same time causing the server to become unavailable or the service to restart.
If you check the Octopus server logs (default location C:\Octopus\Logs) for the times and dates that your build pipeline has failed do you see any errors appearing?
If you’d like us to take a look through those logs feel free to either private message them to me here or email them through to support@octopus.com

The odd part here is that the Octopus UI makes full use of the REST API. So, when you trigger these deployments from within Octopus, the same API calls will be made as the ones that ADO is making.

When this pipeline runs, am I correct that some of the tenant deployments are being successfully initiated before the failure occurs and you are only deploying the remaining tenants from within Octopus?

If so, would it be possible to perform a test deployment from within Octopus that includes all of these tenants? I’d like to see whether the same error occurs when this is run within the UI.

Regarding your question about a retry mechanism for the pending tenants when it fails. With the current configuration the only option I see is what you’ve been doing. Which is to run the remainder of the tenant deployments within Octopus.
One alternative option would be to make use of the ADO Release Pipeline feature rather than the standard pipeline for creating the release and running these deployments.
The benefit of this is that in the event of any failures you can re-trigger any failed deployments from within ADO.
e.g.

I appreciate that this would be a significant change though and would take some time to implement.

Regards,
Paul

[ritvij05]

Hi @paul.calvert ,
Appreciate your Response.

Error from Octopus Server (HTTP 503 ServiceUnavailable) – Yes for Adding More information Created this new Topic.

As mentioned we are checking in octopus server logs and will reach back to you here or on email.

When this pipeline runs, am I correct that some of the tenant deployments are being successfully initiated before the failure occurs and you are only deploying the remaining tenants from within Octopus? – YES Your correct.

If so, would it be possible to perform a test deployment from within Octopus that includes all of these tenants? – We have tried that and it does not fails. We are only getting service unavailable Error when running via ADO.

Also we are making use of ADO RELEASE pipeline feature, but there we have to go on ADO and hit redeploy manually. Issue here is like it will redeploy whole pipeline and we want to deploy only pending domains and not the domains which are already deployed.

CC: @bhagyashreesarode21
Regards,
Ritvij Iyer