Octopus Instances Ignoring Bindings

bob.ready · 28 June 2016 20:36

I’m currently working on upgrading from Octopus 2.6 to 3.latest by standing up a new server and migrating. The problem we have right now is that Octopus, with 3 instances configured, ignores all of its web bindings and only responds on localhost. Virtual directories work fine, but we’d much prefer the bindings to work.

Each one of the instances has a single binding (pinging these addresses resolves the proper IP):

im-deploy-beta.oursite.com
cd-deploy-beta.oursite.com
oo-deploy-beta.oursite.com

Hitting any of these locally or remotely results in a 503 Service Unavailable.

If we add a binding for localhost to any instance, ALL of the addresses above resolve … to the same instance, whichever one localhost was added to.

We’ve checked the http headers, they have the correct host.

For all it doesn’t make any sense, the only thing we can imagine is that Octopus is ignoring the host and sending everything to localhost??

Here’s the Octopus Server config from one of the instances … the prefixes appear to be correct.

<?xml version="1.0" encoding="utf-8"?> 10943 I:\cd-deploy false XXXXX Data Source=XXXXX;Initial Catalog=CD_Octopus;Integrated Security=True XXXXX true true 1 false http://cd-deploy-beta.oursite.com/

Michael_Noonan · 30 June 2016 02:22

Hi Bob,

Thanks for getting in touch! I have just configured a multi-instance Octopus Server setup on my test server using Octopus 3.3.20 and was able to navigate to the two different instances using host headers.

My configuration looks very much like yours:

<?xml version="1.0" encoding="utf-8"?>
<octopus-settings xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <set key="Octopus.Communications.ServicesPort">9001</set>
  <set key="Octopus.Home">C:\Octopus\Instance1</set>
  <set key="Octopus.Server.NodeName">REMOVED</set>
  <set key="Octopus.Storage.ExternalDatabaseConnectionString">Data Source=.;Initial Catalog=OctopusInstance1;Integrated Security=True</set>
  <set key="Octopus.Storage.MasterKey">REMOVED</set>
  <set key="Octopus.Upgrades.AllowChecking">false</set>
  <set key="Octopus.Upgrades.IncludeStatistics">false</set>
  <set key="Octopus.WebPortal.AuthenticationMode">0</set>
  <set key="Octopus.WebPortal.ForceSsl">false</set>
  <set key="Octopus.WebPortal.ListenPrefixes">http://instance1.local/</set>
</octopus-settings>

<?xml version="1.0" encoding="utf-8"?>
<octopus-settings xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <set key="Octopus.Communications.ServicesPort">9002</set>
  <set key="Octopus.Home">C:\Octopus\Instance2</set>
  <set key="Octopus.Server.NodeName">REMOVED</set>
  <set key="Octopus.Storage.ExternalDatabaseConnectionString">Data Source=.;Initial Catalog=OctopusInstance2;Integrated Security=True</set>
  <set key="Octopus.Storage.MasterKey">REMOVED</set>
  <set key="Octopus.Upgrades.AllowChecking">false</set>
  <set key="Octopus.Upgrades.IncludeStatistics">false</set>
  <set key="Octopus.WebPortal.AuthenticationMode">0</set>
  <set key="Octopus.WebPortal.ForceSsl">false</set>
  <set key="Octopus.WebPortal.ListenPrefixes">http://instance2.local/</set>
</octopus-settings>

The HTTP status code 503: Service Unavailable you are seeing can happen when the Octopus Server is not running, or is starting up. It does sound like the Octopus Server services are running when you tested them individually, but are they all happily running at the same time?

Also, is IIS installed on this server? I have seen instances where IIS and Application Request Routing will capture and reroute requests to the wrong place on the server, resulting in a 503.

If none of this helps, could you send me the config files for all three instances?

Hope this helps!
Mike

bob.ready · 8 September 2016 19:16

This fell off my plate for a while, sorry for not responding.

I’ve reinstalled Octopus from scratch, version 3.4.5.
The server does not have IIS installed.

There’s just one instance running on the server right now, and it’s essentially having the same problem.

Here’s the non-functional config:

<?xml version="1.0" encoding="utf-8"?>
<octopus-settings xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <set key="Octopus.Communications.ServicesPort">10943</set>
  <set key="Octopus.Home">I:\CD</set>
  <set key="Octopus.Server.NodeName">OCTOPUSSERVER</set>
  <set key="Octopus.Storage.ExternalDatabaseConnectionString">Data Source=OURSQL;Initial Catalog=CD_Octopus;Integrated Security=True</set>
  <set key="Octopus.Storage.MasterKey">XXX</set>
  <set key="Octopus.Upgrades.AllowChecking">true</set>
  <set key="Octopus.Upgrades.IncludeStatistics">true</set>
  <set key="Octopus.WebPortal.AuthenticationMode">1</set>
  <set key="Octopus.WebPortal.ForceSsl">false</set>
  <set key="Octopus.WebPortal.ListenPrefixes">http://deploy.ourdomain.net/</set>
</octopus-settings>

I get a 503: Service Unavailable either from the box or from my own laptop.

If I simply add localhost, however, it works. The config:

<?xml version="1.0" encoding="utf-8"?>
<octopus-settings xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <set key="Octopus.Communications.ServicesPort">10943</set>
  <set key="Octopus.Home">I:\CD</set>
  <set key="Octopus.Server.NodeName">OCTOPUSSERVER</set>
  <set key="Octopus.Storage.ExternalDatabaseConnectionString">Data Source=OURSQL;Initial Catalog=CD_Octopus;Integrated Security=True</set>
  <set key="Octopus.Storage.MasterKey">XXX</set>
  <set key="Octopus.Upgrades.AllowChecking">true</set>
  <set key="Octopus.Upgrades.IncludeStatistics">true</set>
  <set key="Octopus.WebPortal.AuthenticationMode">1</set>
  <set key="Octopus.WebPortal.ForceSsl">false</set>
  <set key="Octopus.WebPortal.ListenPrefixes">http://deploy.ourdomain.net/,http://localhost/</set>
</octopus-settings>

That works from both the Octopus server itself and from other devices, even externally via deploy.ourdomain.net. I haven’t replicated multiple instances yet, but if behavior was as it was before, whichever instance had localhost assigned would connect, everything else got a 503.

Michael_Noonan · 9 September 2016 00:37

Hi Bob,

Thanks for getting back to me, it has been a while! I’m glad you got it up and running OK by binding to localhost. The most likely candidate for what is going wrong here is that the HOST HEADER of the web request is being rewritten by a reverse proxy. Do you have a device acting as a reverse proxy or doing SSL offloading etc?

Either way, by binding to localhost the Octopus server will be listening for inbound requests on Port 80 regardless of the HOST HEADER which is, I think, the reason why the everything starts working. You could probably remove the binding to deploy.ourcompany.com and it would still work…

Our demo server at https://demo.octopus.com/ and https://demo.octopusdeploy.com/ has multiple bindings (both HTTP and HTTPS) but none to localhost and it is working nicely which is why I think there is some intermediary affecting the web requests before they arrive at the Octopus Server.

At the end of the day, you’ve got it working now which is great, but I can’t reproduce the behaviour you are seeing in our network topology - I’m sorry about that. If you do happen to find out something more please let me know - I’d be grateful.

Don’t hesitate to reach out for help with anything else!
Mike

bob.ready · 27 September 2016 14:59

Followup … it’s not exactly working, because only the instance bound to localhost works. Which doesn’t help us with multiple instances.

I have verified with Wireshark that the incoming request is stripped of all HTTP contents, so it explains why Octopus is giving a 503. I cannot for the life of me figure out where that information is being removed, though, which is making me tear my hair out.

Michael_Noonan · 10 October 2016 00:22

Hi Bob,

I’ve just come back from vacation and wanted to get back in touch and see if you’ve figured out what is removing the host header from the request?

Kind Regards,
Mike

Dave_Tremblay · 24 October 2016 19:45

Hi,
I’m having the same issue as Bob. I have a fresh install of Octopus Server 3.4.12 and whatever I set as binding, when looking at the logs, I always see this:
2016-10-24 19:29:07.6956 7 INFO Browse your Octopus server at: http://localhost:80/
2016-10-24 19:29:07.6956 7 INFO Browse your Octopus server at: https://localhost:443/

It’s strange because there’s no binding to localhost. Looks like a code issue if you want my point of view. It’s like it’s not taking in account the binding we are configuring.

Thanks

Michael_Noonan · 1 November 2016 21:57

Hi All,

Thanks for getting back in touch. I’ve done some more investigating into these issues. It turns out this is just a logging bug: the actual host name was being clobbered with localhost when we log that message. Rest assured the actual binding you’ve configured actually gets passed untouched to the HTTP listener.

I’ve updated the message to look more like this:

Listening for HTTP requests on:
http://localhost:8065/
http://mike-mbp2015:8065/

Previously it would have been:

Listening for HTTP requests on:
http://localhost:8065/
http://localhost:8065/

Could you send through a Fiddler or Wireshark trace because in every case we’ve seen so far the host-header is being tampered with by a networking device between the client and the Octopus server.

Hope that helps!
Mike

Dave_Tremblay · 8 November 2016 18:14

Hi,

I’m now on version 3.5.1 and I don’t have any issue right now. Thanks for helping.

Michael_Noonan · 10 November 2016 01:10

Hi Dave,

Thanks for getting back to me! Great news and Happy Deployments!
Mike