An invalid IP address was specified

mathias.karcher · 7 March 2023 10:03

Since the last Octopus update (to 2023.1.9672) we receive an error on the Login screen when clicking “Sign In”.

Octopus is using IIS as a reverse proxy to make it available at http://octopus.xyz.local. The error occurs when using this URL. It does NOT occur when accessing it using the standard URL, e.g. http://localhost:8112.

“An invalid IP address was specified. An invalid argument was supplied.”

The log entry (full stack at the end of this post):

2023-03-07 10:51:35.1934   3612    276  INFO  "HTTP" "POST" to "octopus.xyz.local""/api/users/login" "completed" with 500 in 00:00:00.0183649 (18ms) by "<anonymous>"
System.FormatException: An invalid IP address was specified.
 ---> System.Net.Sockets.SocketException (10022): An invalid argument was supplied.
   --- End of inner exception stack trace ---
   at System.Net.IPAddressParser.Parse(ReadOnlySpan`1 ipSpan, Boolean tryParse)
   at System.Net.IPAddress.Parse(String ipString)
   at Octopus.Server.Orchestration.Logging.IpNetworkChecker.IsIpAddressInRangeOfAnyIpNetworks(String ip, IEnumerable`1 ipNetworks) in ./source/Octopus.Server/Orchestration/Logging/IpNetworkChecker.cs:line 10
   at Octopus.Server.Orchestration.Logging.OriginalClientIpAddressRetriever.Get(HttpContext httpContext) in ./source/Octopus.Server/Orchestration/Logging/OriginalClientIpAddressRetriever.cs:line 30
   at Octopus.Server.Web.Controllers.Users.LoginUserController.LoginUser(LoginUserCommand command, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Controllers/Users/LoginUserController.cs:line 31

IIS is set up with a Server Farm node with the following configuration in applicationHost.config:

<configuration>
   <system.applicationHost>
      <site name="Octopus" id="8" serverAutoStart="true">
         <application path="/" applicationPool="Octopus">
            <virtualDirectory path="/" physicalPath="C:\Octopus\iis-root" />
         </application>
         <bindings>
            <binding protocol="http" bindingInformation="*:80:octopus.xyz.local" />
         </bindings>
      </site>
   <system.applicationHost>
</configuration>

C:\Octopus\iis-root\web.config contains this configuration:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>					
                <rule name="ReverseProxyInboundRule" stopProcessing="true">
                    <match url="(.*)" />
                    <action type="Rewrite" url="http://Octopus/{R:1}" />
                </rule>
            </rules>
        </rewrite>
        <security>
		<requestFiltering>
			<fileExtensions allowUnlisted="true">
				<clear />
			</fileExtensions>
                <requestLimits maxUrl="16384" maxQueryString="16384" />
		</requestFiltering>
	  </security>
    </system.webServer>
</configuration>

Neither an Octopus reinstall, nor update to the latest version helped. Any ideas?

Full stack trace:

2023-03-07 10:51:35.1934   3612    276  INFO  "HTTP" "POST" to "octopus.xyz.local""/api/users/login" "completed" with 500 in 00:00:00.0183649 (18ms) by "<anonymous>"
System.FormatException: An invalid IP address was specified.
 ---> System.Net.Sockets.SocketException (10022): An invalid argument was supplied.
   --- End of inner exception stack trace ---
   at System.Net.IPAddressParser.Parse(ReadOnlySpan`1 ipSpan, Boolean tryParse)
   at System.Net.IPAddress.Parse(String ipString)
   at Octopus.Server.Orchestration.Logging.IpNetworkChecker.IsIpAddressInRangeOfAnyIpNetworks(String ip, IEnumerable`1 ipNetworks) in ./source/Octopus.Server/Orchestration/Logging/IpNetworkChecker.cs:line 10
   at Octopus.Server.Orchestration.Logging.OriginalClientIpAddressRetriever.Get(HttpContext httpContext) in ./source/Octopus.Server/Orchestration/Logging/OriginalClientIpAddressRetriever.cs:line 30
   at Octopus.Server.Web.Controllers.Users.LoginUserController.LoginUser(LoginUserCommand command, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Controllers/Users/LoginUserController.cs:line 31
   at lambda_method6783(Closure , Object )
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Octopus.Server.Web.Middleware.BoundaryTrailerRewriteMiddleware.Invoke(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/BoundaryTrailerRewriteMiddleware.cs:line 44
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 46
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 46
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 52
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 48
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 59
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, IUserAuthenticator userAuthenticator, IAuthCookieService authCookieService, IWebAuthCache authCache, ILogger logger) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 58
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.Middleware.LegacyRequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/LegacyRequestLoggerMiddleware.cs:line 41
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 76
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 117

clare.martin · 7 March 2023 10:42

Good morning @mathias.karcher,

Thank you for contacting Octopus Support and sorry to hear you are having issues after upgrading to 2023.1.9672.

It looks like you are using a reverse proxy for Octopus, luckily a customer noticed this issue yesterday and we did get to the bottom of it so I will post what I put in that ticket in here and hopefully you are able to resolve the issue from that:

In Octopus 2023.1, we now log the IP address of the client in the audit logs for each action, this is useful for situations such as a rogue API key making calls for example, if that happens you can now track it down to a machine via its IP address as well. So, because Octopus is trying to parse that IP from the header field, and it doesn’t exist, it’s throwing the “invalid IP address” error. Or you are sending it, but it’s blank, or it could be a hostname, not an IP address.

To fix this you need to supply a valid IP address for the x-forwarded-for value in the header as shown here - Application Request Routing Page | Microsoft Learn.

Hopefully, that then allows you to make calls to Octopus again via the UI and you are able to sign in.

The other customer got back to me with what they did change in case it helps you:

The header “X-Forwared-For” was already sent, but the option “Include TCP port from client IP” was enabled. So the IP address is the header field had the format X.X.X.X:YY instead of just X.X.X.X.
It works fine after disabling the option.

So, if you are already sending the X-Forwared-For header I would try and disable the Include TCP port from client IP and see if that helps.

I have already flagged this up to our engineers and asked them to include this as a breaking change on our download comparison pages or at least document it somewhere (a GitHub issue perhaps?). I will let you know the outcome of that when they get back to me.

I look forward to hearing if those options worked for you,

Kind Regards,
Clare

mathias.karcher · 7 March 2023 10:54

Hi Claire,

that was spot on. In my case the “Include TCP port from client IP” checkbox was the culprit. Thank you so much for your quick and helpful reply!

Thanks & best regards,
Mathias

clare.martin · 7 March 2023 10:58

Hey @mathias.karcher,

No problem at all, glad we could help, this forum post will help other customers too as I imagine we will get a few of these issues in over the next few weeks as customers with reverse proxies upgrade so thank you for positing it up and confirming the fix.

If you need anything in future please reach out as we are always happy to help,
Kind Regards,
Clare

clare.martin · 8 March 2023 13:13

Hey @mathias.karcher,

Just a quick update on this in case anyone else finds the same issue or you did not want to leave your proxy without having a TCP port on the IP address in the header.

A few other customers who have proxy servers have notified us of having similar issues after upgrading to 2023.1 so we changed the priority of this issue to High and got our engineers to look at it since ideally we don’t want customers to have to change their proxy settings just for Octopus.

They have created a GitHub issue here which you can subscribe to, and you will be notified when the fix is out.

The engineers are expanding the options for the X-Forwared-For header so you should be able to have localhost (instead of IP) or a port number at the end of the IP address.

Once the fix is out and you upgrade you can enable Include TCP port from client IP again on your proxy and Octopus should allow that.

Kind Regards,
Clare

mathias.karcher · 8 March 2023 13:30

Hi Claire,

thanks a lot for your follow-up, appreciate it and I’ll be looking forward to the fix.

Best regards,
Mathias

p.forristal · 13 March 2023 19:43

We also ran into this issue behind an Azure App Gateway. For anyone else finding this thread, here is a blog post from MS that explains how to set this header to just IP, and not IP:Port :

system · 13 April 2023 19:43

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.