Azure App Service Health Check Proxy Authorization Required

john2 · 8 July 2021 14:28

Hi,

We have on-prem Octopus Server deploying to Azure App Services via an on-prem worker tentacle.

Yesterday morning we upgraded to v2021.1 Build 7379 from v2020.11 or there abouts.

Last night we started getting unhealthy Azure App Service targets across the board with the following in the beginning of the health check task logs.

<HEAD><TITLE>Proxy Authorization Required</TITLE></HEAD> 
<BODY BGCOLOR="white" FGCOLOR="black"><H1>Proxy Authorization Required</H1><HR> 
<FONT FACE="Helvetica,Arial"><B> 
Description: Authorization is required for access to this proxy</B></FONT> 
<HR> 
<!-- default "Proxy Authorization Required" response (407) --> 
</BODY>

This would have been the first health check since the upgrade.

We are however, still able to deploy to Azure.

Our worker tentacles are running as dedicated windows NT accounts, rather than default machine accounts (i.e. local system/network service).

I have been able to upgrade the tentacles today - but still get unhealthy checks to the Azure App services.

Now in all likelyhood it looks like our proxy is causing the issue but my questions are thus:

Why when our targets are unhealthy are we still able to deploy to Azure?

Could the upgrade of Octopus Server affect health check on Azure App Services from the tentacles?

When I have come across this type of thing in the past it has usually been down to the application not providing credentials to the proxy when challenged.

Is there a difference when doing deployments and health checks as to this sort of behaviour in terms of the credentials used?

Cheers

John

garrett.dass · 8 July 2021 19:40

Hi @john2,

Thanks for reaching out and I apologize for the delay in my response.

In regards to your questions, we’ve had some other reports of proxy issues in 2021.1 that we’re currently investigating.

To help us better answer your request I just have a few questions I hope you don’t mind answering first:

How is your Proxy configured on the Tentacle Server?
How is your Tentacle configured to use this Proxy? If this is a System proxy, does it work if the proxy is explicitly defined in the Tentacle config? You can try configuring this here if you haven’t already.
If this is not a system proxy, how is it configured? Are you using static values or is it a .pac file?
Would it be possible for us to get a copy of the raw Task log of this particular health check task?

Looking forward to hearing back from you.

Regards,
Garrett

john2 · 8 July 2021 21:58

Hi Garret,

Here is the HeathCheck Task Log output for one of our targets. I’ll have to get back to you tomorrow with answers to first three bullet points - I don’t have access to the servers myself. Here is what I can provide for now.

Regards

John

Health Check Task Log.txt (4.9 KB)

finnian.dempsey · 9 July 2021 02:07

Hi John,

Just stepping in for Garrett as he’s signed off for the day.

Thanks for sending that log through, we’ll investigate further on our end and will let you know if we require any more info.

Best Regards,

john2 · 9 July 2021 10:26

Thanks Finnian,

Generally the proxy settings in our organisation are controlled by a pac file and I believe set by Group Policy

Our experience with Octopus in our set-up with Azure to say the least have been problematic, hence why they are now set on the Tentacle as well.

These settings have not changed for some time.

Internet Proxy Settings

Tentacle Proxy Settings

These are the current sequence of events:

June 21 2021 8:56 PM Azure App Service Targets become healthy
…
Multiple successful Azure deployments
…
July 7 2021 AM before 9:30 AM Octopus Server upgraded to 2021.1 build 7379
…
Multiple successful Azure deployments
…
July 7 2021 8:24 PM all Azure App Service Targets become unhealthy
July 8 2021 AM Worker Tentacles upgraded via Octopus Server and are healthy
…
Multiple successful Azure deployments
…

The health check interval in Machine Policies is 24 hours.

We still can deploy to Azure, though it is disconcerting the health checks are coming back bad with proxy authentication issues which do not appear to affect actual deployments.

john2 · 9 July 2021 10:46

Another thought on this, although we deploy through workers and pools are the App Service Target checks actually performed from the Octopus Server? So it would be our Octopus Server having difficulty do in the Health Checks rather than the Octopus tentacles?

garrett.dass · 9 July 2021 22:47

Hi @john2,

Sorry to hear the Proxy configuration has been a headache. You are correct that the Health Check runs from the Octopus Server and not the Tentacle.

I am wondering if your Configuration > Settings > Web Request Proxy is pulling settings or is blank?

We might need to set this back up in Octopus Manager by going to the instance and changing Proxy Server Settings to custom as was done with your Tentacle:

Could you test that on your end and see if that resolves the issue?

Regards,
Garrett

john2 · 10 July 2021 05:17

Cheers Garrett,

Will give this a try in a few days.

Regards

John

john2 · 12 July 2021 14:49

Hi,

We have performed some more checks tests including:

Setting custom proxy settings on Tentacle Manager
Setting custom proxy settings on Octopus Manager

Our findings were:

The health check did appear to occur on the worker not the Octopus Server - at least that is what we found in the logs
A line appeared in the logs indicating whether Calamari would use the settings or not. Settings not there… line would appear, proxy settings in, line would not appear
We also found that under Configuration > Settings > Web Request Proxy the setting in the Manager would be reflected (including the actual password!) but not be editable
None of the combinations tried worked - always Pre Authentication Required
Our team which manages proxy - says no credentials are being passed through

Have also reviewed:

https://github.com/OctopusDeploy/Issues/issues/6941

Behaviour still exists.

Regards

John

finnian.dempsey · 19 July 2021 03:01

Hi John,

Apologies for the delay in getting back to you, I’ve been attempting to reproduce the error but haven’t yet been successful. Thanks for providing that info!

The Octopus Server typically uses it’s built-in worker for most tasks, however here it looks like it’s been configured to use a different worker from the ‘Non Production’ pool.

In my opinion, it’s likely the issue is in the configuration for this worker’s proxy for web requests, and not on the Octopus Server itself. The endpoint for the deployments is typically “myapp.azurewebsites.com” whereas the health check is via: “management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Web/sites/{name}”.

Could you please send through a screenshot showing the Tentacle’s proxy configuration?

I’ve also seen that line appear when I change my workers’ proxy configuration, however it’s presence isn’t effecting my Web App health checks and so I don’t believe this is the cause.
Thank you for raising this! I’ve passed this on and the password should not be visible following the next release.
Just confirming you tried the different options for Proxy configuration on the Tentacle instance, as seen in the image below, and all of them are producing the same error “Proxy Authentication Required”?

Are you able to confirm if the Tentacle instance is using the proxy outside of Octopus, i.e. through I.E? Otherwise could you please share the proxy pac file so I can try reproduce your setup on my end? Feel free to redact any sensitive info/passwords etc.

Could you please also test if adding a new Web App Target also fails the Health Check or is it just existing targets? I’ll keep testing on my end to see if I can figure out what’s going on, keen to get to the bottom of this!

Best Regards,

john2 · 19 July 2021 16:39

Hi Finnian,

w.r.t 1) The screen shots of our tentacles proxy set-up can be seen in my third post above, this is unchanged for pre and post upgrade to 2021.1 build 7379

w.r.t 4) We haven’t tried the options Default Proxy, Default Proxy with Custom Credentials, nor would we want to at this stage - mainly because we are quite locked down in our environments and this was working before the upgrade - If I get the opportunity I’ll give them a go.

w.r.t 5) 100% sure we are getting through the proxy as we are able to do app service deployments to Azure - just the healthcheck which is causing the grief at the moment. I’ll send you a “reacted” version of a our proxy.pac file directly - I’ve replaced any sensitive values with nonsense.

I’ve created a new Azure Web App service target - it also is failing the healthcheck

Regards

John

finnian.dempsey · 19 July 2021 23:48

Hi John,

Thanks for confirming that it doesn’t work for new targets either.

I’ve created a link for you to securely upload the pac file here, once received I’ll try to reproduce your exact configuration and hopefully reproduce the issue!

Just to confirm your setup, please correct me if any of the following is wrong:

Octopus Server initially v2020.1.11 approx. upgraded to v2021.1 Build 7379
Azure Web App Targets - see image below
Worker Pool (Non Production) with a configured Worker - This worker performs the Health Check on the Web App Target, configured on the Web App. See image below.
The Octopus Server has been configured to use a Proxy. See image below.
The Octopus Tentacle (Worker from Non Production worker pool) has been configured to use a Proxy. See image below.
Windows is aware of the Proxy configuration. - See image below

Web App Target:

Web App with Configured Worker Pool:

Octopus Server Proxy configured from within the Server Manager:

Octopus Tentacle Proxy configured from within Tentacle Manager:

Proxy Configuration in Windows:

Feel free to let me know if your configuration is slightly different or you have any questions. I’ll keep diving into it on my end and let you know if I need anymore info.

Best Regards,

john2 · 20 July 2021 09:16

Hi Finnian,

Have checked:

the upgrade was from 2020.4.11 to 2021.1 build 7379
The target is set-up as Azure Web App
Each app service target is assigned to one of two pools, each pool has one tentacle worker
The Default proxy is enabled on the Octopus Server (I believe the following mirrors what is set through the Octopus Server Manager)

Octopus Service Proxy Settings 20072021731×324 14.7 KB
The internet proxy settings on the tentacle machine are set as shown in my third post. These are obtained through either; IE > Internet Options or windows command prompt > inetcpl.cpl. Whether they match the latest windows UI, I cannot readily tell as I don’t have direct access to the machines involved, I expect they will.

Regards

John

finnian.dempsey · 22 July 2021 00:22

Hi John,

Thanks for that extra confirmation of the setup!

I have finally managed to reproduce it and have raised an issue here if you would like to track it’s progress while the engineers work on it. Unfortunately I can’t say exactly when this will be resolved and I haven’t yet been able to find a workaround for you to use in the meantime, is this currently blocking you?

As always, feel free to let me know if you have any questions!

Best Regards,

john2 · 22 July 2021 09:02

Hi Finnian,

It is not blocking us at the moment - we can still deploy. A fix in a reasonable time frame (a release in the next quarter?) would be good - more to reassure management than anything. A work round would be nice - I will look at our end as to whether something can be relaxed on our proxy which may help? Would that help from what you have seen of the issue? Otherwise we may just have to live with it for now.

Regards

John

finnian.dempsey · 23 July 2021 00:33

Hi John,

Great to hear you can still deploy, although it will be preventing you from adding any new Web App targets, and of course determining the health status of existing ones which isn’t ideal.

I don’t think there are any changes required to your proxy setup to help resolve this, however I’d like to confirm if the Tentacle and Server are connected directly or are they using a proxy also?

This can be seen by navigating to Infrastructure -> Workers -> Select the desired Worker and you’ll be able to see the proxy setting at the bottom of the page.

While I can’t guarantee that this will be resolved in the next quarter, in my opinion it’s very likely that it will be addressed sooner rather than later. I’ll keep you posted if I have any more info and try to let you know as soon as I’m aware of a release that includes a fix.

Feel free to let me know if you have any questions in the meantime.

Best Regards,

john2 · 23 July 2021 16:05

Hi Finnian,

Our problem is not affecting us adding new Web App Targets, at least I don’t think it is - I have been able to add a new target albeit to an existing azure resource. Can’t really tell for a completely new resource - I don’t have the ability to create one in Azure.

In terms of connectivity between Server and Worker we are connecting directly and not via a proxy.

The following is some more information which may have a bearing on the problem. We have 12 app service targets in Azure, 4 used for production and 8 used for non production stuff.

Today I noticed the Health Checks for the Production app services carried out yesterday were successful, I have however just run them again an they failed with the same error. We are not aware of any changes at out end which might have affected this and we are following up with our third party if any changes have been made over our affected dates on Azure.

Given that we are able to deploy to Azure, is it possible to determine what resource the healthcheck is trying to access when this issue occurs. The log suggests it is trying check the app service exists, but do we know which resource URI it is trying to access. Have attached a log example.

Health Check Task Log2.txt (4.9 KB)

Regards

John

finnian.dempsey · 26 July 2021 06:52

Hi John,

Thanks for clarifying the Server/Tentacle communication method, and that extra info. I have to say that behaviour is very strange and doesn’t line up with what I’m seeing in my testing. I’ll continue to investigate.

Just to confirm, when you add a new Web App deployment target, is this able to pass it’s health check? Is it possible for you to configure the proxy settings manually? I believe that might resolve the health check issue, until a fix is released.

The health check is configured via this library, which constructs a URL like:

https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Web/sites/{name}

I’ve been able to see entries in the Proxy access logs for Web App health checks prior to 2021.1 however I am unable to see them following the upgrade, and I raised this in the Github issue linked previously. The other access log entries relating to this health request have URL’s similar to:
dc.services.visualstudio, login.microsoftonline.com and management.azure.com as linked above. Your networking team should be able to help provide the exact endpoints for a health check via the proxy access logs.

I’ll keep you posted if I have any updates.

Best Regards,

john2 · 26 July 2021 09:17

Hi Finnian,

Thanks for clarifying the endpoints used by the health - I’ll pass these on to our security team.

In terms of a new web app deployment target, it does not pass the health check.

When you say configure the the proxy setting manually are you referring through IE > Internet options…Windows? This is something we could try to see if we can get the health check to work - though not a viable work round. My organisations group policy resets these frequently.

I am taking the case last week when the health check passed on some of our targets was an aberration, it was only successful the once, and has failed the last 4 attempts.

From what I have seen, my current understanding is that post 2021.1, when custom proxy is configured on the tentacle the credentials are not being passed through to the proxy for the health checks. Please let me know if further investigations at your end prove differently.

We can live with this for now, I’ll pass the end point onto various teams at my end to see if any work rounds are possible.

Cheers

John

finnian.dempsey · 27 July 2021 01:06

Hi John,

No problem, thanks for clarifying that new targets also have this issue, that’s the behaviour I’m also seeing.

Apologies, I meant to configure the Custom Web Request Proxy in Octopus Manager. I don’t believe this will resolve the issue, as I think it is a seperate bug, but it could be something to try!

To summarise the two issues:

Web App Health checks aren’t using the Proxy. Issue
Octopus Server isn’t using the Default Proxy configured in Windows. Issue

I agree that the Health Check passing last week might have been a one off ‘bug’. If you are able to reproduce it please let me know! I’ll keep you posted with any further findings on my end.

Feel free to reach out if you have any questions, or would like any extra info.

Best Regards,