Task "AcquireStepPackages" failed with 'No such host is known. (steps-feed.octopus.com:443)'

I observe an error every hour relating to the update of the step packages :

...
2022-09-11 11:26:07.8793  11684     17  INFO  Server task "ServerTasks-134290" waited for 29 ms before starting.
2022-09-11 11:26:07.8793  11684     17  INFO  Acquiring non-throttled cluster-wide mutex "ServerTasks-134290" "succeeded" after 001ms.
2022-09-11 11:26:07.8793  11684     17  INFO  Updating step template library...
2022-09-11 11:26:07.8793  11684     17  INFO  Acquiring step packages from "CompatibilityCheckingStepPackageSource(Octopus.Core.StepPackages.RemoteStepPackageSource)"
2022-09-11 11:26:07.8793  11684     17  WARN  Unable to get step package index from "CompatibilityCheckingStepPackageSource(Octopus.Core.StepPackages.RemoteStepPackageSource)": "Error fetching package index.
Call failed. No such host is known. (steps-feed.octopus.com:443): GET https://steps-feed.octopus.com/catalog

"
2022-09-11 11:26:07.8793  11684     17  INFO  Acquired 0 step package(s).
...

My Octopus server instance use a proxy to access internet and all other web queries work.
I do not detect any attempt to access steps-feed.octopus.com in the proxy logs so I think this task doesn’t seem to be using the configured proxy.

Hi @FMM
Thanks for reaching out.

It does seem that the proxy is not picking up the steps-feed.octopus.com host in question. Do you manually have to add each allowed host to your proxy by any chance? It also doesn’t seem to be resolving via DNS on your Octopus server. To be sure I checked at our end and its definitely up and running.

Could you check with your network team to see if this does need to be manually added? It would be good to see if you have DNS configured on your local Octopus server to check internet hosts and if so, to try and ping that site in a CMD prompt.

Let me know how you get on with that and we can dig deeper if needed.

Kind regards,
Paraic

Hi @paraic.oceallaigh ,
I checked the connection to steps-feed.octopus.com through a browser with the proxy configured, it goes well. I conclude that the proxy is not taken during this request by Octopus Server

Good afternoon @FMM,

Just jumping in for Paraic as he is currently off shift as part of our Australian based team.

Its going to be quite hard to narrow down this one to see where the issue lies but it does seem like a DNS issue. Are you able to confirm you have your Octopus Server set to use your proxy server, you alluded to it in your first comment but I did just want to confirm.

Also, where did you try and get to steps-feed.octopus.com through the browser from? Did you use the Octopus Server to search for that URL, did you try and ping it from the Octopus Server as Paraic suggested? Do you perhaps have an internal DNS that the Octopus server uses that is not configured to point externally?

Once we have that information we can continue to work with you to get a resolution.

Kind Regards,

Clare

Here is my setup :

  • Server A : hosting Octopus Deploy server instance and transparent proxy Glider configured to forward internet URI to a proxy connected to internet on Server B
  • Server B : hosting Squid connected to internet

Local Glider is setup to bypass intranet URI or forward to Squid for Internet URI.

Implementation :
Glider : listening on localhost:3128
Octopus Deploy proxy setup :
image

Testing Proxy chain :

Run locally on Octopus Server the query using the proxy chain :

PS C:\> curl -Uri https://steps-feed.octopus.com/catalog -Proxy http://localhost:3128 | Select-Object -Property StatusDescription , StatusCode

StatusDescription StatusCode
----------------- ----------
OK                       200

Glider logs show :

2022/09/12 17:18:05 server.go:96: [http] 127.0.0.1:62055 <-> steps-feed.octopus.com:443 [c] via SERVERB-squid:8080
2022/09/12 17:18:06 server.go:96: [http] 127.0.0.1:62059 <-> stepsprodpackages.blob.core.windows.net:443 [c] via SERVERB-squid:8080

When Glider receive request whether it is considered to be bypassed or forwarded a log is generated. But when “Acquire Step Packages” task is run by Octopus Deploy instance no logs appear in Glider logs.
All other internet queries made by Octopus are successful (for example library.octopus.com).

Hi @FMM,

Thanks for getting back to us with all those details. It does seem like everything looks to be configured to hit the steps-feed.octopus.com/catalog endpoint, so we’re still not sure what the root cause is here.

After looking further into this on our end, steps-feed.octopus.com/catalog redirects to the following Url: https://stepsprodpackages.blob.core.windows.net/packages-published/catalog.json

I’m not sure if you’ve noticed this and looked through the proxy logs for it, but it might be the next thing to try if you haven’t yet. Let me know what you think and we can go from there.

Best,
Patrick

Hi @patrick.smergut,

Glider logs reflect this redirection when testing with Powershell :

2022/09/12 17:18:05 server.go:96: [http] 127.0.0.1:62055 <-> steps-feed.octopus.com:443 [c] via SERVERB-squid:8080
2022/09/12 17:18:06 server.go:96: [http] 127.0.0.1:62059 <-> stepsprodpackages.blob.core.windows.net:443 [c] via SERVERB-squid:8080

It seems that the call to steps-feed.octopus.com is made through the dependent Flurl library unlike the rest of the Http calls using the HttpClient API?
Maybe the Flurl client configuration doesn’t use the proxy configured in OD?

Hi @FMM,

Thanks for pointing that out, and my apologies I didn’t spot that initially.

That’s an interesting find, and one I’ll need to bring up with our engineers to see if they can determine if there is an issue with the call made through that library, or if they think something else could be causing this. I’ll let you know what I hear from them.

Best,
Patrick

Hey @FMM,

After discussing with my colleagues, I was wondering if I could get just a bit more information on your instance configuration. Specifically, I was hoping to know the following:

  • Which version of Octopus Server are you currently using?
  • Have you configured any proxy settings on the machine hosting Octopus Server, or have you tried applying the proxy on the machine (outside of Octopus) to see if it makes a difference?
  • Would you be willing to provide me a recent server log from your instance? You can upload these to me securely here: FMM | Octopus Support

At the very least I can pass these details on to our developers, but it might help us with troubleshooting in the meantime.

Best,
Patrick

Hi,

  • Version of Octopus Server is 2022.3.10405 but it was previously observed in 2022.2.8011.
  • No other proxy is configured on machine.
  • OD log file sent

Regards,
FMM

I did a new test with Fidler.
Here is the proxy chain :
OD → Fiddler → Glider → Squid on Server B

Fiddler should capture all proxified request.

Fiddler logs :

Request #156 : at startup of OD instance
Request #158,#159,#166 : corresponding to manual update launched of Step template :


Request #164 : manual update of Step Community templates :

Analyzing the OD logs I see a timestamp 09:05:17.9467:

2022-09-13 09:05:17.9467   9532     49  INFO  Acquiring step packages from "CompatibilityCheckingStepPackageSource(Octopus.Core.StepPackages.RemoteStepPackageSource)"
2022-09-13 09:05:17.9467   9532     49  WARN  Unable to get step package index from "CompatibilityCheckingStepPackageSource(Octopus.Core.StepPackages.RemoteStepPackageSource)": "Error fetching package index.
Call failed. No such host is known. (steps-feed.octopus.com:443): GET https://steps-feed.octopus.com/catalog

"

but it’s not present in Fiddler which makes me say that the call to steps-feed.octopus.com is really not proxified

Hey @FMM,

This is all fantastic fault finding information thank you very much for providing us with such detail!

I will pass this new information and your logs onto the engineers, they may have some more questions for you but you have given us a lot to go off already so hopefully the engineers can dig into this.

I will keep you updated on any news they have.

Kind Regards,

Clare

Hey @FMM,

Sorry for the time it has taken to get back to you, our engineers can confirm we are not using the configured proxy correctly for requests to the step package feed.

The engineers have a fix in place for this which should rollout to 2022.2 and 2022.3 versions soon. I will let you know when the fix is out so you can download and install the latest 2022.3 version, this should hopefully fix your issue.

Thank you for your patience whilst we investigated this for you, I will be in touch when the new version is out.

Kind Regards,

Clare

Hey @FMM,

Sorry for the double post but the engineers have just sent me over a GitHub Issue they created so I wanted to pass that onto you. If you subscribe to that you will get updates on when the fix is out.

Kind Regards,

Clare

1 Like