I observe an error every hour relating to the update of the step packages :
...
2022-09-11 11:26:07.8793 11684 17 INFO Server task "ServerTasks-134290" waited for 29 ms before starting.
2022-09-11 11:26:07.8793 11684 17 INFO Acquiring non-throttled cluster-wide mutex "ServerTasks-134290" "succeeded" after 001ms.
2022-09-11 11:26:07.8793 11684 17 INFO Updating step template library...
2022-09-11 11:26:07.8793 11684 17 INFO Acquiring step packages from "CompatibilityCheckingStepPackageSource(Octopus.Core.StepPackages.RemoteStepPackageSource)"
2022-09-11 11:26:07.8793 11684 17 WARN Unable to get step package index from "CompatibilityCheckingStepPackageSource(Octopus.Core.StepPackages.RemoteStepPackageSource)": "Error fetching package index.
Call failed. No such host is known. (steps-feed.octopus.com:443): GET https://steps-feed.octopus.com/catalog
"
2022-09-11 11:26:07.8793 11684 17 INFO Acquired 0 step package(s).
...
My Octopus server instance use a proxy to access internet and all other web queries work.
I do not detect any attempt to access steps-feed.octopus.com in the proxy logs so I think this task doesn’t seem to be using the configured proxy.
It does seem that the proxy is not picking up the steps-feed.octopus.com host in question. Do you manually have to add each allowed host to your proxy by any chance? It also doesn’t seem to be resolving via DNS on your Octopus server. To be sure I checked at our end and its definitely up and running.
Could you check with your network team to see if this does need to be manually added? It would be good to see if you have DNS configured on your local Octopus server to check internet hosts and if so, to try and ping that site in a CMD prompt.
Let me know how you get on with that and we can dig deeper if needed.
Hi @paraic.oceallaigh ,
I checked the connection to steps-feed.octopus.com through a browser with the proxy configured, it goes well. I conclude that the proxy is not taken during this request by Octopus Server
Just jumping in for Paraic as he is currently off shift as part of our Australian based team.
Its going to be quite hard to narrow down this one to see where the issue lies but it does seem like a DNS issue. Are you able to confirm you have your Octopus Server set to use your proxy server, you alluded to it in your first comment but I did just want to confirm.
Also, where did you try and get to steps-feed.octopus.com through the browser from? Did you use the Octopus Server to search for that URL, did you try and ping it from the Octopus Server as Paraic suggested? Do you perhaps have an internal DNS that the Octopus server uses that is not configured to point externally?
Once we have that information we can continue to work with you to get a resolution.
Server A : hosting Octopus Deploy server instance and transparent proxy Glider configured to forward internet URI to a proxy connected to internet on Server B
Server B : hosting Squid connected to internet
Local Glider is setup to bypass intranet URI or forward to Squid for Internet URI.
2022/09/12 17:18:05 server.go:96: [http] 127.0.0.1:62055 <-> steps-feed.octopus.com:443 [c] via SERVERB-squid:8080
2022/09/12 17:18:06 server.go:96: [http] 127.0.0.1:62059 <-> stepsprodpackages.blob.core.windows.net:443 [c] via SERVERB-squid:8080
When Glider receive request whether it is considered to be bypassed or forwarded a log is generated. But when “Acquire Step Packages” task is run by Octopus Deploy instance no logs appear in Glider logs.
All other internet queries made by Octopus are successful (for example library.octopus.com).
Thanks for getting back to us with all those details. It does seem like everything looks to be configured to hit the steps-feed.octopus.com/catalog endpoint, so we’re still not sure what the root cause is here.
I’m not sure if you’ve noticed this and looked through the proxy logs for it, but it might be the next thing to try if you haven’t yet. Let me know what you think and we can go from there.
Glider logs reflect this redirection when testing with Powershell :
2022/09/12 17:18:05 server.go:96: [http] 127.0.0.1:62055 <-> steps-feed.octopus.com:443 [c] via SERVERB-squid:8080
2022/09/12 17:18:06 server.go:96: [http] 127.0.0.1:62059 <-> stepsprodpackages.blob.core.windows.net:443 [c] via SERVERB-squid:8080
It seems that the call to steps-feed.octopus.com is made through the dependent Flurl library unlike the rest of the Http calls using the HttpClient API?
Maybe the Flurl client configuration doesn’t use the proxy configured in OD?
Thanks for pointing that out, and my apologies I didn’t spot that initially.
That’s an interesting find, and one I’ll need to bring up with our engineers to see if they can determine if there is an issue with the call made through that library, or if they think something else could be causing this. I’ll let you know what I hear from them.
After discussing with my colleagues, I was wondering if I could get just a bit more information on your instance configuration. Specifically, I was hoping to know the following:
Which version of Octopus Server are you currently using?
Have you configured any proxy settings on the machine hosting Octopus Server, or have you tried applying the proxy on the machine (outside of Octopus) to see if it makes a difference?
Would you be willing to provide me a recent server log from your instance? You can upload these to me securely here: FMM | Octopus Support
At the very least I can pass these details on to our developers, but it might help us with troubleshooting in the meantime.
Analyzing the OD logs I see a timestamp 09:05:17.9467:
2022-09-13 09:05:17.9467 9532 49 INFO Acquiring step packages from "CompatibilityCheckingStepPackageSource(Octopus.Core.StepPackages.RemoteStepPackageSource)"
2022-09-13 09:05:17.9467 9532 49 WARN Unable to get step package index from "CompatibilityCheckingStepPackageSource(Octopus.Core.StepPackages.RemoteStepPackageSource)": "Error fetching package index.
Call failed. No such host is known. (steps-feed.octopus.com:443): GET https://steps-feed.octopus.com/catalog
"
but it’s not present in Fiddler which makes me say that the call to steps-feed.octopus.com is really not proxified
This is all fantastic fault finding information thank you very much for providing us with such detail!
I will pass this new information and your logs onto the engineers, they may have some more questions for you but you have given us a lot to go off already so hopefully the engineers can dig into this.
Sorry for the time it has taken to get back to you, our engineers can confirm we are not using the configured proxy correctly for requests to the step package feed.
The engineers have a fix in place for this which should rollout to 2022.2 and 2022.3 versions soon. I will let you know when the fix is out so you can download and install the latest 2022.3 version, this should hopefully fix your issue.
Thank you for your patience whilst we investigated this for you, I will be in touch when the new version is out.
Sorry for the double post but the engineers have just sent me over a GitHub Issue they created so I wanted to pass that onto you. If you subscribe to that you will get updates on when the fix is out.
