Help trouble shooting Azure Account setup

reliability
(andrew.w) #1

Hello,

I’m trying to setup an Azure account and when I click “Save and Test” I get an error

“Unable to verify account:
Service returned error. Check InnerException for more details
Inner Exception:
Unable to connect to the remote server”

I’ve followed the steps in https://octopus.com/docs/infrastructure/deployment-targets/azure and watched the deep dive video.

I can’t find any details in the logs, we’ve whitelisted the server to ensure it’s not the proxy.

We just upgraded to 2019.6.0 LTS

Can you help me trouble shoot this error

(Kartik Gupta) #5

Hi Andrew,

Thanks for getting in touch and sorry that you are facing this issue.

I’m unable to determine precisely what’s causing this issue, or if it’s a bug (though it seems to work as expected in my testing, but I may be misinterpreting something). Would you be willing to elaborate a bit further on if you used Service Principles or Management Certificates and if you created the Azure Service Principle Account or Credential using Powershell or the Azure Portal. If you could also please send some screenshots showing the Azure Portal, Resource Permissions and the error on the Octopus Server? This will help me better understand your scenario and give it a test as you have it setup. :slight_smile:

I look forward to hearing back and getting to the bottom of this one!

Cheers,
Kartik

(andrew.w) #6

Hi Kartik,

I am hoping you can give me some tips on trouble shooting the issue, is there any way to work out what server it is trying to connect to? Are there any logs
I can look at, or tracing I can turn on? At the moment all I get is a generic “Cannot connect” message.

Here are the settings in Octopus

And the error

Azure

Subscription

Product

Key

(andrew.w) #7

Forgot this screenshot,

(Kartik Gupta) #8

Hi Andrew,

Thanks for the above info and your patience while I debugged this further.

I tried to recreate the error that you are getting and was able to do so when I switched to a host-only network on my Virtual Machine. This indicated that there is indeed something being blocked by your firewall or proxy. Can I please ask which server did you whitelist? I would suggest trying to access your Azure portal from the server that Octopus is running on.

Let me know how you go.

Regards,
Kartik

(andrew.w) #9

Portal works fine from the server
L

Can you think of any logs that might have more details on what is failing?

(Kartik Gupta) #10

Hi Andrew,

Sorry that you are still facing this issue. I had a discussion with the team about the problem you are facing and we have a couple of steps you could try to help narrow down the root cause of the issue.

Step 1: We suggest running a PowerShell script locally instead of the UI to test the authentication of your Azure credentials. You would need to install Azure PowerShell module using the command Install-Module -Name Az -AllowClobber. (Here is Microsoft’s documentation on how to do this).

Next, please run the TestAzure.ps1 (731 Bytes). You will have to enter your SubscriptionId, TenantId, ClientId and Secret in the script. If it works you should be able to see the Azure Resource Groups you have access to.

Your output should look something like this:

Step 2: If Step 1 works could you please RDP into your server running the Octopus instance and try running the script there? Please note that the script would have to be run on a PowerShell window with the same user permissions used by your Octopus server. By doing this we are eliminating the possibility that there is something blocking on Octopus Server or your Azure credentials.

Step 3: If that works we recommend trying to connect to Azure using a local instance of Octopus Server.

Step 4: If the above steps work, we recommend next logging the outgoing requests rejected by your firewall/proxy. We think there might be an endpoint used by the Azure authentication library that may be blocked.

Hope this helps!
Kartik

(andrew.w) #11

That’s excellent, that will really help me determine the cause, thanks for that I’ll let you know how it goes.

(andrew.w) #12

Hi Kartik,

We’re getting there!

I managed to get your script working on our server, it was super helpful because it returns a meaningful error.

If I try to use those settings in the account page it still doesn’t work
L Same error as before.

If I try to run your powershell script from the Octopus script console it also doesn’t work. But the script console will work if I add this to your script:

[system.net.webrequest]::defaultwebproxy = new-object system.net.webproxy(’’)

[system.net.webrequest]::defaultwebproxy.credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials

[system.net.webrequest]::defaultwebproxy.BypassProxyOnLocal = $true

Is that proxy setting something I need to setup in Octopus somewhere?

Regards

Andrew

(andrew.w) #13

Hi Kartik,

Got the account connecting
J I found the proxy settings in Octopus and tentacle manager. I can now try to get an actual deployment
going

(Kartik Gupta) #14

Hi Andrew,

Thanks so much for updating us with your progress. Nice work on getting it working!

Please do not hesitate to get in touch should anything come up.

Cheers!
Kartik

(andrew.w) #15

Hi Kartik,

I am now having trouble setting up my deployment targets
L

My account works and connects:

Now I need to setup the target to deploy to, I set it to my account, I know this works because it lists the available web apps

However when I view the deployment target I see this:

If I run a health check the log just says

(Kartik Gupta) #16

Hi Andrew,

Sorry that you are facing this issue. I tried reproducing the steps to add an Azure Web App on my local instance and was able to get through the health check successfully. We will need some more information to investigate what is going on there. Can you please attach the full task log? Details on how to do this are available here https://octopus.com/docs/support/get-the-raw-output-from-a-task

Alternatively, you can send it to support@octopus.com. Only Octopus staff will be able to access it.

Cheers,
Kartik

(Kartik Gupta) #17

Hi Andrew

We wanted to confirm if you have modified the default machine policy to use a custom script? If so, can you please send us the script? At the moment, the Azure targets only run the default machine policy. This is a known issue which can be found here: https://github.com/OctopusDeploy/Issues/issues/5341. The script might help us identify why the Azure target is running into trouble here.

Thanks,
Kartik

(andrew.w) #18

Hi Kartik,

Log attached:

ServerTasks-48277.log.txt (3.33 KB)

(andrew.w) #19

Hi Kartik,

I haven’t intentionally done this, the only scripts I’ve been running against azure was the one you sent me

(Michael Noonan) #22

Hi @andrew_w,

This is proving to be quite the mystery, made all the more awkward by our poor timezone overlap.

Good: At this point it sounds like you can set up and test your Azure Service Principal Account through your normal Octopus Server.

Bad: It sounds like Health Checks are failing, which is strange. The health check just checks the Azure Web App actually exists and can be accessed using the Service Principle. What is also strange is that there is no helpful logging nor error message, just an exit code of 1.

Zoom call?

The easiest way forward would be to set up a Zoom call where we can screenshare and get to the bottom of this more quickly, the problem being our time zone overlap again, and we may not be able to get this together until next week. Can you confirm some times which may work for you to do that call? I can join a call most nights of the week after 8PM AEST.

In the meantime

In the meantime there are still some things you can do to isolate what is going wrong:

  1. Upgrade to the latest release of Octopus Server. If you want to stay on the 2019.6 LTS, please install the latest patch of 2019.6.x. If you are happy to go onto the fast lane, upgrade to the latest possible version of Octopus Server. Learn about release lanes. I’m doubtful this will fix the problem, but it will make it much easier for me to support you moving forward.
  2. Install an instance of Octopus Server on your own computer and try doing the same thing which is failing on your “real” Octopus Server. This will help isolate the problem down to the environment or the software or the configuration. You can download the latest MSI and set up a trial of Octopus Server in just a few minutes as long as you have SQL Server (even express) accessible on/from your computer.

Hope that helps!
Mike

(andrew.w) #23

Hi Michael,

Thanks for getting back to me. I’ve booked another session with our network team to make sure they’re not blocking anything when I do the health check.

After that I’m definitely keen on the zoom call. Can we book in an evening later this week - Wednesday or Thursday?

Regards

Andrew

(Derek Campbell) #26

Hi Andrew,

Thanks for your patience on this.

I am a Continuous Delivery Architect based in the UK and only got back from Annual Leave today. If you have time, book in some time with me and we can jump in and get this working for you. You can book some time with me here.

Thanks

Derek

(Derek Campbell) #27

Hi @andrew_w,

Thanks for your time on the call today.

It looks like the proxy in place is blocking the health checks and these IP’s need to be whitelisted in your proxy configuration to allow for the health check to run successfully.

We’d love to hear how you got on.

Thanks

Derek