Authenticating to Azure Service Fabric using certificate

Hi there,
I was trying to set up Azure Service Fabric as a deployment target by following https://octopus.com/docs/infrastructure/deployment-targets/azure/service-fabric-cluster-targets. However I get an exception, which doesn’t give much information or any clues what could potentially be wrong. Here is exception I get:

Connecting with Secure Client Certificate
Operation canceled.
System.Fabric.FabricTransientException
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Calamari.AzureServiceFabric.HealthCheckCommand.Execute()
at Calamari.CalamariFlavourProgram.ResolveAndExecuteCommand(IContainer container, CommonOptions options)
at Calamari.CalamariFlavourProgram.Run(String[] args)
–Inner Exception–
Operation aborted (Exception from HRESULT: 0x80004004 (E_ABORT))
System.Runtime.InteropServices.COMException
at System.Fabric.Interop.NativeClient.IFabricClusterManagementClient11.EndGetClusterManifest2(IFabricAsyncOperationContext context)
at System.Fabric.FabricClient.ClusterManagementClient.GetClusterManifestAsyncEndWrapper(IFabricAsyncOperationContext context)
at System.Fabric.Interop.AsyncCallOutAdapter2`1.Finish(IFabricAsyncOperationContext context, Boolean expectedCompletedSynchronously)
Fatal
The remote script failed with exit code 100

My first thought was certificate is wrong. So I tried to give a different certificate, which would be wrong, just to see what difference it makes. And I got this back:

Connecting with Secure Client Certificate
FABRIC_E_CONNECTION_DENIED: CertificateNotMatched
System.Fabric.FabricConnectionDeniedException
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Calamari.AzureServiceFabric.HealthCheckCommand.Execute()
at Calamari.CalamariFlavourProgram.ResolveAndExecuteCommand(IContainer container, CommonOptions options)
at Calamari.CalamariFlavourProgram.Run(String[] args)
–Inner Exception–
Exception from HRESULT: 0x80071C43
System.Runtime.InteropServices.COMException
at System.Fabric.Interop.NativeClient.IFabricClusterManagementClient11.EndGetClusterManifest2(IFabricAsyncOperationContext context)
at System.Fabric.FabricClient.ClusterManagementClient.GetClusterManifestAsyncEndWrapper(IFabricAsyncOperationContext context)
at System.Fabric.Interop.AsyncCallOutAdapter2`1.Finish(IFabricAsyncOperationContext context, Boolean expectedCompletedSynchronously)
The remote script failed with exit code 100

And this time it makes more sense as it says CertificateNotMatched. It makes sense, because I know this time certificate is wrong.

Is there a way to get more details on the first exceptions as it feels like certificate is correct, connection has been made, but something else fails and I have no clue what?

Thanks

Hi @tadas.poc,

Thank you for contacting Octopus Support.

Are you seeing this on Octopus Cloud or on an on-premise installation of Octopus Deploy? If on-prem, what version of Octopus are you running?

I look forward to hearing back from you.

Regards,
Donny

Hi Donny,

We do have Octopus Server on premises. We use 2020.3.2.0.

Regards,
Tadas

Hi @tadas.poc,

Thank you for getting back to me.

If you haven’t already, I would suggest checking the service fabric logs to see if anything lines up with the connection attempts from Octopus.

Let me know if that turns anything up.

Regards,
Donny

Hi Donny,

I did check the logs now but unfortunately nothing is logged. For further investigation purposes, I tried to connect Service Fabric directly using powershell on Octopus host machine:

Connect-ServiceFabricCluster -ConnectionEndpoint $ClusterName -KeepAliveIntervalInSec 10 -X509Credential -ServerCertThumbprint $CertThumbprint -FindType FindByThumbprint -FindValue $CertThumbprint -StoreLocation LocalMachine -StoreName MY

That worked spot on. Which means Azure Service Fabric is reachable and certificate is correct. That suggests me, that the problem must be on Octopus side. Maybe configuration. Although there is not that much to set up - specify host and certificate.

Hi @tadas.poc,

Thank you for getting back to me.

I am going to consult with our engineering team in Australia on this one.

I will let you know what I find out.

Regards,
Donny

Hi @donny.bell,

Any update on this?

Regards,
Tadas

Hi @tadas.poc,

Apologies for the delay.

I was able to get this working in my test environment. I did need to import the PFX file provided by the Azure Key Vault. This is done in Octopus via Library -> Certificates.

Does my configuration match what you are trying to do on your end?

Regards,
Donny

Hi,

Yes, I did that. If I import certificate from Azure Key Vault, I get “Operation cancelled” back. If I try to import a random certificate, for testing purposes, then I get “CertificateNotMatched” back. This is why I thought it might be Azure blocking further operations after connection is established, so I tried Powershell script with the same certificate and it worked.

Would you be able to tell me what does Octopus tries to do, when it performs Health check? Could I replicate it using Powershell or something, what would help me to narrow down the issue?

Regards,
Tadas

Hi @tadas.poc,

Thank you for getting back to me.

I have a few more questions for you:

  • Have you tried connecting to your cluster via Azure Service Fabric Explorer?
  • Are you behind a proxy or load balancer?
  • What version of Azure Service Fabric SDK are you running?
  • Is Visual Studio installed on the same machine as the Octopus Server? If so, what version?

I look forward to hearing back from you.

Regards,
Donny