Hello,
When using VM extensions to install the tentacle we get this error:
“Enabling Tentacle failed. The OctopusServerUrl ‘https://octopus.phxmn.com’ was not found.”. I have validate this from the Portal, Power Shell, and ARM…
Here is the thing. When I log into the VM and re-run the installer it connects to https://octopus.phxmn.com just fine, and the installation goes as expected.
Hi,
Thanks for getting in touch.
Is there anything unique about your Azure subscription being locked down in terms of networking permissions or alike?
We’re wondering if the account being used to enable the Tentacle is unable to make network requests? (E.g. which may occur if things are running from the system account instead of the network service account)
That may explain why, when you log into the VM with your user account, it all connects just fine, but whatever account is being used to run the installer doesn’t have the necessary permissions to make network requests.
If not, to help us reproduce, could you please let us know all the details around how you’re using the VM extension to install the Tentacle? I.e. Are you just using the Azure portal UI, or have you customised an ARM template, or are you using the Azure PowerShell cmdlets/CLI?
Cheers
Mark
Hi Mark, thanks for the reply…
Nothing unique about our subscription… I should note this started happening when we moved octopus behind the Barracuda WAF. We do enforce TLS 1.1 and 1.2 only at the WAF.
As I said communication is fine on the VM… I did try to track down with Azure if they have weird TLS rules or anything from Vm Extensions, but we couldn’t actually communicate about it.
Before answering your second question, I would like to ask if we can just turn off the ssl redirect on then the TCP endpoints running 5986 and 10943 will work without TLS. In this architecture we would route external communication through the Barracuda and
all internal communication would be over 80, and the other ports, but we don’t want them enforcing and TLS. Is that possible?
To answer your next questions I recreated my problem with the portal, through power shell, and through custom ARM. In each of those cases I can log into the VM, but I then I have to delete the tentacle and reinstall and all is just fine… I even copy and
paste the values out of my scripts into the tentacle installer to make sure they don’t have a problem.
The VM Extension logs didn’t seem to have much more info, but I may have missed it.
$publicSettings = @{
OctopusServerUrl = “https://octopus.phxmn.com”;
Environments = @(“Int-Prep”);
Roles = @(“Web”);
CommunicationMode = “Poll”;
Port = 10934
}
$privateSettings = @{“ApiKey” = “API-MYKEY”}
$rg = “Development”
$vm = “dev-ext-tst1”
$loc = “WestUS”
Set-AzureRmVMExtension -ResourceGroupName $rg `
-Location $loc `
-VMName $vm `
-Name “OctopusDeployWindowsTentacle” `
-Publisher “OctopusDeploy.Tentacle” `
-TypeHandlerVersion “2.0” `
-Settings $publicSettings `
-ProtectedSettings $privateSettings `
-ExtensionType "OctopusDeployWindowsTentacle"
PS C:\dev> .\octo-install.ps1
Set-AzureRmVMExtension : Long running operation failed with status ‘Failed’. Additional Info:'VM has reported a
failure when processing extension ‘OctopusDeployWindowsTentacle’. Error message: "Enabling Tentacle failed. The
OctopusServerUrl ‘https://octopus.phxmn.com’ was not found.".’
ErrorCode: VMExtensionProvisioningError
ErrorMessage: VM has reported a failure when processing extension ‘OctopusDeployWindowsTentacle’. Error message:
“Enabling Tentacle failed. The OctopusServerUrl ‘https://octopus.phxmn.com’ was not found.”.
ErrorTarget:
StartTime: 5/16/2019 10:26:47 PM
EndTime: 5/16/2019 10:27:53 PM
OperationID: 2c5c162b-c8a8-4e37-836a-9007842fdd51
Status: Failed
At C:\dev\octo-install.ps1:19 char:1
Set-AzureRmVMExtension -ResourceGroupName $rg `
CategoryInfo : CloseError: (
[Set-AzureRmVMExtension], ComputeCloudException
FullyQualifiedErrorId : Microsoft.Azure.Commands.Compute.SetAzureVMExtensionCommand
Hey Mark,
I don’t think I heard back after I asked you guys about off-loading ssl at our load balancer and using non-TLS for communication internally.
We are also failing our PCI audits due to how the tcp apps are server up their cert. How does the cert get bound to those apps?
It would really be helpful if we could have a quick design call to discuss our environment and the right way to set-up Octopus server so the VM Extension works as well as we can pass our PCI audits.
Is that something we can get set-up?
Hi,
Very sorry this slipped through the cracks. Can I just double check my understanding of your scenario? My understanding is that you have an Octopus server, that is now behind Barracuda, and you are trying to connect polling Tentacles to it via Barracuda, is that correct?
I’d assume the portal is working ok and the Tentacle connections are the real issue?
This isn’t exactly my area of expertise, but my understanding is that the polling Tentacles must connect directly to the Octopus server node(s), due to the “sticky session” nature of the connections it uses. It may be possible to get this to work to a single server node if sticky sessions are supported, but again this is way outside of my area of expertise.
For customers running On-Premise, especially those with HA configurations (multiple Octopus server nodes), the setup would usually be to run the portal through the load balancer but still have the polling Tentacles connect directly to the server nodes by name.
Would it be possible to configure the Azure network that the VM is connected to such that it could have 10943 open to the server directly? Alternatively, would it be possible to reverse the comms and configure the Azure networking to allow the server to connect to the VM as a listening Tentacle?
I’m not an expert on PCI compliance either, so unsure whether these would be viable options. If you’d still like to set up a call, could I get you to drop a line to support@octopus.com and we’ll organise the details through there.
Regards
Shannon
Thanks Shannon,
Our Network guy is out next week, so the Architect and I will chew on the info and see if can get better questions.
So ya to summarize our situation.
Our Octopus server sits behind a Barracuda load balancer and WAF
We have almost all functionality working except:
Installing the tenacle via VM extension, though I can immediately log onto that server and run the tentacle installer by hand. This is just super painful because all my automation depends on the VM coming up and being able to bootstrap with Octopus
We also cannot upload files from the UI to the Octopus feed.
PCI compliance has strict rules about self signed certs, local certs without the right encryption level and other things… This is where I think a chat would help and we will schedule that when the Network guy gets back.
We did try to set-up all comm path internally, but has a whole range of other issues. I do think we can try that again though, just not sure what certs are needed to make tentacle communication work in that model.
Mark,
The VM extension has detailed diagnostic logging, which you can read about in our docs. Can you send through the log files that you find there, and we can take a look?
I suspect that there is something odd going on where the account that is running the azure vm extension cannot access the internet, or at the very least, access the url provided.
The error message you have seen is logged when this method returns false:
private bool IsOctopusServerContactable(string octopusServerUrl)
{
var request = (HttpWebRequest)WebRequest.Create(octopusServerUrl);
request.Timeout = 15000;
request.Method = "HEAD";
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
return response.StatusCode == HttpStatusCode.OK;
}
}
catch (WebException)
{
return false;
}
}
Another thing that might be interesting to see from the logs is what account is being used. This will be logged in the Tentacle log files. You may be able to try and run the code above (or a powershell variant of it) using something like psexec to run it under the account, and see if there is something odd happening there.
On the bigger picture of PCI… It’s not as clear cut as it could be - it’s often “in the eye of the auditor”. We’ve had many customers pass their PCI audits with the default self signed certificate configuration of Octopus. In the general sense, self signed certificates are considered bad, but as Octopus explicitly checks certificate thumbprints on both sides, this is actually an increased level of security than standard https gives you. Check out our blog post on the subject for a more detailed explanation here.
All that said, if you end up with a super strict, “by the book” auditor, Octopus supports using customer provided certificates for Tentacle communications. This is achieved via the import-certificate command on Tentacle.exe.
Unfortunately, this is not supported by the Azure VM Extension though. Normally, I’d suggest falling back to using the OctopusDSC from your ARM template, however it’s not currently available there either. I’m hoping one of my colleagues will be able to whip something up in the next few days though.
Regarding ssl off-loading, we don’t tend to recommend configuring this, as it leaves your communication channel vulnerable to threats that are already inside your network. We’d advocate end-to-end encryption.
Currently SSL offloading is not supported via polling Tentacles, though you may have success with using websockets.
Hope this helps clarify things - look forward to getting the logs through so we can dig deeper.
Shannon