Deployment Target Unavailable

I am adding a new deployment target that is used for both a SQL DACPAC database deploy and an IIS Website deploy. We currently have several other target servers that all work fine.

The SQL database deploy to the new target is working as expected, however the IIS deploy is not. Th message in the Task Log is:


A transient machine error has been detected
An error occurred when sending a request to ‘https://rmd-sql-rc:10933/’, before the request could begin: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host…
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host…
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host…
An existing connection was forcibly closed by the remote host.

The OctopusTentacle log is showing:


A client at [::ffff:10.0.77.32]:59863 connected, and attempted a message exchange, but it presented a client certificate with the thumbprint ‘258A9531015366536BEBCCA3595D5C0C6EFEBDD3’ which is not in the list of thumbprints that we trust

What does not make sense is that the Database deploy is not getting that message, and the tentacle on the target machine is showing that the thumbprint in the error message is the one that it is accepting requests from.

Any suggestions for me?

More information…

When I look at the deployment targets the new one is listed as Unavailable. Selecting it and going to the Connectivity page and pressing the green check health button. I am attaching the Task Log file from that. As you can see, the log file is telling us that an error occurred when sending https://rmd-sql-rc:10933. If I open a browser and enter that url, I immediately get a message saying:
“Octopus Tentacle configured successfully
If you can view this page, your Octopus Tentacle is configured and ready to accept deployment commands.
This landing page is displayed when no X509 certificate is provided. Only Octopus Servers with a trusted certificate can control this Tentacle.”

This is the same results I get when executing that for other servers that I am fine deploying to.

I will also attach the Octopus Tenacle Log file.

ServerTasks-145946.log.txt (10.7 KB)
OctopusTentacle.txt (97.0 KB)

Hi @WDiamond,

Thanks for reaching out to Octopus Support, and I’m sorry you’re experiencing this connectivity issue with your new deployment target.

Do you happen to have multiple Tentacle instances installed on this same machine? You can easily check by opening the Octopus Tentacle Manager on the machine and selecting the dropdown in the upper right. If you only have the one instance, does the Thumbprint displayed in the Tentacle Manager match what is configured in the Deployment Target settings within the Octopus UI?

I look forward to hearing back, and please let me know if you have any questions.

Thanks!
Dan

There was only one (Default). I deleted it and added it back in. Now I am fine!

1 Like

Dan,

I spoke too soon…

The Web deploy is now fine, but the database deploy is now hanging on the first step, in all environments, not just the one I was having web deploy issues with. I am attaching the task log for your input. The log looks like it

ServerTasks-145979.log.txt (18.7 KB)

Hi @WDiamond,

I’m just stepping in for Dan while he’s offline as a member of our US team. Thanks for attaching the logs, they contain some helpful information about the issue.

You may already know all of this, but I thought I’d first clarify what you’re seeing in the screenshot, in case there’s any confusion around it. The RC environment has been configured with guided failure mode, which will cause the deploy to halt when an error is encountered, rather that fail directly. This feature is essentially a manual intervention designed to give you a chance to take manual action and put some notes on the issue. You can abort the deployment (default behaviour on failure), continue on and ignore the error, retry the deployment, or exclude a failing machine and continue.

Regarding the reason this specific deployment has failed, your logs show the following:

Package #{PackageID} v9.8.2.3917 is required by action 'Get Package'

Then the error:

The package #{PackageID} v9.8.2.3917 could not be downloaded to the package cache from Octopus Server (built-in) after making 5 attempts over a total of 0s.

My initial thoughts are that the package ID variable you’re using is not correctly evaluating. The first thing to check would be that the #{PackageID} variable is correctly configured and scoped to evaluate during your deployment. You can do a quick test to see how Octopus expect this variable to evaluate under VariablesPreview on the project.
image

When you configure the filter to represent the environment, target, etc of the deployment it failed on, you should see what Octopus expect it to evaluate to.

If this doesn’t help, would you be able to attach a screenshot of how the #{PackageID} variable is configured?

Let me know how you go here.

Best regards,
Daniel

You pointed me in the right direction. The #{PackageID} had been had coded and I changed it to use the variable instead. As soon as I changed it back things started working again!

While I have your attention I would like to ask about resources on our Octopus server. IT is a VM with 16 cores and 32G RAM. I am attaching a series of screen shots that show the configuration. My concern is that Task Manager is showing that about 90% of the memory is in use, yet the processes listed do not add up to the 28G necessary to show that that much is in use. Also, the server get very lethargic, often taking about 3 to 5 minutes to fully log on to it. I have also notices a trend for our deploys are taking much linger than usual.

Is this something that you have seen before?

Hi @WDiamond,

I’m glad Daniel could help you with your previous issue!

As for the performance issues with your VM, I cannot offer much advice as it’s outside of the Octopus domain. As I’m sure you know, VM problems can have a wide variety of causes (VM Host, SQL tuning, etc). Generally speaking, we do recommend keeping the Octopus and SQL servers on separate VM’s to help reduce the load and performance issues you may experience when combining them.

Sorry I can’t offer much more there, but please do let us know if you have any questions we may be able to help with.

Thanks,
Dan

Unfortunately, we are stuck in a virtual server environment. For the most part, it works pretty well. I will look into moving the Octo SQL database to a different server.

Hi @WDiamond,

I understand how it goes. You’ll likely see some performance gains by splitting SQL off onto its own VM, but the behavior you mentioned seems to likely be something beyond that. From a resource standpoint, it’s beefy enough I wouldn’t expect you to see significant issues. What are you using for your VM hosting, if you don’t mind me asking? The VM settings screenshot doesn’t look familiar at first glance.

I look forward to hearing back, and please let me know if you have any other questions.

Thanks!
Dan

VMware. What would be your suggestion for added resources? CPU, RAM?

Hi @WDiamond
Just stepping in for Dan who is offline for the day.
Looking at the numbers 16C/32GB is a substantially sized VM and barring very large DB size and large and frequent deployments, should be a good for size for medium octopus systems.

Dan’s option of splitting DB and Octopus is the definitely the recommended one here. One benefit among many would be after splitting DB and Octopus you will be able to see clearly whether its the DB or Octopus is consuming resources.

Apart from that, with VMware, not knowing which version you are running but assuming some version of vSphere, the ESX host the VM is running on may be where the issue lies. ESX hosts will typically overcommit memory to its guests as standard as rarely do the guests consume all of the allocated memory all of the time. This only becomes an issue once the guests ramp up the collective memory use and then some version of swapping may occur. This can dramatically slow down a guest’s performance and it may explain why you are seeing strange numbers in Task Manager.

However since we are not familiar with your environment, I would discuss this with your VMware admin and get some stats on the ESX host and take it from there. Moving this VM to another ESX host might be a good troubleshooting option to see if performance improves.

Let us know if this helps get your performance back.

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.