Restarting Docker based Linux Tentacle gives "Octopus Tentacle is already configured"

Octopus Server
1

I have a Tentacle that is running in Docker. I run it with the following command:

docker run -d --restart always --env Space=“My Target Space” --env ServerPort=“10943” --env TargetWorkerPool=“My Target Worker Pool” --env ACCEPT_EULA=“Y” --env ServerUrl=“http://my-octopus-server.mydomain.net/” --env ServerApiKey=“API-ABCD1234_My_API_Key_Here” my-container-repository.mydomain.net/tooling/library/octo.octopus-tentacle

When I run this, it works perfectly. The Worker starts up and is seen in Octopus Deploy. I can upgrade Calamari, and run runbooks on it.

If you look, you will note that the command included --restart always. This causes the container to restart on a reboot.

And when I restart the Linux Server that this is running on, the container does indeed attempt to restart. But it goes into a crash loop of staring up, and then crashing and trying again.

Each time, it only logs the following: Octopus Tentacle is already configured.

I am guessing that when it re-starts it tries to create the tentacle in Octopus and, failing that, it just exits.

Is there something I can do to get it to just reconnect to the existing tentacle (instead of trying to make a new one when the container restarts).

Update:
Here is my guess of what is happening. When you start up the container it runs:

CMD /scripts/configure-tentacle.sh && /scripts/run-tentacle.sh

This is from line 54 of the Dockerfile.

The first script there (configure-tentacle.sh) checks if an install is needed. Since there is not anything then it just outputs that the tentacle is already installed and calls return.

My best guess is that this return call is somehow returning false. (I am not sure how.)

I come to this conclusion because run-tentacle.sh is not called. (It has echo statements that don’t happen.) The two scripts are joined by an && rather than a ;. The && means that the second one will not run unless the previous one is successful.

Now, assuming that the return statement is valid in this scenario (which I cannot figure how it is, since it is not in a function or called “sourced”), then it should return true (to my understanding), since the previous command was an echo.

But the evidence is quite strong that the second script is not called. (Because the container just exits, it does not have any errors that I can see.)

I would like to suggest that this is a bug.

3

Hi,
Thanks for reaching out with this issue. It does indeed seem like a bug.

I am in the process of reproducing the issue to see if it matches what you are experiencing.

To make sure we have the same configuration, can you let me know the version of Docker and the OS you are running it on and also the version of Octopus server and Tentacle? This will rule out any issues we may already have fixed in later versions.

I will update shortly when I have a full repro done.

Thanks for your patience while we get this sorted for you.

Kind Regards,
Paraic

4

Hi,
Unfortunately in reading the Docker Readme in the Github repo it does actually state that this is not officially supported:

It is used internally and it should work for 99% of your use cases, it is not officially supported. Please do not contact support if you run into issues with this image.

I also notice that you are using your own registry and presumably have your own tentacle image, which we won’t have access to for any troubleshooting.

I will do due diligence in attempting the repro to let the Dev team know if there is an issue with the CMD line in the Dockerfile. This is just good practise anyway but unfortunately we can’t guarantee any fixes or troubleshooting from our Support team.

If you want to keep an eye on the Github repo to see if any fixes show up, it’s probably the best we can do for now.

Kind Regards,
Paraic

5

Hi Paraic,

Thank you for looking into this for me. I was able to find a workaround that will keep me going forward for now. (More on that below.)

However, I would know if the Octopus Tentacle docker container image is truly supported or not. You said:

Unfortunately in reading the Docker Readme in the Github repo it does actually state that this is not officially supported.

I want to be clear that I am not actually “using” that Github repro other than reading it to help me understand the container. I did not build any sources from there or anything. I just read the code to see what was going on “under the hood”.

I got the Tentacle container image from the Octopus Deploy Tentacle repository on Docker Hub.

On that page, it says to contact Octopus Support for support:

image
image1320×165 7.41 KB

I assumed from that statement that the container image was supported.

If the Tentacle container image published to Docker Hub by Octopus is not supported, I would like to know for sure (so I don’t bother Octopus Support with any future issues I have with it).

I also notice that you are using your own registry and presumably have your own tentacle image, which we won’t have access to for any troubleshooting.

True, I am using my own Tentacle image. But it uses the official Octopus Deploy Tentacle Image as its source. I am just adding a few certificates and command line tools (like helm, kubectl, istioctl etc).

I am fairly confident that those additions do not impact this issue. But I could run this without my additions to prove it if needed.

Earlier you asked for a few version numbers, in case you still want to repro this for a bug report, here they are:

  • Docker Version: Docker CE 20.10.6, build 370c289
  • OS: Red Hat Enterprise Linux 7 (RHEL 7)
  • Octopus Server Version: 2020.6.4688
  • Tentacle Version: 6.0.591

Now the Workaround:
As I stated in my original question, I theorized that this issue was due to the && between the two commands run at container startup. As I got to thinking about that, I recalled that a container image is only allowed one CMD clause. If a Dockerfile adds one, it overrides any found in the base container images.

So I just added my own CMD clause swapping out the && for a ; (Semicolon). A simple Dockerfile to work around this issue would look like this:

FROM octopusdeploy/tentacle:latest

CMD /scripts/configure-tentacle.sh ; /scripts/run-tentacle.sh

I tested this workaround in my Tentacle image. I started a container using my image, verified that it was found by my Octopus Server, then I rebooted the host machine. When it came back online the container restarted just fine and resumed its status as an Octopus Deploy Worker (and passed a Health Check).

6

Hi @OctopusSchaff

Thanks for circling back around on this and providing the workaround.
In the meantime I have talked with several of my colleagues in the Product team and they have amended the Readme for the OctopusDeployLabs repo and clarified the Support we provide from Octopus for Docker images.

In summary our OctopusDeployLabs repo is officially not supported and is provided as is. It’s there as a guide and can be taken and run with as our customers wish but we don’t go any further than that.

Our officially supported docker images come from Docker Hub and we fully support that. However this comes from a private repo on GitHub and therefore items like the official Dockerfile etc is not available. This applies to both Octopus Server and Tentacle.

I will continue to test the use case above and verify if we have the issue and we can provide an official fix as needed. We usually provide a public GitHub issue link and I can post that here when it’s available.

Apologies again for the confusion around our Docker support. Let me know if you need anything else from us.

Kind Regards,
Paraic

1 Like
7

Hi @OctopusSchaff
Just following up on this one, our Devs have let us know this issue is now fixed, and as you suspected the issue was around the restart feature.

You can see the GitHub notes here: https://github.com/OctopusDeploy/OctopusTentacle/issues/297

Thanks again for bringing it to our attention.

Kind Regards,
Paraic

8

Glad to hear it! Thank you for circling back.

By the way, the URL to the GitHub notes is a 404 link for me. Since there is no repository I can see in the OctopusDeploy project called OcotpusTentacle, I assume this is a private repro.

9

Hi @OctopusSchaff

The changes are fairly self-explanatory in the shell script.
Just adding a variable and making sure it doesn’t reconfigure.

   7  docker/linux/scripts/configure-tentacle.sh 

@@ -19,8 +19,9 @@ mkdir -p $configurationDirectory
mkdir -p $applicationsDirectory

if [ -f "$alreadyConfiguredSemaphore" ]; then
echo "Octopus Tentacle is already configured."
exit 1
echo "Octopus Tentacle is already configured. Skipping reconfiguration."
echo "If you want to force reconfiguration, please delete the file $alreadyConfiguredSemaphore and re-launch the container."
exit 0
fi

if [ ! -f /usr/bin/tentacle ]; then
@@ -196,4 +197,4 @@ registerTentacle
touch $alreadyConfiguredSemaphore

echo "Configuration successful."
echo "" 
echo ""

Kind Regards,
Paraic

10

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.