Octopus attempts to make a Namespace before running Helm

I am attempting to get an Upgrade a Helm Chart step working. I set it up and it is very close. But I am seeing this in log output:

Creating kubectl context to https://myKubeCluster:443 (namespace default) using a Token 
User "octouser" set. 
Error from server (Forbidden): namespaces is forbidden: User "myLimitedUser@mydomain.com" cannot create resource "namespaces" in API group "" at the cluster scope 
Invoking target script "/etc/octopus/Work/20210428024226-122445-125/staging/Calamari.HelmUpgrade.sh" with  parameters 
Release "my-kube-release-name-here" does not exist. Installing it now. 

The key line in here is the third one. (The error attempting to create a namespace.) I have setup my user that runs deployments to not have permissions to create namespaces to restrict damage it can cause. So I see why any attempt to create a namespace would fail.

But I am confused as to what is causing this to happen as my Helm chart does not create a namespace.

It seems that this is a action that Octopus Deploy is taking on its own. I arrive at this conclusion because of the following:

  1. The error log message precedes the line saying that it is going to invoke the script that runs the Helm Chart. (And the line saying that it is going to create the release).

  2. When I login as my reduced permissions user and manually run the helm chart (using the same command as is output in the logs), it runs without issues. (No namespace related errors.)

So it seems that the failure to create the namespace is before the helm chart update is started. But I am confused as to why Octopus would need to create a namespace in order to run the upgrade step. (The only step in my deployment is the one Upgrade Helm Chart step.)

What is going on and how can I get around this error?

Hey @OctopusSchaff

I wanted to let you know that we’re looking into this for you. We’ve reached out for developer input so we should have an explanation for you soon.

Thank you for your patience!

OK, I got to the bottom of this one. Here is what is happening:

  1. Octopus Deploy runs a script called Octopus.KubectlPowershellContext.ps1 during deployment (this is an Octopus Deploy created script).
  2. On line 315 it calls the method CreateNamespace
  3. The goal of CreateNamespace is to ensure that the namespace listed on the Deployment Target really exists. If it does not exist then it tries to create it.
    • I left the namespace on the deployment target blank. I have specified my namespace in my Upgrade a Helm Chart step.
    • Because I left the namespace on the deployment target blank, the Octopus.KubectlPowershellContext.ps1 script sets this to the namespace called default.
    • The user associated with my Deployment Target has permissions to work in very few namespaces (and default is NOT one of them). And it does not have permissions to list ANY namespaces.
  4. CreateNamespace does a kubectl get namespace $K8S_Namespace where $K8S_Namespace is the namespace for the Deployment Target (default in this case). It does this in a Try/Catch block.
    • The call to check the namespace fails due to insufficient permissions, which causes the execution to move to the catch block.
  5. The catch block then tries to create the namespace (which in my case it does not have permissions to do).

My workaround to this issue is:

  1. Create a namespace called octopus-deploy-target
  2. Create a Role in that namespace that has the permissions to get the namespaces (See this Stack Overflow question for details)
  3. Create a RoleBinding in that namespace that attaches that permission to the deployment target’s user.
    • This allows the user to list just that one namespace (and no others).
  4. Set the Deployment Target to use the octopus-deploy-target namespace.
  5. Now when CreateNamespace tries to check if the namespace exists, the permissions are there for it to check it.

In my opinion, the Octopus.KubectlPowershellContext.ps1 script should check to see if it got a permissions error when it checks the namespace, and if so, it should trust that the namespace is there. (If it needs to be there and is not, other things will fail later.)

As an additional feature, you could add a “Test it Out” kind of button to the Deployment Target that will ensure that the user defined there has the permissions to check the namespace defined there (default or whatever is entered). That way you could warn the user that they need to make a change permissions change with the account they are using.

1 Like

Hey @OctopusSchaff

Thank you for such a thorough investigation along with a viable workaround, I would agree that definitely seems to be what’s going on.

Our engineers are still discussing the best option for handling this but your suggestion of a “Test It Out” button definitely seems like a viable option. Thanks again for the suggestion!

We are also looking at changing our logging to distinguish between “a namespace not existing” vs “insufficient permissions” so that the logging message is more informative, as well as failing more appropriately depending on the result.

Sorry for not being super helpful but we appreciate you bringing this to our attention. Hopefully this will be resolved in the near future.

Regards,

1 Like