Kubectl apply -o json” returned invalid JSON

Octopus Server
,
1

Hi,

We are trying to configure a ‘DEPLOY RAW KUBERNETES YAML’ . The step fails with error:

“kubectl apply -o json” returned invalid JSON:
This can happen with older versions of kubectl. Please update to a recent version of kubectl.

Running the command manually in powershell works without a problem.
kubectl version
Client Version: version.Info{Major:“1”, Minor:“23”, GitVersion:“v1.23.7”, GitCom
mit:“42c05a547468804b2053ecf60a3bd15560362fc2”, GitTreeState:“clean”, BuildDate:
“2022-05-24T12:30:55Z”, GoVersion:“go1.17.10”, Compiler:“gc”, Platform:“windows/
amd64”}
Log attached:
ServerTasks-184576.log.txt (27.0 KB)

3

Hey @james.cook,

Thanks for reaching out to Octopus Support and welcome to our community forums!

Would you be able to upload a copy of your process JSON so that we can see how the step is filled out and generating that JSON?
You can upload to our secure file hosting platform here: Support Files | james.cook.
There’s currently a GitHub issue that shows a similar output to the error you’re receiving: Yaml Editor - Deploy Kubernetes Step - Export with empty runAsGroup causes validation error · Issue #7437 · OctopusDeploy/Issues · GitHub, however I do not see any empty values in the JSON, I wondered if you could verify the same for me, please?

Let me know if you’re able to send over the process JSON, or if the above issue link triggers any ideas.

Kind Regards,
Adam

4

Hi @adam.hollow,
I have uploaded the OCL process to your dropbox.
We don’t have any null parameters in our deployment yaml.

Thanks
James

5

Hi @james.cook,

Thanks for sending that over.

I’ve had a look and I can see that one of the YAMLs for the deployment is obtained via a package, would you be able to upload the YAML from the package also, please?

Kind Regards,
Adam

6

Hi Adam,

That has been uploaded.

Thanks
James

9

Hi @james.cook,

Just stepping in for Adam from the Australia based team to confirm we have received the files ok!

I’ll dig into this and see if I can figure out exactly what’s going on, but my initial suspicion is that it’s the following section at the end of the deployment.yml file:

status:
	loadBalancer: {  }

It seems that it might not be valid for Octopus as I’m unable to save my deployments when adding this section. Could you please try removing it and let us know if it’s still giving the error?

I’ll keep you posted with any updates on my end and if there are any issues raised for this, let me know if you have any questions at all!

Best Regards,

10

Hi Finnian,

We have tried removing the lines from the deployment.yml
We have also tried setting up as an embedded script
We have attempted with different deployment.yml which are very simple, a single pod.
All return the same error ‘Invalid JSON’ This seems like an issue with octopus not being able to parse the JSON output.
We are running 2022.1 (Build 2661)

Thanks
James

11

Hey James,

Are you able to upload the deployment process for the simple yaml project as it sits in the database so we can take a look please? It will be in table dbo.DeploymentProcess and you can match on the OwnerID for the project. We would need the JSON portion of the latest “Version” that’s in the table.

You can either DM it to me or upload it to the same location and let us know. We’re mainly looking to see if there is anything in the DB that is unexpected data, like extra spaces, nonprintable characters, basically anything that isn’t exactly what we expect in that yaml step.(if you want to take a look yourself as well.)

Looking forward to hearing back.

Best,
Jeremy

12

Hi @james.cook,

Just a quick update, I’ve confirmed it’s not because of my previous suggestion, and I haven’t been able to reproduce this and on my end even when using the same Octopus version. For me the deployment.yml file worked ok so I definitely think something strange is going on.

I’ll keep exploring and see if I can get to the bottom of this, however if you could send through that dbo.DeploymentProcess table’s JSON as @jeremy.miller mentioned, that should allow us to copy your exact process and hopefully reproduce this.

It seems to be implying the error is because of invalid JSON when it translates the seperate resources into a List, I wonder if you could potentially workaround the issue by separating each resource into its own Step?

I’ll keep you posted with any further updates, let me know if you have any issues or questions!

Best Regards,

13

Hi,

I work with @james.cook. I’ve set up a demo project which just deploys a simple nginx container. This works on the command line without issue with kubectl but Octopus still complains with the same error on parsing the output.

I’ve captured the output from a command line run and can send it over if it’s any help.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-demo
  namespace: ournamespace
  labels:
    app: nginx-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-demo
  template:
    metadata:
      labels:
        app: nginx-demo
    spec:
      imagePullSecrets:
        - name: ourregistry
      containers:
        - name: nginx-demo
          image: ourregistry.ourcompany.local/nginx:1.23.1
          imagePullPolicy: Always
          ports:
            - containerPort: 80
              name: http
14

Hey @beirtipol,

Welcome to the Octopus forums and thank you for posting up your demo project you have been working on. This is definately strange if Finnian cant replicate it but you can with two separate projects.

Any information we have is always beneficial so if you have the output from the command line run we would love to see it!

I have created you a secure link - Support Files | beirtipol

If you wanted to upload your file there we can take a look, are you able to let us know when its been uploaded as we are not notified when a customer uploads to our secure file site.

Also, as mentioned if we can get the dbo.DeploymentProcess table’s JSON for both projects, that would be really handy too, I realise this is happening on two separate projects but it would be good to see how those were set up in Octopus.

Kind Regards,

Clare

15

Hi Clare,

I have uploaded the deployment process table json for the original project to the dropbox.

Thanks
James

16

Hi @clare.martin I’ve uploaded the deployment process table json for the nginx demo project. I’ve also included the raw k8s.yaml and output.json from the command line run of kubectl

Thanks,
B

17

Hey @beirtipol,

Thank you for uploading those, Finnian is currently offline as part of our Australian team and I know he has worked on this and has a reproduction, or at least he has tried to reproduce without success.

Would you be happy to wait until he comes back online later on tonight so he can take a look, it may be easier to have one dedicated Support person working on this with you?

I think it would also be beneficial for us to have that deployment process JSON so we can really dig into this and try to recreate the problem. @james.cook if you are not busy at some point today are you able to get that to us along with the one from Beirtipol’s test? You would need to use your support files link Adam posted above though to upload it.

Let me know if its possible to get the JSON and we can gather all the logs together and hopefully have a better idea of how to reproduce this.

Kind Regards,

Clare

18

Hi @clare.martin. I think we have both uploaded what’s needed already?

19

Hey @beirtipol,

You are correct I do apologise, thank you hopefully we have all we need now in order to hopefully try and reproduce this, I will let Finnian know all the files are there.

We will keep you updated on any progress,

Kind Regards,

Clare

20

Hi @beirtipol,

Thanks for sending those latest files through!

Unfortunately I haven’t had any luck reproducing any issue on my end, even when using the exact same process and versions of Octopus/kubectl it works ok on my end. This is really strange indeed!

Would you be able to please also send through the latest task logs showing the error for that simple example ‘raw-k8s.yml’ file?

I’ve raised this with the devs and will keep you posted with any updates or suggestions from them, however it might be best if we jump on a call together to see if we can figure out what’s going on live.

Let us know a time/timezone that would suit you best and I’ll get it scheduled!

Best Regards,

22

Hi,

Are you able to process the output.json that I attached earlier? I’m wondering if there’s something in the response from our K8s cluster which Octopus doesn’t like. Perhaps the JSON embedded in the JSON?

23

Good afternoon @beirtipol,

Thank you for that suggestion, Finnian will be online later and will look into that, he has been racking his brain trying to get this to break (as yours does) and is pretty stumped so hopefully your information may lead him into something.

He will get you an update when he is next online.

Kind Regards,

Clare

25

Hi @beirtipol,

Are you able to process the output.json that I attached earlier?

Just confirming that file is coming from running a command like kubectl apply -f raw-k8s.yml -o json ran from outside of Octopus? This seems fine to me and is valid JSON, however it isn’t indicating a success like I’m seeing on my end with the additional line "availableReplicas": 1,

I’m wondering if there’s something in the response from our K8s cluster which Octopus doesn’t like. Perhaps the JSON embedded in the JSON?

I thought something like this could be going on too, however all my testing and attempts to break it are either working fine or giving more informative errors.

Looking at the code, the error is being thrown due to the output from the kubectl apply -f <file>.yml -o json command not being able to be successfully parsed by ConvertFrom-Json, however when I use the JSON supplied with the error message, it’s able to parse it ok.

A teammate managed to find a previous forum post with what looks like it could be a very similar issue: Failure deplying Kubernetes CronJob Yaml

The resolution there was found by using the Run a Kubectl Script step from inside Octopus to deploy the YAML, which ended up revealing what the actual error was. This issue with the error not being shown should have been fixed in v2019 however, but I definitely think it would be worth trying it to see if another error is shown.

Let me know how you get on or if you have any questions!

Best Regards,