Hi
I recently connected a second worker in my test environment but their seems to be a problem when executing the the runbook. Both workers show in the logs but the details of worker 2 is a duplicate of the first. I confirmed this by enabling the env variable that stores a copy of the scrips locally, the scripts of both were on the first worker
Hi Shaun,
Thanks for reaching out, and sorry to hear that youâre having issues with your worker.
Can you send through a task log showing this in action, so we can take a look and try to gather some more information? Feel free to upload it HERE, and weâll investigate further.
Look forward to hearing from you soon!
Hey @shaun.a.havelaar,
Just jumping in for Justin who wont be online until later on today, thank you for uploading those logs, I cant see much that sticks out but there is an error here:
14:39:42 Verbose | User "xxxxxx" set.
14:39:42 Verbose | "C:\\ProgramData\\chocolatey\\bin\\kubectl.exe" get namespace xxxxxx --request-timeout=1m
14:39:42 Verbose | error: You must be logged in to the server (Unauthorized)
14:39:42 Verbose | "C:\\ProgramData\\chocolatey\\bin\\kubectl.exe" create namespace xxxxx --request-timeout=1m
14:39:42 Verbose | error: You must be logged in to the server (Unauthorized)
14:39:42 Verbose | Could not create namespace. Continuing on, as it may not be working directly with the target.
error: You must be logged in to the server (the server has asked for the client to provide credentials)
From what I can see you are correct in that worker 1 and worker 2 are the exact same machines, which I assume is why you are getting the logon error. In the first task you are logged in with your user and it gets the namespace, this is acquired at 14.39.43 and you can see in the above error that its trying to get the namespace at 14:39:42 so I assume the tasks are clashing with each other and causing it to error out as you are already using the user in step 1 to perform the tasks whilst the same user in step 2 is trying to do the same.
It looks like you are using a worker pool called Local Machine, how many workers are in that pool, is your new worker in that pool? It is a bit odd that Octopus is running two tasks to the same worker.
I can see you are running Octopus 2023.2.5010 which means you are on our Octopus Cloud offering, would you be willing to allow us to create a support user on your instance please so we can take a look at how this runbook is setup and the worker pool and I can see if I can figure out why the runbook is running twice to the same worker. If you would prefer you could send the process json to the same link Justin sent to our support files site and we can take a look but it would be easier for us to have a look at your instance to troubleshoot.
If we can access your instance are you able to post up the URL of the runbook so we can make sure we are looking at the right runbook.
I look forward to hearing from you,
Kind Regards,
Clare
Yes that is correct, I was testing this locally and wanted to demo it to a coworker by adding his machine as a second worker.
Sure that wonât be a problem for me, just let me know how you want to proceed in sharing credentials for the support user. Please just note that I have disabled the second worker for now, but whoever looks at it can change things as they please (Iâm mostly doing research at this point)
Kind regards
Shaun
Hey @shaun.a.havelaar,
Great thanks for confirming we can create a support user on your instance, you do not have to do anything regarding that, we can do it all from our backed system but I did need your permission to logon to your instance.
With that in mind can you post up the URL of the runbook please (this will give me the runbook I need to look at and also the URL of the instance so I can create the support user on it).
I look forward to hearing from you,
Kind Regards,
Clare
Hey @shaun.a.havelaar,
Thanks for the runbook link, I will create a support user after my lunch break and see if I can spot what the issue is, I will get back to you once I have more information.
Kind Regards,
Clare
Hey @shaun.a.havelaar,
Thank you for your patience whilst I looked into this for you, it seems the logs are a tad misleading in that both tasks are using the same worker but they are executing on different targets, you can see this in your UI here:
It is actually fine for Octopus to use the same worker for two deployments and the issue you are seeing is not because of that.
It looks like the credentials on the Stephan target are incorrect for your kubectl token as you can see the error I posted in my previous post and also this in a previous failed runbook deployment in the Stephen logs:
14:37:39 Info | Creating kubectl context to https://kubernetes.docker.xxxxxxx (namespace xxxxxxx) using a Token
14:37:39 Verbose | "C:\\ProgramData\\chocolatey\\bin\\kubectl.exe" config set-credentials xxxxx --token=<token> --request-timeout=1m
14:37:39 Verbose | User "xxxxxxxxx" set.
14:37:39 Verbose | "C:\\ProgramData\\chocolatey\\bin\\kubectl.exe" get namespace honeycomb --request-timeout=1m
14:37:39 Verbose | E0421 16:37:40.282921 39620 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
I would check the Shaun target against the Stephen target and ensure the kube config files and tokens are the same as it looks like your Shaun target has always deployed successfully. I did find this on google which may help. If you run the kubectl get nodes command directly in PowerShell on the Stephen target does it bring back the same unauthorised error?
As for the deployment using the same worker, if you wanted to change this behaviour you would need to split your workers into different pools and then set your Deployment.Pool variable in your deployment variable set to have it use a different worker pool for different environments:
What will happen now is when you deploy and you want to use the Stephen worker you deploy the runbook to the Env-Stephen environment which will then use the Stephen worker and will run on the Stephen target and the Shaun target as you have the deployment running on behalf of the âserverâ role which both machines are part of.
Let me know if that helps, I am not a K8 expert so am unable to direct you to a better website for the errors you are seeing but it does look like there is some different settings between your Shaun target that is working and your Stephen target which is not. The error does not seem to be related to having the same worker run both deployments as I previously thought.
On another note, it looks like your Stephen target is currently disabled just in case you were not aware. Also, we do not recommend using workers as deployment targets as there can be complications if they are setup to use both, it can be done, its just not recommended so I thought I would mention it in case you are wanting to stick with this setup. We have some documentation on this here if you have not seen it yet, it mentions listening tentacles and yours are K8 targets so I am not sure if we support the use of those as workers.
Kind Regards,
Clare
Okay, that makes a lot of sense (now that I thoroughly went through the documentation - insert slap face emoji), I completely misunderstood the purpose of a âworkerâ.
So just for the sake of coming to a conclusion for anyone reading this in the future: I finally realized that I needed to create a worker pool for every machine that I want an instance of my deployment to run on. I could easily link my new deployment pool to the second target and everything just worked as expected from there on.
Thank you for taking me through all the steps @clare.martin, really appreciate the detailed support you provided.
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.
