I’m seeing intermittent failures when running PowerShell script steps in Azure. I’ve noticed that there are a couple of existing issues on the forum but can’t see from those that there is a definitive answer about what causes this issue.
When running Azure PowerShell steps, I’m seeing them intermittently fail with the following error: Calamari.exe : Invoke-AzureRmResourceAction : Your Azure credentials have not been set up or have expired, please run
Retrying the deployment will successfully run the step, but will then fail running a later script in the process. It’s currently taking around 4-7 retries (excluding previously completed steps each time) in order to complete a deployment.
We’ve got a couple of solutions that face this issue on almost every deployment and another that seems to not face the issue at all any more (having experienced it previously). One of the steps is identical across two of these pipelines, one has the issue and one doesn’t, so I don’t think it’s entirely an issue with the script.
I’d be grateful for any assistance in diagnosing and resolving this issue.
When you look at the task logs of the deployment with the successful retries, do the workers used differ from the ones used when you the step fails? If that’s the case, we may need to figure out how these workers differ to isolate what might be causing the issue.
I’m by no means an expert with Octopus, so apologies if I’m being dim…
I’ve had a look through the verbose task log and I can’t see anything relating to a worker on the affected steps. I’ve had a quick look at the Workers documentation and I would assume that all of these steps are being handled by the built in worker because I can see that we don’t have any of our own workers configured.
Does that make any sense or am I missing something obvious?
If you haven’t configured any extra workers, then you are correct it would just be the built in worker that comes with the server itself. Would you be able to privately message me a task log of a failure and a task log of a success so I can take a look?
For reference you can check if you have any workers by going to the Infrastructure tab, then Workers on the left. You can also tell which worker is being used in a task log by searching on the word “lease” and it will give you the ID of the worker doing the task, but this won’t be too helpful if you don’t have any dedicated workers.
I think the way we may need to go is have you install the most up to date version of az , set up aliasing, then configure the step to not use the bundled azure tools.
Do all of this as the service account that runs the Octopus Server service, that way we are sure that the worker will have access to everything.