Unable to use "Calamari.AzureScripting" with custom tentacle package directory

Hello all,

We’ve run into some trouble recently when using a custom build of Calamari on our servers. It seems that we’re unable to use any Azure scripting when at the same time using a custom calamari package directory.

We’ve currently got two outstanding PRs, one for adding a timeout for Calamari scripts, and the other to ensure the ‘netfx’ Calamari package is created by the cake script. So far this has served us well in our environments.

We’ve recently updated our servers to v2020.5.5, and when running Azure scripts, we get the failure
…failed with error 'No versions of the package ‘Calamari.AzureScripting’ could be found under the custom Tentacle package directory…

We noticed that this functionality is being migrated into the Sashimi repo, but even after building this locally and putting the calamari flavoured package into the custom package directory, the server requires ‘Calamari.AzureScripting.linux-x64’ (running a test docker image).

Is it possible to use a fallback in the server, so that if the AzureScripting package cannot be found, it can revert to the default server functionality instead? Any suggestions or help is greatly appreciated.

Thanks in advance,

Leon

Hey Leon,

Thanks for reaching out.

I am going to reach out to engineering and get you some answers as soon as I can. As they are in Australia we likely won’t hear back til tomorrow or early tomorrow morning.

I will update you as soon as I hear back or one of them will respond themselves.

Please feel free to reach out in the meantime with any other questions or concerns.

Thanks,
Jeremy

Hi Leon,

Thanks for getting in touch! Firstly I’d like to apologise for the inconvenience we’ve caused by what you’ve correctly pointed out is our migrating of this functionality to Sashimi. After briefly discussing this with an engineer here, I think the best way forward to come up with a plan would be to get a copy of your task log to give us a better idea of the context here.

To be as thorough as possible, could you add two variables, OctopusPrintVariables and OctopusPrintEvaluatedVariables, both with a value of True to your project, then create and deploy a new release? To send us the resulting task log, you can mark this thread as private, or email it to support@octopus.com where we can pick it up from there.

I look forward to hearing back!

Best regards,

Kenny

Thanks @jeremy.miller and @Kenneth_Bates - log have been attached, we’ve also uploaded our process json as well.

In terms of variables, they didn’t make it out to the logs indicate that the step fails while trying to send the AzureScripting package to the tentacle.

For this test we’ve got two defined (we’re just checking if the resource groups has been created, if not, we create it via Get-AzureRmResourceGroup and New-AzureRmResourceGroup).

  • ResourceGroupLocation: eastus
  • ResourceGroupName:octo-test-rg

In terms of the setup, we’re simply using the output from pull request #689 and mapping that a custom modules directory for Calamari.

ServerTasks-28.log.txt (14.8 KB)
server-process.json (4.5 KB)

Hi Leon,

Thanks for the update and additional information. Apologies for the delay over the weekend.

I raised this internally with an engineer here and I just wanted to give you a quick update that we’re still looking into this one. I’ll make sure to keep in touch, and please don’t hesitate to reach out if you have any questions in the meantime. :slight_smile:

Best regards,

Kenny

1 Like

No worries @Kenneth_Bates - appreciate the quick feedback, if there is any further testing or info we can provide please let me know.

Hi Leon,

Thanks for your patience on this one. I’m sorry to give you some bad news, but after looking into this one in more depth with a couple engineers, it looks like this just won’t be possible. Sashimi is essentially pulling a lot of the guts out of Calamari, and with everything moving around, custom Calamari doesn’t do anything now, and it would have to be a custom Sashimi. Problem there is it’s not currently possible to use a custom Sashimi at the moment, and a solution to that problem will still be some ways off.

However, I’m hoping we can eventually give you some good news. Am I correct in assuming the Add timeout argument to run-script command PR was the one that initially caused you to require a custom Calamari? We’re having a good look at that PR currently, so my hope is that we might be able to simply negate the need for any custom Calamari/Sashimi that way. Please let us know if there’s anything else relevant here that required your custom Calamari.

I look forward to hearing back!

Best regards,

Kenny

Thanks @Kenneth_Bates - yes you’re right, the timeout is our only customization that we require.

my hope is that we might be able to simply negate the need for any custom Calamari/Sashimi that way

That would be great for us too as it would allow us to drop the overhead of the custom package. I’ve seen that one of the engineers has already responded to the PR so we’ll follow up.

Thanks again.

Hi Leon,

That’s not a problem! Please let us know if you have any questions or concerns moving forward. :slight_smile:

Best regards,

Kenny

Hi Leon,

Thanks so much for your patience. I have reviewed the PR you’ve submitted regarding adding a timeout to the script execution.
As part of that review, I think I may have a better solution for you to deal with this problem you are facing.

So, it seems you are calling the old Azure RM cmdlets, specifically the Get-AzureRmResourceGroup which according to its documentation does not support passing a “timeout” to it :slightly_frowning_face:
But I think we can solve this with PS using jobs.

Here is an example:

$Job = Start-Job -ScriptBlock {

    $resourceGroup = Get-AzureRmResourceGroup | ? { $_.ResourceGroupName -eq $ResourceGroupName }

    if(!$resourceGroup)
    {
      Write-Host \"Creating Resource Group: $resourceGroupName in $resourceGroupLocation\"
      $resourceGroup = New-AzureRmResourceGroup -Name $resourceGroupName -Location $resourceGroupLocation
    }
}

$Job | Wait-Job -Timeout 30
$Job | Stop-Job

What do you think? Will this solve the issue you are facing?

Regards
John

Hey @John_Simons - thanks for the investigation, unfortunately I think there is a little bit of confusion here.

The reason we’ve authored the timeout script is due to some deployments getting stuck during high load, the investigation and details are in PR #468.

We re-worked this fix in our latest PR #689 and as a result of using a custom Calamari modules directory, now have a separate issue with Azure deployments (this one). So the two are not related per-se, but since we need the custom timeout functionality, we’re faced with the Calamari.AzureScripting dependency.

I’ve been on holidays this week but return to work tomorrow and hope to be able to review the comments on PR #698.

Thanks again to all who are helping and the quick responses.

Hi Leon,

Thanks for the follow-up. Just a quick acknowledgement to let you know I’ve passed this along, and we’ll await your response to the PR review. :slight_smile:

Kind regards,

Kenny

Thanks again all - have responded in the PR. As discussed, the source of the issue that required us to write a timeout argument can’t be resolved in script. Have put some more details of our investigation in the GitHub PR.

Just to update this thread, thanks for creating this issue to deal with the timeout. Once this is resolved it allow us to go back to using the out-of-the-box Calamari that are deployed with the server.

In the meantime, is there a possibility to also have a fallback on the server, so that when using custom Calamari packages, if a “default” package cannot be found in the custom modules directory (like the netfx or Calamari.AzureScripting), that the server deploys it’s version as usual.

This could allow using customized packages while the Sashimi solution is built out.

Many thanks

Hi Leon,

Thank you for keeping in touch! I’m sorry to say that I don’t think it will be possible to fall back to the out-of-the-box, non-custom version of Calamari, since this is set at the server level. My apologies that’s not the news you’re after, but please let us know if you have any further questions moving forward.

Best regards,

Kenny

No worries @Kenneth_Bates - thanks again for all the time/help on this one.