We’ve run into some trouble recently when using a custom build of Calamari on our servers. It seems that we’re unable to use any Azure scripting when at the same time using a custom calamari package directory.
We’ve recently updated our servers to v2020.5.5, and when running Azure scripts, we get the failure …failed with error 'No versions of the package ‘Calamari.AzureScripting’ could be found under the custom Tentacle package directory…
We noticed that this functionality is being migrated into the Sashimi repo, but even after building this locally and putting the calamari flavoured package into the custom package directory, the server requires ‘Calamari.AzureScripting.linux-x64’ (running a test docker image).
Is it possible to use a fallback in the server, so that if the AzureScripting package cannot be found, it can revert to the default server functionality instead? Any suggestions or help is greatly appreciated.
I am going to reach out to engineering and get you some answers as soon as I can. As they are in Australia we likely won’t hear back til tomorrow or early tomorrow morning.
I will update you as soon as I hear back or one of them will respond themselves.
Please feel free to reach out in the meantime with any other questions or concerns.
Thanks for getting in touch! Firstly I’d like to apologise for the inconvenience we’ve caused by what you’ve correctly pointed out is our migrating of this functionality to Sashimi. After briefly discussing this with an engineer here, I think the best way forward to come up with a plan would be to get a copy of your task log to give us a better idea of the context here.
To be as thorough as possible, could you add two variables, OctopusPrintVariables and OctopusPrintEvaluatedVariables, both with a value of True to your project, then create and deploy a new release? To send us the resulting task log, you can mark this thread as private, or email it to support@octopus.com where we can pick it up from there.
Thanks @jeremy.miller and @Kenneth_Bates - log have been attached, we’ve also uploaded our process json as well.
In terms of variables, they didn’t make it out to the logs indicate that the step fails while trying to send the AzureScripting package to the tentacle.
For this test we’ve got two defined (we’re just checking if the resource groups has been created, if not, we create it via Get-AzureRmResourceGroup and New-AzureRmResourceGroup).
ResourceGroupLocation: eastus
ResourceGroupName:octo-test-rg
In terms of the setup, we’re simply using the output from pull request #689 and mapping that a custom modules directory for Calamari.
Thanks for the update and additional information. Apologies for the delay over the weekend.
I raised this internally with an engineer here and I just wanted to give you a quick update that we’re still looking into this one. I’ll make sure to keep in touch, and please don’t hesitate to reach out if you have any questions in the meantime.
Thanks for your patience on this one. I’m sorry to give you some bad news, but after looking into this one in more depth with a couple engineers, it looks like this just won’t be possible. Sashimi is essentially pulling a lot of the guts out of Calamari, and with everything moving around, custom Calamari doesn’t do anything now, and it would have to be a custom Sashimi. Problem there is it’s not currently possible to use a custom Sashimi at the moment, and a solution to that problem will still be some ways off.
However, I’m hoping we can eventually give you some good news. Am I correct in assuming the Add timeout argument to run-script command PR was the one that initially caused you to require a custom Calamari? We’re having a good look at that PR currently, so my hope is that we might be able to simply negate the need for any custom Calamari/Sashimi that way. Please let us know if there’s anything else relevant here that required your custom Calamari.
Thanks @Kenneth_Bates - yes you’re right, the timeout is our only customization that we require.
my hope is that we might be able to simply negate the need for any custom Calamari/Sashimi that way
That would be great for us too as it would allow us to drop the overhead of the custom package. I’ve seen that one of the engineers has already responded to the PR so we’ll follow up.
Thanks so much for your patience. I have reviewed the PR you’ve submitted regarding adding a timeout to the script execution.
As part of that review, I think I may have a better solution for you to deal with this problem you are facing.
So, it seems you are calling the old Azure RM cmdlets, specifically the Get-AzureRmResourceGroup which according to its documentation does not support passing a “timeout” to it
But I think we can solve this with PS using jobs.
Hey @John_Simons - thanks for the investigation, unfortunately I think there is a little bit of confusion here.
The reason we’ve authored the timeout script is due to some deployments getting stuck during high load, the investigation and details are in PR #468.
We re-worked this fix in our latest PR #689 and as a result of using a custom Calamari modules directory, now have a separate issue with Azure deployments (this one). So the two are not related per-se, but since we need the custom timeout functionality, we’re faced with the Calamari.AzureScripting dependency.
I’ve been on holidays this week but return to work tomorrow and hope to be able to review the comments on PR #698.
Thanks again to all who are helping and the quick responses.
Thanks again all - have responded in the PR. As discussed, the source of the issue that required us to write a timeout argument can’t be resolved in script. Have put some more details of our investigation in the GitHub PR.
Just to update this thread, thanks for creating this issue to deal with the timeout. Once this is resolved it allow us to go back to using the out-of-the-box Calamari that are deployed with the server.
In the meantime, is there a possibility to also have a fallback on the server, so that when using custom Calamari packages, if a “default” package cannot be found in the custom modules directory (like the netfx or Calamari.AzureScripting), that the server deploys it’s version as usual.
This could allow using customized packages while the Sashimi solution is built out.
Thank you for keeping in touch! I’m sorry to say that I don’t think it will be possible to fall back to the out-of-the-box, non-custom version of Calamari, since this is set at the server level. My apologies that’s not the news you’re after, but please let us know if you have any further questions moving forward.
The developers are wondering if you’re interested in scheduling a screen-share to dig a bit deeper into this issue on your end?
Whilst I have reproduced some sort of issue with the specific set-itemproperties cmdlet, the developers are not entirely certain that it’s the cause here. They would like to get some eyes on the issue in person to confirm.
Would you be free to schedule a screen-share? If so, let me know and I’ll email you a link to schedule a time that works for you.
I agree, it would seem Set-ItemProperties isn’t related to this issue (at least not directly for us). From our investigations it seems to be tied directly to the IIS libraries (i.e. IISWebSite_BeforePostDeploy).
I’ve spoken with my team lead, would it be possible to get idea of what specifics around the issue you’d like to get eyes on before setting up a screen share? E.g. proc dump info from a blocked deployment? something else ect?
Hi Leon,
I’m jumping in to pass along our engineer’s request as Daniel’s off for the rest of the week.
Initially, our engineers would like to take a step back and review your environment’s configuration and verify the evidence that has been compiled to date. They’re hoping that walking through all the details with you will surface something that’s getting overlooked. Indeed, a proc dump from a blocked deployment could be helpful.
I hope that helps, but please don’t hesitate to let us know if you need more information beyond what I have provided.
