When building calamari using the build.cmd as described here (https://github.com/OctopusDeploy/Calamari), the artifacts directory doesn’t contain a package for Calamari.netfx
Octopus Server v2020.3.2 want’s this version of Calamari to deploy to a Windows Tentacle.
Calamari :
SHA: dc46e552b134ffe38f7cae1cc0c5eddd8ad749da
Tag: 13.0.10
Branch: release/2020.3
Hi @estubbs
Thank you for contacting Octopus Support. I’m sorry you are running into this problem.
Allow me to raise this issue internally and see if anyone has run across this yet. If not, I’ll attempt to reproduce the issue on my end.
Thank you for your patience while I investigate this issue for you.
Regards,
Donny
Hi @estubbs,
Thanks for getting in touch! I figured I’d jump in this thread quickly after getting some thoughts from an engineer here, and to assist Donny given the timezone differences between Australia and the rest of the world. 
In 2020.3 we changed the packaging for Calamari, where server now interanlly repackages the Calamari NuGet packages to optimize storage (due to NetCore support the installer would be pushing GBs without this).
Calamari.nupkg => Calamari.netfx.zip (and Calamari.Cloud => Calamari.Cloud.netfx) when it gets pushed to the target now. My understanding is that if you’re using the custom folder, and you put the Calamari and Calamari.Cloud nupkgs in there, then it should work as before.
I hope this helps, and let me know how you go or if you have any further questions going forward!
Best regards,
Kenny
Thanks for getting back to me. The explanation you provided makes sense. I checked the packages that we placed in the custom calamari folder, and they include both Calamari.nupkg and Calamari.Cloud.nupkg. Below is a list of files that we put there
Calamari.13.0.11-calmari-fix0001.nupkg
Calamari.Cloud.13.0.11-calmari-fix0001.nupkg
Calamari.CloudAccounts.13.0.11-calmari-fix0001.nupkg
Calamari.Common.13.0.11-calmari-fix0001.nupkg
Calamari.linux-arm.13.0.11-calmari-fix0001.nupkg
Calamari.linux-arm64.13.0.11-calmari-fix0001.nupkg
Calamari.linux-x64.13.0.11-calmari-fix0001.nupkg
Calamari.osx-x64.13.0.11-calmari-fix0001.nupkg
Calamari.portable.13.0.11-calmari-fix0001.nupkg
Calamari.win-x64.13.0.11-calmari-fix0001.nupkg
After placing those files in the proper location, and configuring octopus server to use that location, all of our Windows based deployments immediately began to fail with the error message very similar to “Unable to find Calamari.netfx.nupkg” (I don’t have the original error anymore, we had to roll back Octopus Server to a previous backup due to this problem).
Our other deployments to Linux targets worked fine and were using the custom built calamari packages (clearly seen in their deployment logs).
Hi @estubbs,
Thanks for keeping in touch! That doesn’t sound nice at all that this required you to roll back. My apologies for the unexpected issues you’ve hit here.
Another thing to note here from the same engineer is that as we move forward with migration of steps from Calamari to a separate module (which we’re calling Sashimi) custom versions will not be possible. I.e. the Calamari flavours that are being built out of Sashimi will get embedded into the server build and cannot be overridden. It won’t be possible to build your own custom versions of Sashimi packages.
I’m sorry it’s probably not the best news you were after, but please let us know if we can try to help with anything else in the future.
Best regards,
Kenny
Appreciate the offer to help in the future, but what about the current issue?
To be clear, I am reporting a bug which causes 100% of deployments to fail to any Windows target using the CURRENT version of Octopus Deploy. This bug does not occur in 2020.1.10, but it does occur in 2020.3.2.
We are only using a custom build of Calamari, due to another bug in Calamari which was reported on January 22nd, and is still not fixed. If we go back to the standard version of Calamari we have other problems which lead to production outages. I would link you to my original support request, but I can’t since it was communicated by email.
However another of your customers submitted a merge request to Calamari back in February (which hasn’t been approved yet). Their Merge request adds a feature to Calamari which enables a work around in some scenarios for the underlying issue. This wouldn’t solve the problem for us at all, but the feature they are implementing addresses the same underlying issue. https://github.com/OctopusDeploy/Calamari/pull/468
I would submit our own code change to Calamari that fixes this, but it does so by moving the critical part of the IIS steps to a background powershell job which can be polled, canceled, and retried. It’s a hack, and you wouldn’t want it in your code.
Hi @estubbs,
Thanks for following up and I appreciate you providing those details. I think I have dug up your original support request email. I see a couple of developers were looking into this one, which I’ll need to have a thorough look through and reach out specifically to the dev team here to get eyes back on this one. I’m sorry this being unresolved has caused you to go through these workaround hoops, and if we can get to the bottom of it directly I’m hoping that will prevent the need of the custom Calamari build altogether. We’ll certainly let you know soon of any progress going forward.
Best regards,
Kenny
Ok thank you, let me know if we can provide you with anymore information.
Hi @estubbs,
Thanks for keeping in touch, and my sincere apologies for the drop in communication here. I actually took the last few days off due to sickness.
Just to confirm we’re on the same page in regards to the issue you first reported on Jan 22, the email chain I’m looking at was in regards to deployments to IIS where Deploy a Package steps were intermittently spinning forever which was requiring you to cancel and redeploy at these times. Is this the underlying root issue you were referring to?
If so, I see the last correspondence was from my colleague Tom Williams who asked:
“Would you be able to enable metabase auditing for those machines so we can audit what actions to IIS are being performed. When it happens again, we should be able to view all the actions in Event Viewer. If you can export those events after the next occurrence and send them through we can investigate further.”
If not, would you be able to forward the email chain to support@octopus.com? I’m sorry for the hiccup here, I’d just like to make sure I get the facts and history straight to get to resolving this issue accurately.
Best regards,
Kenny
Thanks for getting back to us. It looks like the right email thread. Can we move this conversation to support email to discuss further?
Hi @estubbs,
Thanks for keeping in touch! Yes of course, if you’re able to export those events after the next time you hit this issue, as previously mentioned, feel free to email those through to support@octopus.com. I’ll be more than happy to dive into that one, and probably pull in some extra help from my team if needed.
Best regards,
Kenny
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.