Thanks for that, confirming I have received it ok!
It looks like this is happening when running scripts on the Built-In Worker and the script is exiting just prior with code 0:
10:29:36 Verbose | Process C:\WINDOWS\system32\WindowsPowershell\v1.0\PowerShell.exe in d:\Octopus\Work\20221122232934-1547-1 exited with code 0
10:29:38 Verbose | Bootstrapper did not return the bootstrapper service message
I think it’s likely the new version of Calamari (by default under C:\Octopus\OctopusServer\Tools\Calamari.win-x64) is being quarantined by AV, which is likely what’s changed. Otherwise it could be that the Octopus Work folder has changed (to d:\) and not been added to the AV exclusion list?
If you’d like to also send through the Octopus Server Logs for this timeframe, I’ll check to see if I can spot any obvious issues, otherwise it might require checking over Windows Events, or taking a dump of the Octopus.Server and Calamari processes to see exactly what’s going on.
Let me know how you get on or if you have any questions at all!
No problem at all, looking forward to hearing how you get on!
It will depend on your specific Antivirus, but as long as the Tool and Work folders have been excluded it should just work, there won’t be any additional configuration required in Octopus.
Bonus: Our Script Console feature allows for quickly running on off scripts, which I find great for testing!
Feel free to reach out if you have any questions or run into any issues at all!
I work with Andrew. We have verified with our network team to confirm that our AV is not excluding Calamari packages or anything relating to Octopus Deploy.
However, I’ve managed to upload the dump files for Octopus Server and Tentacles.
Can we also confirm the end point that’s used to download Calamari package during the upgrade process?
Thanks for stepping in and uploading those dump files, confirming we’ve received them ok!
Just to confirm, your Network team have added the following folder exclusions to your AntiVirus and are still seeing this same issue with the Bootstrapper? Are there any AV or Windows events indicating that a process was blocked?
Thanks for that, it definitely seems likely that the Calamari extraction has been quarantined if it’s not present in that folder. It gets extracted from a zip which resides in the C:\Program Files\Octopus Deploy\Octopus folder when a task is run on the Server directly or Tentacle.
Looking forward to hearing what your Network team have to say, feel free to reach out with any questions!
Thanks for the update, were they able to locate any events in the AV logs indicating that any executables are being blocked or files being quarantined?
Just to clarify, Calamari shouldn’t require installation as a service, just to be extracted to the C:\Octopus\OctopusServer\Tools folder. This error regarding the Bootstrapper is implying that something is preventing that extraction from occurring, often because of blocking an executable.
We’ve found that the best steps for testing this is to disable AV completely to see if re-running the script allows it to execute successfully. If it’s still giving the same error after disabling AV, we will need to figure out what is blocking the extraction of Calamari e.g. by consulting windows events.
Is the Calamari Zip present in the C:\Program Files\Octopus Deploy\Octopus folder?
Let me know how you get on or if you have any questions at all, if it’s still giving issues after disabling AV we’d be happy to jump in a call to see if we can help spot whats’ going on!
That Calamari zip will get extracted when a Deployment is run against a Target or Worker rather than when the Windows Service is started.
Octopus will check for the same Calamari version it’s bundled with and extract it if it’s not present, using the account that Octopus/Tentacle is running under. This uses both the Work and Tools folders and subfolders but as long as same Calamari version is present in the Tools folder as is configured for that version of Octopus it should work without needing to extract it, allowing you to manually copy Calamari into that location if required.
Logs showing Calamari Extraction :
Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory 'C:\Octopus\Work\20221201013631-159-8' using 'OEM United States' encoding running as 'NT AUTHORITY\SYSTEM'
Checking for C:\Octopus\Tools\Calamari.win-x64\24.0.24\Success.txt
Package: C:\Octopus\Work\20221201013631-159-8\Calamari.win-x64.24.0.24.nupkg
Destination: C:\Octopus\Tools\Calamari.win-x64\24.0.24
411 files extracted
Calamari.win-x64 24.0.24 extracted to C:\Octopus\Tools\Calamari.win-x64\24.0.24
Removing write permissions on C:\Octopus\Tools\Calamari.win-x64\24.0.24 for BUILTIN\Users
Cleaning up old Calamari.win-x64 versions...
Keeping only Calamari.win-x64 version 24.0.24...
Found no old versions of Calamari.win-x64 to delete...
Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20221201013631-159-8 exited with code 0
Version 24.0.24 of the Calamari.win-x64 tool has been extracted successfully
Let me know if I can help with any further questions!
Back in 2020.6 the Calamari folder was called Calamari.netfx rather than calamari.win-x64 and so there doesn’t look like there were any issues with the old installer.
Sorry I should have specified the extracted Calamari folder can be copied from one instance to another instance, but the extraction will need to be done by an Octopus Server originally in order to map the files correctly. If you have another instance/tentacle which does have a calamari.win-x64 folder then you can copy it however unfortunately I don’t think that will resolve the issue and you’d likely see the same error message as it will still try to run the bootstrap script to check if it needs extracting.
If after disabling AV it’s still not working then that will indicate that there is some other environmental problem preventing it which we’ll most likely need to help hunt out during a call.
Thanks for your help today, we’ve progressed a bit further after updating some registry permissions that event log error pointed us at.
We will probably rebuild the server regardless as that error really indicates something is up with the server.
But after making the fix we found our tentacle correctly updated and has the calamari x64 folder. And scripts run by the tentacle work!
But scripts run on the Octopus server still have the issue. We tried copying the folder from the tentacle server to the octopus server. But it didn’t help L is there a trick to making it work after you copy in the folder manually?
Just confirming that this is a Tentacle installed on another instance? If so and it’s using the same AV configuration, that’s great news and implies that there aren’t any issues with your AV!
As long as the calamari version is the on that’s expected, then there shouldn’t be anything special required, however I think it will hit the same issue even if you copy the folder over as I believe the same issue would be hit executing calamari as it is getting trying to extract it (e.g. the bootstrapping of scripts).