Bootstrapper error after upgrade

We are doing a test upgrade on our onprem dev server. From 2020.6 to the latest.

The upgrade completed fine, but now when we run any scripts we get an error

‘Bootstrapper did not return the bootstrapper service message’

i can see one post with the same error, they said it was a virus scanner issue? I’m not sure about this, we didn’t have this issue until we upgraded.

2 Likes

Hi @andrew_w,

Thanks for reaching out, great to hear from you again!

You are correct that we usually see this error caused by AntiVirus, however we did recently see a case where there were duplicate Tentacles configured causing this error. I’d be happy to take a look into exactly what’s going on, could you please send through the Raw Task logs?

You should be able to securely upload them at this location but please reach out if there are any issues with it or you have any questions at all!

Best Regards,

Hi finnian,

logs uploaded as requested

regards

Andrew

1 Like

Hi @andrew_w,

Thanks for that, confirming I have received it ok!

It looks like this is happening when running scripts on the Built-In Worker and the script is exiting just prior with code 0:

10:29:36   Verbose  |       Process C:\WINDOWS\system32\WindowsPowershell\v1.0\PowerShell.exe in d:\Octopus\Work\20221122232934-1547-1 exited with code 0
10:29:38   Verbose  |       Bootstrapper did not return the bootstrapper service message

I think it’s likely the new version of Calamari (by default under C:\Octopus\OctopusServer\Tools\Calamari.win-x64) is being quarantined by AV, which is likely what’s changed. Otherwise it could be that the Octopus Work folder has changed (to d:\) and not been added to the AV exclusion list?

If you’d like to also send through the Octopus Server Logs for this timeframe, I’ll check to see if I can spot any obvious issues, otherwise it might require checking over Windows Events, or taking a dump of the Octopus.Server and Calamari processes to see exactly what’s going on.

Let me know how you get on or if you have any questions at all!

Best Regards,

Hi Finnian,

Ok thanks for that, we’ll investigate at our end.

One question, once we’ve fixed up our AV exclusion list, should it just start working? Or is there anything we need to do.

Regards

Andrew

Hi @andrew_w,

No problem at all, looking forward to hearing how you get on!

It will depend on your specific Antivirus, but as long as the Tool and Work folders have been excluded it should just work, there won’t be any additional configuration required in Octopus.

Bonus: Our Script Console feature allows for quickly running on off scripts, which I find great for testing!

Feel free to reach out if you have any questions or run into any issues at all!

Best Regards,

Hi,

I work with Andrew. We have verified with our network team to confirm that our AV is not excluding Calamari packages or anything relating to Octopus Deploy.

However, I’ve managed to upload the dump files for Octopus Server and Tentacles.

Can we also confirm the end point that’s used to download Calamari package during the upgrade process?

Thanks.

Joseph.

Hi @vaananart,

Thanks for stepping in and uploading those dump files, confirming we’ve received them ok!

Just to confirm, your Network team have added the following folder exclusions to your AntiVirus and are still seeing this same issue with the Bootstrapper? Are there any AV or Windows events indicating that a process was blocked?

"C:\Octopus\Tools"
"C:\Octopus\Tools\*"
"C:\Octopus\Work"
"C:\Octopus\Work\*"

The Calamari package is bundled with the Octopus Server executable and pushed from the Octopus Server to Tentacle so there shouldn’t be any endpoints that need to be whitelisted. Check out our docs regarding Outbound Requests for more info about what requests are made and the Machine Policy docs for how Calamari is updated!

I’ll see if I can spot anything odd with those dump files and keep you posted, let me know if you have any questions at all!

Best Regards,

Hi Finnian,

ok we’ll pass it on to our network team.

A couple of things we’ve noted…

  • Our initial testing is running a hello world powershell script set to run on the Octopus server. I don’t think tentacles are involved at all.
  • We can see in the OctopusServer\Tools folder, there are no recent updates, and the folder you mention - calamari.win-x64 is not there

Hi @andrew_w,

Thanks for that, it definitely seems likely that the Calamari extraction has been quarantined if it’s not present in that folder. It gets extracted from a zip which resides in the C:\Program Files\Octopus Deploy\Octopus folder when a task is run on the Server directly or Tentacle.

Looking forward to hearing what your Network team have to say, feel free to reach out with any questions!

Best Regards,

Hi Finnian,

We’ve chased it up with our infrastructure team, they are certain the virus software is not blocking the install of calamari.

What do you recommend we do next. Is it possible to do an online session? Are there any other logs we can look at?

regards

Andrew

Hi Andy,

Thanks for the update, were they able to locate any events in the AV logs indicating that any executables are being blocked or files being quarantined?

Just to clarify, Calamari shouldn’t require installation as a service, just to be extracted to the C:\Octopus\OctopusServer\Tools folder. This error regarding the Bootstrapper is implying that something is preventing that extraction from occurring, often because of blocking an executable.

We’ve found that the best steps for testing this is to disable AV completely to see if re-running the script allows it to execute successfully. If it’s still giving the same error after disabling AV, we will need to figure out what is blocking the extraction of Calamari e.g. by consulting windows events.

Is the Calamari Zip present in the C:\Program Files\Octopus Deploy\Octopus folder?

Let me know how you get on or if you have any questions at all, if it’s still giving issues after disabling AV we’d be happy to jump in a call to see if we can help spot whats’ going on!

Best Regards,

Oh yes I do see that zip file in the folder.

So when the Octopus service starts up it should take that zip file and extract it to d:\Octopus\OctopusServer\Tools ?

The permissions on that folder look good, I assume it will be running as the account the Octopus server service is running under?

Hi @andrew_w,

That’s great, thanks for confirming that!

That Calamari zip will get extracted when a Deployment is run against a Target or Worker rather than when the Windows Service is started.

Octopus will check for the same Calamari version it’s bundled with and extract it if it’s not present, using the account that Octopus/Tentacle is running under. This uses both the Work and Tools folders and subfolders but as long as same Calamari version is present in the Tools folder as is configured for that version of Octopus it should work without needing to extract it, allowing you to manually copy Calamari into that location if required.

Logs showing Calamari Extraction :

Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory 'C:\Octopus\Work\20221201013631-159-8' using 'OEM United States' encoding running as 'NT AUTHORITY\SYSTEM'
Checking for C:\Octopus\Tools\Calamari.win-x64\24.0.24\Success.txt
Package: C:\Octopus\Work\20221201013631-159-8\Calamari.win-x64.24.0.24.nupkg
Destination: C:\Octopus\Tools\Calamari.win-x64\24.0.24
411 files extracted
Calamari.win-x64 24.0.24 extracted to C:\Octopus\Tools\Calamari.win-x64\24.0.24
Removing write permissions on C:\Octopus\Tools\Calamari.win-x64\24.0.24 for BUILTIN\Users
Cleaning up old Calamari.win-x64 versions...
Keeping only Calamari.win-x64 version 24.0.24...
Found no old versions of Calamari.win-x64 to delete...
Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20221201013631-159-8 exited with code 0
Version 24.0.24 of the Calamari.win-x64 tool has been extracted successfully

Let me know if I can help with any further questions!

Best Regards,

I don’t see the folder calamari.win-x64 on either c or d drives

I thought about manually extracting it but that zip doesn’t contain anything I recognise

Hi @andrew_w,

Thanks for sharing that!

Back in 2020.6 the Calamari folder was called Calamari.netfx rather than calamari.win-x64 and so there doesn’t look like there were any issues with the old installer.

Sorry I should have specified the extracted Calamari folder can be copied from one instance to another instance, but the extraction will need to be done by an Octopus Server originally in order to map the files correctly. If you have another instance/tentacle which does have a calamari.win-x64 folder then you can copy it however unfortunately I don’t think that will resolve the issue and you’d likely see the same error message as it will still try to run the bootstrap script to check if it needs extracting.

If after disabling AV it’s still not working then that will indicate that there is some other environmental problem preventing it which we’ll most likely need to help hunt out during a call.

Best Regards,