Activities failed with errors ‘Object reference not set to an instance of an object.’, ‘Bootstrapper did not return the bootstrapper service message’, ‘Bootstrapper did not return the bootstrapper service message’
ServerTasks-667610.log.txt (48.5 KB)
Hi @karan.work,
Thanks for getting in touch!
Have deployments worked for this target previously? Or is it a new target?
If it has previously worked, have any changes been made to the target outside of Octopus?
There are several different possible causes for this error.
The most common is AV or security software preventing the package from being transferred successfully.
Regards,
Paul
Hi @paul.calvert ,
Deployments have worked previously on this server. This server has windows defender installed from inception and nothing has changed recently.
Hi @karan.work,
Just stepping in for Paul while he’s offline, cheers for that extra info!
I’d just like to confirm if this is currently blocking you or has it succeeded with a retry?
It appears to be able to download the packages from the feed to the Octopus Server ok, but then it’s failing to use Delta Compression to upload them to the target:
14:15:57 Fatal | Bootstrapper did not return the bootstrapper service message
…
Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageAcquisitionExecutionHandlers.<>c__DisplayClass5_0.<b__0>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageAcquisitionExecutionHandlers.cs:line 70
Timing wise it looks like the issue happened just after the first package was uploaded to the target and the delta was being applied:
14:15:57 Info | Original package was 6.083 MB, delta file is 59 B (100% size reduction).
14:15:57 Info | Uploading and applying delta Massivision.Domain.Query.Migrations.24.40.0-alphafeatureaxp-0603.604_12715FF7_to_24.40.0-alphafeatureaxp-0603.604_12715FF7.octodelta
14:15:59 Verbose | Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-667610
It’s possible the Calamari process was interrupted (which there might be a Windows Event describing what happened) but it sounds like it was possible a corrupt delta for that package. Are you able to repeat this issue?
If so, a process dump of Calamari when it happens should tell us exactly what’s going on to investigate otherwise if you clear the Tentacle package cache that should resolve it otherwise maybe consider disabling delta compression until retention clears out the potentially corrupt package!
Hope that helps but feel free to reach out with any questions at all!
Best Regards,
Hi @finnian.dempsey ,
Thank you for your response. I started re looking at the issue today. I did not change anything but this time “Acquiring Packages” task worked good.
ServerTasks-669442.log.txt (57.3 KB)
Can you please check and let me know if you could find something?
Hi @karan.work,
I don’t see anything concerning in the task log, if this is working as expected now then it could have been a transient issue with either the network or on the target.
If the issue reoccurs please let us know.
Regards,
Paul
Hi @paul.calvert ,
Today again the deployment failed.
I tried disabling deltacompression but it resulted in another error. Logs attached.
ServerTasks-670248.log.txt (63.9 KB)
I do not see Octopus Server on our EC2 instance, I can only see Tentacle.exe
Can you please help.
Thanks.
The failure looks to be occurring later in the process this time.
As mentioned before, when we’ve seen this error occur before, it has been chiefly caused by AV or endpoint protection software.
Some users have resolved the issue by removing the Tentacle software and reinstalling it.
It would be worth checking the Event Logs on the target (dev2-core) to see if anything was generated at the time of the error. The tentacle logs (default location C:\Octopus\Logs) from this target would also be useful.
Hi @paul.calvert ,
Tried to reinstall Tentacle using Octopus Tentacle Manager but it did not help.
ServerTasks-670764.log.txt (53.6 KB)
ServerTasks-670753.log.txt (61.6 KB)
Please find the logs of services failing. Also did not notice any log in event viewer during this timeline.
Octopus Log.txt (94.3 KB)
Also attached logs from Octopus.
Thanks,
Am I right in saying that this issue is occurring on more than one target/machine?
If so, assuming there have been no updates to Octopus, the only other explanation is that something has changed on these machines.
It could be a change in the group policy somewhere, windows updates or, most likely, the addition or alteration of the security software running on these targets.
If you’re able to get a process dump of the tentacle.exe and child calamari.exe processes during the deployment we can pass this to our engineers to review.
Hi @paul.calvert , This is occuring only on single machine/instance. Yes, I think you can pass this to your engineers review meanwhile I shall also check with our DevOps.
I noted two different machine hostnames in the task logs you sent.
Most of the logs you sent were showing this error on machine DEV2-COR however log 670764 shows the error occurring on DEV2-ESB0.
Hi @paul.calvert , You are right these 2 are 2 different EC2 instance but both for dev2 environment.
Hi @paul.calvert,
I have forwarded you memory dump.
Also, I tried to upgrade tentacle to version 7 still no luck. Can you please check on this issue on urgent basis?
Hi Karan,
Thanks for following up and sending through that information. I’ll jump in for Paul as he’s currently offline to pass along some input from the engineers.
Would you be able to grab a process dump of the child calamari processes (link to docs)? In the meantime, are you able to also try re-installing the Tentacle manager to re-install the Tentacle?
I look forward to hearing back and getting to the bottom of this issue!
Best regards,
Kenny
Hi @Kenneth_Bates,
I have reinstalled Tentacle Manager as well but did not help. Current Tentacle version is 7.0.1
I am not able to take dump of Calamari process as It does not stay long enough to take dump.
Best Regards,
Karan
Hi Karan,
Thanks for following up, and apologies that the re-install didn’t help. If you add a sleep (Start-Sleep -Seconds 120 for example) to the process, does that allow the Calamari process to stay long enough to take a dump of it?
Best regards,
Kenny
Hi @Kenneth_Bates ,
Please find the log file attached zip containing more logs
Work.zip (4.6 KB)
0.5 KB)
Also, Can you help me with detailed steps to prolong Calameri process execution as the Deployment process fails at acquiring packages stage.
Thanks.
Hi Karan,
Thanks for following up and for your continued help digging into this one. I’ll have a look through the files provided and look at a specific way to prolong Calamari process in this scenario.
For now I’ll pass along another update from the engineers. They have looked into this one again, and their suspicion is that PowerShell is not being invoked successfully, leading to these bootstrapper messages. Their thinking is there are external factors preventing this from happening, and mentioning a couple things to check to see if you can find anything that might prevent PowerShell from running.
- Windows Event Viewer logs
- Defender (or AV if you have any) logs
Best regards,
Kenny
HI @Kenneth_Bates,
I tired disabling AV for a while still the deployments did not work. I set “$env:TentacleHome” variable value in Powershell.
Deployment Failed for a Project A.txt (34.9 KB)
Deployment Succeed for a Project B.txt (26.8 KB)
Can we book sometime tomorrow. We can go through the setup and the issue in detail?
Thanks,
Karan Savla