I have got a weird issue on Octopus, Actually we have a powershell script and it ran fine on almost all the time but today it just got terminated in middle of running but it didn’t show any error on octopus UI. when i check the worker pool i found the calamari error.
Here is the error on server:
Application: Calamari.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.OutOfMemoryException at Calamari.Common.CalamariFlavourProgram.Run(System.String[]) at Calamari.Program.Main(System.String[])
Note: As pet octopus the process completed successfully which is a problem for us. Please let me know if you need server logs.
Which version of Octopus Server are you currently running?
Would you be able to send me a full task log of the deployment, along with the timestamp of that error in your event log? It would also be helpful to have the server/tentacle log of the machine running the task in the deployment to line up as well.
Please find attached logs and provide some input ASAP.
Note: Step no. 12 didn’t completed the full script but it just got terminated and as per octopus tasks it was completed successfully and due to that other steps didn’t ran as we set octopus variable during step 12.
Octopus Version : 2020.4.2
Attached file Description:::::
ServerTask Log.Zip ----Octopus tasks logs
Deploy_Releases.ps1 ---- Step no 12 script (you can find there is a last line which set octopus variable during run time that line also didn’t ran and neither octopus throw any error).
eventvwr.zip ---- Evet viewer logs of that time
OctopusTentacle.4.zip — Octopus Server tentacle logs
Thanks for all of the logs. I took a quick look and this does appear like strange behavior. I’m going to pass this along to the engineering team to get some more eyes on it.
Please let me know if you have any questions or concerns in the meantime.
Let me know when i can expect an update on this as this is a blocker for us because if we get this error in prod then we won’t be even know that there is any error because octopus shows it is in green (success).
This is currently with our engineering team but I will pass along the urgency to them. Are you able to up the memory on your tentacle/worker doing this task to mitigate the chances of it happening? It looks to be caused by the machine running out of memory.
do we have any update on this as it’s been 7 days and add the memory to the server is not a solution as we are running a script and we don’t know how much memory it requires as there are other programs and services which are install over the server. Ideally it should throw an error.
I spoke with the engineers and they believe the memory issue is a coincidence. It’s probably not a bad idea to give it some more memory as this will cause issues going forward with everything on the server if it maxes out.
However, they think in this case your script is swallowing the actual error in the catch block on line 75.