We started to have an issue with Octopus after we did an inplace update to Windows Server 2019 Datacenter. Since then when an error is triggered we can only see this:
“NotSpecified:”
and the rest of the error message is eaten by something. While trying to fix that we found this post:
So we understood the fact that there was a problem because we had Windows Server 2019 and .NET 4.7.2 and we updated it to .NET 4.8, but this didn’t changed anything in the behaviour of Octopus.
Current config that we have is:
Windows Server 2019 Datacenter;
Octopus 2023.1.9781;
.NET 4.8;
Powershell 5.1;
We were wondering what else can we do in order to resolve this.
Thank you for looking into this.
For the moment I can provide only this piece of the raw log:
13:25:40 Error | NotSpecified:
13:25:40 Error | At E:\OctopusHome\Work\20230526132528-496740-1482\Script.ps1:24 char:9
13:25:40 Error | + $temp | kubectl apply -f - 2>&1
13:25:40 Error | + ~~~~~~~~~~~~~~~~~~~~~~~
13:25:40 Error | at , E:\OctopusHome\Work\20230526132528-496740-1482\Script.ps1: line 24
13:25:40 Error | at , : line 1
13:25:40 Error | at , E:\OctopusHome\Work\20230526132528-496740-1482\Octopus.FunctionAppenderContext.ps1: line 211
13:25:40 Error | at , E:\OctopusHome\Work\20230526132528-496740-1482\Bootstrap.Octopus.FunctionAppenderContext.ps1: line 4331
13:25:40 Error | at , : line 1
13:25:40 Error | at , : line 1
13:25:40 Verbose | Process C:\WINDOWS\system32\WindowsPowershell\v1.0\PowerShell.exe in E:\OctopusHome\Work\20230526132528-496740-1482 exited with code 1
13:25:40 Verbose | Updating manifest with output variables
13:25:40 Verbose | Updating manifest with action evaluated variables
13:25:40 Fatal | The remote script failed with exit code 1
As you can see, at the beginning is an error saying “13:25:40 Error | NotSpecified:” but there isn’t any error output next to it, so we can’t understand what’s happening and where the problem is.
These kind of behaviour with “NotSpecified” appeared after we did an update from Windows Server 2012 to Windows Server 2019, we didn’t experienced such a thing before.
Hi Mihal,
Thanks for sending through that log snippet.
On first glance it appears that $temp is not being set correctly and therefore not being evaluated, giving an error. Has anything changed with the script since it last ran successfully?
To dig a bit deeper, could you send us the process JSON and the raw task log for the task? You can upload them here securely:
I’ve just uploaded the raw output and Json file, as requested. These 2 files are from a diferent project but the behaviour is the same, we can not see the error messages.
At this stage since it doesn’t seem related to the script or the variables its setting, we need to do some more analysis.
Could you run the following from the above Github issue to see if we can clarify exactly what is the environment? We may have a regression that we need to address.
If the workarounds don’t work, please run the following powershell commands and send the output to support referencing this issue:
Get-ItemProperty “HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion”
Get-ItemProperty “HKLM:SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full”
Also please retrieve and build GitHub - droyad/PSProcessTerminateDemo: Demo app to demonstrate a powershell/process execution problem and run the exe on the server and let us know what the output is. (We get you to build it so you can see what is going to be run).
Include deployment logs if possible.
That would be very helpful and give us more exact info.
Hi Mihai,
Just to keep this one moving, in the case its not a regression on that GH issue, could you do the following for us?
Can you run the AWS command outside of Octopus and see if the Paths are correct and variables are set correctly? the same for the kubectl command on the other script.
Are you able to upload the original task log for the first error? We can compare both logs to see if there are correlating issues.
there is a line showing some potential variable substitution on the k8s secrets script - was anything changed in the script right before it executed? Performing variable substitution on 'E:\OctopusHome\Work\20230523190214-495850-240\kubernetes\k8s-secrets.ps1'
I have this error message when trying to upload a new file: “Access has expired, please contact support@octopus.com”.
Can you please provide a new link?
We will take this into consideration.
There were no changes done on our scripts. These issues appeated right after we did Windows server upgrade.
I have some info that may help you understand better what is going on. So, for example, before the upgrade, when there was an error in the deploy we could see an error like this (that was treated as a warning, was not stopping the process):
"[INFO] Configuring HTTPS Proxy
May 9th 2023 19:54:38Error
An error occurred (AccessDeniedException) when calling the ListSecrets operation: User: arn:aws:iam::[REDACTED]:[REDACTED]/[REDACTED] is not authorized to perform: secretsmanager:ListSecrets because no identity-based policy allows the secretsmanager:ListSecrets action "
Which helped us understand what is happening.
And now, after the upgrade, we can see something like (which is stopping the whole process):
"[INFO] Configuring HTTPS Proxy
May 31st 2023 17:16:29Error
NotSpecified:
May 31st 2023 17:16:29Error
At E:\OctopusHome\Work\20230531141609-498042-6676\kubernetes\k8s-secrets.ps1:445 char:22
May 31st 2023 17:16:29Error
May 31st 2023 17:16:29Error
at setSecretPaths, E:\OctopusHome\Work\20230531141609-498042-6676\kubernetes\k8s-secrets.ps1: line 445
May 31st 2023 17:16:29Error
at , E:\OctopusHome\Work\20230531141609-498042-6676\kubernetes\k8s-secrets.ps1: line 937
May 31st 2023 17:16:29Error
at , : line 1
May 31st 2023 17:16:29Error
at , E:\OctopusHome\Work\20230531141609-498042-6676\kubernetes\Octopus.FunctionAppenderContext.ps1: line 211
May 31st 2023 17:16:29Error
at , E:\OctopusHome\Work\20230531141609-498042-6676\kubernetes\Bootstrap.Octopus.FunctionAppenderContext.ps1: line 3864
May 31st 2023 17:16:29Error
at , : line 1
May 31st 2023 17:16:29Error
at , : line 1
May 31st 2023 17:16:29Fatal
The remote script failed with exit code 1 "
This kind of error messages are not helping us at all to undestand what is wrong and what needs to be fixed. It looks like something is eating the error output…
Thanks for getting back in touch and sorry to hear you’re still experiencing the issue.
Have you been able to try building and running the diagnostic tool described in the Diagnostics area of the GitHub issue you linked in your original post?
I’ve build the exe, and I’ve run it on the server. This is the output:
PS E:\PSProcessTerminateDemo\PSProcessTerminateDemo-master\bin> .\Demo.exe
Starting child process
curl: no URL specified!
curl: try ‘curl --help’ for more information
PS E:\PSProcessTerminateDemo\PSProcessTerminateDemo-master\bin> & ‘c:\windows\system32\curl.exe’ -gwefgew
curl: no URL specified!
curl: try ‘curl --help’ for more information
PS E:\PSProcessTerminateDemo\PSProcessTerminateDemo-master\bin>
Now that we know what appears when run outside of Octopus, would you be able to try running the exe from within Octopus either using a deployment process to target the exe or the script console?
Hopefully we can see a different/better output from there, apologies for the back and forth on this one.
Thank you for sending that over, I have sent it to our engineers for review, we have contacted our enginners about this issue as its a bit of a head scratcher so we wanted their input and they did manage to take a look at this last night.
They think the issue stems from this line in your script kubectl apply -f 2>&1 , where you are redirecting errors to stdout.
They wanted to ask what you were trying to achieve with this so that an we may be able to find an alternative solution and see if that fixes this issue.
I look forward to hearing from you, I will let you know if the engineers find anything from that test project file though but it looks like this may be down to how the error handling is being redirected possibly.
We used this redirecting “2>&1”only as a test while trying to fix the issues that we recently encountered with Octopus (I read this post: Calamari.exe swallowing errors - #4 by Matt.Richardson and it was saying that this may help fix the problem - but it didn’t in our case). So we are not using that in our nonprod or prod scripts and “NotSpecified” issue is still there.
Thank you for getting back to us so quickly with that information, I have passed that onto the engineers, hopefully they find something useful in the test project build you sent over but I have asked them if there is anything else they would need to diagnose this if the build project log doesn’t show us anything.
I will be in touch when I have some more details from the engineers,
Kind Regards,
Clare
