Is there a way to set the $ErrorActionPreference on Calamari?

nsaia1 · 29 July 2020 18:37

This article says the Calamari always sets the $ErrorActionPreference = ‘Stop’ before it runs.

Is there a way to override that at the Calamri configuration so I don’t have to keep putting $ErrorActionPreference = ‘Continue’ at the top of my scripts?

Thanks,

jeremy.miller · 29 July 2020 19:58

Hi Nsaia,

If you want to make a permanent change to Calamari, you’d need to build your own version, then tell octopus to use the custom build. If you do go this route, and you want to stay up to date, you will need to build a new version every time calamari updates.

You would want to change this part of the code:

github.com

OctopusDeploy/Calamari/blob/master/source/Calamari.Common/Features/Scripting/WindowsPowerShell/Bootstrap.ps1#L3


param([string]$OctopusKey="")
{{StartOfBootstrapScriptDebugLocation}}
$ErrorActionPreference = 'Stop'

# All PowerShell scripts invoked by Calamari will be bootstrapped using this script. This script:
#  1. Declares/overrides various functions for scripts to use
#  2. Loads the $OctopusParameters variables
#  3. Sets a few defaults, like aborting scripts when an error is encountered
#  4. Invokes (using dot-sourcing) the target PowerShell script

# -----------------------------------------------------------------
# Functions
# -----------------------------------------------------------------

Please let me know if that helps.

Thanks,
Jeremy

nsaia1 · 29 July 2020 21:33

I need to just set the $ErrorActionPreference at the top of the bootstrap to $ErrorActionPreference = ‘Continue’. Can I just change the bootstrap source file? Where is that located?

jeremy.miller · 29 July 2020 21:42

Hi Nsaia,

Unfortunately, it’s not that easy. You will have to go to https://github.com/OctopusDeploy/Calamari and follow the instructions for setting up your own version of Calamari.

Please take a look at the readme at that location, it explains how the process works. Please let me know if you have questions or if you get it going.

Thanks,
Jeremy

nsaia1 · 31 July 2020 22:25

+@Robert Harris

Hi Jeremy,

I’ve troubleshot more into the issue. We are trying to deprecate a Windows 2016 server that had our worker tentacles installed and our new servers are Windows 2019. We are running gcloud commands in Octopus via powershell. When run on Windows 2016 worker tentacles, the commands run fine. When ran on the Windows 2019 worker tentacles, the command fail with a NativeCommandError:

We are running the following:

gcloud config set project myProject

gcloud auth activate-service-account myServiceAccount --key-file=C:\myKeyFile.json

gcloud config list account --format “value(core.account)”

Even though powershell normally shows a NativeCommandError on all gcloud commands, the commands are running and returning from gcloud. This is expected behavior. Here is the output when running on powershell terminal on the VM. As you can see, the gcloud messages are returning success responses.

gcloud : Updated property [core/project].

At line:2 char:1

gcloud config set project myProject
CategoryInfo : NotSpecified: (Updated property [core/project].:String) [], RemoteException
FullyQualifiedErrorId : NativeCommandError

gcloud : Activated service account credentials for: [myServiceAccount]

At line:3 char:1

gcloud auth activate-service-account myServiceAccount …
CategoryInfo : NotSpecified: (Activated myServiceAccount]:String) [], RemoteException
FullyQualifiedErrorId : NativeCommandError

myServiceAccount

Running the same command in a custom script in powershell on a Windows 2016, here is the output. Notice that the calamari is not outputting the NativeCommandError error message wrapper, but only giving the gcloud success message output:

Updated property [core/project].

Activated service account credentials for: [myServiceAccount]

myServiceAccount

Running the same command in a custom script in powershell on a Windows 2019, here is the output. Notice that the calamari is not outputting the NativeCommandError error message wrapper on the first message, but fails on the second NativeCommandError and fails the octopus job.

Updated property [core/project].

NotSpecified: Activated service account credentials for: [myServiceAccount]

At C:\Octopus\Worker7\Work\20200731221346-878030-7\Script.ps1:3 char:1

gcloud auth activate-service-account myServiceAccount …

at , C:\Octopus\int-nprod-mng-1-Worker7\Work\20200731221346-878030-7\Script.ps1: line 3

at , C:\Octopus\int-nprod-mng-1-Worker7\Work\20200731221346-878030-7\Bootstrap.Script.ps1: line 1358

at , : line 1

The remote script failed with exit code 1

The action Test GCP Script on Worker Pool on a Worker failed

I have seen that this issue has been documented with Windows 2019 servers.

https://github.com/OctopusDeploy/Issues/issues/5755

We were only able to apply 1 of the recommended workaround as we were able. But the error continues to happen:

.NET 4.8 is installed

When will this be fixed? We are on a scheduled to deprecate the Windows 2016 server and need this working on Windows 2019. Are there other work arounds we can apply to get this issue?

jeremy.miller · 3 August 2020 14:32

Hi @nsaia1

Thanks for the information. Are you on our cloud instances? If so could I please have permission to log in and look at some logs? Which Project has this issue so i can go take a look?

Please let me know.

Thanks,
Jeremy

nsaia1 · 3 August 2020 16:51

Hi Jeremy,

Yes. We are on the cloud and you have permissions to work with project in the Production space:

Project Group: Test Feature Deployment

Project: Test GCP

I am testing this by changing the worker pool:

Production Worker Pool (Windows 2016) Success
Prod-Mng-Worker (Windows 2019) Failing
Non-Prod-Mng-Worker (Windows 2019) Failing

Please let me know if you’d like to schedule a Zoom or Teams call to take a look as well. I can be available.

jeremy.miller · 3 August 2020 20:24

Hi Nsaia,

Unfortunately, the way the gcloud CLI(among many other applications) reports its success/progress/etc messages is to send them to the standard error stream. This causes some issues within Octopus script steps and require workarounds.

Here is some documentation on logging messages in scripts.

A colleague showed me his workaround for gcloud CLI, which is available on our samples page for you to see here. If you go into step 2, you will see the following code:

Write-Host "##octopus[stderr-progress]"
$ipAddress=(& gcloud compute addresses list --project=$projectName --filter="name=($loadbalancerIPName)" --format="get(address)" --quiet)
Test-LastExit "gcloud compute addresses list"

The first line tells octopus to not error out when it gets the progress of the gcloud command on the standard error stream from that point on in the script.

The second line then sends the gcloud command which would normally fail the Octopus step.

The final line is a custom function that will read the last exit code and error out if it actually failed. That will be defined within a Script Module, which will be included on the process. The code is as follows:

function Test-LastExit($cmd) {
if ($LastExitCode -ne 0) {
    Write-Host "##octopus[stderr-error]"
    write-error "$cmd failed with exit code: $LastExitCode"
}
}

function CheckForGCPSDK() {
  if (-not (Test-Path -Path $OctopusParameters["GCP.InstallDir"])) {
  throw "GCP SDK Not found!"
  }
}

I know this is probably not the answer you were hoping to hear, but we unfortunately can’t stop companies from writing non errors to the standard error stream, we can only give tools to circumvent that behavior within Octopus to not treat them as errors.

Please let me know if that works for you.

Thanks,
Jeremy

nsaia1 · 3 August 2020 23:17

Jeremy,

Thank you. I will look at these steps. I’m not super excited about coding around this when this works in Windows 2016, but not in Windows 2019. Can I have more details on why this doesn’t work with Windows 2019? Is there a fix coming in a Calamari update?

jeremy.miller · 4 August 2020 13:25

Hi Nsaia,

On a non working worker could you please do the following for me:

Run the following and let me know the output:

Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion"
Get-ItemProperty "HKLM:SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full"

Please retrieve and build https://github.com/droyad/PSProcessTerminateDemo/tree/master/Demo
and run the exe on the server and let us know what the output is. (We get you to build it so you can see what is going to be run).

Thanks,
Jeremy

nsaia1 · 4 August 2020 13:46

+@Robert Harris

Thank you Jeremy. I’m jumping into meeting shortly, but will try to get this returned to you asap

Nick Saia | Sr. DevOps Engineer

TITAN School Solutions | www.titank12.com

SOCIALLY INCLINED? Connect with us on for access to our latest information.

DISCLAIMER: This e-mail contains proprietary information, some of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to it. If you are not the intended recipient, you may not use, disclose, copy, print, or rely on this e-mail.

nsaia1 · 4 August 2020 15:34

`Get-ItemProperty “HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion”```

``

Output:

SystemRoot : C:\Windows

BuildBranch : rs5_release

BuildGUID : ffffffff-ffff-ffff-ffff-ffffffffffff

BuildLab : 17763.rs5_release.180914-1434

BuildLabEx : 17763.1.amd64fre.rs5_release.180914-1434

CompositionEditionID : ServerDatacenter

CurrentBuild : 17763

CurrentBuildNumber : 17763

CurrentMajorVersionNumber : 10

CurrentMinorVersionNumber : 0

CurrentType : Multiprocessor Free

CurrentVersion : 6.3

EditionID : ServerDatacenter

EditionSubManufacturer :

EditionSubstring :

EditionSubVersion :

InstallationType : Server

InstallDate : 1587044293

ProductName : Windows Server 2019 Datacenter

ReleaseId : 1809

SoftwareType : System

UBR : 1339

PathName : C:\Windows

ProductId : 00430-00000-00000-AA682

DigitalProductId : {164, 0, 0, 0…}

DigitalProductId4 : {248, 4, 0, 0…}

InstallTime : 132315178939453329

PSPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows

NT\CurrentVersion

PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT

PSChildName : CurrentVersion

PSDrive : HKLM

PSProvider : Microsoft.PowerShell.Core\Registry

jeremy.miller · 4 August 2020 17:22

Hi Nsaia,

Thanks for the info. Please let me know when you are able to run the other 2 tests and I will have our engineers look at the results.

Please let me know if you have any questions.

Best,
Jeremy

nsaia1 · 4 August 2020 18:27

Hi Jeremy,

I had included all the results for what you asked me to run for all tests

Nick Saia | Sr. DevOps Engineer

TITAN School Solutions | www.titank12.com

SOCIALLY INCLINED? Connect with us on for access to our latest information.

DISCLAIMER: This e-mail contains proprietary information, some of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to it. If you are not the intended recipient, you may not use, disclose, copy, print, or rely on this e-mail.

jeremy.miller · 4 August 2020 19:01

Hi Nsaia,

I’m not seeing results from Get-ItemProperty "HKLM:SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full", which should kick out something like this:

CBS           : 1
Install       : 1
InstallPath   : C:\Windows\Microsoft.NET\Framework64\v4.0.30319\
Release       : 528372
Servicing     : 0
TargetVersion : 4.0.0
Version       : 4.8.04084
PSPath        : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full
PSParentPath  : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4
PSChildName   : Full
PSDrive       : HKLM
PSProvider    : Microsoft.PowerShell.Core\Registry

The engineers also needed the output from you running the exe generated by building this project:
Please retrieve and build https://github.com/droyad/PSProcessTerminateDemo/tree/master/Demo
and run the exe on the server and let us know what the output is. (We get you to build it so you can see what is going to be run).

Could you please provide both of those?

Thanks,
Jeremy

nsaia1 · 4 August 2020 19:16

Hmm. I sent this earlier this morning. Sending again:

Get-ItemProperty “HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion”

Output:

SystemRoot : C:\Windows
BuildBranch : rs5_release
BuildGUID : ffffffff-ffff-ffff-ffff-ffffffffffff
BuildLab : 17763.rs5_release.180914-1434
BuildLabEx : 17763.1.amd64fre.rs5_release.180914-1434
CompositionEditionID : ServerDatacenter
CurrentBuild : 17763
CurrentBuildNumber : 17763
CurrentMajorVersionNumber : 10
CurrentMinorVersionNumber : 0
CurrentType : Multiprocessor Free
CurrentVersion : 6.3
EditionID : ServerDatacenter
EditionSubManufacturer :
EditionSubstring :
EditionSubVersion :
InstallationType : Server
InstallDate : 1587044293
ProductName : Windows Server 2019 Datacenter
ReleaseId : 1809
SoftwareType : System
UBR : 1339
PathName : C:\Windows
ProductId : 00430-00000-00000-AA682
DigitalProductId : {164, 0, 0, 0…}
DigitalProductId4 : {248, 4, 0, 0…}
InstallTime : 132315178939453329
PSPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows
NT\CurrentVersion
PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT
PSChildName : CurrentVersion
PSDrive : HKLM
PSProvider : Microsoft.PowerShell.Core\Registry

Get-ItemProperty “HKLM:SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full”

Output:

CBS : 1
Install : 1
InstallPath : C:\Windows\Microsoft.NET\Framework64\v4.0.30319
Release : 528049
Servicing : 0
TargetVersion : 4.0.0
Version : 4.8.03761
PSPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework
Setup\NDP\v4\Full
PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4
PSChildName : Full
PSDrive : HKLM
PSProvider : Microsoft.PowerShell.Core\Registry

Here is the exe output:

Starting child process
curl: no URL specified!
curl: try ‘curl --help’ for more information

jeremy.miller · 4 August 2020 19:32

Hi Nsaia,

I see it all now, thank you for the information. I will pass it off to the engineers and update you when I get more information.

Thanks,
Jeremy

nsaia1 · 4 August 2020 19:43

Thank you. Please let me know. I can be available today as soon as needed.

nsaia1 · 5 August 2020 13:18

Hi Jeremy!

Do you have a update?

Nick Saia | Sr. DevOps Engineer

TITAN School Solutions | www.titank12.com

SOCIALLY INCLINED? Connect with us on for access to our latest information.

DISCLAIMER: This e-mail contains proprietary information, some of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to it. If you are not the intended recipient, you may not use, disclose, copy, print, or rely on this e-mail.

jeremy.miller · 5 August 2020 13:52

Hi Nsaia,

This ticket/issue has been passed to the Product team. It might be a little bit before we hear anything back so unfortunately, if you need to get this going very quickly, you will have to use the workaround I listed above to get around this until we hear back.

Please let me know if you have any questions or concerns in the meantime.

Thanks,
Jeremy