Package deployment failing on one particular server

We have a problem where we are unable to deploy a package to one tentacle though it can deploy it to another. On one server where it works I see this in the deployment raw log:

                |     == Success: App2  (x.x.x.32) ==

19:13:44 Verbose | Octopus Server version: 2021.1.7149+Branch.release-2021.1.Sha.86015aa2c54e611d1a22583b46b4be047031a803
19:13:44 Verbose | Environment Information:
| IsRunningInContainer: False
| OperatingSystem: Microsoft Windows 10.0.14393
| OsBitVersion: x64
| Is64BitProcess: True
| CurrentUser: NT AUTHORITY\SYSTEM
| MachineName: machine11
| ProcessorCount: 4
| CurrentDirectory: C:\Windows\system32
| TempDirectory: C:\Windows\TEMP
| HostProcessName: Octopus.Server
| PID: 2604
19:13:44 Verbose | Executing Deploy a Package (Test) (type Deploy a Package) on App2 (x.x.x.32)
19:13:47 Verbose | Using Calamari.netfx 18.1.1
19:13:49 Verbose | Calamari Version: 18.1.1
19:13:49 Verbose | Environment Information:
19:13:49 Verbose | OperatingSystem: Microsoft Windows NT 10.0.14393.0
19:13:49 Verbose | OsBitVersion: x64
19:13:49 Verbose | Is64BitProcess: True
19:13:49 Verbose | CurrentUser: NT AUTHORITY\SYSTEM
19:13:49 Verbose | MachineName: VOR1WFIPUPAP16
19:13:49 Verbose | ProcessorCount: 4
19:13:49 Verbose | CurrentDirectory: C:\Octo\Work\20220523231347-52
19:13:49 Verbose | TempDirectory: C:\Windows\TEMP
19:13:49 Verbose | HostProcess: Calamari (52012)
19:13:50 Info | Deploying package: C:\Octo\Files\package@S2022.2.00.0412-166@209C65BAA25F1643A9339E6AB50909DC.zip
19:13:50 Verbose | Extracting package to: C:\Octo\Apps\PROD\package\2022.2.00.0412-166
19:13:52 Verbose | Extracted 499 files
19:13:52 Verbose | Installing package to custom directory C:\Temp\metadata
19:13:52 Info | Copying package contents to ‘C:\Temp\metadata’
19:13:54 Info | Copied 499 files

In the same deployment, the same package deployment to another machine it fails with this error:

                |     == Failed: App1 (x.x.x.26) ==

19:13:44 Verbose | Octopus Server version: 2021.1.7149+Branch.release-2021.1.Sha.86015aa2c54e611d1a22583b46b4be047031a803
19:13:44 Verbose | Environment Information:
| IsRunningInContainer: False
| OperatingSystem: Microsoft Windows 10.0.14393
| OsBitVersion: x64
| Is64BitProcess: True
| CurrentUser: NT AUTHORITY\SYSTEM
| MachineName: VOR1WFIPGPAP11
| ProcessorCount: 4
| CurrentDirectory: C:\Windows\system32
| TempDirectory: C:\Windows\TEMP
| HostProcessName: Octopus.Server
| PID: 2604
19:13:44 Verbose | Executing Deploy a Package (Test) (type Deploy a Package) on App1 (x.x.x.26)
19:13:47 Verbose | Using Calamari.netfx 18.1.1
19:13:49 Verbose | Octopus Deploy: Calamari version 3.7.105
19:13:49 Verbose | Environment Information:
19:13:49 Verbose | OperatingSystem: Microsoft Windows NT 10.0.14393.0
19:13:49 Verbose | OsBitVersion: x64
19:13:49 Verbose | Is64BitProcess: True
19:13:49 Verbose | CurrentUser: NT AUTHORITY\SYSTEM
19:13:49 Verbose | MachineName: machine15
19:13:49 Verbose | ProcessorCount: 4
19:13:49 Verbose | CurrentDirectory: C:\Octo\Work\20220523231347-14
19:13:49 Verbose | TempDirectory: C:\Windows\TEMP
19:13:49 Verbose | HostProcessName: Calamari
19:13:49 Error | System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
19:13:49 Error | at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
19:13:49 Error | at System.Reflection.Assembly.GetTypes()
19:13:49 Error | at System.Linq.Enumerable.d__172.MoveNext() 19:13:49 Error | at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext()
19:13:49 Error | at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext() 19:13:49 Error | at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable1 source)
19:13:49 Error | at Calamari.Commands.Support.CommandLocator.Find(String name)
19:13:49 Error | at Calamari.Program.Execute(String[] args)
19:13:49 Error | System.IO.FileLoadException: Could not load file or assembly ‘NuGet.Core, Version=2.11.1.812, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
19:13:49 Error | File name: ‘NuGet.Core, Version=2.11.1.812, Culture=neutral, PublicKeyToken=31bf3856ad364e35’
19:13:49 Error | WRN: Assembly binding logging is turned OFF.
19:13:49 Error | To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
19:13:49 Error | Note: There is some performance penalty associated with assembly bind failure logging.
19:13:49 Error | To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
19:13:49 Error | System.IO.FileLoadException: Could not load file or assembly ‘NuGet.Core, Version=2.11.1.812, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
19:13:49 Error | File name: ‘NuGet.Core, Version=2.11.1.812, Culture=neutral, PublicKeyToken=31bf3856ad364e35’
19:13:49 Error | WRN: Assembly binding logging is turned OFF.
19:13:49 Error | To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
19:13:49 Error | Note: There is some performance penalty associated with assembly bind failure logging.
19:13:49 Error | To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
19:13:49 Error | System.IO.FileLoadException: Could not load file or assembly ‘Microsoft.Web.XmlTransform, Version=2.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
19:13:49 Error | File name: ‘Microsoft.Web.XmlTransform, Version=2.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’
19:13:49 Error | WRN: Assembly binding logging is turned OFF.
19:13:49 Error | To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
19:13:49 Error | Note: There is some performance penalty associated with assembly bind failure logging.
19:13:49 Error | To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
19:13:49 Verbose | Updating manifest with output variables
19:13:49 Verbose | Updating manifest with action evaluated variables
19:13:49 Fatal | The remote script failed with exit code 43
19:13:49 Fatal | The action Deploy a Package (Test) App1 (x.x.x.26) failed

I suspect that it has something to do with the Calamari version detected. On servers where it deploys to, it detects Calamari 18.1
19:13:49 Verbose | Calamari Version: 18.1.1

For the failed cases, it’s detecting 3.7.105
19:13:49 Verbose | Octopus Deploy: Calamari version 3.7.105

However, Health Checks on both servers (listening tentacles) shows these same versions:
Running As: PROD\SYSTEM (Local Administrator: True)
19:27:29 Info | Running Tentacle version 3.7.11
19:27:29 Info | Tentacle communication uses a ‘’ certificate
19:27:29 Info | Running latest version of Calamari: 18.1.1

Any suggestions?

Thanks,
Gary

Hi Gary,

Thanks for getting in touch! I think you’re probably on the right track that it will come down to the big version discrepancy, but probably with the Tentacle version as your health check is reporting 3.7.11, which is quite old at this point.

The first step, which probably will resolve this outright, is to attempt an update these components. You can do this straight from the web portal in your deployment target’s Connectivity tab.

Let me know how you go!

Best regards,

Kenny

Hi Kenneth - thanks for the reply and help.

We are limited in what we can upgrade as that is the current version we have in our Prod environment and we only have this issue on one server / Tentacle. The odd thing as I mentioned is that both the working and non-working server report the exact same versions in the Health Check. Tentacle version 3.7.11 and running the latest version of Calamari (18.1.1). Do you know why for the deployment package step on that server it reports using Calamari Version: 3.7.105? Other non deployment-package steps on the same server reflect 18.1.1 but the package deployment step says 3.7.105. Do you know where that would even come from or why the same server would report both 3.7.105 and 18.1.1 Calamari versions?

Our current Octopus Server version is v2021.1 (Build 7149).

Thanks,
Gary

Hi Gary,

Thanks for following up! I imagine the difference in versions being reported in your health check vs. your deployments would be that in the health check, being run on the Octopus Server, it’s writing out the version of Calamari on your server. Whereas during a deployment it’s writing out the version of Calamari installed on the target that is being run on. That might be the case with your package steps being run on your targets, whereas other steps might be running on the Octopus Server with a different Calamari version. I think the best first step, if possible, is to update the version of Calamari on the single target that is reporting 3.7.105 when deploying to it, as I strongly suspect that will resolve this.

Please let me know how you go, or if I’ve misunderstood anything about your situation and I’d be happy to jump back into it!

Best regards,

Kenny

Hi Kenneth,
Thanks again for your suggestion. The overnight team reinstalled the Tentacle on that target server which I think essentially did what you suggested and it worked.

Thanks again!
Gary

Hi Gary,

Great to hear, and thanks for the update! Please let us know anytime if you come across any other questions or concerns in the future. :slight_smile:

Best regards,

Kenny

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.