Hi, today I have noticed we are unable to upgrade tentacles on some of our servers.
Windows 2012R2 with .Net 4.6.2
Octopus v2018.4.12
Calamari version 4.5.8
From the log:
Could not load file or assembly ‘System.ValueTuple, Version=4.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
System.IO.FileLoadException
at Calamari.Integration.Scripting.WindowsPowerShell.PowerShellBootstrapper.PrepareBootstrapFile(Script script, CalamariVariableDictionary variables)
at Calamari.Integration.Scripting.WindowsPowerShell.PowerShellScriptEngine.Execute(Script script, CalamariVariableDictionary variables, ICommandLineRunner commandLineRunner, StringDictionary environmentVars)
at Calamari.Commands.RunScriptCommand.InvokeScript(CalamariVariableDictionary variables)
at Calamari.Program.Execute(String[] args)
–Inner Exception–
Could not load file or assembly ‘System.ValueTuple, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
System.IO.FileLoadException
Could you send through the full deployment log from the Tentacle upgrade that failed with that error? I’ve just attempted to replicate the issue on a local W2K12R2 server with .NET 4.6.2 installed and it worked as expected.
Perhaps re-installing .NET 4.6.2 could help, but I can’t say for certain that it would.
Thanks for testing this out. I will reinstall .net and check there the state of these DLL’s in GAC.
Full log is below:
| == Failed: Upgrade SERVERNAME Tentacle ==
12:21:52 Info | Starting Tentacle upgrade for a limited set of deployment targets
12:21:52 Info | 1 machines will be upgraded to the latest Tentacle version.
12:21:54 Verbose | Checking for Tentacles to update
12:21:54 Fatal | At least one machine was not successfully upgraded.
|
| == Failed: SERVERNAME ==
12:21:52 Verbose | Updating Tentacle
12:21:54 Verbose | Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory 'd:\Octopus\Work\20180522102153-110710-2' using 'OEM United States' encoding running as 'NT AUTHORITY\SYSTEM' with the same environment variables as the launching process
12:21:54 Verbose | Octopus Deploy: Calamari version 4.5.8
12:21:54 Verbose | Environment Information:
12:21:54 Verbose | OperatingSystem: Microsoft Windows NT 6.3.9600.0
12:21:54 Verbose | OsBitVersion: x64
12:21:54 Verbose | Is64BitProcess: True
12:21:54 Verbose | CurrentUser: NT AUTHORITY\SYSTEM
12:21:54 Verbose | MachineName: SERVERNAME
12:21:54 Verbose | ProcessorCount: 8
12:21:54 Verbose | CurrentDirectory: D:\Octopus\Work\20180522102153-110710-2
12:21:54 Verbose | TempDirectory: C:\Windows\TEMP\
12:21:54 Verbose | HostProcessName: Calamari
12:21:54 Verbose | Executing 'D:\Octopus\Work\20180522102153-110710-2\Script.ps1'
12:21:54 Error | Could not load file or assembly 'System.ValueTuple, Version=4.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
12:21:54 Error | System.IO.FileLoadException
12:21:54 Error | at Calamari.Integration.Scripting.WindowsPowerShell.PowerShellBootstrapper.PrepareBootstrapFile(Script script, CalamariVariableDictionary variables)
12:21:54 Error | at Calamari.Integration.Scripting.WindowsPowerShell.PowerShellScriptEngine.Execute(Script script, CalamariVariableDictionary variables, ICommandLineRunner commandLineRunner, StringDictionary environmentVars)
12:21:54 Error | at Calamari.Commands.RunScriptCommand.InvokeScript(CalamariVariableDictionary variables)
12:21:54 Error | at Calamari.Program.Execute(String[] args)
12:21:54 Error | --Inner Exception--
12:21:54 Error | Could not load file or assembly 'System.ValueTuple, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
12:21:54 Error | System.IO.FileLoadException
12:21:54 Verbose | Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in d:\Octopus\Work\20180522102153-110710-2 exited with code 100
12:21:54 Fatal | The remote script failed with exit code 100
12:21:54 Verbose | The remote script failed with exit code 100
| Octopus.Server.Orchestration.Targets.Tasks.ActivityFailedException: The remote script failed with exit code 100
| at Octopus.Server.Orchestration.Targets.Scripting.ScriptResult.EnsureSuccessful()
| at Octopus.Server.Orchestration.Targets.Tentacles.TentacleUpgradeMediator.PerformUpgrade(TargetManifest targetManifest, IEnumerable`1 preconditions)
| at Octopus.Server.Orchestration.ServerTasks.Upgrade.UpgradeTaskController.InvokeWorkerAction(ITarget target, Machine machine)
| at Octopus.Server.Orchestration.ServerTasks.HealthCheck.HealthCheckService.InvokeHealthCheck(Machine machine, Action`2 customAction)
| at Octopus.Server.Orchestration.ServerTasks.HealthCheck.HealthCheckService.PerformHealthCheck(Machine machine, ExceptionHandling exceptionHandling, Action`2 customAction)
| Octopus.Server version 2018.4.12 (2018.4.12+Branch.master.Sha.0cbebac012d5bebecbd81b25274da00c794037b6)
12:21:54 Verbose | Recording health check results
|
| == Failed: Summary ==
12:21:54 Fatal | One or more deployment targets were not upgraded. Please see the output Log for details.
|
I can see that on all of these servers (both the working servers and the failing ones) we have the System.ValueTuple.dll with following properties:
Version=4.0.2.0
PublicKeyToken=7CEC85D7BEA7798E
The failing servers are looking instead for System.ValueTuple.dll with these properties:
Version=4.0.2.0 or 4.0.0.0
PublicKeyToken=cc7b13ffcd2ddd51
I guess what’s happened is that the not working servers have an older version of the code that references the system.tuple.dll with PublicKeyToken=cc7b13ffcd2ddd51.
Do you have the system.tuple.dll with the properties below that you can send to us?
Version=4.0.2.0 or 4.0.0.0
PublicKeyToken=cc7b13ffcd2ddd51
I assume once we get it, we can just place it in
D:\Octopus\Calamari\4.5.8\
Thanks for the extra information, could you zip up the Calamari folder from one of the servers that are failing and send that to us. We suspect that we’re hitting this issue as we’re targeting both .net code and full .net framework with Calamari.
We enabled Fusion logging and saw a redirect to version 4.0.0.0 of the library. This does not happen on working servers. I have attached this log but the example is below.
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Version redirect found in framework config: 4.0.2.0 redirected to 4.0.0.0.
Comparing the machine.configs to a working server I did not see any differences (or any specific reference to a redirect for System.ValueTuple).
The Calamari files all seems to be fine so I wonder what the differences in .NET frameworks are installed on the servers that don’t work compared to the ones that work.
Hi, initially when we first saw this issue, out of the 6 servers with the same configuration (Windows 2012R2 with .net 4.6.2), 4 were showing this error.
After reading some articles where System.ValueTuple was included in .Net 4.7 we have upgraded all of these to this. This may not have been the correct path to take as it was only after we this realised System.ValueTuple is also part of the Calarari directory (as we were troubleshooting the idea that .net is forcing a version on the whole system). It did however fix 2 out of the 4 problem servers. Why the remaining two servers are still in a bad state is a mystery.
Hi, we were able to upgrade Calamari by reinstalling the Octopus tentacle (Uninstall tentacle / delete calamari directory / re-install Octopus tentacle). This allowed Calamari to install without issue when we did the next deployment however it still fails during a deployment step in our software with the same issue.
So we are not able to test the Calamari issue until it is required to be upgraded. I will check if our developers can add the assembly binding to our step to see if it resolves this.
Hi, well the good news for us is that the servers with this issue no longer appear to be a problem, the bad news is we cannot replicate the issue now.
After taking these servers out of our production environment we attempted to trigger this error but so far deployments and calamari upgrades have been without issue.
Not too sure how to proceed from here, but thanks for your assistance.