The archive entry was compressed using an unsupported compression method

jeremy.miller · 28 October 2020 20:21

Just to confirm, when you are running the command locally and it succeeds, are you using the same msi that was downloaded by the deployment process, in addition to being logged in as the service account that runs the tentacle?

Thanks,
Jeremy

hardKOrr · 28 October 2020 22:19

The service runs as a virtual user, which makes it tricky as I can’t log him in. However if I use the Task Console I can execute the “Deploy” script and it installs the MSI without issue. It’s hard to really make sure everything is the same but here is what I have attempted (successfully) :

Goto Tasks in Octopus Server UI and Select Script Console
Manually create and populate an appropriate OctopusParameters hashtable with variables and values
Execute the local Deploy.ps1 from the extraction location of the failing deployment
Modify the Deploy.ps1 in 2 places, which was just to change relative pathing to absolute (since the task didnt run from the same place as the deployment apparently?)
Run the Task

The log from the MSI reports as being under the virtual user, and succeeds installation without any issues.

hardKOrr · 28 October 2020 23:31

Finally got security guy around to remove McAfee as another check, rebooted and still a failure.

jeremy.miller · 29 October 2020 14:35

Hey @hardKOrr,

We’re still investigating. I didnt want you to think we forgot about you. Our engineers are going to look into it again tonight as I’ve given them more information to dig into.

Please let me know if you have any questions in the meantime.

Thanks,
Jeremy

hardKOrr · 29 October 2020 14:47

Thanks for all the work, its luckily just this 1 package on this 1 machine having the problem right now. We’d had a few others throw similar issues, but a reboot had resolved those.

I understand how hard this could be to track down, my research into the errors and what was happening at the time got me almost nowhere.

jeremy.miller · 29 October 2020 14:55

Hey @hardKOrr,

You’re very welcome.

How cumbersome would it be to re-install the tentacle on the machine? Would that disrupt your ecosystem at all? It could be worth a shot at this point. I typically don’t like to suggest that but I feel like we’ve thrown the kitchen sink at it thus far.

Let me know what you think. Regardless, the engineers will look into it again tonight.

Best,

hardKOrr · 29 October 2020 15:16

I will give that a shot later today when the machine is less busy and see if that helps.

jeremy.miller · 29 October 2020 15:18

Sounds great. If you can manually confirm all the files are deleted (and if not, delete them), and reboot to make sure we have a truly clean slate regarding the Tentacle.

Please let me know how it goes.

hardKOrr · 29 October 2020 20:21

Tentacle Service completely wiped out and renamed the Octopus folder to be 100% fresh.
Still fails.

There is another interested tidbit that I suppose wasn’t directly pulled up :
The log shows output from the Deploy.ps1 running the “Install MSI” script, as if it DID start the msi. msiexec IS running on the machine… however nothing “happens” with it as best I can tell as it doesn’t even create the log.

This is the same “Install MSI” script that runs for the previous process in the same project without issue, and the same package works on other machines still. Soooo, I’m still lost.

/e : If I kill msiexec process Octopus registers a failure within seconds…

jeremy.miller · 29 October 2020 20:25

Thanks for the new information. It’s unfortunate that the reinstall didnt fix it. I’ll pass it along to the engineers.

Best,
Jeremy

jeremy.miller · 2 November 2020 14:11

Hey @hardKOrr,

I was able to get a similar(but not identical) issue on a 2008 R2 SP1 tentacle and resolved it by updating windows. Is the box that’s having the issue fully updated within Windows? I’ve got 2 reports of this now and it seems like every instance of it is 2008 R2 SP1, do you have any outside of that?

Thanks,
Jeremy

hardKOrr · 2 November 2020 14:34

The machine we are having issues with is a Windows Server 2016 machine.

jeremy.miller · 2 November 2020 15:11

Hi,

Is that machine fully updated? 2008 R2 was having a similar issue til it was updated.

Please let me know.

Thanks,
Jeremy

hardKOrr · 2 November 2020 16:26

Yes it is currently up-to-date with Windows Updates

jeremy.miller · 2 November 2020 16:29

Hi,

Do you by chance know the count of the OS’ you’re running with the machines that are having issues? Are they all the same or do they differ?

Thanks,
Jeremy

hardKOrr · 2 November 2020 16:47

This is the only remaining machine that exhibit this issue. The other 2 that happened intermittently do not have any further issues, they were resolved quickly on their own.

I do not recall what was different for those machines that they are now working. It could be that the weekend let windows update run through.

jeremy.miller · 2 November 2020 17:37

Hi,

Let me go back to the engineers with this new information. My apologies, for some reason I had remembered your machines as being 2008 R2 SP1. Our other user with this issue seems to only have that OS with the problem so I thought we were on to something.

Thanks,
Jeremy

hardKOrr · 4 November 2020 18:36

This issue has cropped up on another server with a different package and is happening on upload there. Server 2012 this time

Here is the log snippet :

12:14:57 Info | 12:14:57 Info | 12:14:57 Verbose | 12:14:57 Info | 12:14:58 Info | |
| Failed: FO-BEEF-WS1V
12:14:41 Fatal | | Server exception:
| System.IO.Inv | at System.IO. | at System.IO. | at System.IO. | at System.IO. | at System.IO. | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se | at Octopus.Se |
| Success: 12:13:04 Verbose | 12:13:04 Verbose | 12:13:06 Verbose | 12:13:06 Verbose | 12:13:06 Verbose | 12:13:06 Verbose | 12:13:10 Verbose | 12:13:10 Verbose | 12:13:10 Verbose | 12:13:10 Verbose | 12:13:10 Verbose | 12:13:10 Verbose | 12:13:10 Verbose | 12:13:10 Verbose | 12:13:10 Verbose | 12:13:10 Verbose | 12:13:10 Verbose | 12:13:11 Verbose | 12:13:11 Verbose | 12:13:11 Verbose | 12:13:11 Verbose | 12:13:11 Verbose | 12:13:11 Verbose | 12:13:11 Verbose | 12:13:11 Verbose | 12:13:12 Verbose | 12:13:12 Verbose | 12:13:12 Verbose | 12:13:12 Verbose | 12:13:17 Verbose | 12:13:17 Verbose | 12:13:17 Verbose | 12:13:17 Verbose | 12:13:17 Verbose | 12:13:17 Verbose | 12:13:17 Verbose | 12:13:17 Verbose | 12:13:17 Verbose | 12:13:17 Verbose | 12:13:17 Verbose | 12:13:18 Verbose | 12:13:18 Verbose | 12:13:18 Verbose | 12:13:18 Verbose | 12:13:18 Verbose | 12:13:18 Verbose | 12:13:18 Info | 12:13:18 Info | 12:13:18 Verbose | 12:13:18 Info | 12:13:20 Info | |
| Failed: 12:14:41 Verbose | 12:14:41 Verbose | No earlier versions of WindowsInstallerExecute-PS1 were found on the target so we can’t perform a delta upload.
Uploading package WindowsInstallerExecute-PS1 (6.288 KB)…
Requesting upload…
Beginning streaming transfer of WindowsInstallerExecute-PS1@S2.0.2.0@8C03D3E534143745B0EE6A21B1CF1E71.nupkg
Stream transfer complete
The archive entry was compressed using an unsupported compression method.
alidDataException: The archive entry was compressed using an unsupported compression method.
Compression.Inflater.Inflate(FlushCode flushCode)
Compression.Inflater.ReadInflateOutput(Byte* bufPtr, Int32 length, FlushCode flushCode, Int32& bytesRead)
Compression.Inflater.ReadOutput(Byte* bufPtr, Int32 length, Int32& bytesRead)
Compression.Inflater.InflateVerified(Byte* bufPtr, Int32 length)
Compression.DeflateStream.ReadCore(Span1 buffer) | at System.IO.Compression.DeflateStream.Read(Byte[] array, Int32 offset, Int32 count) | at System.IO.BinaryReader.InternalRead(Int32 numBytes) | at System.IO.BinaryReader.ReadInt32() | at Newtonsoft.Json.Bson.BsonDataReader.ReadNormal() | at Newtonsoft.Json.Bson.BsonDataReader.Read() | at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter) | at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) | at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) | at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader) | at Halibut.Transport.Protocol.MessageExchangeStream.ReadBsonMessage[T]() | at Halibut.Transport.Protocol.MessageExchangeStream.Receive[T]() | at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessReceiverInternalAsync(IPendingRequestQueue pendingRequests, RequestMessage nextRequest) 12:14:41 Verbose | at Halibut.ServiceModel.HalibutProxy.EnsureNotError(ResponseMessage responseMessage) | at Halibut.ServiceModel.HalibutProxy.Invoke(MethodInfo targetMethod, Object[] args) | --- End of stack trace from previous location where exception was thrown --- | at System.Reflection.DispatchProxyGenerator.Invoke(Object[] args) | at generatedProxy_1.StartScript(StartScriptCommand ) | at Octopus.Server.Orchestration.Targets.Tentacles.TentacleRemoteEndpointFacade.ExecuteCommand(StartScriptCommand command) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\Targets\Tentacles\TentacleRemoteEndpointFacade.cs:line 61 | at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ExecutionTargets.TentacleExecutionTarget.Execute(ScriptCollection bootstrapperScripts, IReadOnlyList1 boostrapperArguments, IReadOnlyList1 files, Nullable1 forceIsolationLevel, Boolean raw, Nullable1 isolationMutexTimeout) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionExecution\Immediate\ExecutionTargets\TentacleExecutionTarget.cs:line 48 | at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.RunOnTarget(ScriptCollection finalBootstrapperScripts, IReadOnlyList1 bootstrapperArguments, IReadOnlyList1 allFiles, Nullable1 isolation, Nullable1 isolationMutexTimeout, CalamariFlavour flavour, IReadOnlyList1 deploymentTools) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionExecution\Immediate\ImmediateExecutor.cs:line 206
rver.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.ExecuteCalamari(CalamariFlavour calamariFlavour, String calamariCommand, IReadOnlyList1 calamariArguments, IReadOnlyList1 files, IReadOnlyList1 deploymentTools, VariableCollection extraVariables, TargetManifest targetManifest, Nullable1 isolationMutexTimeout) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionExecution\Immediate\ImmediateExecutor.cs:line 162
rver.Orchestration.ServerTasks.Deploy.ActionExecution.CommandBuilders.CalamariCommandBuilder.Execute() in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionExecution\CommandBuilders\CalamariCommandBuilder.cs:line 143
rver.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageAcquisitionExecutionHandlers.<>c__DisplayClass3_0.b__0(IActionHandlerContext ctx) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\Steps\Acquire\PackageAcquisitionExecutionHandlers.cs:line 32
rver.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.InvokeActionHandler(Machine target, ActionHandlerInvocation actionHandler, ActionAndTargetScopedVariables actionAndTargetScopedVariables, IExecutor executor, TargetManifest targetManifest) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\AdHocActionDispatcher.cs:line 167
rver.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.ExecuteOnDeploymentTarget(DeploymentTarget deploymentTarget, ActionHandlerInvocation actionHandler, TargetManifest targetManifest) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\AdHocActionDispatcher.cs:line 145
rver.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation actionHandler, VariableCollection variables) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ActionDispatch\AdHocActionDispatcher.cs:line 62
rver.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageAcquisitionExecutionHandlers.FindPackage(String packageId, IVersion version, String packageHash, PackageSearchOptions packageSearchOptions) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\Steps\Acquire\PackageAcquisitionExecutionHandlers.cs:line 40
rver.Orchestration.Targets.Common.PackageStagingMediator.TryFindPackageInCache(StoredPackage packageToPush, PackageAcquisitionExecutionHandlers execs, StagedPackage& stagedPackage) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\Targets\Common\PackageStagingMediator.cs:line 118
rver.Orchestration.Targets.Common.PackageStagingMediator.PushPackage(PackageIdentity packageIdentity, StoredPackage packageToPush, IDeploymentPackageDownloader deploymentPackageDownloader, DeltaCompressionOptions deltaCompressionOptions, PackageCacheOptions packageCacheOptions, PackageAcquisitionExecutionHandlers execs) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\Targets\Common\PackageStagingMediator.cs:line 99
rver.Orchestration.Targets.Common.PackageStagingMediator.DownloadPackageAndPushToAgent(PackageIdentity packageIdentity, IDeploymentPackageDownloader deploymentPackageDownloader, DeltaCompressionOptions deltaCompressionOptions, PackageCacheOptions packageCacheOptions, PackageAcquisitionExecutionHandlers execs) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\Targets\Common\PackageStagingMediator.cs:line 68
rver.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireMachinePackageTask.StagePackage(PackageAcquisitionExecutionHandlers packageAcquisitionHandlers) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\Steps\Acquire\AcquireMachinePackageTask.cs:line 156
rver.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireMachinePackageTask.Acquire(AcquiredPackageMap acquiredPackageMap, ITaskContext taskContext, LogContext logContext) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\Steps\Acquire\AcquireMachinePackageTask.cs:line 124
rver.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(Action callback, String actionName, Boolean actionIsRequiredToRun) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\Guidance.cs:line 132
rver.Orchestration.ServerTasks.Deploy.ExecutionTaskController1.<>c__DisplayClass38_1.<AcquirePackages>b__1() in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ExecutionTaskController.cs:line 347 | at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionTaskController1.ExecuteWithTransientErrorDetection(Action action, DeploymentTarget deploymentTarget) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\Deploy\ExecutionTaskController.cs:line 683
rver.Orchestration.ServerTasks.OctoThreadClosure`1.Execute() in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Orchestration\ServerTasks\OctoThreadClosure.cs:line 71
Upload package BrowserControl v3.1.6.0
Using Calamari.netfx 13.0.12
Using Calamari.linux-x64 13.0.12
Executable directory is C:\Windows\system32\WindowsPowershell\v1.0
Executable name or full path: C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe
No user context provided. Running as current user.
Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory ‘C:\Octopus\Work\20201104181304-65596-8’ using ‘OEM United States’ encoding running as ‘NT SERVICE\OctopusDeploy Tentacle’ with the same environment variables as the launching process
Calamari Version: 13.0.12
Environment Information:
OperatingSystem: Microsoft Windows NT 6.3.9600.0
OsBitVersion: x64
Is64BitProcess: True
CurrentUser: NT SERVICE\OctopusDeploy Tentacle
MachineName: FO-BEEF-WS1V
ProcessorCount: 2
CurrentDirectory: C:\Octopus\Work\20201104181304-65596-8
TempDirectory: C:\SPCentral\Temp
HostProcess: Calamari (108)
Package BrowserControl version 3.1.6.0 hash fa7bcda450843a66dedb36fa0426321d7b4108d1 has not been uploaded.
Finding earlier packages that have been uploaded to this Tentacle.
No earlier packages for BrowserControl has been uploaded
Found 0 earlier versions of BrowserControl on this Tentacle
Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20201104181304-65596-8 exited with code 0
Exit code: 0
Using Calamari.netfx 13.0.12
Using Calamari.linux-x64 13.0.12
Executable directory is C:\Windows\system32\WindowsPowershell\v1.0
Executable name or full path: C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe
No user context provided. Running as current user.
Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory ‘C:\Octopus\Work\20201104181311-65596-9’ using ‘OEM United States’ encoding running as ‘NT SERVICE\OctopusDeploy Tentacle’ with the same environment variables as the launching process
Calamari Version: 13.0.12
Environment Information:
OperatingSystem: Microsoft Windows NT 6.3.9600.0
OsBitVersion: x64
Is64BitProcess: True
CurrentUser: NT SERVICE\OctopusDeploy Tentacle
MachineName: FO-BEEF-WS1V
ProcessorCount: 2
CurrentDirectory: C:\Octopus\Work\20201104181311-65596-9
TempDirectory: C:\SPCentral\Temp
HostProcess: Calamari (8484)
Package BrowserControl version 3.1.6.0 hash fa7bcda450843a66dedb36fa0426321d7b4108d1 has not been uploaded.
Finding earlier packages that have been uploaded to this Tentacle.
No earlier packages for BrowserControl has been uploaded
Found 0 earlier versions of BrowserControl on this Tentacle
Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20201104181311-65596-9 exited with code 0
Exit code: 0
No earlier versions of BrowserControl were found on the target so we can’t perform a delta upload.
Uploading package BrowserControl (59.47 MB)…
Requesting upload…
Beginning streaming transfer of BrowserControl@S3.1.6.0@5AFCE5FC2D7D0447A39CC736D1B79717.nupkg
Stream transfer complete
Upload package WindowsInstallerExecute-PS1 v2.0.2.0
Using Calamari.netfx 13.0.12
Using Calamari.linux-x64 13.0.12

jeremy.miller · 4 November 2020 18:40

Hi,

Thanks for the extra data point. I’ll pass it along to the engineers.

Best,
Jeremy

jeremy.miller · 9 November 2020 15:11

Hey @hardKOrr,

Our engineers have been digging in on this and had some followup questions, if a question isn’t applicable to you feel free to skip it:

Are they restarting the deployment target before they do a health check? And is there any kind of wait/monitoring that checks that the deployment target is back online before they health check it? (if it fails on a health check)
Can we get their deployment process JSON? (if you have one that is failing due to this error currently)
Are they using Azure VMs?
Can we get their tentacle logs? Can they enable trace level logging on their tentacle logs first? (Documentation: https://octopus.com/docs/support/log-files#Logfiles-Changingloglevelstentacle)

Anything confidential just please private message the attachments.

Please let me know if you have any questions.

Thanks,
Jeremy