Octopus 3.10.1 wiped many deployed applications

After upgrading from 3.10.0 to 3.10.1, Octopus is now wiping many of our deployed applications when deploying them. Instead of the contents of the NuGet package there is only the bin directory and one of the files that were supposed to be inside it. Redeploying gives the same result, but reverting to 3.10.0 and redeploying fixes the issue.

One thing I noticed is that 3.10.1 tries some delta stuff against an older version nuget file and downloads 11MB instead of 68MB. 3.10.0 does not try this trick.

The log from 3.10.1:

|   == Success: Acquire packages ==
09:40:05   Info     |     Acquiring packages
09:40:05   Info     |     Making a list of packages to acquire
09:40:05   Verbose  |     No packages are required on the Octopus Server
09:40:05   Info     |     Delta compression is enabled for package transfers from the Octopus Server to deployment targets
09:40:05   Verbose  |     Upload Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2 to mer-web02
09:40:05   Verbose  |     Checking package cache for package Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2
09:40:05   Info     |     Package Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2 was found in cache. No need to download.
09:40:05   Verbose  |     Using file: C:\Octopus\OctopusServer\PackageCache\feeds-1\Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2_6FF1991BB2A7F046ABA4937EAF948C8B.nupkg
09:40:22   Info     |     All packages have been acquired
09:40:22   Verbose  |     Acquire Packages completed
                    |   
                    |     Success: mer-web02
                    |     
                    |       Success: Upload package Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2
09:40:06   Verbose  |         Octopus Deploy: Calamari version 3.6.48+Branch.master.Sha.d858383672653f2c991a15d2e618cf84c99dad39
09:40:06   Verbose  |         Environment Information:
09:40:06   Verbose  |         OperatingSystem: Microsoft Windows NT 6.1.7601 Service Pack 1
09:40:06   Verbose  |         OsBitVersion: x64
09:40:06   Verbose  |         Is64BitProcess: True
09:40:06   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
09:40:06   Verbose  |         MachineName: MER-WEB02
09:40:06   Verbose  |         ProcessorCount: 2
09:40:06   Verbose  |         CurrentDirectory: D:\Octopus\Work\20170222084005-259
09:40:06   Verbose  |         TempDirectory: C:\Windows\TEMP\
09:40:06   Verbose  |         HostProcessName: Calamari
09:40:06   Verbose  |         Executing 'D:\Octopus\Work\20170222084005-259\Script.ps1'
09:40:07   Verbose  |         Name                           Value
09:40:07   Verbose  |         ----                           -----
09:40:07   Verbose  |         PSVersion                      5.0.10586.117
09:40:07   Verbose  |         PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
09:40:07   Verbose  |         BuildVersion                   10.0.10586.117
09:40:07   Verbose  |         CLRVersion                     4.0.30319.42000
09:40:07   Verbose  |         WSManStackVersion              3.0
09:40:07   Verbose  |         PSRemotingProtocolVersion      2.3
09:40:07   Verbose  |         SerializationVersion           1.1.0.1
09:40:07   Verbose  |         PowerShell Environment Information:
09:40:07   Verbose  |         OperatingSystem: Microsoft Windows NT 6.1.7601 Service Pack 1
09:40:07   Verbose  |         OsBitVersion: x64
09:40:07   Verbose  |         Is64BitProcess: True
09:40:07   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
09:40:07   Verbose  |         MachineName: MER-WEB02
09:40:07   Verbose  |         ProcessorCount: 2
09:40:07   Verbose  |         CurrentDirectory: D:\Octopus\Work\20170222084005-259
09:40:08   Verbose  |         CurrentLocation: D:\Octopus\Work\20170222084005-259
09:40:08   Verbose  |         TempDirectory: C:\Windows\TEMP\
09:40:08   Verbose  |         HostProcessName: powershell
09:40:08   Verbose  |         TotalPhysicalMemory: 6143544 KB
09:40:08   Verbose  |         AvailablePhysicalMemory: 1206952 KB
09:40:08   Verbose  |         Octopus Deploy: Calamari version 3.6.48+Branch.master.Sha.d858383672653f2c991a15d2e618cf84c99dad39
09:40:08   Verbose  |         Environment Information:
09:40:08   Verbose  |         OperatingSystem: Microsoft Windows NT 6.1.7601 Service Pack 1
09:40:08   Verbose  |         OsBitVersion: x64
09:40:08   Verbose  |         Is64BitProcess: True
09:40:08   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
09:40:08   Verbose  |         MachineName: MER-WEB02
09:40:08   Verbose  |         ProcessorCount: 2
09:40:08   Verbose  |         CurrentDirectory: D:\Octopus\Work\20170222084005-259
09:40:08   Verbose  |         TempDirectory: C:\Windows\TEMP\
09:40:08   Verbose  |         HostProcessName: Calamari
09:40:08   Verbose  |         Package Mercatus.Future.Aqua.Web version 5.0.2.1697-release-5.0.2 hash 4044bfa644dac89e308b773c82d26717c1125357 has not been uploaded.
09:40:08   Verbose  |         Finding earlier packages that have been uploaded to this Tentacle.
09:40:16   Verbose  |         Found 5 earlier versions of Mercatus.Future.Aqua.Web on this Tentacle
09:40:16   Verbose  |         - 5.0.1.1710: D:\Octopus\Files\Mercatus.Future.Aqua.Web.5.0.1.1710.nupkg-ea3132bb-66d7-4c39-a1d5-e0567d5e5f23
09:40:16   Verbose  |         - 4.1.0.1037: D:\Octopus\Files\Mercatus.Future.Aqua.Web.4.1.0.1037_A6AF40EB1EA0B14BBA757B10E49378BD.nupkg-6fee666b-b7c0-4474-8fa4-a3598bf95763
09:40:16   Verbose  |         - 4.0.10.1712-support-4.0: D:\Octopus\Files\Mercatus.Future.Aqua.Web.4.0.10.1712-support-4.0.nupkg-19800254-eeba-49d5-bb48-a759b1cf7e50
09:40:16   Verbose  |         - 4.0.1.1713-support-4.0.1: D:\Octopus\Files\Mercatus.Future.Aqua.Web.4.0.1.1713-support-4.0.1.nupkg-163e2bc7-3acf-4187-811e-fd5cb68b0e98
09:40:16   Verbose  |         - 4.0.1.1019: D:\Octopus\Files\Mercatus.Future.Aqua.Web.4.0.1.1019.nupkg-3a5abf54-fedc-4a7e-97b1-045e45be06d1
09:40:17   Info     |         Found matching version 5.0.1.1710: C:\Octopus\OctopusServer\PackageCache\feeds-1\Mercatus.Future.Aqua.Web.5.0.1.1710_E1FF3B2D97BEE642AE5A54285C6C4715.nupkg
09:40:22   Info     |         Delta for package Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2 successfully uploaded and applied.
                    |       
                    |         Success: Building delta for Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2
09:40:17   Info     |           Using package C:\Octopus\OctopusServer\PackageCache\feeds-1\Mercatus.Future.Aqua.Web.5.0.1.1710_E1FF3B2D97BEE642AE5A54285C6C4715.nupkg with hash 1ed8b9577c1057a79f717e34815e9fb8a3464984 for creating delta.
09:40:17   Verbose  |           Building signature file: C:\Octopus\OctopusServer\PackageCache\Mercatus.Future.Aqua.Web.5.0.1.1710_E1FF3B2D97BEE642AE5A54285C6C4715.nupkg.octosig
09:40:17   Verbose  |           - Using nearest package: C:\Octopus\OctopusServer\PackageCache\feeds-1\Mercatus.Future.Aqua.Web.5.0.1.1710_E1FF3B2D97BEE642AE5A54285C6C4715.nupkg
09:40:17   Verbose  |           Building delta file: C:\Octopus\OctopusServer\PackageCache\Mercatus.Future.Aqua.Web.5.0.1.1710_to_5.0.2.1697-release-5.0.2.octodelta
09:40:17   Verbose  |           - Using package: C:\Octopus\OctopusServer\PackageCache\feeds-1\Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2_6FF1991BB2A7F046ABA4937EAF948C8B.nupkg.
09:40:17   Verbose  |           - Using signature: C:\Octopus\OctopusServer\PackageCache\Mercatus.Future.Aqua.Web.5.0.1.1710_E1FF3B2D97BEE642AE5A54285C6C4715.nupkg.octosig
09:40:20   Info     |           Original package was 68,4 MB, delta file is 11,057 MB (83,83% size reduction).
                    |         
                    |           Success: Uploading and applying delta Mercatus.Future.Aqua.Web.5.0.1.1710_to_5.0.2.1697-release-5.0.2.octodelta
09:40:21   Verbose  |             Octopus Deploy: Calamari version 3.6.48+Branch.master.Sha.d858383672653f2c991a15d2e618cf84c99dad39
09:40:21   Verbose  |             Environment Information:
09:40:21   Verbose  |             OperatingSystem: Microsoft Windows NT 6.1.7601 Service Pack 1
09:40:21   Verbose  |             OsBitVersion: x64
09:40:21   Verbose  |             Is64BitProcess: True
09:40:21   Verbose  |             CurrentUser: NT AUTHORITY\SYSTEM
09:40:21   Verbose  |             MachineName: MER-WEB02
09:40:21   Verbose  |             ProcessorCount: 2
09:40:21   Verbose  |             CurrentDirectory: D:\Octopus\Work\20170222084020-260
09:40:21   Verbose  |             TempDirectory: C:\Windows\TEMP\
09:40:21   Verbose  |             HostProcessName: Calamari
09:40:21   Info     |             Applying delta to D:\Octopus\Files\Mercatus.Future.Aqua.Web.5.0.1.1710.nupkg-ea3132bb-66d7-4c39-a1d5-e0567d5e5f23 with hash 1ed8b9577c1057a79f717e34815e9fb8a3464984 and storing as D:\Octopus\Files\Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2.nupkg-38c238ce-1ea2-43ef-8354-750670ea6831
09:40:21   Info     |             Applying delta: 0%
09:40:21   Info     |             Applying delta: 10%
09:40:21   Info     |             Applying delta: 30%
09:40:22   Info     |             Applying delta: 40%

The log from 3.10.0:

|   == Success: Acquire packages ==
09:49:54   Info     |     Acquiring packages
09:49:54   Info     |     Making a list of packages to acquire
09:49:54   Verbose  |     No packages are required on the Octopus Server
09:49:54   Info     |     Delta compression is enabled for package transfers from the Octopus Server to deployment targets
09:49:54   Verbose  |     Upload Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2 to mer-web02
09:49:54   Verbose  |     Checking package cache for package Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2
09:49:54   Info     |     Package Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2 was found in cache. No need to download.
09:49:54   Verbose  |     Using file: C:\Octopus\OctopusServer\PackageCache\feeds-1\Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2_6FF1991BB2A7F046ABA4937EAF948C8B.nupkg
09:50:00   Info     |     All packages have been acquired
09:50:00   Verbose  |     Acquire Packages completed
                    |   
                    |     Success: mer-web02
                    |     
                    |       Success: Upload package Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2
09:49:55   Info     |         This server does not have version 3.6.46 of the Calamari package. It will be pushed automatically.
09:49:56   Verbose  |         Checking for D:\Octopus\Calamari\3.6.46\Success.txt
09:49:57   Verbose  |         Package: D:\Octopus\Work\20170222084955-270\Calamari.3.6.46.nupkg
09:49:57   Verbose  |         Destination: D:\Octopus\Calamari\3.6.46
09:49:57   Verbose  |         Octopus Deploy: Tentacle version 3.10.1 (3.10.1+Branch.master.Sha.a11eb4828f3e500637cc6966c8a62c6c6c7f3571) instance Default
09:49:57   Verbose  |         Environment Information:
09:49:57   Verbose  |         OperatingSystem: Microsoft Windows NT 6.1.7601 Service Pack 1
09:49:57   Verbose  |         OsBitVersion: x64
09:49:57   Verbose  |         Is64BitProcess: True
09:49:57   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
09:49:57   Verbose  |         MachineName: MER-WEB02
09:49:57   Verbose  |         ProcessorCount: 2
09:49:57   Verbose  |         CurrentDirectory: D:\Octopus\Work\20170222084955-270
09:49:57   Verbose  |         TempDirectory: C:\Windows\TEMP\
09:49:57   Verbose  |         HostProcessName: Tentacle
09:49:58   Verbose  |         110 files extracted
09:49:58   Verbose  |         Calamari 3.6.46 extracted to D:\Octopus\Calamari\3.6.46
09:49:58   Verbose  |         Cleaning up old Calamari versions...
09:49:58   Verbose  |         Keeping only Calamari version 3.6.46...
09:49:58   Verbose  |         Removing D:\Octopus\Calamari\3.6.48...
09:49:58   Info     |         Calamari upgrade successful
09:49:58   Verbose  |         Octopus Deploy: Calamari version 3.6.46+Branch.master.Sha.6df0a1ca3dc9380cdee828e0d21425b4374797fc
09:49:58   Verbose  |         Environment Information:
09:49:58   Verbose  |         OperatingSystem: Microsoft Windows NT 6.1.7601 Service Pack 1
09:49:58   Verbose  |         OsBitVersion: x64
09:49:58   Verbose  |         Is64BitProcess: True
09:49:58   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
09:49:58   Verbose  |         MachineName: MER-WEB02
09:49:58   Verbose  |         ProcessorCount: 2
09:49:58   Verbose  |         CurrentDirectory: D:\Octopus\Work\20170222084958-271
09:49:58   Verbose  |         TempDirectory: C:\Windows\TEMP\
09:49:58   Verbose  |         HostProcessName: Calamari
09:49:59   Verbose  |         Executing 'D:\Octopus\Work\20170222084958-271\Script.ps1'
09:49:59   Verbose  |         Name                           Value
09:49:59   Verbose  |         ----                           -----
09:49:59   Verbose  |         PSVersion                      5.0.10586.117
09:49:59   Verbose  |         PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
09:49:59   Verbose  |         BuildVersion                   10.0.10586.117
09:49:59   Verbose  |         CLRVersion                     4.0.30319.42000
09:49:59   Verbose  |         WSManStackVersion              3.0
09:49:59   Verbose  |         PSRemotingProtocolVersion      2.3
09:49:59   Verbose  |         SerializationVersion           1.1.0.1
09:49:59   Verbose  |         PowerShell Environment Information:
09:49:59   Verbose  |         OperatingSystem: Microsoft Windows NT 6.1.7601 Service Pack 1
09:49:59   Verbose  |         OsBitVersion: x64
09:49:59   Verbose  |         Is64BitProcess: True
09:49:59   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
09:49:59   Verbose  |         MachineName: MER-WEB02
09:49:59   Verbose  |         ProcessorCount: 2
09:49:59   Verbose  |         CurrentDirectory: D:\Octopus\Work\20170222084958-271
09:49:59   Verbose  |         TempDirectory: C:\Windows\TEMP\
09:49:59   Verbose  |         HostProcessName: powershell
09:49:59   Verbose  |         TotalPhysicalMemory: 6143544 KB
09:49:59   Verbose  |         AvailablePhysicalMemory: 1057376 KB
09:49:59   Verbose  |         Octopus Deploy: Calamari version 3.6.46+Branch.master.Sha.6df0a1ca3dc9380cdee828e0d21425b4374797fc
09:49:59   Verbose  |         Environment Information:
09:49:59   Verbose  |         OperatingSystem: Microsoft Windows NT 6.1.7601 Service Pack 1
09:49:59   Verbose  |         OsBitVersion: x64
09:49:59   Verbose  |         Is64BitProcess: True
09:49:59   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
09:49:59   Verbose  |         MachineName: MER-WEB02
09:49:59   Verbose  |         ProcessorCount: 2
09:49:59   Verbose  |         CurrentDirectory: D:\Octopus\Work\20170222084958-271
09:49:59   Verbose  |         TempDirectory: C:\Windows\TEMP\
09:49:59   Verbose  |         HostProcessName: Calamari
09:50:00   Verbose  |         Package Mercatus.Future.Aqua.Web 5.0.2.1697-release-5.0.2 hash 4044bfa644dac89e308b773c82d26717c1125357 has already been uploaded
09:50:00   Info     |         Package Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2 (68,4 MB) found in cache. No need to upload this 68,4 MB package. Using D:\Octopus\Files\Mercatus.Future.Aqua.Web.5.0.2.1697-release-5.0.2.nupkg-38c238ce-1ea2-43ef-8354-750670ea6831

My guess is that this delta between the old version and the new, goes very very wrong.

Hi Halstein,

Thanks for getting in touch! I don’t think we have changed anything in relating to delta compression between those two versions.

I can see however from your second log that delta compression was not required as the package was already present on the destination server.

It is possible that the delta compression messed up the remote file, but I would have expected it to fail to unzip. If it does happen again, please check the package on the destination server (rename .nupkg to .zip).

I’d still like to look into this, so are you able to provide the full log of the 3.10.1 deployment (you can mark this issue as private)?

In case you are not aware, it is possible to move between patch versions of Octopus without needing to restore the database.

Hope that helps!

Robert W