Apply retention policy on Tentacles failing after upgrade to 3.12.6

After we upgraded to Octopus version 3.12.6 today deployments started failing all over the place. They all fail in the “Apply retention policy on Tentacles” step with the same error. Reading the release notes it looks like this is related to the changes around long directory paths. The relevant paths don’t can’t be considered long, but they fail anyway. This is a blocking issue for us so I hope you can provide a quick fix

13:21:38   Verbose  |         Octopus Deploy: Calamari version 3.7.53+Branch.master.Sha.416708ce8cf8a2361f341285ce3655453a1be6f6
13:21:38   Verbose  |         Environment Information:
13:21:38   Verbose  |         OperatingSystem: Microsoft Windows NT 6.2.9200.0
13:21:38   Verbose  |         OsBitVersion: x64
13:21:38   Verbose  |         Is64BitProcess: True
13:21:38   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
13:21:38   Verbose  |         MachineName: CO-OSL-TENTA145
13:21:38   Verbose  |         ProcessorCount: 4
13:21:38   Verbose  |         CurrentDirectory: C:\Octopus\Work\20170421112137-75
13:21:38   Verbose  |         TempDirectory: C:\Windows\TEMP\
13:21:38   Verbose  |         HostProcessName: Calamari
13:21:38   Info     |         Keeping this deployment and the previous 1 successful deployments
13:21:38   Verbose  |         Keeping C:\Applications\master.Build.Testlab\Confirmit.TaskSystem\22.0.384 and C:\Octopus\Files\feeds-1\Confirmit.TaskSystem.22.0.384_E50F7F74E941694E996427BF20E94312.nupkg as it is the most recent successful release
13:21:38   Verbose  |         Keeping C:\Applications\master.Build.Testlab\Confirmit.TaskSystem\22.0.374_1 and C:\Octopus\Files\feeds-1\Confirmit.TaskSystem.22.0.374_7C9B93DD0281FE41A10B296A8B45D516.nupkg as it is the 2nd most recent successful release
13:21:38   Info     |         Removing directory 'C:\Applications\master.Build.Testlab\Confirmit.TaskSystem\22.0.374'
13:21:39   Error    |         (87) The parameter is incorrect: [\\?\C:\Applications\master.Build.Testlab\Confirmit.TaskSystem\22.0.374\Tasks\customcode]
13:21:39   Error    |         System.IO.IOException
13:21:39   Error    |         at Alphaleonis.Win32.NativeError.ThrowException(UInt32 errorCode, String readPath, String writePath)
13:21:39   Error    |         at Alphaleonis.Win32.Filesystem.Directory.DeleteDirectoryCore(FileSystemEntryInfo fileSystemEntryInfo, KernelTransaction transaction, String path, Boolean recursive, Boolean ignoreReadOnly, Boolean continueOnNotExist, PathFormat pathFormat)
13:21:39   Error    |         at Calamari.Integration.FileSystem.LongPathsDirectory.Delete(String path, Boolean recursive) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\FileOperations.cs:line 77
13:21:39   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, Predicate`1 exclude, FailureOptions options, CancellationToken cancel, Boolean includeTarget) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 414
13:21:39   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, Predicate`1 exclude, FailureOptions options, CancellationToken cancel, Boolean includeTarget) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 403
13:21:39   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, FailureOptions options) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 369
13:21:39   Error    |         at Calamari.Deployment.Retention.RetentionPolicy.DeleteExtractionDestination(JournalEntry deployment, List`1 preservedEntries) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Deployment\Retention\RetentionPolicy.cs:line 85
13:21:39   Error    |         at Calamari.Deployment.Retention.RetentionPolicy.ApplyRetentionPolicy(String retentionPolicySet, Nullable`1 days, Nullable`1 releases) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Deployment\Retention\RetentionPolicy.cs:line 62
13:21:39   Error    |         at Calamari.Commands.CleanCommand.Execute(String[] commandLineArguments) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Commands\CleanCommand.cs:line 44
13:21:39   Error    |         at Calamari.Program.Execute(String[] args) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Program.cs:line 45

Hi Jan,

Thanks for reporting this issue. As a temporary workaround could you disable retention policies on your Lifecycle? It will not require a new release to be created.
Is it happening for all environments or only a few?

Vanessa

This is happening on several environments and is blocking our build pipeline.
I can try to disable retention policies for short while, but disks will fill up quickly if I do this

I’ve an Octopus installation on my dev machine and there I don’t see this problem.

Hi Jan,

You’re right, that looks like an issue to do with the changes around long paths, but not because that’s specifically a long path. It looks like a library we’re using to deal with long file paths is failing on a recursive delete.

To help diagnose, can you have a look at the folder it’s trying to delete and let us know what’s in it? The one in the log you pasted is: C:\Applications\master.Build.Testlab\Confirmit.TaskSystem\22.0.374.

There appears to be a folder tree under there that’s causing the exception when Calamari tries to delete it: Tasks\customcode.

Could you try deleting the whole 22.0.374 folder manually (ideally using the same account that Calamari is using) and see if that succeeds?

Thanks,
Damian

This has been deployed on hundreds of machines and hundreds of releases prior to 3.12.6 so I don’t believe it is an issue with the directory itself

I think I’ve found what is triggering the problem: All the directories that fail on delete are actually junctions. The junctions should not be traversed, but just deleted. This has worked fine in the past.

We typically use junctions to add some shared libraries/folders to the applications. Has worked fine for years.

I’ve another one for you here:

                    |       Running: Applying retention policy using 1 Items to set Environments-120/Projects-7/Step-SurveyEngine/Machines-524/<default>
13:58:44   Verbose  |         Octopus Deploy: Calamari version 3.7.53+Branch.master.Sha.416708ce8cf8a2361f341285ce3655453a1be6f6
13:58:44   Verbose  |         Environment Information:
13:58:44   Verbose  |         OperatingSystem: Microsoft Windows NT 6.2.9200.0
13:58:44   Verbose  |         OsBitVersion: x64
13:58:44   Verbose  |         Is64BitProcess: True
13:58:44   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
13:58:44   Verbose  |         MachineName: CO-OSL-TENTA185
13:58:44   Verbose  |         ProcessorCount: 2
13:58:44   Verbose  |         CurrentDirectory: C:\Octopus\Work\20170421115844-76
13:58:44   Verbose  |         TempDirectory: C:\Windows\TEMP\
13:58:44   Verbose  |         HostProcessName: Calamari
13:58:44   Info     |         Keeping this deployment and the previous 1 successful deployments
13:58:44   Verbose  |         Keeping C:\Applications\Master.Selenium.Testlab\Firmglobal.Confirmit.SurveyEngine.Web\22.0.384 and C:\Octopus\Files\feeds-1\Firmglobal.Confirmit.SurveyEngine.Web.22.0.384_AE37837D50526C43903387A2667AC644.nupkg as it is the most recent successful release
13:58:44   Verbose  |         Keeping C:\Applications\Master.Selenium.Testlab\Firmglobal.Confirmit.SurveyEngine.Web\22.0.374_2 and C:\Octopus\Files\feeds-1\Firmglobal.Confirmit.SurveyEngine.Web.22.0.374_02765924AB75C640B0F5AB9C6FE86D5D.nupkg as it is the 2nd most recent successful release
13:58:44   Info     |         Removing directory 'C:\Applications\Master.Selenium.Testlab\Firmglobal.Confirmit.SurveyEngine.Web\22.0.374_1'
13:58:45   Error    |         (87) The parameter is incorrect: [\\?\C:\Applications\Master.Selenium.Testlab\Firmglobal.Confirmit.SurveyEngine.Web\22.0.374_1\wix\bin]
13:58:45   Error    |         System.IO.IOException
13:58:45   Error    |         at Alphaleonis.Win32.NativeError.ThrowException(UInt32 errorCode, String readPath, String writePath)
13:58:45   Error    |         at Alphaleonis.Win32.Filesystem.Directory.DeleteDirectoryCore(FileSystemEntryInfo fileSystemEntryInfo, KernelTransaction transaction, String path, Boolean recursive, Boolean ignoreReadOnly, Boolean continueOnNotExist, PathFormat pathFormat)
13:58:45   Error    |         at Calamari.Integration.FileSystem.LongPathsDirectory.Delete(String path, Boolean recursive) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\FileOperations.cs:line 77
13:58:45   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, Predicate`1 exclude, FailureOptions options, CancellationToken cancel, Boolean includeTarget) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 414
13:58:45   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, Predicate`1 exclude, FailureOptions options, CancellationToken cancel, Boolean includeTarget) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 403
13:58:45   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, FailureOptions options) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 369
13:58:45   Error    |         at Calamari.Deployment.Retention.RetentionPolicy.DeleteExtractionDestination(JournalEntry deployment, List`1 preservedEntries) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Deployment\Retention\RetentionPolicy.cs:line 85
13:58:45   Error    |         at Calamari.Deployment.Retention.RetentionPolicy.ApplyRetentionPolicy(String retentionPolicySet, Nullable`1 days, Nullable`1 releases) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Deployment\Retention\RetentionPolicy.cs:line 62
13:58:45   Error    |         at Calamari.Commands.CleanCommand.Execute(String[] commandLineArguments) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Commands\CleanCommand.cs:line 44
13:58:45   Error    |         at Calamari.Program.Execute(String[] args) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Program.cs:line 45
                    |       
                    |     Failed: CO-OSL-TENTA183
13:58:52   Fatal    |       The remote script failed with exit code 100

Hi Jan,

Ah, yes, that will definitely be it.

I’ve had a look and this will require some work to resolve. Bear with me, I’m trying to work out how to downgrade your Calamari to the previous version - that’s likely to be the safest option until we can create a fix.

Damian

It turns out to be easy to reproduce. Just create a junction in one of the deplyoment directories that will be cleaned up. Retention policy will always fail.

In the fix, make sure that the retention policy don’t delete the content of the junction target directory. This is the behavior prior to version 3.12.6. Deleting the content of the junction target directory will have VERY bad effects

A downgrade of Calamari sounds like the best short term option. I’ve blocked all attempts for deployment to production now until this is resolved or Calamari is downgraded on all hundreds of machines

Hi Jan,

It looks like you may just be able to deploy the previous version of Octopus Server over the top, and it will downgrade the Calamari version for you when the next deployment occurs. The version is checked as part of the deployment, so it should have immediate effect.

Are you able to try this and let me know if it resolves the issue for the time being?

I’m really sorry for the problems here - it’s an edge case we didn’t consider. And yes, definitely noted that we don’t want to delete the contents of the junction.

Damian

Hi Damian,

We have installed Octopus using chocolatey. What procedure should I follow to downgrade from version 3.12.6 to 3.12.5?

All tentacles have also been upgraded to 3.12.6. Will/can they still stay on 3.12.6?

/Jan

I’ll try with this one:
choco install OctopusDeploy --version 3.12.5 -yf

and keeping the tentacles on version 3.12.6

Hi Jan,

I’m not an expert on Chocolately, but I think you can downgrade an installation using these instructions. I’d recommend not doing an uninstall and reinstall (although that should still work), but doing an install and specifying the previous version instead.

The tentacles should be fine running a version higher. Because it’s a patch release, the versions should be compatible.

I’ve created an issue in GitHub that you can follow along with as well: https://github.com/OctopusDeploy/Issues/issues/3432

Thanks,
Damian

Hi Damian,

Downgrade to 3.12.5 on the server fixed the issue. Calamari was downgraded also and are now working again.

Thanks for your help

We like to stay current on the Octopus version so hopefully the next version will have a fix for this issue

Thanks,
Jan