Nuget Push fails - Internal server error - The directory is not empty

When pushing an octopack to the nuget feed of octopus it quite often results in “Internal Server Error (500)”

There is an (seemingly identical) issue here from another user which got closed:
http://help.octopusdeploy.com/discussions/problems/49803-internal-nuget-server-push-fails-with-500-internal-server-error

When looking into the logs I get this the message: “The directory is not empty.”

Best regards

// Henrik

Full log from the occasion:

2017-04-18 12:24:43.4060 12 WARN Outstanding Retention tasks were not completed before the next task was due to be scheduled. If this error persists, check the Tasks tab for any running Retention tasks, and cancel them manually.
2017-04-18 12:24:46.3889 29 INFO Reader took 2264ms (132ms until the first record): SELECT ALIAS_VariableSet_0.* FROM (SELECT * FROM dbo.[VariableSet]) ALIAS_VariableSet_0 INNER JOIN (SELECT * FROM dbo.[Deployment] WHERE ([TaskId] IN (SELECT [Id] FROM [ServerTask] WHERE [State] IN (‘Success’, ‘Failed’)))) ALIAS_Deployment_1 ON ALIAS_VariableSet_0.OwnerId = ALIAS_Deployment_1.Id ORDER BY ALIAS_VariableSet_0.Id
2017-04-18 12:25:17.3521 32 INFO Reader took 350ms (59ms until the first record): SELECT * FROM dbo.[DeploymentProcess] WHERE ([IsFrozen] = 0) ORDER BY Id
2017-04-18 12:25:23.0539 32 INFO Insert took 1654ms: INSERT INTO dbo.[EventRelatedDocument] (EventId, RelatedDocumentId) values (@0__EventId, @0__RelatedDocumentId)
2017-04-18 12:25:23.0539 32 INFO Insert took 1659ms: INSERT INTO dbo.[Event] (RelatedDocumentIds, ProjectId, EnvironmentId, TenantId, Category, UserId, Username, Occurred, Message, Id, Json) values (@RelatedDocumentIds, @ProjectId, @EnvironmentId, @TenantId, @Category, @UserId, @Username, @Occurred, @Message, @Id, @Json)
2017-04-18 12:25:25.4817 32 ERROR Unhandled error on request: http://byggserver:8080/nuget/packages/ by TeamcityPublisherAccount : The directory is not empty.

System.IO.IOException: The directory is not empty.

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound)
at System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive, Boolean checkHost)
at Octopus.Shared.Util.OctopusPhysicalFileSystem.DeleteDirectory(String path) in Y:\Work\refs\tags\3.4.12\source\Octopus.Shared\Util\OctopusPhysicalFileSystem.cs:line 85
at Octopus.Server.Web.Api.NuGet.PackagePushActionBase1.Execute() in Y:\Work\refs\tags\3.4.12\source\Octopus.Server\Web\Api\NuGet\PackagePushActionBase.cs:line 105 at Octopus.Server.Web.Infrastructure.Api.Responder1.Respond(TDescriptor options, NancyContext context) in Y:\Work\refs\tags\3.4.12\source\Octopus.Server\Web\Infrastructure\Api\Responder.cs:line 162
at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
at Octopus.Server.Web.Api.OctopusRestApiModule.<>c__DisplayClass0_0.<.ctor>b__0(Object o) in Y:\Work\refs\tags\3.4.12\source\Octopus.Server\Web\Api\OctopusRestApiModule.cs:line 46
at Nancy.Routing.Route.<>c__DisplayClass4.b__3(Object parameters, CancellationToken context)

Hi there,

Thanks for reaching out! Its very possible that the temp folder in the Octopus Server is a full, causing this problem.

If your Octopus Server service is running under a custom account, try deleting the contents of C:\Users\[user]\AppData\Local\Octopus.Server\Temp. If the service is running under Local System, then you’ll have to clean up C:\Windows\Temp.

Let me know if that works.

Thanks!
Dalmiro

Hello!

The C:\Windows\Temp contained 6mb files which are now deleted. There is 24GB available on C:. The folder can’t really be full? If files are locked causing this behavior it should be considered a bug, don’t you agree?

Let’s see if this fixed the error, we cannot know for sure until we have had some releases.

// Henrik

HENRIK CARLSSON
Utvecklare

Växel: 031-83 36 70
Mobil: 0725 – 63 00 65
henrik.carlsson@aroskapital.semailto:henrik.carlsson@aroskapital.se

Johan På Gårdas Gata 5A
412 50 Göteborg
[cid:0662D5A3-F41E-464F-9D02-9AA2E265D603]

image001.png

Hello,

This did not fix the issue, which is not really a surprise.

Please let me know what to do next.

// Henrik

HENRIK CARLSSON
Utvecklare

Växel: 031-83 36 70
Mobil: 0725 – 63 00 65
henrik.carlsson@aroskapital.semailto:henrik.carlsson@aroskapital.se

Johan På Gårdas Gata 5A
412 50 Göteborg
[cid:0662D5A3-F41E-464F-9D02-9AA2E265D603]

From: Henrik Carlsson
Sent: den 19 april 2017 08:27
To: ‘Dalmiro Grañas’ tender2+db4e9f0076@tenderapp.com
Subject: RE: Nuget Push fails - Internal server error - The directory is not empty [Problems #53692]

Hello!

The C:\Windows\Temp contained 6mb files which are now deleted. There is 24GB available on C:. The folder can’t really be full? If files are locked causing this behavior it should be considered a bug, don’t you agree?

Let’s see if this fixed the error, we cannot know for sure until we have had some releases.

// Henrik

HENRIK CARLSSON
Utvecklare

Växel: 031-83 36 70
Mobil: 0725 – 63 00 65
henrik.carlsson@aroskapital.semailto:henrik.carlsson@aroskapital.se

Johan På Gårdas Gata 5A
412 50 Göteborg
[cid:0662D5A3-F41E-464F-9D02-9AA2E265D603]

image001.png

Hi Henrik,

Sad to hear that didn’t help.

Could you please confirm:

  • Under which account is your Octopus Server service running?
  • Which version of the Octopus server are you using? From the error log I can see 3.4.12, but I just want you to confirm that for me if possible.

Regards,
Dalmiro

Hello

  • Local System
  • Yes, version 3.4.12

Regards,
Henrik

HENRIK CARLSSON
Utvecklare

Växel: 031-83 36 70
Mobil: 0725 – 63 00 65
henrik.carlsson@aroskapital.semailto:henrik.carlsson@aroskapital.se

Johan På Gårdas Gata 5A
412 50 Göteborg
[cid:0662D5A3-F41E-464F-9D02-9AA2E265D603]

image001.png

Hi Henrik,

Sorry for the delay here. I wanted to consult this one with the dev team before replying again.

So the 2 thoughts that came from the discussion where:

  1. In 3.7.6 we introduced the below fix to do a clean retry when we encounter a problem deleting temp files after a package push. Upgrading to at least this version (“latest” version is always recommended) would be the best thing you could do at the moment.
  1. Is there any chance you are running an AV scanner on your filesystem (particularly on any parent directory of the temp folder)? 9/10 times we’ve seen that AV scanners can get files locked up, causing all sorts of things like the one you are experiencing. If you upgrade as mentioned in (1), rather than failing, we’ll display a warning when the temp file cannot be deleted after a few tries.

Best regards,
Dalmiro