Octo.exe fails to push package because it is being used by another process

Summary

Part of our deployment process is to use octo.exe to pack a deployment package and then push it to our octopus server. I keep getting an error whenever we try to push the package.

Due to the sensitive data I have I replaced some of the contents with tags (i.e. <tag_name>

Also I should note that this has worked in the past. We updated to 2018.2.1 from 2018.1.2 on March 20th.

Versions:
octo.exe: 4.31.2
Octopus Server: 2018.2.1

I understand that our server is a few versions behind but I need to know if this will fix it before I can begin to plan an upgrade. After looking through the release notes that doesn’t seem to be the case.

As always, if you need more information to help me fix this issue just ask.

Thanks

Procedure

Write-Output "=============================================================="
Write-Output "====================== Creating package ======================"
Write-Output "=============================================================="
& "$OctopusExe" pack --id=<package_name> --format="nupkg" --version=<version> --basePath=<base_dir> --include=<content> --overwrite

Write-Output "=============================================================="
Write-Output "====================== Pushing package ======================="
Write-Output "=============================================================="
& "$OctoExe" push --server=<octopus_server> --apiKey=<api_key> --package=<package_path> --replace-existing

Error and Logs

octo.exe Error

Octopus Deploy Command Line Tool, version 4.31.2

Handshaking with Octopus server: http://<our_octopus_server>/
Handshake successful. Octopus version: 2018.2.1; API version: 3.0.0
Authenticated as: <service_account>
Pushing package: <package_path.nupkg>
Octopus Server returned an error: The process cannot access the file '<octopus_server_package_path.nupkg>' because it is being used by another process.
Error from Octopus server (HTTP 500 InternalServerError)
Exit code: -7

Octopus Deploy Server Log

Unhandled error on request: http://<octopus_server>/api/packages/raw?replace=True 8fb99776dae347a7a4b9553d8cd3de6c by <service_account> : The process cannot access the file '<octopus_server_package_path.nupkg>' because it is being used by another process.
System.IO.IOException: The process cannot access the file '<octopus_server_package_path.nupkg>' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at Octopus.Core.BuiltInFeed.BuiltInPackageRepository.AddPackage(IndexedPackage package, String srcPath)
   at Octopus.Server.Web.Api.NuGet.PackagePushActionBase`1.Execute()
   at Octopus.Server.Web.Infrastructure.Api.Responder`1.ExecuteRegistered()
   at Octopus.Server.Web.Infrastructure.Api.Responder`1.Respond(TDescriptor options, NancyContext context)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
   at CallSite.Target(Closure , CallSite , Object , Object , NancyContext )
   at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass14_0.<get_Routes>b__1(Object x)
   at CallSite.Target(Closure , CallSite , Func`2 , Object )
   at Nancy.Routing.Route.<>c__DisplayClass4.<Wrap>b__3(Object parameters, CancellationToken context)

Hi Jacob,

Thanks for getting in touch.

Unfortunately we’ve been unable to reproduce this problem on version 2018.1.2 and we don’t believe upgrading will fix this issue.

Can you confirm if this is only occurring for a specific package? If you restart your Octopus Server then retry, does the issue still occur? (you can test without Octo.exe by manually uploading the package via Library > Packages > Upload Package with “Replace this version if it exists” checked).

If restarting your Octopus Server does not release the lock on the file, you may need to investigate manually on the server to determine what process is locking the file.

As a workaround, are you able to consider bumping your version numbers so you upload a fresh package version each time, rather than replacing/overwriting an existing package?

Cheers
Mark

We were finally able to restart our Octopus Deploy server this morning and it seems like the error had been resolved but after a few more deployments it happened again

Some notes on the issue:

It is not package specific, but it appears to only happen on our build servers. Also, the logs make it look like it is happening when the package lands in the Octopus Deploy temp directory on our Octopus Deploy server. If I run the same script from my workstation I don’t get this issue. Even after releasing all all file locks with something like SysInternals it still fails to push the package. Uploading the package with the web portal works just fine.

Hi Jacob,

Thanks for the additional info. It’s strange that it appears to only happen when calling Octo.exe from your build servers.

Could you please confirm if any Anti Virus software is running on your Octopus server? We typically see cases where the AV software is scanning the package folder and locking package files like this.

If you do have any AV software running on your Octopus server, we recommend adding the Octopus packages folder as an exclusion.

If you do not have any AV software running, are you able to identify what process is locking the file when this problem occurs?

Cheers
Mark

The solution was updating our AV white-list. Our AV was hogging the package.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.