Error 500 when pushing package

Hi,
I have a problem where I can’t push a specific package to our octopus server.
We just updated our server to version 2019.9.10. It was working fine before, I do not know the version we were before as I was not the one doing the update.

We have a Jenkins CI/CD creating a package and pushing it to the octopus server.
When pushing the package, I get this error :

Octo.exe push --package App.1.148.zip --replace-existing --server "http://octopusserver/" --apiKey **** 
Octopus Deploy Command Line Tool, version 7.4.1

Detected automation environment: "Jenkins"
Space name unspecified, process will run in the default space context
Handshaking with Octopus Server: http://octopusserver/
Handshake successful. Octopus version: 2019.9.10; API version: 3.0.0
Authenticated as: xxxxxxx
Pushing package: D:\xxxxxxxx\App.1.148.zip...
Requesting signature for delta compression from the server for upload of a package with id 'App' and version '1.148'
Something went wrong while performing a delta transfer: Octopus Server returned an error: Object reference not set to an instance of an object.
Falling back to pushing the complete package to the server
Octopus Server returned an error: Object reference not set to an instance of an object.
Error from Octopus Server (HTTP 500 InternalServerError)

I tried packaging the package with octo pack, 7zip and powershell compress-archive. Nothing changed.

Then, when I go on the server in Library -> Package, I don’t see my package. If I open the explorer on the server, I can see the package inside the C:\Octopus\Packages\Spaces-1\feeds-builtin\App folder.

I tried to re-index but I get an error when re-indexing the package;

August 13th 2020 09:35:02Warning
Error inserting package details into database. Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
   at Octopus.Server.Web.Api.ModelRestrictionMap.PackageUsages.<>c__DisplayClass1_0.<UsesPackage>b__2(PackageReference p) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Api\ModelRestrictionMap\PackageUsages.cs:line 30
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
   at System.Linq.Enumerable.<DistinctIterator>d__64`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Octopus.Server.Web.Api.ModelRestrictionMap.PackageRestriction.CheckPermissionInProjects(IndexedPackage model, ISpacePartitionPrincipal principal, IOctopusQueryExecutor queryExecutor, Permission permissionToCheck)
   at Octopus.Server.Web.Infrastructure.Security.DocumentRestrictions.CompositeAndRestriction`2.CombineWithAnd(Func`2 check) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Infrastructure\Security\DocumentRestrictions\CompositeAndRestriction.cs:line 44
   at Octopus.Server.Web.Infrastructure.Security.DocumentRestrictions.CompositeAndRestriction`2.CombineWithAnd(Func`2 check) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Infrastructure\Security\DocumentRestrictions\CompositeAndRestriction.cs:line 44
   at Octopus.Server.Web.Infrastructure.Security.AccessChecker`1.CanCreate(TModel model)
   at Octopus.Core.Security.Permissions.AccessCheckerExtensions.AssertCanCreate[TModel](IAccessChecker`1 accessChecker, TModel model) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Security\Permissions\IAccessChecker.cs:line 46
   at Octopus.Server.Web.Infrastructure.OctopusQueryExecutor.Insert[TDocument](TDocument instance, Nullable`1 commandTimeout) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Infrastructure\OctopusQueryExecutor.cs:line 200
   at Octopus.Core.BuiltInFeed.BuiltInPackageIndex.AddPackage(IndexedPackage package) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\BuiltInFeed\BuiltInPackageIndex.cs:line 29
   at Octopus.Server.Orchestration.ServerTasks.SynchronizeBuiltInPackageRepositoryIndex.SynchronizeBuiltInPackageRepositoryIndexTaskController.AddPackagesToIndex(ICollection`1 packages) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\SynchronizeBuiltInPackageRepositoryIndex\SynchronizeBuiltInPackageRepositoryIndexTaskController.cs:line 207
   at Octopus.Server.Orchestration.ServerTasks.SynchronizeBuiltInPackageRepositoryIndex.SynchronizeBuiltInPackageRepositoryIndexTaskController.IndexPackages() in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Orchestration\ServerTasks\SynchronizeBuiltInPackageRepositoryIndex\SynchronizeBuiltInPackageRepositoryIndexTaskController.cs:line 181
Octopus.Server version 2019.9.10 (2019.9.10+Branch.tags-2019.9.10.Sha.ca4f2468aae1d241b9680055dec48ab3aca34a4a)

I also tried disabling the antivirus and it still do not work.

We have multiple project with different package, and every ones works except this one. When I rename the package, it works but as soon as I try to push a second version, it crash.

Anybody have an idea on what might be the problem?

Hi @jean-philippe.boudreau,

Thanks for getting in touch!

Could you try removing the package from the C:\Octopus\Packages\Spaces-1\feeds-builtin\App folder and then running the re-index task.
If you can then check the re-index task to see if the package was removed from the database or if an error occurs?

Assuming it runs ok, can you then try re-adding the package and see if the same error occurs?

Regards,
Paul

Hi @paul.calvert,
If I remove the package from the app folder, the re-index works.

August 14th 2020 07:08:32Info
Built-in package repository re-indexed with 0 new packages. 
August 14th 2020 07:08:32Info
Removed 0 packages from built-in repository. 
August 14th 2020 07:08:32Info
Built-in package repository successfully re-indexed. 

But, as soon as I push the package again, the error occurs again.

Space name unspecified, process will run in the default space context
Handshaking with Octopus Server: http://octopusserver/
Handshake successful. Octopus version: 2019.9.10; API version: 3.0.0
Authenticated as: xxxxxx 
Pushing package: D:\xxxx\App.1.149.nupkg...
Requesting signature for delta compression from the server for upload of a package with id 'App' and version '1.149'
Something went wrong while performing a delta transfer: Octopus Server returned an error: Object reference not set to an instance of an object.
Falling back to pushing the complete package to the server

On the octopus server I see this in octopuserver.txt log

   
2020-08-14 07:14:27.2783   7064      3 ERROR  Unhandled error on request: http://octopusserver/api/Spaces-1/packages/raw?overwriteMode=OverwriteExisting 281749fad80a434c90f287ff103e22fe by <anonymous> : Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
   at Octopus.Server.Web.Api.ModelRestrictionMap.PackageUsages.<>c__DisplayClass1_0.<UsesPackage>b__2(PackageReference p) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Api\ModelRestrictionMap\PackageUsages.cs:line 30
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
   at System.Linq.Enumerable.<DistinctIterator>d__64`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Octopus.Server.Web.Api.ModelRestrictionMap.PackageRestriction.CheckPermissionInProjects(IndexedPackage model, ISpacePartitionPrincipal principal, IOctopusQueryExecutor queryExecutor, Permission permissionToCheck)
   at Octopus.Server.Web.Infrastructure.Security.DocumentRestrictions.CompositeAndRestriction`2.CombineWithAnd(Func`2 check) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Infrastructure\Security\DocumentRestrictions\CompositeAndRestriction.cs:line 44
   at Octopus.Server.Web.Infrastructure.Security.DocumentRestrictions.CompositeAndRestriction`2.CombineWithAnd(Func`2 check) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Infrastructure\Security\DocumentRestrictions\CompositeAndRestriction.cs:line 44
   at Octopus.Server.Web.Infrastructure.Security.AccessChecker`1.CanCreate(TModel model)
   at Octopus.Core.Security.Permissions.AccessCheckerExtensions.AssertCanCreate[TModel](IAccessChecker`1 accessChecker, TModel model) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\Security\Permissions\IAccessChecker.cs:line 46
   at Octopus.Server.Web.Infrastructure.OctopusQueryExecutor.Insert[TDocument](TDocument instance, Nullable`1 commandTimeout) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Infrastructure\OctopusQueryExecutor.cs:line 200
   at Octopus.Core.BuiltInFeed.BuiltInPackageIndex.AddPackage(IndexedPackage package) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\BuiltInFeed\BuiltInPackageIndex.cs:line 29
   at Octopus.Core.BuiltInFeed.BuiltInPackageRepository.AddPackage(IndexedPackage package, String srcPath) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Core\BuiltInFeed\BuiltInPackageRepository.cs:line 247
   at Octopus.Server.Web.Api.NuGet.PackagePushActionBase`1.Process(String temp) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Api\NuGet\PackagePushActionBase.cs:line 130
   at Octopus.Server.Web.Api.NuGet.PackagePushActionBase`1.Execute() in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Api\NuGet\PackagePushActionBase.cs:line 92
   at Octopus.Server.Web.Infrastructure.Api.Responder`1.ExecuteRegistered() in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Infrastructure\Api\Responder.cs:line 297
   at Octopus.Server.Web.Infrastructure.Api.Responder`1.Respond(TDescriptor options, NancyContext context) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Infrastructure\Api\Responder.cs:line 281
   at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
   at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass14_0.<get_Routes>b__1(Object o, CancellationToken x) in C:\buildAgent\work\abb2fbfce959a439\source\Octopus.Server\Web\Infrastructure\OctopusNancyModule.cs:line 122
   at Nancy.Routing.Route`1.<Invoke>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Nancy.Routing.DefaultRouteInvoker.<Invoke>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Nancy.Routing.DefaultRequestDispatcher.<Dispatch>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Nancy.NancyEngine.<InvokeRequestLifeCycle>d__22.MoveNext()

Would you be able to check the Octopus database and search the dbo.packages table to see if there is an entry for this package? My guess is that there won’t be, but would be good to be certain.

Hi,
No the package is not in the dbo.NugetPackage. (I don’t have a dbo.packages, so I assume you meant NugetPackage?)

I figured as much.

Been discussing this internally and we found a couple similar reports on this version but no clear solution.
The best option may be to try upgrading the server to a more recent version and see if the issue resolves itself.

Another possible option to try would be to remove the --replace-existing flag from the octo.exe push command. As the error is initially appearing when it attempts to generate the delta transfer, I’m wondering if that flag may be causing the problem.

Also, when you say that you renamed the package and it worked, but then failed on a second version. Do you mean that you added package App.1.149.zip fine, but then when you tried to re-add this same package again and overwrite it, it failed?

Hi,
I tried without the --replace-existing and it does the same error.

What I meant by renaming the package is, let say I have App.1.149.nupkg that crash. If I rename to App2.xxxx.nukg, the first push works. But as soon as I push a second one, it crash with the same error.

With our current license we can’t update to newest version. I’ll check what I can do.

If you are able to upload a backup of your database we can take a deeper look to see if we can find a fix for this.

If you have an account at Octopus.com there is a Support section where you can upload files. If that isn’t available you can upload to our smartfile location here: https://file.ac/PaAU0o04YB8/

We don’t need your master key, that will keep sensitive data you store in Octopus secure as we won’t be able to decrypt it. We also scrub any personal identifiable information from the database before we start working on the database.

Hi,
I don’t think I will be able to upload a backup of our database.

I tried to create a new project and add the same steps and it seems to be working. The only difference I can see is I tried to add a Flyway community steps a couple of week ago in the old project.

When I was looking at similar issues to this one, there was one that was being caused by a null value buried in a project’s packageid field. They were experiencing a slightly different form of the error but it could be something similar which would explain why the new project has resolved this.

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.