OctopusDeploy: Create release step fails with error “feed returned an unexpected status code '403 '”

ruf-r
(kira.resari) #1

Hello everyone.

I have a TeamCity/OctopusDeploy integration that used to work just fine before, but suddenly fails without anything having be changed in the configuration.

The only thing that changed is that both TeamCity and OctopusDeploy were updated in the meantime. Does anyone have any idea what causes this, and how can I fix it?

Here’s a log of the last successful build:

Step 5/5: OctopusDeploy: Create release (22s)
[Step 5/5] Octopus Deploy (21s)
[Octopus Deploy] Running command: octo.exe create-release --server http://localhost:8888/octopus --apikey SECRET --project Test Configuration Dev --enableservicemessages --version 0.0.0-develop.10 --deployto Test Environment --progress --package DemoFetcher:0.0.0-develop.10
[Octopus Deploy] Creating Octopus Deploy release
[Octopus Deploy] Octopus Deploy Command Line Tool, version 5.2.6
[Octopus Deploy] 
[Octopus Deploy] Detected automation environment: "TeamCity/2018.2.4"
[Octopus Deploy] Space name unspecified, process will run in the default space context
[Octopus Deploy] Handshaking with Octopus Server: http://localhost:8888/octopus
[Octopus Deploy] Handshake successful. Octopus version: 2019.3.1; API version: 3.0.0
[Octopus Deploy] Authenticated as: Admin <admin@company.com> 
[Octopus Deploy] This Octopus Server supports channels
[Octopus Deploy] Finding project: Test Configuration Dev
[Octopus Deploy] Automatically selecting the best channel for this release...
[Octopus Deploy] Building a release plan for Channel 'Default'...
[Octopus Deploy] Finding deployment process...
[Octopus Deploy] Finding release template...
[Octopus Deploy] The package version for some steps was not specified. Going to try and resolve those automatically...
[Octopus Deploy] Finding latest package for step: Install Company Test App Config Service
[Octopus Deploy] Selected 'TestApp.test' version '0.0.0-develop.10' for 'Install Company Test App Config Service'
[Octopus Deploy] Selected the release plan for Channel 'Default' - it is a perfect match
[Octopus Deploy] Using version number provided on command-line: 0.0.0-develop.10
[Octopus Deploy] Release plan for Test Configuration Dev 0.0.0-develop.10
[Octopus Deploy] Channel: 'Default' (this is the default channel)
[Octopus Deploy] # Name Version Source Version rules 
[Octopus Deploy] --- ------------------------------------------------------- ------------------ ------------------ -------------------
[Octopus Deploy] 1 Install Company Test App Config Service 0.0.0-develop.10 Latest available Allow any version 
[Octopus Deploy] 
[Octopus Deploy] Creating release...
[Octopus Deploy] Release 0.0.0-develop.10 created successfully!
[Octopus Deploy] ##teamcity[setParameter name='octo.releaseNumber' value='0.0.0-develop.10']
[Octopus Deploy] Deploying Test Configuration Dev "0.0.0-develop.10" to: Test Environment (Guided Failure: Not Enabled)
[Octopus Deploy] Waiting for 1 deployment(s) to complete....
[Octopus Deploy] Success: Acquire packages
[Octopus Deploy] Success: Step 2: Install Company Test App Config Service
[Octopus Deploy] Success: Apply retention policy on Tentacles
[Octopus Deploy] Deploy Test Configuration Dev release 0.0.0-develop.10 to Test Environment: Success
[Octopus Deploy] Done!
[Octopus Deploy] Octo.exe exit code: 0

By contrast, here’s a log of the first failing build:

Step 6/6: OctopusDeploy: Create release (2s)
[Step 6/6] Octopus Deploy (1s)
[Octopus Deploy] Running command: octo.exe create-release --server http://localhost:8888/octopus --apikey SECRET --project Test Configuration Dev --enableservicemessages --version 0.0.0-develop.17 --deployto Test Environment --progress --package DemoFetcher:0.0.0-develop.17
[Octopus Deploy] Creating Octopus Deploy release
[Octopus Deploy] Octopus Deploy Command Line Tool, version 5.2.6
[Octopus Deploy] 
[Octopus Deploy] Detected automation environment: "TeamCity/2019.1"
[Octopus Deploy] Space name unspecified, process will run in the default space context
[Octopus Deploy] Handshaking with Octopus Server: http://localhost:8888/octopus
[Octopus Deploy] Handshake successful. Octopus version: 2019.3.3; API version: 3.0.0
[Octopus Deploy] Authenticated as: Admin <admin@company.com> 
[Octopus Deploy] This Octopus Server supports channels
[Octopus Deploy] Finding project: Test Configuration Dev
[Octopus Deploy] Automatically selecting the best channel for this release...
[Octopus Deploy] Building a release plan for Channel 'Default'...
[Octopus Deploy] Finding deployment process...
[Octopus Deploy] Finding release template...
[Octopus Deploy] The package version for some steps was not specified. Going to try and resolve those automatically...
[Octopus Deploy] Finding latest package for step: Install Company Test App Config Service
[Octopus Deploy] Octopus Server returned an error: The V2 feed at 'http://collaboration.company.de/httpAuth/app/nuget/feed/_Root/default/v2/FindPackagesById()?id='TestApp.test'&semVerLevel=2.0.0' returned an unexpected status code '403 '.
[Octopus Deploy] Server exception: 
[Octopus Deploy] The V2 feed at 'http://collaboration.company.de/httpAuth/app/nuget/feed/_Root/default/v2/FindPackagesById()?id='TestApp.test'&semVerLevel=2.0.0' returned an unexpected status code '403 '.
[Octopus Deploy] NuGet.Protocol.Core.Types.FatalProtocolException
[Octopus Deploy] at NuGet.Protocol.V2FeedParser.<>c__DisplayClass53_0.<LoadXmlAsync
[Octopus Deploy] at NuGet.Protocol.HttpSource.ProcessResponseAsync
[Octopus Deploy] at NuGet.Protocol.V2FeedParser.LoadXmlAsync
[Octopus Deploy] at NuGet.Protocol.V2FeedParser.QueryV2Feed
[Octopus Deploy] at NuGet.Protocol.V2FeedParser.FindPackagesByIdAsync
[Octopus Deploy] at NuGet.Protocol.PackageMetadataResourceV2Feed.GetMetadataAsync
[Octopus Deploy] at Octopus.Shared.Util.TaskExtensions.PrepareForRethrow(Exception exception)
[Octopus Deploy] at Octopus.Shared.Util.TaskExtensions.SafeResult[T](Task`1 task, CancellationToken cancellationToken)
[Octopus Deploy] at Octopus.Core.Packages.NuGet.ExternalHttpNuGetPackageFeed.FetchPackageMetadataForAllVersionsExpensive(String packageId, Boolean allowPreRelease, CancellationToken cancellationToken)
[Octopus Deploy] at Octopus.Core.Packages.NuGet.ExternalHttpNuGetPackageFeed.ListVersions(String packageId, CancellationToken cancellationToken, PackageVersionSearchOptions searchOptions, Int32& total, Int32 skip, Int32 take, Boolean includeReleaseNotes)
[Octopus Deploy] at Octopus.Server.Web.Api.Actions.PackageSearchActionOld.Search(String feedId, String packageId, Boolean partialMatch, Boolean includeMultipleVersions, Boolean includeNotes, Int32 take, PackageVersionSearchOptions searchOptions, Boolean descriptionsOptional)
[Octopus Deploy] at Octopus.Server.Web.Api.Actions.PackageSearchActionOld.ExecuteRegistered()
[Octopus Deploy] at Octopus.Server.Web.Infrastructure.Api.Responder`1.Respond(TDescriptor options, NancyContext context)
[Octopus Deploy] at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
[Octopus Deploy] at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass14_0.<get_Routes>b__1(Object x)
[Octopus Deploy] at Nancy.Routing.Route.<>c__DisplayClass4.<Wrap>b__3(Object parameters, CancellationToken context)
[Octopus Deploy] -----------------------
[Octopus Deploy] 
[Octopus Deploy] Error from Octopus Server (HTTP 500 InternalServerError)
[Octopus Deploy] Exit code: -7
[Octopus Deploy] Octo.exe exit code: -7
[Step 6/6] Unable to create or deploy release. Please check the build log for details on the error.
[Step 6/6] Unable to create or deploy release. Please check the build log for details on the error.
[Step 6/6] Step OctopusDeploy: Create release failed

Any advice on this would be greatly appreciated.

Thanks in advance,
Kira Resari

(Justin Walsh) #3

Hi @kira_resari!

From the looks of this error, it seems like the issue is stemming from Octopus not being able to pull the package from the feed. We generally see a HTTP 403 error (Forbidden) when the account credentials configured in the feed in Octopus do not have permission to access the feed/folder/file from the NuGet feed.

First step would be to verify that the feed is configured in Octopus (Library -> External Feeds -> DemoFetcher -> Credentials) in a way that lines up with what is configured in TeamCity / your external feed server.

Please let me know how this works out for you, and please don’t hesitate to let me know if you have any further questions.

(kira.resari) #4

Hi Justin,

I just wanted to add that when trying to create a release directly in Octopus, I also get the Error Message The V2 feed at 'http://collaboration.company.de/httpAuth/app/nuget/feed/_Root/default/v2/FindPackagesById()?id='TestApp.test'&semVerLevel=2.0.0' returned an unexpected status code '403 '..

I found the configuration for the feed credentials in OctopusDeploy where you pointed them out (well, almost, it’s called Library -> External Packages-> TeamCity Packages -> Credentials for me). However, I have no idea where I would configure these in TeamCity. It’s not in the build steps. I already checked there.

However, your input was already a great help! I’ll now see and check if I can find the configuration for the credentials in TeamCity, and post it here if I do.

Thank you!
Kira
=^,^=

(kira.resari) #5

Okay, I now figured it out, and it turned out to be a two-layer problem, that was easily resolved once I figured out what was the problem.

The superficial problem was that the “octopus” login in TeamCity that OctopusDeploy used to access the TeamCity Packages feed was no longer working.

However, the actual problem was that the build configuration in OctopusDeploy tried to use the TeamCity Packages feed in the first place, even though the package was pushed to the internal OctopusDeploy feed during an earlier build step.

I now managed to fix this by changing the Package feed in the Package Details of the affected deployment steps in the Process of the “Test Configuration Dev” project from “TeamCity Packages” to “Octopus Server (built-in)”.

Now it works.

So summed up: If you have this problem, try using the internal octopus feed as a package source since it’s easier to configure.