Azure DevOps 2020 NuGet Feed Unhandled Exception

Robert_Livermore · 3 November 2021 18:40

Using Octopus Server 2021.2.7727, adding a NuGet feed to an on-premise Azure DevOps 2020 server. Receive the following exception.

2021-11-03 12:02:58.3813   6908     24 ERROR  Unhandled error on request: GET https://vmdmzdeldev.actcosys.com/OctopusDeploy/api/Spaces-1/feeds/Feeds-1021/packages/search?term=d&take=10 80000094-0000-f200-b63f-84710c7967bb by SimTeam : Package feed returned 401 Unauthorised, check the username and password for this feed. Also attempt to connect using NuGet.exe.
System.Exception: Package feed returned 401 Unauthorised, check the username and password for this feed. Also attempt to connect using NuGet.exe.
   at Octopus.Core.Packages.NuGet.ExternalHttpNuGetPackageFeed.SearchForPackagesNamedLike(String packageId, Int32 take, CancellationToken cancellationToken, Boolean exactPackageIdMatch) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Core\Packages\NuGet\ExternalHttpNuGetPackageFeed.cs:line 143
   at Octopus.Core.Packages.NuGet.ExternalHttpNuGetPackageFeed.SearchPackages(String term, CancellationToken cancellationToken, ILog log, Int32& total, Int32 skip, Int32 take) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Core\Packages\NuGet\ExternalHttpNuGetPackageFeed.cs:line 71
   at Octopus.Server.Web.Api.Actions.PackageSearchAction.SearchPackages(IPackageFeed repository, String term, Int32 skip, Int32 take, String feedId, ILog log) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Api\Actions\PackageSearchAction.cs:line 91
   at Octopus.Server.Web.Api.Actions.PackageSearchAction.ExecuteRegistered() in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Api\Actions\PackageSearchAction.cs:line 58
   at Octopus.Server.Web.Infrastructure.Api.CustomResponder`1.Respond(TDescriptor options, NancyContext context) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Infrastructure\Api\CustomResponder.cs:line 288
   at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass14_0.<get_Routes>b__1(Object o, CancellationToken x) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Infrastructure\OctopusNancyModule.cs:line 82
   at Nancy.Routing.Route`1.Invoke(DynamicDictionary parameters, CancellationToken cancellationToken)
   at Nancy.Routing.DefaultRouteInvoker.Invoke(Route route, CancellationToken cancellationToken, DynamicDictionary parameters, NancyContext context)
   at Nancy.Routing.DefaultRequestDispatcher.Dispatch(NancyContext context, CancellationToken cancellationToken)
   at Nancy.NancyEngine.InvokeRequestLifeCycle(NancyContext context, CancellationToken cancellationToken, IPipelines pipelines)
   at Octopus.NancyOwin.NancyMiddleware.RequestComplete(NancyContext context, IDictionary`2 environment, CancellationToken cancellationToken) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Owin\NancyMiddleware.cs:line 171
   at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Owin\NancyMiddleware.cs:line 111
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.HttpOverrides.HttpMethodOverrideMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\OctopusClientOldVersionWarningMiddleware.cs:line 38
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\DynamicContentHeadersMiddleware.cs:line 48
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\MaintenanceModeMiddleware.cs:line 55
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\OctopusAuthenticationMiddleware.cs:line 67
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.Middleware.RequestLoggerMiddleware.InvokeAsync(HttpContext context) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\RequestLoggerMiddleware.cs:line 42
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\TelemetryMiddleware.cs:line 74
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in C:\BuildAgent\work\eb99e602a96dd63\source\Octopus.Server\Web\Middleware\ErrorHandlingMiddleware.cs:line 100

Attempted to reproduce with NuGet commandline

.\nuget.exe sources update -ConfigFile .\nuget.config -Name SIM-SQLSERVER -ValidAuthenticationTypes basic -Source https://.../nuget/v3/index.json -UserName "SIM-SQLServer" -Password [PATHERE]

.\nuget.exe list -ConfigFile .\nuget.config -Source SIM-SQLServer -Verbosity detailed

This works. Tested with NuGet Explorer. This works too. Use same PAT token for each test.

I did find some similar problem in 2017; but is marked as fixed. I did edit credentials and endpoint a few times. So I repeated with a server restart

Get-Service OC* | Restart-Service

Not sure what to do next.

sergei.dubovsky · 3 November 2021 19:43

Hi, @Robert_Livermore,

Thank you for contacting Octopus support. We are sorry you have an issue with NuGet.
Can you try to connect to the v2 feed instead of v3? here: https://…/nuget/v3/index.json
We have an Azure DevOps feed setup that works with no additional configuration, so it should be possible.
Where was that nuget.exe test performed? Was it the Octopus server machine or some other computer?

Thank you.
Sergei

Robert_Livermore · 8 November 2021 12:46

Performed on the same host as the Octopus deploy server. Same endpoint, same PAT token.
Not a networking problem, Not azure devops access issue, when trying to get a list of packages.

Already tried the combination with/without nuget version 3. same unhandled error.
Already tried with/without credential.

Did an Octopus Server Service restart between each test. (Just in case the caching bug is back)

sergei.dubovsky · 8 November 2021 15:19

Hi, @Robert_Livermore,

Is Octopus Server set up with a proxy? It is configured in the Octopus manager:

We suspect this 401 is not coming from the feed, but from something that is between the Octopus server and ADO2020 feed.

Thank you.
Sergei

Robert_Livermore · 8 November 2021 16:03

Hi Sergei,

Thank you for your response. Web proxy is a good idea.

The Web Request Proxy Settings was configured to Use the proxy server configured in Internet Explorer. Changed to Do not use a proxy.

Same result. 401 error in Octopus Deploy Server.

Is there a way to give verbose output on the Octopus Server for more details on what request is causing HTTP 401 error? I can start a conversation with IT; but they will ask for more details on the request Octopus Server is sending.

Repeat Test with Nuget command line
NuGet.exe works from same host

PS D:\nuget> .\nuget.exe list -ConfigFile .\nuget.config -Source SIM-SQLServer -Verbosity detailed
NuGet Version: 5.11.0.10
  GET https://vmdevops2020.actcosys.com:4443/tfs/WebFleetAssistant/05e43684-222c-4363-9f0a-ceaffea10e2d/_packaging/17d38
b4c-f90e-4510-9dc8-b03b0e42d501/nuget/v2/$metadata
  OK https://vmdevops2020.actcosys.com:4443/tfs/WebFleetAssistant/05e43684-222c-4363-9f0a-ceaffea10e2d/_packaging/17d38b
4c-f90e-4510-9dc8-b03b0e42d501/nuget/v2/$metadata 22ms
  GET https://vmdevops2020.actcosys.com:4443/tfs/WebFleetAssistant/05e43684-222c-4363-9f0a-ceaffea10e2d/_packaging/17d38
b4c-f90e-4510-9dc8-b03b0e42d501/nuget/v2/Search()?filter=IsLatestVersion&$orderby=Id&searchTerm=''&targetFramework=''&includePrerelease=false&$skip=0&$top=30&semVerLevel=2.0.0
  OK https://vmdevops2020.actcosys.com:4443/tfs/WebFleetAssistant/05e43684-222c-4363-f0a-ceaffea10e2d/_packaging/17d38b4c-f90e-4510-9dc8-b03b0e42d501/nuget/v2/Search()$filter=IsLatestVersion&$orderby=Id&searchTerm=''&targetFramework=''&includePrerelease=false&$skip=0&$top=30&semVerLevel=2.0.0 28ms
CET.DACPAC
 11.9.57238
 Multi-version Microsoft SQL Server dacpac for SQL sqlpackage.exe tooling.

Microsoft.SqlPackage
 8.8.57211
 Repackage of the SQLPackage deployment tool.

sergei.dubovsky · 8 November 2021 16:14

Hi Robert,

The log level verbosity is adjustable. Here is a documentation link: Log files - Octopus Deploy

Just to check - The server was restarted after the proxy setting change, right?

There is one more thing we can try. Let’s check if it’s dependent on the user account, under which nuget.exe runs. Usually, Octopus server installs under NT AUTHORITY\SYSTEM, so to check this, the easiest way would be to use a script console, with the target set to Octopus Server:

Thank you.
Sergei

Robert_Livermore · 8 November 2021 19:10

Tried the Script Console and reproduced the exception. Tested with relative and specified paths the nuget.config file – same error. Not sure where nuget is looking for credentials.

Also check encrypted socket. TLS 1.2 is supported on both ends.

The version 19.4.9 of the Calamari.win-x64 tool has not been extracted, it will be extracted automatically. 
November 8th 2021 13:29:31Info
Version 19.4.9 of the Calamari.win-x64 tool has been extracted successfully 
November 8th 2021 13:29:43Info
NuGet Version: 5.11.0.10 
November 8th 2021 13:29:45Info
MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin\amd64'. Use option -MSBuildVersion to force nuget to use a specific version of MSBuild. 
November 8th 2021 13:29:45Info
Please provide credentials for: https://vmdevops2020.actcosys.com:4443/tfs/WebFleetAssistant/WFA343/_packaging/SIM-SQLServer/nuget/v3/index.json 
November 8th 2021 13:34:44Error
Unable to load the service index for source https://vmdevops2020.actcosys.com:4443/tfs/WebFleetAssistant/WFA343/_packaging/SIM-SQLServer/nuget/v3/index.json. 
November 8th 2021 13:34:44Error
  The HTTP request to 'GET https://vmdevops2020.actcosys.com:4443/tfs/WebFleetAssistant/WFA343/_packaging/SIM-SQLServer/nuget/v3/index.json' has timed out after 100000ms. 
November 8th 2021 13:34:44Error
NuGet.Protocol.Core.Types.FatalProtocolException: Unable to load the service index for source https://vmdevops2020.actcosys.com:4443/tfs/WebFleetAssistant/WFA343/_packaging/SIM-SQLServer/nuget/v3/index.json. ---> System.TimeoutException: The HTTP request to 'GET https://vmdevops2020.actcosys.com:4443/tfs/WebFleetAssistant/WFA343/_packaging/SIM-SQLServer/nuget/v3/index.json' has timed out after 100000ms.

Robert_Livermore · 8 November 2021 19:36

I did find something, can you let me know if this is worth pursuing. There is integration with Windows Credential Manager microsoft/artifacts-credprovider: The Azure Artifacts Credential Provider enables dotnet, NuGet.exe, and MSBuild to interactively acquire credentials for Azure Artifacts feeds. (github.com)

Does this need to be configured for the Octopus Deploy Server to manage credentials? I’m suspicious because the stack trace contains MSBuild and this credential managers also depends on MSBUILD.

sergei.dubovsky · 8 November 2021 20:00

Hi @Robert_Livermore,

It is certainly worth trying. As I understand it would be a little tricky to install since usually, credentials helpers rely on a user profile to store the data. By default, Octopus Server runs under the NT AUTHORITY\SYSTEM and that account doesn’t have a user home folder. When we run a script from the Console, it starts the Powershell in the same context as Octopus Server. If we can install the credential helper using Script Console, it might solve the authentication issue

Thank you.
Sergei

Robert_Livermore · 9 November 2021 12:53

Hi Sergei,

Indeterminate result. The Azure Artifacts Credential Provider nuget plug-in fixed the Script Console. It was able to authenticate and list the packages. Octopus deploy Server still raises an unhandled exception when testing the feed.

Console Script Result

The Azure Artifacts Credential Provider is included in the visual studio build tools include the nuget target and build tasks, Nuget package Manager, and Net SDK options. NuGet.exe will search for plug-ins by file convention; so no worries about NT AUTHORITY\SYSTEM assuming the account has read and execute rights. There is a one time call to nuget.exe call to add the credential into the WIndows Credentials store

D:\nuget\nuget.exe sources add -Name SIM-SQLSERVER -ValidAuthenticationTypes basic -Source https://{devops2020}/_packaging/SIM-SQLServer/nuget/v3/index.json -UserName "SIM-SQLServer" -Password [PATTOKEN]

After That the List command works.

D:\nuget\nuget.exe list -Source SIM-SQLServer

Octopus Deploy Server Test Result

Still errors with the same unhandled exception. The extra tracing in logging did not add any addition information.

Conclusion

Octopus Server NuGet feeds do not support nuget plug-ins.

Thanks for the help Sergei. I’m out of ideas on how the authenticate with DevOps 2020 Server.

sergei.dubovsky · 9 November 2021 20:22

Hi Robert,

I am recreating the whole thing here in my Hyper-V. One thing I noted so far - if I use my Username/Password for the Nuget feed credentials, it works. If I use PAT it fails with 401.

To your question, where NuGet stores the credentials, it’s in here: C:\Users<YOURACCOUNT>\AppData\Roaming\NuGet

However, it doesn’t seem to affect the thing that we are unable the use the PAT.

Please allow me more time to troubleshoot this. If you can do me a favor and try to authenticate with your username/password instead of PAT. This will eliminate the networking aspect of it.

Thank you
Sergei

Robert_Livermore · 11 November 2021 13:06

Hi Sergei,

The Active Directory user credentials worked. Not ideal because of group policy restrictions .
Please let me know if you figure out the PAT token.

Thank you
Robert

P.S. I did learn about NuGet plug-ins. Come in handy with docker container builds with team builds oops/azure devops build pipelines.

sergei.dubovsky · 11 November 2021 14:49

Hi Robert,

Thank you for the update. I opened a bug for this issue. Hopefully, it won’t take long to fix it.
You can keep an eye on the progress here: On-prem Azure DevOps fails the PAT authentication of Nuget feed · Issue #7163 · OctopusDeploy/Issues · GitHub

Thank you.
Sergei

system · 12 December 2021 14:49

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.