Azure App Service deployments no longer working

As of this morning our Azure App Service step deployments are no longer working. The error is:

Hyak.Common.CloudException: <string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">Request to a downlevel service failed.</string> 
December 18th 2018 12:21:02Error
   at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
December 18th 2018 12:21:02Error
   at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task) 
December 18th 2018 12:21:02Error
   at Microsoft.WindowsAzure.Management.WebSites.WebSpaceOperationsExtensions.List(IWebSpaceOperations operations) 
December 18th 2018 12:21:02Error
   at Calamari.Azure.Integration.Websites.Publishing.ServiceManagementPublishProfileProvider.GetPublishProperties(AzureAccount account, AzureTargetSite targetSite) 
December 18th 2018 12:21:02Error
   at Calamari.Azure.Deployment.Conventions.AzureWebAppConvention.GetPublishProfile(VariableDictionary variables) 
December 18th 2018 12:21:02Error
   at Calamari.Azure.Deployment.Conventions.AzureWebAppConvention.Install(RunningDeployment deployment) 
December 18th 2018 12:21:02Error
   at Calamari.Deployment.ConventionProcessor.RunInstallConventions() 
December 18th 2018 12:21:02Error
   at Calamari.Deployment.ConventionProcessor.RunConventions() 

We were running an older 4.x version, so decided to upgrade to 2018.8.6 (the latest supported by our license). The error was not resolved by this upgrade.

We rely heavily on Azure deployments and this is stopping almost all of our CI integration from working.

How can we resolve this?

Update

We are using a management certificate rather than a service principal - I have since changed to use a service principal and this is now working. We have a number of other projects using management certificates so I’m hoping this is a short term problem as I don’t like the thought of updating all of our projects (not to mention we have limited access to our client Azure accounts to be able to do this)

We’re having the same issue. Also on 4.0.X started around the same time yesterday.

Additionally, the same error happens when editing the Process and attempting to access the Azure section dropdown.

Octopus v4.1.5

<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">Request to a downlevel service failed.</string>
Hyak.Common.CloudException
   at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
   at Microsoft.WindowsAzure.Management.WebSites.WebSpaceOperationsExtensions.List(IWebSpaceOperations operations)
   at Octopus.Server.Web.Api.Actions.AzureWebSitesListAction.GetSites(AzureAccount certificateAccount)
   at Octopus.Server.Web.Api.Actions.AzureWebSitesListAction.ExecuteRegistered()
   at Octopus.Server.Web.Infrastructure.Api.Responder`1.Respond(TDescriptor options, NancyContext context)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
   at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass14_0.<get_Routes>b__1(Object x)
   at Nancy.Routing.Route.<>c__DisplayClass4.<Wrap>b__3(Object parameters, CancellationToken context)

Having this same error in our deployment. Unable to use a service principal at this time also so I hope this can be resolved quickly. Here’s the error I get when trying to load the list of our web apps in any deployment process:

<string xmlns=“http://schemas.microsoft.com/2003/10/Serialization/”>Request to a downlevel service failed.</string>

Microsoft.WindowsAzure.CloudException: <string xmlns=“http://schemas.microsoft.com/2003/10/Serialization/”>Request to a downlevel service failed.</string> at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task) at Microsoft.WindowsAzure.WebSpaceOperationsExtensions.List(IWebSpaceOperations operations) at Octopus.Server.Web.Api.Actions.AzureWebSitesListAction.GetSites(AzureAccount certificateAccount) in Y:\Work\refs\tags\3.4.11\source\Octopus.Server\Web\Api\Actions\AzureWebSitesListAction.cs:line 60 at Octopus.Server.Web.Api.Actions.AzureWebSitesListAction.Execute() in Y:\Work\refs\tags\3.4.11\source\Octopus.Server\Web\Api\Actions\AzureWebSitesListAction.cs:line 46 at Octopus.Server.Web.Infrastructure.Api.Responder`1.Respond(TDescriptor options, NancyContext context) in Y:\Work\refs\tags\3.4.11\source\Octopus.Server\Web\Infrastructure\Api\Responder.cs:line 162 at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2) at Octopus.Server.Web.Api.OctopusRestApiModule.<>c__DisplayClass0_0.<.ctor>b__0(Object o) in Y:\Work\refs\tags\3.4.11\source\Octopus.Server\Web\Api\OctopusRestApiModule.cs:line 46 at Nancy.Routing.Route.<>c__DisplayClass4.<Wrap>b__3(Object parameters, CancellationToken context)

Hi All,

We’ve gotten reports of this error from people using a wide variety of Octopus Deploy versions, which makes us think something changed in Azure. If Azure decided to stop supporting management certificates, or they started disabling API calls which use Management Certificates, there is not much we can do on our end.

You can tell if you are using management certificates by going to infrastructure -> account and then clicking on the account you are using. If it looks like this then you are using management certs.
image

If it looks like this then you are using service principal.
image

Here are some instructions on how to create a service principal account. https://octopus.com/docs/infrastructure/accounts/azure#azure-service-principal

Our Azure account in Octopus was authenticating with Management Certificates.

I followed Bob’s steps and moved us over to using Azure Resource Management with Service Principals and it’s all up and working again.

Hi all,

Azure have previously announced that from June 30th 2018 they are retiring support for Service Management API (which refers to Management Certificates for App Services). The announcement link has apparently been removed by Microsoft now, but it was here for reference: https://blogs.msdn.microsoft.com/appserviceteam/2018/03/12/deprecating-service-management-apis-support-for-azure-app-services/ (Edit: we found where they moved the link to: https://azure.microsoft.com/en-au/updates/deprecating-service-management-apis-support-for-azure-app-service/)

Azure customers are now required to use Service Principals.

Since this June 30 deadline, Azure have been gradually stopping support, so it sounds like your subscriptions were updated by Azure very recently to meet this objective.

Sorry for the bad news. If you are using Service Principals and are still having issues, please submit a new ticket, including your raw task log showing the relevant error details and we will be happy to investigate.

Cheers
Mark

We have enabled this temporary for now to unblock octopus deployment. Please do the migration as described by the blog.