Service Fabric Package Deployment on On-Prem Cluster

Hi All

We are suing Octopus for deployment in our company. For our service Fabric I am trying to use Octopus as well. Actaully we have On-Prem cluster on a physical server using GMSA security (Windows Security).
I am using powershell to connect and copy the package using Octopus Powershell Script step on target machine. We are getting Unauthorised access while copying the package on Image store.
Same script works fine if I RDP on target Machine and run the script with the Octopus user.

Any idea.

Copy-ServiceFabricApplicationPackage -ApplicationPackagePath $packageSourcePath
-ApplicationPackagePathInImageStore $ImageStoreName_$appVersion -ImageStoreConnectionString $ImageConnectionString.Value
-TimeoutSec 2700 -Verbose

This is step where it fails.

Thanks

Hi Sunil,

Thanks for getting in touch and confirming it works when running directly on the target machine. I’m sorry for the inconvenience this issue is causing you! I’ve seen another issue recently with Service Fabric which may or may not be related. Could I ask for some additional details on your situation to help us dig deeper into this?

Which version of Octopus are you currently running?
Is this a transient error, or is it occurring on all of your deployments?
Would you be willing to send through a full task log from a deployment that’s exhibiting this issue? Feel free to mark this thread as private, or email this to us at support@octopus.com. :slight_smile:

I look forward to hearing back and getting to the bottom of this one!

Best regards,

Kenny

Hi Kenny

Thanks for your response.

The Octopus Version We are using is “v2018.9.7”.
It is failing in each deployment. Here is the Powershell script running in Octopus PS Script step on Target Machine where Cluster is is installed.

Import-Module ServiceFabric
Write-Output “Start Connecting to the Cluster”
Connect-ServiceFabricCluster -ConnectionEndpoint ‘:19000’ -WindowsCredential -ClusterSpn $ClusterSpn
Write-Output “Successfully Connected to Cluster”

Write-Output “Get Image Store”
$manifest = Get-ServiceFabricClusterManifest
#$manifest
[xml] $xml = $manifest

$node = $xml.ClusterManifest.FabricSettings.Section | ? {$.Name -eq “Management”}
if($node)
{
$ImageConnectionString = $node.Parameter | ? {$
.Name -eq “ImageStoreConnectionString”} | select Value
Write-output “Image Connection String is “$ImageConnectionString.Value
Write-Output “Get Application Version From Manifest File in Package”
$xml = Select-Xml -Path $PackageLocation$PackageName”\ApplicationManifest.xml” `
-XPath “/” | Select-Object –ExpandProperty “node”

$appVersion = $xml.GetElementsByTagName("ApplicationManifest").ApplicationTypeVersion

Write-Output "AppVersion in Manifest file is $appVersion"
$packageSourcePath = $PackageLocation + $PackageName
Write-Output "Package SourcePath is $packageSourcePath"
Write-Output "Start: Copy Service Fabric Application Package in Image Store $ImageStoreName_$appVersion with Image Connection String $ImageConnectionString.Value"

Copy-ServiceFabricApplicationPackage `
    -ApplicationPackagePath $packageSourcePath `
    -ApplicationPackagePathInImageStore $ImageStoreName_$appVersion `
    -ImageStoreConnectionString $ImageConnectionString.Value `
    -TimeoutSec 2700 -Verbose
#-ImageStoreConnectionString (Get-ImageStoreConnectionStringFromClusterManifest(Get-ServiceFabricClusterManifest)) -TimeoutSec 1800

Write-Output "Package Successfully Copied"


"Get Application Parameter from Environment Param File"
$envParamFilePath = "$PackageLocation$PackageName\ApplicationParameters\$AppParameterFileName"
$applicationParameter = @{}
if(Test-Path $envParamFilePath)
{
    [xml] $content = Get-Content -Path $envParamFilePath
    $parameters = $content.GetElementsByTagName("Parameter")
    
    foreach($param in $parameters)
    {
        $applicationParameter.Add($param.Name,$param.Value)
    }

    Write-Output "Application Parameters Are $applicationParameter"
}

    Write-Output "Start: Registering the service Fabric Package"
    Register-ServiceFabricApplicationType `
        -ApplicationPathInImageStore $ImageStoreName_$appVersion `
        -TimeoutSec 2700 `
        -ApplicationPackageCleanupPolicy Automatic 
    Write-Output "Package Registered Successfully"

    Write-Output "Start: Removing the Application package from Image Store"
    Remove-ServiceFabricApplicationPackage `
        -ApplicationPackagePathInImageStore '$ImageStoreName_$appVersion'
    Write-Output "Package removed successfully from Image Store"

    Write-Output "Check if Application exist"
    $application = Get-ServiceFabricApplication -ApplicationName $ApplicationName  #-ApplicationTypeName "Messaging.CharlesRiver.CrBroadcastServiceType"

    if($application -eq $null)
    {
        Write-Output "Start and Create New Application"
        if(($applicationParameter) -and ($applicationParameter.Count -gt 0))
        {
            New-ServiceFabricApplication `
                -ApplicationName $ApplicationName `
                -ApplicationTypeName $ApplicationTypeName `
                -ApplicationTypeVersion $appVersion -ApplicationParameter $applicationParameter
        }
        else
        {
            New-ServiceFabricApplication `
                -ApplicationName $ApplicationName `
                -ApplicationTypeName $ApplicationTypeName `
                -ApplicationTypeVersion $appVersion
        }
       

        Write-Output "New Application Created and started Successfully"
    }
    else
    {
        Write-Output "Start upgarding the existing Application with new version $appVersion"
        Start-ServiceFabricApplicationUpgrade -ApplicationName $ApplicationName -ApplicationTypeVersion $appVersion -UnmonitoredAuto
    }

}
else
{
throw “Image Store not defined in Manifest”
}

Here is the Error Log:-

== Failed: Step 4: Connect To Cluster And Create Application ==
17:06:20 Fatal | The step failed: Activity Connect To Cluster And Create Application on “Target Server” failed with error ‘The remote script failed with exit code 1’.
17:06:20 Verbose | Connect To Cluster And Create Application completed
|
| Failed: “Target Server”
17:06:16 Verbose | Octopus Server version: 2018.9.7+Branch.master.Sha.bc1aa5ccec678935363ecf0dc4c018f3a4bafb82
17:06:16 Verbose | Environment Information:
| OperatingSystem: Microsoft Windows NT 6.3.9600.0
| OsBitVersion: x64
| Is64BitProcess: True
| CurrentUser: BLUEBAY\svc_OctopusDeployPRD
| MachineName: BBAMOCTAPPPRD01
| ProcessorCount: 2
| CurrentDirectory: C:\Windows\system32
| TempDirectory: C:\Users\svc_OctopusDeployPRD\AppData\Local\Temp
| HostProcessName: Octopus.Server
| PID: 2100
17:06:16 Verbose | Executing Connect To Cluster And Create Application (type Run a Script) on “Target Server”
17:06:16 Verbose | Using Calamari 4.9.18
17:06:17 Verbose | Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory ‘C:\Octopus\Work\20200506160616-110643-259’ using ‘Western European (DOS)’ encoding running as ‘NT AUTHORITY\SYSTEM’ with the same environment variables as the launching process
17:06:17 Verbose | Octopus Deploy: Calamari version 4.9.18
17:06:17 Verbose | Environment Information:
17:06:17 Verbose | OperatingSystem: Microsoft Windows NT 10.0.14393.0
17:06:17 Verbose | OsBitVersion: x64
17:06:17 Verbose | Is64BitProcess: True
17:06:17 Verbose | CurrentUser: NT AUTHORITY\SYSTEM
17:06:17 Verbose | MachineName: “Target Server”
17:06:17 Verbose | ProcessorCount: 2
17:06:17 Verbose | CurrentDirectory: C:\Octopus\Work\20200506160616-110643-259
17:06:17 Verbose | TempDirectory: C:\Windows\TEMP
17:06:17 Verbose | HostProcessName: Calamari
17:06:18 Verbose | Performing variable substitution on ‘C:\Octopus\Work\20200506160616-110643-259\Script.ps1’
17:06:18 Verbose | Executing ‘C:\Octopus\Work\20200506160616-110643-259\Script.ps1’
17:06:18 Verbose | Name Value
17:06:18 Verbose | ---- -----
17:06:18 Verbose | PSVersion 5.1.14393.3471
17:06:18 Verbose | PSEdition Desktop
17:06:18 Verbose | PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
17:06:18 Verbose | BuildVersion 10.0.14393.3471
17:06:18 Verbose | CLRVersion 4.0.30319.42000
17:06:18 Verbose | WSManStackVersion 3.0
17:06:18 Verbose | PSRemotingProtocolVersion 2.3
17:06:18 Verbose | SerializationVersion 1.1.0.1
17:06:19 Verbose | PowerShell Environment Information:
17:06:19 Verbose | OperatingSystem: Microsoft Windows NT 10.0.14393.0
17:06:19 Verbose | OsBitVersion: x64
17:06:19 Verbose | Is64BitProcess: True
17:06:19 Verbose | CurrentUser: NT AUTHORITY\SYSTEM
17:06:19 Verbose | MachineName: “Target Server”
17:06:19 Verbose | ProcessorCount: 2
17:06:19 Verbose | CurrentDirectory: C:\Octopus\Work\20200506160616-110643-259
17:06:19 Verbose | CurrentLocation: C:\Octopus\Work\20200506160616-110643-259
17:06:19 Verbose | TempDirectory: C:\Windows\TEMP
17:06:19 Verbose | HostProcessName: powershell
17:06:19 Verbose | TotalPhysicalMemory: 8388016 KB
17:06:19 Verbose | AvailablePhysicalMemory: 3849532 KB
17:06:19 Info | Start Connecting to the Cluster
17:06:19 Info | True
17:06:19 Info | ConnectionEndpoint : {“IPAddressOfCluster”:19000}
17:06:19 Info | FabricClientSettings : {
17:06:19 Info | ClientFriendlyName :
17:06:19 Info | PowerShell-06cf3c46-c74c-42a8-a6bc-f874ba05c18e
17:06:19 Info | PartitionLocationCacheLimit : 100000
17:06:19 Info | PartitionLocationCacheBucketCount : 1024
17:06:19 Info | ServiceChangePollInterval : 00:02:00
17:06:19 Info | ConnectionInitializationTimeout : 00:00:02
17:06:19 Info | KeepAliveInterval : 00:00:20
17:06:19 Info | ConnectionIdleTimeout : 00:00:00
17:06:19 Info | HealthOperationTimeout : 00:02:00
17:06:19 Info | HealthReportSendInterval : 00:00:00
17:06:19 Info | HealthReportRetrySendInterval : 00:00:30
17:06:19 Info | NotificationGatewayConnectionTimeout : 00:00:30
17:06:19 Info | NotificationCacheUpdateTimeout : 00:00:30
17:06:19 Info | AuthTokenBufferSize : 4096
17:06:19 Info | }
17:06:19 Info | GatewayInformation : {
17:06:19 Info | NodeAddress :
17:06:19 Info | “IPAddressOfCluster”:19000
17:06:19 Info | NodeId :
17:06:19 Info | 85772935593a0315f92e3293832c5fe9
17:06:20 Info | NodeInstanceId :
17:06:20 Info | 132332446296034785
17:06:20 Info | NodeName : vm0
17:06:20 Info | }
17:06:20 Info | Successfully Connected to Cluster
17:06:20 Info | Get Image Store
17:06:20 Info | Image Connection String is
17:06:20 Info | fabric:ImageStore
17:06:20 Info | Get Application Version From Manifest File in Package
17:06:20 Info | AppVersion in Manifest file is 20200506.11
17:06:20 Info | Package SourcePath is C:\BlueBay\ServiceFabric\Messaging.CharlesRiverBroadcastService
17:06:20 Info | Start: Copy Service Fabric Application Package in Image Store 20200506.11 with Image Connection String @{Value=fabric:ImageStore}.Value
17:06:20 Info | VERBOSE: System.UnauthorizedAccessException: Access is denied. (Exception from
17:06:20 Info | HRESULT: 0x80070005 (E_ACCESSDENIED))
17:06:20 Info | at System.Fabric.Interop.Utility.RunInMTA(Action action)
17:06:20 Info | at System.Fabric.ImageStore.NativeImageStoreClient.UploadContent(String
17:06:20 Info | remoteDestination, String localSource, IImageStoreProgressHandler
17:06:20 Info | progressHandler, TimeSpan timeout, CopyFlag imageCopyFlag, Boolean
17:06:20 Info | acquireSourceReaderLock)
17:06:20 Info | at
17:06:20 Info | Microsoft.ServiceFabric.Powershell.CommonCmdletBase.UploadToImageStore(String
17:06:20 Info | imageStoreConnectionString, String src, String des, UploadProgressHandler
17:06:20 Info | progressHandler)
17:06:20 Info | at Microsoft.ServiceFabric.Powershell.CopyApplicationPackage.ProcessRecord()
17:06:20 Error | Copy-ServiceFabricApplicationPackage : Access is denied. (Exception from
17:06:20 Error | HRESULT: 0x80070005 (E_ACCESSDENIED))
17:06:20 Error | At C:\Octopus\Work\20200506160616-110643-259\Script.ps1:34 char:5
17:06:20 Error | + Copy-ServiceFabricApplicationPackage `
17:06:20 Error | + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
17:06:20 Error | + CategoryInfo : SecurityError: (:slight_smile: [Copy-ServiceFabricApplicatio
17:06:20 Error | nPackage], UnauthorizedAccessException
17:06:20 Error | + FullyQualifiedErrorId : CopyApplicationPackageErrorId,Microsoft.ServiceF
17:06:20 Error | abric.Powershell.CopyApplicationPackage
17:06:20 Verbose | Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20200506160616-110643-259 exited with code 1
17:06:20 Verbose | Updating manifest with output variables
17:06:20 Verbose | Updating manifest with action evaluated variables
17:06:20 Fatal | The remote script failed with exit code 1
17:06:20 Fatal | The action Connect To Cluster And Create Application on “Target Server” failed

In Above I just change the MAchine name to Target Server and Actual IpAddress of Cluster to IPAddressOfCluster.

As I mentioned earlier this works perfectly fine if I RDP on Machine and run PS Script

Thanks
Sunil Bindra

Hi Sunil,

Thanks for keeping in touch and sending through that additional information. After some more research into this one, I came across a Service Fabric bug report that I noticed you previously commented on. It looks to me like you’re hitting this same issue, and I see a couple comments with some potential workarounds. Do either of these apply and help in your situation?

  • Update the Service Fabric SDK
  • “Start the PowerShell script from a directory to which you have write permissions.”

I hope that helps! Let me know how you go.

Best regards,

Kenny

I already tried this, even changing the octopus User as admin on the machine. But it still gives the same error

Hi Sunil,

Thanks for keeping in touch! Sorry to hear those options didn’t help unblock this issue. At this point, I’d suggest running process monitor (that comes as part of Microsoft’s Sysinternals Suite) during the deployment, which should point towards the culprit of this Access Denied error. Here’s a handy guide on using Process Monitor, if you’re not familiar with it.

I hope that helps, and let me know how you go or if you have any further questions!

Best regards,

Kenny

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.