Export/Import a Project fails with 'Machine' scoped variables error?

[EM1L (Emil)]

Hi Folks,

I am trying project export in version 2021.3.8275, but it fails with stop error (see below), when the project contains Machine scoped Library Variable Set variables.

The items you are exporting are not currently supported: Variables with the following scopes are not currently supported: Machine. Found variables named <Variable-Names, ...>

I noticed there was a feature request to add “Ignore” option when a project contains Machine scoped variables. Was this feature ever implemented in more recent versions?

I have a requirement to migrate large number of projects from one OD instance to another (both self-hosted) with those projects sharing very large number of global variable sets. I could get away with migrating only the library sets but that also turned out to be a challenge.

Is there a way to force the export excluding the machine scope?

Alternatively, what is the recommended way to tackle this migration problem? Please note we can not modify the source projects as they are all use actively in production.

Thank you in advance.
Emil

[Kenneth_Bates]

Hi Emil,

Thank you for getting in touch, and this is a great question. As you pointed out, machine-scopes on any variables included in what you’re attempting to export is a limitation of this feature, with no current way to ignore these.

How we’ve handled similar issues in the past is a script that removes the machine scopes from the source (storing the changes in a hashtable for easier reference when needing to re-add the scopes on the destination). However this is a modification of the source before the export, which you mention is not possible since the source is production-active. Also, since these are in global variable sets, you can’t simply clone the project as they would just be attached to the same static variable set, so approaching this situation will be tricky.

I’ve been attempting to come up with a workaround solution to export/import variable sets like you have mentioned in your other thread (we can continue this as one discussion in this thread to keep it in one place, if that’s okay!), though it’s proving to be taking time and effort to figure out, and I think this is similar to what it may end up looking like:

Export variable set:

$ErrorActionPreference = "Stop";

# Define working variables
$octopusURL = ""
$octopusAPIKey = ""
$header = @{ "X-Octopus-ApiKey" = $octopusAPIKey }
$ErrorActionPreference = "Stop";

$spaceName = ""
$varSetName = ""

# Get space
$space = (Invoke-RestMethod -Method Get -Uri "$octopusURL/api/spaces/all" -Headers $header) | Where-Object {$_.Name -eq $spaceName}

# Get project
$varSet = (Invoke-RestMethod -Method Get -Uri "$octopusURL/api/$($space.Id)/libraryvariablesets/all" -Headers $header) | Where-Object {$_.Name -eq $varSetName}

# Get project variables
#$projectVariables = Invoke-RestMethod -Method Get -Uri "$octopusURL/api/$($space.Id)/libraryvariablesets/$($varSet.Id)" -Headers $header 
$projectVariables = Invoke-RestMethod -Method Get -Uri "$octopusURL/api/$($space.Id)/variables/variableset-$($varSet.Id)" -Headers $header 
#$projectVariables.Variables

$varTable = @{}

foreach($var in $projectVariables.Variables) {

    $var.Scope.Machine
    if($var.Scope.Machine -ne $null){
      $varTable += @{$var.Id = @($var.Name, $var.Scope.Machine)}
      $var.Scope.PSObject.Properties.Remove("Machine")
    }

    #$var.Name
    #$var.Value
   
    #$var.isSensitive
}

$projectVariables | ConvertTo-Json -depth 10 | Out-File "path\to\variables.json"

Import variable set:

# Define octopus variables
$octopusURL = ""
$octopusAPIKey = ""
$header = @{ "X-Octopus-ApiKey" = $octopusAPIKey }

# Define working variables
$spaceName = ""
$variableSetFilePath = "path\to\variables.json"
$destinationVarSetName = ""

# Get space
$space = (Invoke-RestMethod -Method Get -Uri "$octopusURL/api/spaces/all" -Headers $header) | Where-Object {$_.Name -eq $spaceName}

# Get project
$varSet = (Invoke-RestMethod -Method Get -Uri "$octopusURL/api/$($space.Id)/libraryvariablesets/all" -Headers $header) | Where-Object {$_.Name -eq $destinationVarSetName}

# Get project variables from file
$projectVariablesFromFile = Get-Content -Raw -Path $variableSetFilePath | ConvertFrom-Json

write-host $projectVariablesFromFile

# Update the collection
Invoke-RestMethod -Method Put -Uri "$octopusURL/api/$($space.Id)/variables/variableset-$($varSet.Id)" -Headers $header -Body ($projectVariablesFromFile)

I hope this is helpful for the time being. I’m going to keep looking into this one shortly and will update with anything I find.

Best regards,

Kenny

[EM1L (Emil)]

Hi @Kenneth_Bates,

I managed to export the globally shared variable sets and I posted the code in the other thread, before reading this one. One problem solved, however now I need to focus on projects migration and I am not sure how I can minimise any manual work or in other words what automation script I can come-up with that will take care of most project variables while working around the scope limitations.

Any ideas welcome and thank you for your comments and code - very helpful indeed!

Thanks,
Emil

[Kenneth_Bates]

Hi Emil,

That’s awesome news, thanks for the update. Since the library variable set migration was the trickiest part. since they’re global, the project migration should be the easier part (though some manual work will be required). The idea is to clone the project, remove the variable set connections from the cloned project, then export/import that. If you have machine-scoped project variables, the same error will occur on export, but the following script will remove the machine-scopes from any of these variables in the project, allowing you then to export.

$octopusURL = "https://octopusURL"
$octopusAPIKey = "API-XXXXXXXXXXXXXXXXXXXXXXXXXXXX"
$header = @{ "X-Octopus-ApiKey" = $octopusAPIKey }
$spaceName = "Default"
$projectName = ""

# Get space
$spaces = Invoke-RestMethod -Uri "$octopusURL/api/spaces?partialName=$([uri]::EscapeDataString($spaceName))&skip=0&take=100" -Headers $header 
$space = $spaces.Items | Where-Object { $_.Name -eq $spaceName }

# Get project
$projects = Invoke-RestMethod -Uri "$octopusURL/api/$($space.Id)/projects?partialName=$([uri]::EscapeDataString($projectName))&skip=0&take=100" -Headers $header 
$project = $projects.Items | Where-Object { $_.Name -eq $projectName }

# Get variables
$variables = Invoke-RestMethod -Method Get -Uri "$octopusURL$($project.Links.Variables)" -Headers $header

#Construct Hashtable for Machine Scoping
$varTable = @{}

foreach($vars in $variables.Variables){
    if($vars.Scope.Machine -ne $null){
        $varTable += @{$vars.Id = @($vars.Name, $vars.Scope.Machine)}
        $vars.Scope.PSObject.Properties.Remove("Machine")
    }
}

#Store Machines references from Variables.ScopeValues
$machineStorage = $variables.ScopeValues.Machines

#Remove Machine references from Variables.ScopeValues
$variables.ScopeValues.PSObject.Properties.Remove("Machines")

#Remove Machine Scoping
Invoke-RestMethod -method Put -Uri "$octopusURL$($project.Links.Variables)" -Body ($variables | ConvertTo-JSON -Depth 10) -Headers $header

Write-Host "For reference, the following scopings were removed from individual variables, the syntax of the table is:"
Write-Host "Variables-Id {Variable Name, Machines-Id}"
$varTable
write-host "-------------------------------------------------------------------"
write-host "For reference, the following ScopeValues were removed from the projects variable endpoint:"
write-host $machineStorage

After importing on your other instance/space, you can use a combination of the data from $varTable and $machineStorage to re-apply the scoping once the machines have been added on the import environment.

I hope this helps!

Best regards,

Kenny

[EM1L (Emil)]

Thanks Kenny,

I appreciate the script example, this is the best approach I could think of too…

Regards,
Emil