Structured variable replacement not working

I’m trying to use structured variable replacement on an XML file and it doesn’t seem to be working the way I would expect. Wondering if you might have some pointers as to what I’m doing wrong…

I set up a new variable /*:ServiceDefinition/*:WebRole/@vmsize with value medium

I’m using a step based on the “Deploy an Azure Cloud Service” step and have set thusly:

The logs show the following output:

No structures have been found that match variable names, so no structured variable replacements have been applied. 
Structured variable replacement succeeded on file C:\Octopus\Work\20230302221225-210450-36315\staging\ServiceDefinition\ServiceDefinition.csdef with format Xml 
No structures have been found that match variable names, so no structured variable replacements have been applied. 
Structured variable replacement succeeded on file C:\Octopus\Work\20230302221225-210450-36315\staging\ServiceConfiguration.Cloud.cscfg with format Xml 

The content of the XML file is this:

<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="XX" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" schemaVersion="2015-04.2.6">
	<WebRole name="WebRole" vmsize="Small">
...
	</WebRole>
</ServiceDefinition>

I pasted the file and the XPath into an online XPath validator and got success:Free Online XPath Tester / Evaluator - FreeFormatter.com

Hi Josh,

Thanks for getting in touch! Great question, and I appreciate the very helpful level of detail provided. I gave this a test using the exact same details, and have confirmed that locally in my test it works as we both were expecting it to. I.e. the vmsize is replaced during deployment to medium, being the variable value in the project.

Apologies if this is a very basic question, but have you created a new release of this project after configuring this feature and variable in question? This error message is thrown when no variables are found, and since your variable is unscoped that should be applicable on any deployment.

If that isn’t applicable here, would you be willing to turn on debugging variables in your project, then create and deploy a new release? This will add verbosity to the resulting task log, and hopefully looking through that might point us in the right direction. If you’re able to provide that, you can upload it here.

I look forward to hearing back and getting to the bottom of this one.

Best regards,

Kenny

Kenneth-

Yes, I’ve created several releases to try different perumutations (with namespace wild cards, without, file name specified/wildcards, etc). None appear to have worked.

We have not used structured variable replacement on XML files on cloud services before but we do use PreDeploy and PostDeploy powershell scripts to manipulate the ServiceDefinition.csdef and ServiceConfiguration.Cloud.cscfg files. That would be my fallback plan if this doesn’t work.

I’ve uploaded a deploy log with the relevant steps per your request with debugging variables turned on.

Thanks,

Josh

Hey @josh_centerx,

Just jumping in for Kenny as he is currently off shift as part of our Australian based team, we have reached out to one of our Solutions team who is quite savvy with XPath to see if he can help here, unfortunately he has signed off for the day but he will be back in on Monday so hopefully he gets that message and can respond.

If Kenny does not respond before then as he should pick this up Sunday night I will update you Monday with any information if Solutions have some thoughts.

Kind Regards,
Clare

Thanks Clare.

I’m kind of suspecting that the transforms might not work with the (obsolete) Cloud Service step…

Hey @josh_centerx,

Funnily enough our Solutions team just got this working on a deploy a package step and did wonder the exact same thing (if this was because it was using a Cloud Service step - which is now obsolete).

They ran this on Octopus Server not Cloud and did not have a Cloud Service target to test this with so mentioned we would need to repro this on Cloud to see if this was a bug with cloud but it does look like it might be that step thats the issue.

I am due to leave now but if Kenny can pick this up when he gets in on Sunday he may be able to try replicate this on Cloud and see if it is that step or not.

We will be back with updates when we have tested.
Kind Regards,
Clare

We’re self-hosted, not Cloud.

Hey @josh_centerx,

Thanks for letting us know, sorry out AUS based team did not get a chance to look at this for you last night, our solutions engineer just tested using a Cloud Service target on Cloud (2023.2) and this does work.

I can see you are running 2022.3.10936 so I will need to spin up a test instance and a cloud service target and test this for you, if this does not work I will upgrade to 2023.1 and see if this then works, if it does it may be you need to upgrade your instance to get this working.

The other thing our engineer mentioned was that our tests aren’t the same as yours, for example the xml values are not as stated/other environmental factors affecting the result.

I cannot confirm that though until I have done the testing my end, hopefully I will have some time today to do this for you, I will be in touch.

Kind Regards,
Clare

Hey @josh_centerx,

So unfortunately I am unable to replicate this issue in 2022.3.10936.

I have a very basic setup, we cant create an azure classic or extended service in our Azure portal as it wont let us. I managed to create a dummy Azure Subscription with a managed certificate in Octopus and then created a dummy Azure Cloud service target where I forced a healthy health check by editing the DB (this does not affect the variable replacement but was needed so I could set that project up with the Deploy an Azure Cloud Service step).

I downloaded a dummy cspkg file from the internet - Download Azure Cloud Service Package (I got the AO extra small one.
I then unzipped that and took the cspkg file out of there, I then created a ServiceDefinition.csdef file and pasted your XML in there. I then zipped both of those files up and uploaded them into Octopus.

I setup my project the way you have with the below as the target files for XML replacement:

 **/*.csdef
 **/*.cscfg

I then set the variable up in my project variables as you have.

When I go to deploy it does fail because of the fact my target is not actually present in Azure but I get this in my logs:

13:58:58   Verbose  |       Updating configuration settings...
13:58:58   Info     |       No settings that match provided variables were found.
13:58:58   Verbose  |       The file at C:\Octopus\Work\20230306135852-85-8\staging\ServiceDefinition\ServiceDefinition.csdef does not match any known filename patterns. The file will be tried as multiple formats and will be treated as the first format that can be successfully parsed.
13:58:58   Verbose  |       The registered replacers we will try, in order, are: JsonFormatVariableReplacer,XmlFormatVariableReplacer,YamlFormatVariableReplacer,PropertiesFormatVariableReplacer
13:58:58   Verbose  |       Attempting structured variable replacement on file C:\Octopus\Work\20230306135852-85-8\staging\ServiceDefinition\ServiceDefinition.csdef with format Json
13:58:58   Verbose  |       The file at C:\Octopus\Work\20230306135852-85-8\staging\ServiceDefinition\ServiceDefinition.csdef couldn't be parsed as Json: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
13:58:58   Verbose  |       Attempting structured variable replacement on file C:\Octopus\Work\20230306135852-85-8\staging\ServiceDefinition\ServiceDefinition.csdef with format Xml
13:58:58   Verbose  |       Structure found matching the variable '/*:ServiceDefinition/*:WebRole/@vmsize'. Replacing its content with the variable value.
13:58:58   Info     |       Structured variable replacement succeeded on file C:\Octopus\Work\20230306135852-85-8\staging\ServiceDefinition\ServiceDefinition.csdef with format Xml
13:59:01   Verbose  |       Re-packaging cspkg.

This suggests this does indeed work in 2022.3.10936 with the Deploy an Azure Cloud Service step.

I am not sure why yours does not work and is not finding the structure for /*:ServiceDefinition/*:WebRole/@vmsize.

The only thing I can think here is you mentioned you use pre and post deployment scripts for those files. I am wondering if that is interfering with the structured variable replacement not being able to see the <WebRole name="WebRole" vmsize="Small"> value to change it to Medium.

Would it be possible to get that pre and post deployment script, you can use the same upload link Kenny sent you in his 2nd March response. If I had that I can see if thats affecting the variable replacement. Let me know if you are able to send me that and if you are when you have sent it as we are not notified when a user uploads to that secure site.

Kind Regards,
Clare

Clare-

Thanks for you detailed investigation. I’ve uploaded the pre/post scripts for your review. Based on your comment, I ran another deploy but turned on Octopus.Calamari.CopyWorkingDirectoryIncludingKeyTo to log the file packages.

That left me with the following structure of the service def file after everything was done:

Which it turns out does not match the XPath :frowning:.

Seems like this issue can be closed. Thanks again for all your help.

-Josh

1 Like

Hey @josh_centerx,

Great news you managed to narrow down the issue, if the XPath did not match that would make the error message you got make sense so glad you found that.

Let me know if there is anything else I can help you with as we are always on hand to tackle any issues or answer any questions!

Kind Regards,
Clare

I think we tracked it down… The “vmsize” attribute defaults to “Small” & is not included in the build process by default. I changed it to “Medium” and it was then included as expected.

From this issue, it appears adding an attribute isn’t supported at this time: https://help.octopus.com/t/strucutre-configuration-variables-xpath-add-new-attribute-to-element/28076/4

But I think I can update the default config to be “Medium” and then overwrite it with “Small” or whatever instead using an XPath mapping.

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.