Substitute Variables in Files Recursive - Octopus 3.1.2

After Octopus 3.1.2 upgrade the step template broke because of Octopus.Platform.dll missing.
https://library.octopusdeploy.com/#!/step-template/actiontemplate-variables-substitute-in-files

So we switched the process to use the built in substitute variables as documented here:
http://docs.octopusdeploy.com/display/OD/Substitute+Variables+in+Files

It states, “You do not need to state the directory paths as Octopus will iterate through directories and find the files.”

All our files follow the convention .nustache. so we added that. But the problem is the recursive functionality is not working. However, it is successfully replacing variables in the top-level directory of the package.

We need help on this immediately as it’s impacting our ability to deploy.

NOTE: FYI, we just upgraded to from 2.6 to 3.1.2 last night. We have 400 machines and without restricting history, it took ~10 hours for the migration script.

Thanks in advance,

Joey Barrett
Chief Software Architect, Development
DealerSocket, Inc.


P:949-900-0300 x654
C:714-248-6204
W: dealersocket.com

Hi Joey,

Thanks for reaching out. We have a github issue submitted for this: https://github.com/OctopusDeploy/Issues/issues/2032 . This is actually a change we have to make in Calamari, which was done already on this other issue linked to the first one: https://github.com/OctopusDeploy/Calamari/pull/61 . The change was made but we still have to merge the pull request with master. This will most likely happen tonight during our devs daytime (they are in Australia)

Sorry for the inconveniences

Dalmiro

Thanks for the quick response. Do you have any other solutions or hacks we could use in the meantime? Could we hack the original script to use the Octopus.Client.dll? How about grab an old copy of the Octopus.Platform.dll and add it to the deployment package to use?

We can’t be down for a day. We have to find a work around.

Thanks again,

Joey Barrett
Chief Software Architect, Development
DealerSocket, Inc.


P:949-900-0300 x654
C:714-248-6204
W: dealersocket.com

This doesn’t look like the same issue you are identifying. This is not a config transform issue, just a variable replacement issue which is a different feature in Octopus. See screenshot.

I don’t think the pull request you linked addresses the issue.

I think the issue is originating here (Line 26), it’s calling EnumerateFiles on the ICalamariFileSystem which in this case is going to be the CalamariPhysicalFileSystem.
https://github.com/OctopusDeploy/Calamari/blob/master/source/Calamari/Deployment/Conventions/SubstituteInFilesConvention.cs

Instead of calling EnumerateFiles (Line 177) it should call EnumerateFilesRecursively (Line 186) of this file:
https://github.com/OctopusDeploy/Calamari/blob/master/source/Calamari/Integration/FileSystem/CalamariPhysicalFileSystem.cs

Thanks,

Joey Barrett
Chief Software Architect, Development
DealerSocket, Inc.


P:949-900-0300 x654
C:714-248-6204
W: dealersocket.com

Hi Joey,

This is indeed related to another thing like you mentioned. It seems to be with the fact that we are not supporting wildcard values for nested config files. I’m gonna consult with the team to see if this is something we plan on supporting or not.

In the meantime, if you specify the full path of the config file, the substitution will work.

Regards,

Dalmiro

Thanks Dalmiro.

Full path didn’t seem to work right, so I’ve included the details of what we did below. I really hope the Octopus developers can get this fixed, and get it right because this issue is repeatedly talked about and core to the functionality. Hacking work arounds since the beginning of using Octopus has added a lot of headache to what otherwise would be a great experience. Please pass this along.

Thanks,
Joey

Here’s what we did as a work around.

#1 Created a script:

function FindFilesToSubstitute([string]$RootPath, [string]$SearchPattern = “.nustache.”) {

Write-Output "Finding files to substitue matching $SearchPattern in $RootPath"

Set-Location $RootPath

$filesToSubstitute = ""

Get-ChildItem -Path $RootPath -Filter $SearchPattern -Recurse | % {

    $path = $_ | Resolve-Path -Relative
    $path = $path.replace(".\", "")
    $path = $path + [Environment]::NewLine
    Write-Output "Found $path"
    
    $filesToSubstitute = $filesToSubstitute + $path
}

Set-OctopusVariable -name "FilesToSubstitute" -value $filesToSubstitute

}

#2 To each step that needs variables changed we added 2 features:
-Custom deployment scripts
-Substitute variables in files

#3 In Pre-deployment script we added

 FindFilesToSubstitute (Get-Location)

#4 Then in the “Substitute Variables in Files” feature we added:

 #{FilesToSubstitute}

#5 We also added the variable FilesToSubstitute with a default value of MUST_BE_SET_BY_OCTOPUS_PROCESS so we could see if something wasn’t set right in the log as it will warn about not finding files that match “MUST_BE_SET_BY_OCTOPUS_PROCESS”.

Hi Joey,

I’ve discussed this with the team, and we agreed that the documentation was misleading. Apologies for that.

I’ve removed the line that said: You do not need to state the directory paths as Octopus will iterate through directories and find the files.

And instead I placed the correct instruction

You need to state the full path of the file, relative to the installation directory. So, if you need to replace variables on a file called *app.config* that is inside of a *config* folder on the root of your package, you need to put *config\app.config* on the Target files field.

Please check this setup and let me know if you did exactly as shown here and it still doesnt work for you:

  1. Package has a folder structure as shown in screenshot 1.
  2. Target files field references both files as shown in screenshot 2.
  3. Deployment log shows how variables were substituted in both files. See screenshot 3

Regards,

Dalmiro

Any chance of adding the functionality back that will allow substitute variables to find files recursively? We have dozens of projects and all the structures have nested folders with .config files in them.

As Joey pointed out it would be as simple as calling EnumerateFilesRecursively here: https://github.com/OctopusDeploy/Calamari/blob/master/source/Calamari/Deployment/Conventions/SubstituteInFilesConvention.cs#L26

Even if this was conditional or based on some specific text in the pattern it would be great. Ex: “*.config" could strip the "” and call the recursive instead of non-recursive enumerate.

Otherwise we have to resort to either listing out the dozens of directories the .config file could live in or hacking around it as joey did, both in dozens of projects …

Thanks!

PS - Would be happy to submit a pull request with any change decided upon complete with unit tests. No need for extra work on yalls part. :slight_smile:

Hi Adam,

What would you exactly like this to work? Do you want for example Web.test.config => **\ and have that test config file full of transforms to be applied on all config files recursively? I’d like to get a better understanding of the expected behavior to be able to give you a proper answer.

That said, feel free to create your own branch of Calamari and test it in your project. If later on the PR makes sense with our vision for Calamari, I can’t see why we wouldn’t think on merging it. But for that to happen, we need to understand exactly what you are looking for, and then evaluate if that’ll be a valuable change for the rest of the users.

Best regards,
Dalmiro

Hi Dalmiro,

Im speaking specifically about the variable substitutions feature on a package deployment, and Im pretty flexible on the HOW part.

What I would like as the end result is a one liner of some kind (in variable substitution) to recursively pick up all the files matching the pattern (ex: *.config).
What I am trying to avoid (and having to do now) is configure it like this:
*.config
Config*.config
Config\Feature1*.config
Config\Feature1\Xml*.config
Config\Feature1\Json*.config
Config\Feature2*.config
Config\Feature2\Xml*.config
Config\Feature2\Json*.config
Config\Feature3*.config
Config\Feature3\Xml*.config
Config\Feature3\Json*.config
Config\Feature4*.config
Config\Feature4\Xml*.config
Config\Feature4\Json*.config
Config\Feature5*.config
Config\Feature5\Xml*.config
Config\Feature5\Json*.config
Config\Feature6*.config
Config\Feature6\Xml*.config
Config\Feature6\Json*.config

I agree with Adam, that’s exactly how it should work - by specifying a matching pattern - potentially the same one that is used in DirectoryInfo.GetFiles method documented here:
https://msdn.microsoft.com/en-us/library/ms143327(v=vs.110).aspx

And to reiterate:
I would just change the following line to call “fileSystem.EnumerateFilesRecursively”

It doesn’t seem like limiting the variable substitution pattern to the top level folder has much utility.

Hi Adam,

Thanks for putting it down in such a clear example. Unfortunately I have to tell you that the fact that we don’t do recursive search like that is absolutely a design decision.

Having recursive search like that can lead to terrible scenarios if something goes wrong. In our experience helping teams deliver their code to production, we learned that its a lot safer to ask users to be super explicit about which files they want to target.

That said, even though we explicitly decided to make this work the opposite way that you expected, we are definitely open to get feedback to make the product better. In our experience this recursive search just isn’t a good practice, but if enough users would like to see this happen, we are more than glad to open the conversation.

You seem to be pretty confident in implementing this change in Calamari. If you can create a branch with the change, and then reference it in a Uservoice request, users looking for that feature can test it right away and provide feedback there. That’ll be by far the best way for us to know how many people would like this to happen (which so far have been less than 3-4 from what I could see).

Best regards,
Dalmiro

Dalmiro,

How about adding a checkbox that says “recursive” that allows the user to select the behavior? That way you can handle both cases and control terrible scenarios.

Best,
Joey Barrett

Also, I’d like to point out that an entire script module was put in place to deal with this and hosted in the library (https://library.octopusdeploy.com/listing) which indicates to me a wider need than 3 or 4 people.

Add to that, all our other build and test tools already have a syntax for this.
*.config = all config files in this directory
**.config =all config files in this or direct subdirectories
***.config = all config files in this or any descendant directory

For testing we use patterns like **\bin**test.dll to discover all test DLLs that reside anywhere beneath any bin folder.

Using that syntax, I (as a user) am forced to explicitly state I want to search all subdirectories, but given the flexibility to not add two dozen lines of patterns.

The problem with using it in a separate step is now i have to modify my package deployments. Example:

Right now we have a two step process for windows service deployments.

  1. deploy the package with the service features turned on
  2. notify me of the result via slack.

The substitutions need to happen before the service is started, which is before the step is finished. So do I turn off the service deploy features and do all of that manually?

  1. deploy package
  2. substitutions step
  3. stop service
  4. install/update service
  5. start service
  6. notify …

That seems like a bad trade-off.

@Jbarrey - Apologies for my previous expression when I said they were “less than 3-4”. I only wanted to express that we are not being asked for this feature very often. The expression totally came out wrong and I apologize for that.

I’m really sorry the current behavior is giving you guys trouble. But at the moment the best way to get the conversation going is to create a Uservoice suggestion for this and get as many people behind it as possible. Only that way we can prioritize it and eventually do something about it.

Best regards,
Dalmiro

HI,

has anyone created a Uservoice request for this?

Having many config files in structure makes this really painful. Just like Adams example above.