Octopus Upgrade Broke Deployment Parameters for Bash Script

Octopus Server
, ,
1

We recently upgraded Octopus Deploy from 2021.1 to 2021.3. When we did that we had problems with a bunch of our projects.

These projects use a custom step template called “Bash Deploy Script” to run the deploy. That step is based on the “Script” step. It was setup to have the script parameters entered on separate lines (for readability). It looked similar to this:

--repo-version #{PackageVersion} 
--environment #{Octopus.Environment.Name | ToLower} 
--consul-address #{DEPLOY_CONSUL_ADDRESS} 
--consul-token #{DEPLOY_CONSUL_TOKEN} #{ExtraArgs}

After the upgrade, this would not work. The line breaks had to be remove for it to work. Like this:

--repo-version #{PackageVersion} --environment #{Octopus.Environment.Name | ToLower} --consul-address #{DEPLOY_CONSUL_ADDRESS} --consul-token #{DEPLOY_CONSUL_TOKEN} #{ExtraArgs}

Error Details

The error message we got when the parameters were on individual lines was:

/etc/octopus/default/Work/20211213175835-150711-4098/deploy.sh: line 47: ./.teamcity/build.sh: No such file or directory

Once the line breaks were removed, it went back to working as it did before the upgrade to 2021.3.

While not ideal, we can live with it needing to be on one line.

The problem is that we cannot use any of our existing releases now. This is because the process change is not registered on any releases that already exist.

Things that were in process to be released now need a new build. But far more concerning to us is that our 30 applications that use this process can no longer be rolled back. We would have to make a new build and deploy it out. For one or two, that is doable, but for 30 applications it becomes a big deal (new builds, paperwork, approvals, possible downtimes, deployments etc).

We need a way to update those existing releases to use the new process.

How can I update an existing release to use the process change?

5

Hi @OctopusSchaff,

Looking into this, it appears as if script parameters that were were newline delimited always needed the newline escaped to work correctly - is it possible that these were previously escaped, but perhaps the upgrade process and a data migration performed as part of that, removed the escaping?

I tested it locally on 2020.6 instance that I had handy and if I had a script in a package with just and echo blah, then I was passing the script parameters as

-n
-e

It would evaluate the -e as a distinct command, with the same sort of message you were seeing (basically a command-not-found error).

Escaping the newline, so my script parameters are

-n \
-e

resulted in the command running successfully as expected.

What I haven’t tested yet, but will shortly, is setting this with the escaped newlines on 2021.1 and then performing an upgrade to 2021.3, to ensure they persist. Can you let me know what build of 2021.1 you upgraded from, as well as what build of 2021.3 you upgraded to?

To answer your direct question, sadly there’s no way of updating an existing release - but leave it with me and I will try to figure out a workaround.

Look forward to hearing from you soon.

6

Just tested the upgrade process from 2021.1.7316 to 2021.3.8275 and it did preserve the newline escaping, so it looks like there’s no bug down that code path.

7

It took some time to hunt down the exact version (I did not have permissions to the DB). We upgraded from 2021.1.7608 to 2021.3.8275.

I went to the history of that step template and it does not show any changes except the one that was made after the upgrade. (Where we removed the newlines to get it working again.) Here is a screenshot of the audit page:

image
image1891×1618 353 KB

As shown in the screenshot, the newlines were \n encoded.

I can show some histories of the deploys working before the upgrade and failing after the upgrade if it will be helpful.

If there is anymore information I can provide please let me know!

8

Thanks for the updated info, @OctopusSchaff!

Would you be able to send through a copy of your custom step template, I want to try that specifically to see if there’s some bts data transformation that happened as part of the upgrade process (these typically wouldn’t be logged to the audit log, as they’re not a user action) - from there, we can test the repro of 2021.1 and 2021.3 post-upgrade.

If it contains any sensitive information, please feel free to email it through to support@octopus.com referencing this thread.

9

Here is the exported template. Using the audit log, I replaced the line that changed so that it is what was present before we upgraded and then had to modify the script parameters.

{
  "Id": "b62bcbbe-0df8-42d4-aafe-b3c30cd5c2d2",
  "Name": "Bash Deploy Script",
  "Description": "Created from step 'Deploy' in project 'sqewer'",
  "ActionType": "Octopus.Script",
  "Version": 1,
  "CommunityActionTemplateId": null,
  "Packages": [
    {
      "Id": "14052843-70d6-422c-ad96-3f90832f3390",
      "Name": "",
      "PackageId": "#{PackageName}",
      "FeedId": "feeds-builtin",
      "AcquisitionLocation": "Server",
      "Properties": {
        "SelectionMode": "immediate"
      }
    }
  ],
  "Properties": {
    "Octopus.Action.RunOnServer": "true",
    "Octopus.Action.Script.ScriptSource": "Package",
    "Octopus.Action.Script.ScriptFileName": "deploy.sh",    
	"Octopus.Action.Script.ScriptParameters": "--repo-version #{PackageVersion}\n--environment #{Octopus.Environment.Name | ToLower}\n--consul-address #{DEPLOY_CONSUL_ADDRESS}\n--consul-token #{DEPLOY_CONSUL_TOKEN}\n#{ExtraArgs}",
    "Octopus.Action.Package.PackageId": "#{PackageName}",
    "Octopus.Action.Package.FeedId": "feeds-builtin",
    "Octopus.Action.Package.DownloadOnTentacle": "False"
  },
  "Parameters": [
    {
      "Id": "9c7f77a7-818c-41a4-a21e-cb7bcf0f5573",
      "Name": "PackageName",
      "Label": "Package Name",
      "HelpText": "The name of the package to deploy that should contain a deploy.sh script.",
      "DefaultValue": "#{Octopus.Project.Name}",
      "DisplaySettings": {
        "Octopus.ControlType": "SingleLineText"
      }
    },
    {
      "Id": "c1f0abb2-c136-4f26-9242-3fe1e4be7126",
      "Name": "PackageVersion",
      "Label": "Package Version",
      "HelpText": "Version of Package to be fetched in the deploy script.",
      "DefaultValue": "#{Octopus.Release.Number}",
      "DisplaySettings": {
        "Octopus.ControlType": "SingleLineText"
      }
    },
    {
      "Id": "97ae20db-038f-494c-aadb-73bdafb56f08",
      "Name": "ExtraArgs",
      "Label": "Extra Arguments",
      "HelpText": "Arguments to add to the call into the deploy script.",
      "DefaultValue": "",
      "DisplaySettings": {
        "Octopus.ControlType": "MultiLineText"
      }
    },
    {
      "Id": "60823727-c8c0-42d5-8dad-f1ff8a74ce7d",
      "Name": "DEPLOY_CONSUL_TOKEN",
      "Label": "",
      "HelpText": null,
      "DefaultValue": "#{CONSUL_TOKEN}",
      "DisplaySettings": {
        "Octopus.ControlType": "SingleLineText"
      }
    },
    {
      "Id": "a47f0f3a-6161-4cf8-95f1-e16c304590fb",
      "Name": "DEPLOY_CONSUL_ADDRESS",
      "Label": "",
      "HelpText": null,
      "DefaultValue": "#{CONSUL_ADDRESS}",
      "DisplaySettings": {
        "Octopus.ControlType": "SingleLineText"
      }
    }
  ],
  "StepPackageId": "Octopus.Script",
  "$Meta": {
    "ExportedAt": "2021-12-15T21:56:55.523Z",
    "OctopusVersion": "2021.3.8275",
    "Type": "ActionTemplate"
  },
  "LastModifiedBy": "Your GitHub Username",
  "Category": "other"
}
11

Hi @OctopusSchaff,

Thanks for sending that through, unfortunately I haven’t been able to reproduce the same issue you are facing. It’s all working as expected for me and I’m unable to have newlines in ScriptParameters without a \ ​

The past releases have a deployment process snapshot which aren’t meant to be modified but during my tests I could repair missing \. I would like to confirm if these previous releases will require modifying before they can be re-deployed.

Could you please run the following sql query? This will output all the Deployment Process Snapshots for any project that contains the name of the Step Template shared above, feel free to redact and trim any results or even modify the query to return some more specific results.

Please make sure to backup the DB and master key prior to running any queries:

select * from DeploymentProcess where JSON like '%Bash Deploy Script%'

The output can be securely uploaded to this location, please reach out if there are any issues!

Best Regards,

12

I uploaded the results of that query. If there is anything else I can provide to work this out please let me know!

1 Like
13

Hi @OctopusSchaff,

Thanks for uploading the results of the query!

Unfortunately this has confirmed that there will be an issue with those older releases. I’ll reach out internally to confirm the best approach for resolving this and will likely have some more SQL queries to run.

I’ll keep you posted if I have any updates!

Best Regards,

15

@finnian.dempsey - Just wanted to ping you on this issue to see if you have any ideas for us? I have some nervous people over here. There is talk of just releasing all 30+ apps again, but that is a LOT of paperwork for us (we work in a very regulated industry and every version change has to be very well documented).

I am hoping you have an idea that could bail us out of this. Would downgrading our Octopus Deploy version fix this?

16

Hi @OctopusSchaff,

No problem at all, I understand the concern!

I’ve been discussing this with the Devs and we haven’t yet been able to pinpoint how you got into this state. Unfortunately with my testing I wasn’t able to get the script to work with it configured how you had it previously working on any version. e.g. \n instead of \\\n and so downgrading version I don’t think will help resolve this.

I’ve created a GH issue for the devs to further explorer what occurred to get you into this state, however the options going forward are to either ‘re-release’ the apps as you’ve mentioned, creating new DeploymentProcess Snapshots which contain the required changes (single line script or \\\n) and deploying those new releases.

The alternative is to edit the Deployment Process JSON snapshots via SQL queries for each of those releases, however this is modifying data that isn’t meant to be changed and this can cause auditing issues, especially if you are in a very regulated industry. I would not recommend this solution if there are alternatives but let me know if you would like the required queries constructed.

I’m sorry I don’t have a better answer for you, feel free to reach out if you have any questions!

Best Regards,