I’m having trouble with how Octopus handles environment variables that contain ampersands (and possibly other special characters). For example, the connection strings generated by Entity Framework look something like this
After Octopus replaces the values however, the two ampersands get turned into “&” in my transformed config file. The resulting file looks like …App=EntityFramework "" instead of …App=EntityFramework ""
This seems like a bug with the HTML encodingdecoding from the octopus UI. Any workarounds to handle this behavior?
Also, please make the columns in the variables screen resizable. Managing long connection strings in the UI is rather burdensome.
It may seem surprising but this isn’t a bug; it’s by design. What would you expect to happen if you created a variable named WelcomeMessage with the value:
Hello, "world!"
If we didn’t encode the values, you’d end up with:
Which would be broken (and would equally be considered a bug). So, we assume that anything you type in the variables editor is unencoded (raw), and we encode it when inserting it into XML.
If you had a PowerShell script like this:
Write-Output $WelcomeMessage
You’d also expect it to output:
Hello, "world!"
Rather than:
Hello, "world!"!
So Octopus expects that any value you give us is unencoded, and we encode it. That way your variables can be agnostic of the encoding used by wherever it might be used.
The problem was because we stored connection strings with escaped symbol "
=> &guot; inside connection string and after transformation &guot; we
receives "
my concern was that you maybe don’t need to do encoding for & symbol.
By I already resolve this issue by using ’ instead " and now I it works ok.
I politely disagree that this is a feature rather than a bug - I think the least surprising outcome is that Octopus directly substitutes the contents of a given variable when it does its substitutions.
Maybe it treats XML files specially, but I’m not seeing the behavior you’ve specified, Paul (if I understand correctly, quotes should automatically be encoded). I’m working with Azure and am inserting values into a .cscfg file.
A variable with the following value: "inside quotes"
Gets inserted into my .cscfg as: “inside quotes” (as I’d expect)
A variable with the following value: "inside quotes"
Gets insert into my .cscfg as: &“inside quotes&” (as I wouldn’t expect)
What I actually need inserted is the one above - "inside quotes"
Based on the above, I can’t find a way to insert the encoded value that I need. I think single quotes will probably work in this case, but, to me, it’s a bug that I can’t insert an encoded value directly via a variable.
Thanks for the follow-up Eric! We’ve spent a bit of time thinking about this one previously; the challenge (and reason we’ve chosen the current behaviour) is that we want to make sure Octopus always leaves configuration files etc. as valid XML.
It’s tricky to do literal substitution like you’ve suggested because let’s say you have a variable defined in the Octopus UI like:
ConnectionString = blah;Password=#{Pass};blah
So, the connection string is bound for an XML config file; but Pass may be a variable used in several places, some of them XML, some not.
Should now the password A&B be entered into the Octopus UI as A&B or A&B?
It’s an open choice, but we need to choose between them for the Octopus default behaviour. We chose the literal form A&B because in more complex variable substitution scenarios we think it will be less surprising. (E.g. imagine Pass is a prompted variable - it would be strange to request the user enter their password using correct XML encoding
There are logical arguments each way, but forced with a choice we’re happy with the current method. Hope this helps explain our reasoning a little better.
Thanks for the quick response, it will work fine as you’ve specified in most conditions. It would be nice if the functionality could be toggled, as I can see needs to do replacements in files other than XML.
Thanks for the update; we’re considering something like this (e.g. in an HTML email template):
<p>Hello, #{HtmlEscape(Name)}</p>
In that case, using #{...} style substitutions in an XML document would work along the lines you’re suggesting. No timeline on this currently, but expecting to address it before very long
I have a substitution variable with a double ampersand.
So, p&ssword => p&ssword
but, p&&ssword => p&&ssword
HTML encoding seems to ignore the double ampersand, breaking my config file. Is this a bug, or is there something I can do to force the encoding to happen?
I’m trying to following this conversation because I seem to be seeing different behavior than explained here.
Yesterday I had a java keystore item where I had a password that I had to pass along. It had an ampersand in it. So as an example, it showed as “p&ssword”. Well that was an issue because it was supposed to show up as “p&ssword”. So I tried passing “p&ssword”. It showed up as “p&amp;ssword”. I also tried “p\&ssword” (ended up with “p&ssword”), because I found that in a previous variable that if I had a path that I passed as a variable “c:\temp\myfiles” that ended up showing up as c:tempmyfiles, so in the end I had to pass it as “c:\\temp\\myfiles\”. (Note: I’ve looked at this in the preview window, so what shows here in my note should be correct. IE in the last example I typed 2 backwards slashes not the 3 I had to use to get 2 to show here)
In the end I changed the password to not have the ampersand to get around this because I couldn’t find a way to just pass a single ampersand as an ampersand. However as I use variables more and more, I may not have that option.
Reading the comments here I’m not clear on how I would actually pass a single ampersand even if that’s what I wanted to do.
Also, is there documentation on what to plan on escaping and what the strategy is to do it?
Note: None of these were web.configs. Maybe the behavior is different for app settings. Guess I’ll find out today.
Second note: Has there ever been thought to a checkbox that says “Don’t touch the contents of this variable. I accept all liability for any bad mojo jojo that ends up happening because of bad encoding?”
I thought about this some more and I still think it’s silly to do any sort of encoding behind the scenes, both as the default and especially because it cannot be turned off.
With the current auto-encoding behavior, there are certain strings which cannot be inserted into files during variable substitution because of the encoding behavior (assuming it hasn’t changed since my first message). At least without encoding, I can type what I want and Octopus can just do a literal, non-fancy substitution.
From Nick’s response above:
"(E.g. imagine Pass is a prompted variable - it would be strange to request the user enter their password using correct XML encoding :)" - it would be, but it’s also strange to ask them to write something such as msallmen stated, this still fails because with “&&” you do have to manually enter it as &&. So now I have to say “as long as your password doesn’t have two subsequent &s, you don’t have to XML encode it, but if it does, then you have to.” Additionally, if you have “Pass” used in one context that requires encoding and another that doesn’t, you just specify two variables with different values. I think that’s easy enough to do - there’s some minor duplication but that’s a fact I’m willing to accept for full control over my variable values.
All that said, it appears there is conflicting information - this entire thread is about automatic encoding during variable substitution, but Vanessa has stated that no encoding is performed on variable substitution. This can’t be true based on the behaviors reported here, as it appears encoding is performed differently based on Octopus’s best guess of the variable’s target use.
I just don’t understand why any substitution other than literal substitution is the default. This behavior to “do what is right” will never have enough context to really do what is right in all situations. In a world with no context, it is doing what is wrong - no one says string xx = “jkl&”; in any programming language and expects it to be XML-encoded when they read it later unless they explicitly call something to convert it. I think that stands here.
I know you can’t change the default behavior for backwards compatibility, but please, please, give us an option (even Octopus-wide) to just turn off the encoding behavior.
I actually think what was stated is probably correct (in intent). I think what is being said is that there are two different behaviors.
If you are doing this as part of the “Replace appSettings and connectionString entries in any .config files” labeled as Configuration Variables in the Deploy a Package step than I think what is being said is that it has one behavior. From what I can tell this doesn’t seem to escape characters.
On the other hand if you “Substitute variables in a file” it appears that it does escape the characters.
I guess there’s a “third behavior”. If you use it as part of a script of your own (or a step template) it also doesn’t appear to escape it.
At least so far (in my limited experience) that appears to be the case. So the specific wording you mention may be correct in intent, but overall it’s still confusing and it seems like in certain situations, it’s not possible to output specific values.
I am going to restate what I said above because it probably seemed like I was joking. Rather than give us a tag, could it be a checkbox. HTMLEncode this field, or not. (I can’t think of a good reason to HTMLEncode only a portion of a variable, but I may be missing something)
