XML from Reporting resource in the API - encoding issues

I am trying to pull the xml file from the reporting endpoint (…api/reporting/deployments/xml?apikey) and i cannot use the resulting xml because the encoding characters in the file are set to UTF-16 even though the content is UTF-8.

$foo = invoke-webrequest -uri “path to xml file w/ apikey” -method Get
$foo | output-file -path .\foo.xml
Try to open the xml file in IE or Excel and get an error “invalid at the top level of the document”

Attached is a screenshot where the background is a correct encoding (UTF-8) and the foreground is the UTF-16 encoding.

This exists with 3.2.24 and 3.13.0.

encodingDifference.png

Hi Kristofer,

Thanks for getting in touch! I had a go at this on my server and noticed the same behavior. I can confirm that Octopus is outputting the file correctly, so it looks like the Powershell out-file default encodes to UTF-16, I ran the following command to export the XML with the correct UTF-8 headers and can confirm that it worked for me.

$header = @{ "X-Octopus-ApiKey" = $OctopusAPIKey }
$foo = (Invoke-WebRequest -Uri "http://octopusserver/api/reporting/deployments/xml" -Method Get -Headers $header).content  | Out-file -Encoding utf8 .\foo.xml

Could you try the above command and let me know if it solves your issue?

Looking forward to hearing from you.

Best regards,
Daniel

Hello,

It is not the file encoding, that does not seem to do the trick. The issue is the Byte order mark in the xml stream. https://en.wikipedia.org/wiki/Byte_order_mark. It looks like your stream, shown in the foreground, is trying to be UTF-16, but the byte order mark is repeated.
[cid:image001.png@01D320A0.5281A280]

image001.png

Hi Kristofer,

Thanks for getting back. Sorry that my previous script didn’t work. I ran it past the team and we did some digging. We can confirm that Octopus is outputting the file correctly. Powershell is deciding to encode it with the UTF-16 when using Out-File. We tested the following lines on multiple PS sessions and have confirmed that it is working to output the .xml with the correct UTF-8 encoding.

$OctopusAPIKey = 'API-'
$header = @{ "X-Octopus-ApiKey" = $OctopusAPIKey }

$MyPath = "C:\Users\daniel\foo.xml"
$response = (Invoke-WebRequest -Uri "http://yourserver/api/reporting/deployments/xml" -Method Get -Headers $header).Content
[IO.File]::WriteAllLines($MyPath, $response)

The [IO.File]::WriteAllLines($MyPath, $response) line will output the file while enforcing our encoding. It appears the issue lies with Out-File.
I’m not too sure why the code I initially linked you appeared to work on my end during testing. Let me know how you go, or if you have any issues with the above.

Best regards,
Daniel

Hi,

More information. The output seems to differ because of the method calling the URL

$OctopusAPIKey = “your key here”
$OctopusURL = “your url here”
$Myfolder = “C:\somefolder”

OD Way

$header = @{ “X-Octopus-ApiKey” = $OctopusAPIKey }
$MyPath = $myfolder + “kris13A.xml”
$response = (Invoke-WebRequest -Uri $OctopusURL -Method Get -Headers $header).Content
[IO.File]::WriteAllLines($MyPath, $response)

My Way

$mypath2 = $Myfolder + “kris13B.xml”
$myuri = $OctopusURL + “?apikey=” + $OctopusAPIKey
$response2 = (Invoke-WebRequest -Uri $myuri -Method Get).content
[IO.File]::WriteAllLines($MyPath2, $response2)

If I grab the xml with the key in the header, like you have been doing, the file looks good.
[cid:image001.png@01D32163.66514F40]

If I grab the xml with the key in the URI, I get extra characters.

[cid:image002.png@01D32163.821DB420]

Dumping the file out with out-file or [io.file]::writealllines produces the same result.

Furthermore, and what’s strange, is that if I pipe the responses to Format-Hex and output those to a file or otherwise compare, they are the same values.

In case its relevant, I’m using PSVersion 5.0.10586.117.

I’ve also done some testing with this option
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[IO.File]::WriteAllLines($MyPath3, $response2, $Utf8NoBomEncoding)

And this seems to help get the response that comes from the url with the key in the uri into a readable format. However, this also seems to cloud the results I get when I don’t use this switch.

Anyway, I’m going to do some further testing constructing the api calls with the key in the header and see where that gets me.

Thanks for the time,

-Kris

Hi Kris,

Thanks for the detailed information here! The results I initially had were quite perplexing to me. I was easily able to replicate the problem, then had it working with my initial response. Then was unable to reproduce the successful request the next day. My most recent response seemed to be a consistent method here. I have passed this information onto the devs, however it seems to be an issue isolated to PowerShell.

If you happen to find any further information here that could potentially shed some more light on this, please feel free to send it though. :slight_smile:

Best regards,
Daniel