File encoding changed after publishing

Hi,

I’ve upgraded octopus to 3.x (finally), and now started using gulp-octo to pack/publish packages. After publishing we have issues with file encoding. For example a € (euro) sign is not being displayed properly anymore. If I open the file in notepad and save as utf-8, the issue is solved.
I tried changing from tar.gz to zip, bot there’s no difference.

Hi Ruben,

I’m sorry to hear you’re experiencing encoding issues.

Is this problem occurring after deployment? Or are you looking within the pushed package before it has been deployed?

Regards,
Michael

Hi,

I’ve done some further checks, and it seems the problem exists already in the package created. I’ve tried adding an extra step in my gulp file to convert to utf8 using gulp-utf-convert, but the result remains the same.

This is the gulp task I’m using:

return gulp.src(['**/*.*', '!**/*.{vb,cs,vbproj,user,vspscc,nuspec,asax,nupkg,gz,pdb,yml,ts,scss}', '!gulpfile.js', '!node_modules/**/*', '!Scripts/iisnode/**/*'])
    .pipe(utf())
    .pipe(octo.pack('zip'))
    .pipe(gulp.dest('./built/'));

Kind regards,

Ruben
Ruben Laureys
Developer
Research & Development
Hudson
t: +32 9 242 54 21
be.hudson.comhttp://be.hudson.com
About Hudson
Hudson is a global talent solutions company. We help transform the workplace and unleash the full potential of organisations and individuals. Our expert team and proprietary tools provide you with unique insights and services that help you maximise your success. Across 20 countries, we deliver a range of recruitment, talent management and recruitment process outsourcing solutions to get you and your business where you want to be.
This e-mail is sent by Hudson Global, Inc., or one of its subsidiaries, and may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.

That’s interesting.

Are you certain the file is initially UTF8? We have seen problems before where input files are actually a different encoding (usually Windows-1252), and converting them to UTF8 causes issues such as the one you are seeing.

We depend on the archiver NPM package to create the archives. It may be worthwhile trying it in isolation to see if the problem persists? The Octopus Gulp tasks are open-source, as is the octopackjs library they depend on, just for reference.

Hi,

Further tests have shown that just copying a vbhtml file with gulp (just gulp.src and gulp.dest) create the same problem. I can use gulp-header to add the utf8 bom, which solves the problem. It seems IIS needs the utf8 bom, and gulp strips this from files by default (as it is deemed unnecessary).

Although it is not a problem in octpack itself, it is probably usefull to investigate this further and fix it in the octopackjs gulp wrapper. Or at least document the issue and propose a nice solution.

To reproduce the problem doing the following should suffice:

return gulp.src('view.vbhtml')
        .pipe(gulp.dest('./test/'));

If you see the problem there, try this:

Var header = require(‘gulp-header’),

return gulp.src(‘view.vbhtml’)
.pipe(header(‘’))
.pipe(gulp.dest(’./test/’));

To get the entire thing working, there’s some work still to be done. Right now, gulp-header will fail if the stream contains directories. I’ve forked the project, and sent a pull-request. We’re currently discussing the exact implementation but it will be accepted eventually, and a new release will be created.

The next problem is that we don’t want to add the bom to every file. Dll’s, resx, images, … shouldn’t have the bom. Currently it seems only vbhtml (and possibly cshtml) should have the bom. I tried creating 2 streams (one for vbhtml, one for all the rest) and merging these using event-stream. But, after merging the paths are not correct anymore and octopack fails because of none existing files.
I’ll discuss with the gulp-header owner if we can add an optional parameter to limit the file types where the bom should be added to, and in the meantime try to get the event-stream owners to fix the file paths in merged streams.

If you have any other suggestions, or if you think I’m going down the wrong path, please let me know :slight_smile:

Kind regards,

Ruben

Ruben Laureys
Developer
Research & Development
Hudson
t: +32 9 242 54 21
be.hudson.comhttp://be.hudson.com
About Hudson
Hudson is a global talent solutions company. We help transform the workplace and unleash the full potential of organisations and individuals. Our expert team and proprietary tools provide you with unique insights and services that help you maximise your success. Across 20 countries, we deliver a range of recruitment, talent management and recruitment process outsourcing solutions to get you and your business where you want to be.
This e-mail is sent by Hudson Global, Inc., or one of its subsidiaries, and may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.

Ruben,

Thank-you for the update. That’s rather surprising that gulp strips the BOM.

I can imagine this would be a frustrating issue. I assure you, text-encoding is a recurring nightmare for us.

However, in this situation I’m not sure how we could better handle it in octopack?