Calamari: Unsupported file extension `.nupkg`

Legacy Execution
1

Getting this error in the logs from Octopus Server:

14:10:01   Info     |       Extracting package: C:\Octopus\Packages\TST101.01.Test.Deploy.Project\TST101.01.Test.Deploy.Project.1.0.0.21.nupkg
14:10:01   Error    |       Unsupported file extension `.nupkg`
14:10:01   Error    |       System.IO.FileFormatException
14:10:01   Error    |       at Calamari.Integration.Packages.GenericPackageExtractor.GetExtractor(String packageFile)
14:10:01   Error    |       at Calamari.Commands.RunScriptCommand.ExtractPackage(VariableDictionary variables)
14:10:01   Error    |       at Calamari.Commands.RunScriptCommand.Execute(String[] commandLineArguments)
14:10:01   Error    |       at Calamari.Program.Execute(String[] args)
14:10:01   Verbose  |       Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20180223030959-18300-5 exited with code 100

Basically in short the issue is the regex used to parse the filename in Calamari.

From Calamari.Integration.Packages.PackageName.cs Line 74

static bool TryParseUnsafeFileName(string fileName, out string packageId, out IVersion version, out string extension)
{
    packageId = null;
    version = null;
    extension = null;
    const string packageIdPattern = @"(?<packageId>(\w+([_.-]\w+)*?))";
    const string semanticVersionPattern = @"(?<semanticVersion>(\d+(\.\d+){0,3}" // Major Minor Patch
                                          + @"(-[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?)" // Pre-release identifiers
                                          + @"(\+[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?)"; // Build Metadata
    const string extensionPattern = @"(?<extension>(\.([a-zA-Z0-9])+)+)"; //Extension (wont catch two part extensions like .tar.gz if there is a pre-release tag)

    var match = Regex.Match(fileName, $@"^{packageIdPattern}\.{semanticVersionPattern}{extensionPattern}$", RegexOptions.IgnoreCase);

    var packageIdMatch = match.Groups["packageId"];
    var versionMatch = match.Groups["semanticVersion"];
    var extensionMatch = match.Groups["extension"];

    if (!packageIdMatch.Success || !versionMatch.Success || !extensionMatch.Success)
        return false;

    if (!VersionFactory.TryCreateSemanticVersion(versionMatch.Value, out version, true))
        return false;

    packageId = packageIdMatch.Value;
    extension = extensionMatch.Value;

    return true;
}

Basically the regular expression parses the filename “TST101.01.Test.Deploy.Project.1.0.0.21.nupkg” into:

  • packageId = “TST101”
  • version = “01”
  • extension = “.Test.Deploy.Project.1.0.0.21.nupkg”

Leading to the unsupported file extension error since “.Test.Deploy.Project.1.0.0.21.nupkg” =/= “.nupkg”

I’m not familiar enough with the naming rules allowed by Calamari to suggest a change to the regex, but this is going to be a fairly significant headache for us as a lot of our projects use this naming convention. Is this able to be fixed in a future release, or is this simply an illegal name for a package?

3

Hi Matthew,

Thanks for getting in touch.

For our naming convention we rely on a variant of SemVer 2.0 called GitVersion, so according to those rules your package name is invalid (which is why it’s tripping up the regex in Calamari). Apart from re-naming your packages to conform to SemVer (e.g. TestDeployProject.1.0.0.21.nupkg) I can’t come up with a valid solution for you. For more information on naming conventions in Octopus I would recommend having a look at our docs page on the topic.

I know that this isn’t the answer that you are looking for, sorry for the bad news.

Regards
Alex

4

Hi,

This is was an issue for us too since upgrading to v2018.5.2. We had to change our build process to output .tgz instead of .tar.gz, which works as a workaround.

Conforming to the versioning docs, our packages were matching 4-digit versions (ex.: 0.3.0-feat-somebranch.33), but the Calamari regex should still be able to parse 2-part extensions as mentioned in your list of supported formats here: https://octopus.com/docs/packaging-applications/supported-packages.

So current documentation does not reflect the actual behavior, since 0.3.0-feat-somebranch.33.tgz works and 0.3.0-feat-somebranch.33.tar.gz does not, while both versions respect 4-digit semantics.

An issue should be raised for this, if I’m correct. Please let me know if I need to open one myself.

Thanks

Vath

5

Hi Vath,

Are you able to give me some more information on the issue you are seeing as I haven’t been able to replicate it (yet) using 0.3.0-feat-somebranch.33.tar.gz as a package source. Can you let me know which steps you are using (is it just a package transfer step?) along with which OS you are using.

I will also need to know if you are using the Mono or .NET Core versions of Calamari.

Just so you know I tested with Octopus 2018.5.1 deploying 0.3.0-feat-somebranch.33.tar.gz to an Ubuntu 16.04 box via SSH (Calamari .NET Core).

Thanks Vath, I look forward to hearing from you shortly.

Regards,
Alex

6

Hi Alex,

We’re packaging with Cake and pushing the package to the Octopus Built-In Package Repository with curl to this endpoint: /api/packages/raw.

Then we execute a step directly on the Octopus Server (Windows Server 2012) to download the package and execute a script inside it. This step failed with the following error:

Unsupported file extension .gz
May 23rd 2018 14:15:06Error
System.IO.FileFormatException
May 23rd 2018 14:15:06Error
at Calamari.Integration.Packages.GenericPackageExtractor.GetExtractor(String packageFile)
May 23rd 2018 14:15:06Error
at Calamari.Commands.RunScriptCommand.ExtractPackage(VariableDictionary variables)
May 23rd 2018 14:15:06Error
at Calamari.Commands.RunScriptCommand.Execute(String[] commandLineArguments)
May 23rd 2018 14:15:06Error
at Calamari.Program.Execute(String[] args)
May 23rd 2018 14:15:06Fatal

For this example our package name was: LANDR.Notifications.0.12.0-feat-MG-24995-TerraformDocker.1.tar.gz

LANDR.Notifications being our Package Id/Project Name and 0.12.0-feat-MG-24995-TerraformDocker.1 being the version.

Our version of Octopus Server is 2018.5.2.

After changing from .tar.gz to .tgz, it went fine.

Let me know if you need more info.

8

Hi Vath,

Thanks for the extra information, that helped track down what the issue is. The issue directly relates to how we are sourcing the packages from the cache. Rather than explain it again here are the comments directly from the code base:

/// <summary>
///  Parser for those file not yet sourced directly from the cache.
/// Expects pattern {PackageId}.{Version}.{Extension}
/// Issue with extension parser if pre-release tag is present where two part extendsions are incorrectly split.
/// e.g. MyApp.1.0.0-beta.tar.gz  => .tar forms part of pre-release tag and not extension
/// </summary>

This only applies for Run on Server steps, extracting/using these packages on a Tentacle works correctly as it goes through a different code path.

I’ve checked with the team and at this point in time we have no plans on fixing this issue, as the engineering effort far outweighs the problems caused (especially since there is a workaround for it).

Apologies again for the inconvenience that this has caused,

Regards,
Alex

9

Hi Alex,

Where is that code excerpt located? If it’s on a public repo, I would not mind taking a look and submitting a PR if I can.

Thanks

10

HI Varth,

Unfortunately that’s on one of our few private repo’s for the core Octopus Server product so I can give you access unfortunately.

However, I should have mentioned the reason why it’s in the no fix pile. We are currently working on a new feature called workers which will eliminate the need for this code path. All “run on server” steps will end up as “run on worker”, which is the same code path as what we use for Tentacles which already don’t have this issue (so run on server == run on Tentacle with some cool add ons).

This is all due to ship in the next month or two, we are pretty much feature complete and are in the middle of bug bash/documentation and UI tweaks now.

I hope that helps, any questions please let me know.

Regards
Alex

1 Like