"fake" packages show up in new releases after 3.16.2 upgrade

See attached files. The “fake” packages (two of them always) have the names of actual steps in the process.

This started happening after upgrading the latest 3.15.x to 3.16.2. We are using the built-in Octopus nuget package feed to store the releases.

CAN NOT DEPLOY ANYMORE BECAUSE OF THIS

;-( <---- SUPER DUPER SAD FACE

We are in New York, USA (EST time zone)

FYI, upgraded to the 3.16.3 release this morning… no dice… problem still exists after a new release is created.

I’m adding another screenshot that shows what the “edit release” screen shows.

Adding screenshot of Audit screen showing only the real package getting pushed to Octopus for release creation.

Attaching log file with nlog configuration minlevel set to Trace. During the trace I performed a release (Releases-961).

I don’t see anything in the log file that leads me to why this is happening.

OctopusServer.txt (1 MB)

In the database, for this release in the releases table, this is the JSON value:

{"ReleaseNotes":null,"ReleaseDefects":[],"LibraryVariableSetSnapshots":[{"LibraryVariableSetId":"LibraryVariableSets-1","VariableSetSnapshotId":"variableset-LibraryVariableSets-1-s-1-GQHNS"},{"LibraryVariableSetId":"LibraryVariableSets-2","VariableSetSnapshotId":"variableset-LibraryVariableSets-2-s-2-6SLT3"},{"LibraryVariableSetId":"LibraryVariableSets-4","VariableSetSnapshotId":"variableset-LibraryVariableSets-4-s-1-Z3FRP"},{"LibraryVariableSetId":"LibraryVariableSets-5","VariableSetSnapshotId":"variableset-LibraryVariableSets-5-s-1-JLS8G"}],"SelectedPackages":[{"StepName":"Deploy Web Application","Version":"1.7.1.5+Branch.hotfix.Sha.9ae9a778ba24a9b88a054f07646d4277e287abb8"},{"StepName":"IIS Application - Create","Version":"1.7.1.5+Branch.hotfix.Sha.9ae9a778ba24a9b88a054f07646d4277e287abb8"},{"StepName":"Configuration - Encrypt Section - Web","Version":"1.7.1.5+Branch.hotfix.Sha.9ae9a778ba24a9b88a054f07646d4277e287abb8"}]}

ARGHHH!!!

What the hell. Why on earth are those two steps getting added to the “SelectedPackages” node in the JSON?!

Just a suggestion, but maybe you can hire a remote worker for support that works in the opposite timezone as you guys in Australia… like someone in New York?

I just know that the Hong Kong timezone is 12 hours difference. When I used to work for Y&R they used to have support split between Hong Kong and New York. That would allow them good coverage globally at any time of the day.

Just my two cents :wink:

Hi Jason,

Thank you for getting in touch, even if it is with a super duper sad face. The problem looks like the create release page thinks that those two steps require packages (showing those two extra lines with a blank package ID). Everything else downstream assumes that page did the right things (hence the extra SelectedPackages in the release). Which packages actually exist does not matter.

I’ve tried reproducing the problem by going from 3.15.1 to 3.16.2, but no dice. I’ve also checked the code changes and nothing stands out.

Could you please send me the following:

  • In your browser go to http://yourserver/api/projects?name=Exam%20Proctoring%System, that should give you some JSON output (if I got the project name right). Then follow the link in the first Links object named DeploymentProcess. Please send me the result
  • Also send me the result when you add /template to the above url (eg http://yourserver/api/deploymentprocesses/deploymentprocess-Projects-1/template
  • Go to Library -> Step templates and go into the two templates in error and export them

What I’m looking for in the first result is whether Octopus.Action.Package.PackageId exists and what it’s value is for those two steps (it should be missing or null). I can also import the JSON to see if I can reproduce it.

Once you have the above information, could you try recreating one or both of the steps in error and see if that helps?

Regards,

Robert W

Attached JSON files.

deploymentprocess-Projects-141.json (33 KB)

deploymentprocess-Projects-141-template.json (680 Bytes)

projects.json (2 KB)

Attached the step templates. They came from the community library.

Configuration_-_Encrypt_Section.json (6 KB)

IIS_Application_-_Create.json (17 KB)

I looked at both of the step templates and the Octopus.Action.Package.PackageId is already set to null.

I can reproduce this same issue in every single one of our projects on new builds/releases.

It’s always these two “phantom” package/steps.

I mean… did these two community steps break because of the 3.16.x release?

Just curious… but do you guys do any sort of automated analysis of the community step templates library on each release of Octopus? Like there should be something like that that way you guys can display what versions of Octopus each step template is compatible with. Atlassian does this with all of their products on the community plugins.

I noticed that inside the step template export JSON… these two problem step templates are missing two properties that other step templates have:

"Octopus.Action.Package.NuGetFeedId": "",
"Octopus.Action.Package.NuGetPackageId": "",

Are those now required?

Ok, so this is weird… I don’t think it has anything to do with those step templates. For shits and giggles I made a clone of one of them, not both, and added those two missing properties… then updated the Process in the project to use the clone step template (instead of one of the community ones.) Mind you, the process still has one of the problem step templates in it that shows the phantom package.

I ran a build and BAM it’s working. I thought… that doesn’t make any sense.

So I went to another project… did a build… still has that problem of two phantom packages for those two steps… Guess what fixes the problem? I went into the Process and re-ordered the steps… clicked save… reordered them back to the correct sequence and saved… did a build and now that project works.

So IMHO, I think there is some sort of database corruption that happened when we did the upgrade from 3.15.x to 3.16.2.

I’m not going to fix the other ones just yet. How do you want to proceed? I can’t be the only one this is going to be a problem for… or was it fluke data corruption?

Ok, I lied… I tried this “reorder and save” trick on the process steps in another project because I just can’t believe it… yup… that fixes this database corruption problem. It’s like the upgrade corrupted the step process and somehow that translates to two of the steps becoming phantom packages.

So I’m definitely not going to muck with anymore of the borked projects in case you want to do further analysis to find the cause of the problem… and maybe also parse for the problem in a future version of Octopus to help fix the problem for other people.

Or maybe the “System Integrity” checker button you guys have in Configuration > Diagnostics should check for this type of problem?

Hi Jason,

Thanks for all that, but it all looks good so far. I tried importing it into 3.16.2 and it all works :(.

However, I realised you showed me the Edit Release page. Could you try creating a release and seeing if the extra packages show up there? If they do, could you, go back to another page, open up the browser dev tools and click Create Release again and send me all the requests it makes?

If it doesn’t, how are you creating the releases? It sounds like they are being created from your CI server. Which one are you using and are you using a plugin, octo.exe or a custom script? Which version? What parameters do you provide?

To answer your other question, no we do not run regression against community templates, but we generally keep backwards compatibility as our built-in steps work in the same way.

The Octopus.Action.Package.NuGetPackageId property does not affect anything, however Octopus.Action.Package.PackageId does. We create a entry in the template API call for every step that has the Octopus.Action.Package.PackageId property. Despite the template having that property, it is removed when the step is saved (as showed in your deployment process JSON) and backed up by the template JSON.

Jason,

Also, could you send me the JSON from the DeploymentProcess table for the fixed project for now and when it was broken?

SELECT Release.Version, DeploymentProcess.JSON
FROM DeploymentProcess
	INNER JOIN Release on Release.ProjectDeploymentProcessSnapshotId = DeploymentProcess.Id
WHERE OwnerId= 'Projects-141'
ORDER By Version Desc

I wonder whether the API response is filtering out any null values.

Robert W