Config as Code + Add a new param to a step template with no default leaves me stuck

lbrody · 8 September 2022 15:22

We are using Octopus version 2022.2.8011 in an on-prem HA configuration. In a project that is Config as Code enabled I added a step template into the deployment on committed it on a branch. I then edited the step template and added a new parameter (a package) and saved it. When I went to the branch in the UI and updated the step template it asked me to verify the parameters (as expected) then when I click the confirm button

it greys out for a few seconds

and then goes back to being clickable

It isn’t a blocker, as I can delete the step and re-add it, but it is annoying

brent_kinney · 8 September 2022 16:22

Hey @lbrody,

Thanks for letting us know about this issue, I will pass that along to our engineering team so they can take a look. Let us know if you have any other questions.

Best,
Brent

brent_kinney · 9 September 2022 15:39

Hey @lbrody,

Our engineering team thought you might be running into the same issue as this help poster did. That issue was with a Git migration trying to happen simultaneously with the step update. You can verify by looking at your .octopus/schema_version.ocl file. If it’s v4 or less, it could be that same issue.

If that file isn’t v4 or less, any server logs, browser console logs, and any details on failed browser network requests will help us continue investigating.

Thanks,
Brent

lbrody · 12 September 2022 05:04

Hi Brett,

that file has version = 3 in it. It has been that value since we created the project and Config As Code enabled it. What is the Git migration you are talking about?

Thanks

clare.martin · 12 September 2022 08:32

Good morning @lbrody,

Just jumping in for Brent who is currently offline as part of our US Based team. From the ticket Brent linked it looks like the Git Migration mentioned is referenced in that ticket as a Schema Migration. In one of the first comments the user updated a step template and there was a message in his logs showing there was a commit to migrate the Octopus Schema from 3 → 4.

Because he updated his step template before the Schema had chance to update this is why the issue was happening for him.

It looks like you added a step template to your project. This would have similar effects to updating a step template. Are you able to look at your logs and see if there are any messages stating the schema for that project is updating or a migration has occured?

From what I can see in the other forum post:

Our engineers working on this have successfully reproduced the same issue locally. They’ve identified that updating the step version is also triggering a schema version update from v4 to v5. The step version update process doesn’t set things up properly for the schema version update and it is failing.

So it may be the case that adding that step template has triggered the schema version to update. In the forum post the user was originally on schema version 3 as you are so it may be the same issue.

When you update the step template are you able to go into your project and do an update and commit. Whilst doing so are you able to go into your dev tools in your browser and see if you see any errors such as "ErrorMessage": "Stack empty."

If so are you able to follow the below process and see if it works for you?

1. Go into the Deployment Process.
2. Make a trivial change, for example, change the name of a step or add a note to it. Anything really.
3. Commit the change. This should trigger the migration of the schema. There should be a commit with a message 'Octopus Schema Migration to vx' that changes the `.octopus/schema_version.ocl` file to `version = 5`. It may have other changes as well.
4. Now the Update and Commit button should work on the step.
5. The trivial change change can now be undone and committed, if required.

If this doesn’t work let us know and we can continue to fault find, any server logs, browser console logs, and any details on failed browser network requests will help us continue investigating.

Kind Regards,

Clare

lbrody · 12 September 2022 15:28

Claire,

The Schema version is still at 3. I have made a non-trivial change, I added the step template in again, we have also made other changes to the project. The version of Octopus running is 2022.2.8011. What I did was to cancel my change, and re-add the step template to the project’s process. Once I did this, the additional variable showed and allowed me to save it. The change here is that I was trying to upgrade to a newer version of the step template while I was doing the initial add of the step template to the project.

brent_kinney · 12 September 2022 16:04

Hey @lbrody,

Thanks for that clarification. I’m glad that you’re able to save your template changes. I will pass that info on to our engineering team and see if they have any other questions for you.

Best regards,
Brent

clare.martin · 13 September 2022 07:35

Hey @lbrody,

To move this along what the engineers really need is your Octopus Server logs if you can provide those? I have created you a secure link you can send the report to - Lbrody | Support Files. If you can make one more trivial change to that step template and then try and update the CaC project. Can we get the logs after you do that please so that action is definitely in them.

Also, right before you try and update the project with the new trivial value of the step template can you get us a HAR file please. You would need to start the capture before clicking on the confirm button in the project, then carry on capturing throughout the process until the button goes back to being clickable. That way we can see what is happening in the browser whilst the complete action of updating the template in the project is being performed.

If you need further clarification on this please let me know. Hopefully we can get some good information from your logs and get this fixed quickly.

Kind Regards,

Clare

lbrody · 13 September 2022 18:35

Diagnostics are uploaded as OctopusDeploy-637986908439678428OctopusDeploy-637986908439678428config-as-code-add-a-new-param-to-a-step-template-with-no-default-leaves-me-stuck.zip. Will work on getting a HAR file

brent_kinney · 13 September 2022 19:03

Great, thanks @lbrody. Ill pass that along and be on the lookout for the HAR

lbrody · 29 September 2022 21:15

FromTaskListToRunningTask.har (2.8 MB)
Sorry for the delay. here is the har file.
here is the error I am seeing

brent_kinney · 29 September 2022 21:46

Awesome thanks @lbrody.

I will send that HAR file to engineering for review and let you know what I hear back.

Best,
Brent

clare.martin · 30 September 2022 08:11

Hey @lbrody,

Just following up on this one, the engineers have seen in your HAR file that its erroring out on https://octopus.imo-online.com/api/Spaces-1/actiontemplates/ActionTemplates-3601/actionsUpdate , resulting in a 400 response with error message:

  "Errors": [
    "Remote repository returned an error: unpacking the sent packfile failed on the remote"

The error they are seeing is similar to this one here which turned out to be permission issues on the Git Repro.

You may have checked this already but our engineers have asked if you would not mind checking user with the PAT has the appropriate permissions to read/write to the git repo. They did say you can commit other changes so it might not be that but we wanted to make sure. The engineers are still investigating but are you able to check permissions and just confirm they are good.

Kind Regards,

Clare

lbrody · 30 September 2022 14:06

It isn’t a permission issue. Other actions like adding a new step in the deploy process or editing a variable work flawlessly

clare.martin · 30 September 2022 15:10

Hey @lbrody,

Thank you for confirming that, I just wanted to make sure your permissions were correct.

In a previous response in this thread Brent mentioned another user was having this issue. He was having it at schema version 4 though, not 3 like you are on. A private issue was created by the engineers for that which is still being looked into.

I asked you to perform the workaround in that private issue, however, this did not work for you as your schema version was still set to 3 even though you changed the template, meaning a forced schema update to 4 or 5 should have been done.

Since you are on 2022.2 I think you will be stuck on schema version 3. I know I had to upgrade my instance to a 2022.3 version to be able to update my CaC schema version so it may be the workaround will not work for you since changing templates wont force an upgrade unless you upgrade Octopus.

I am going to try and get the engineers to create a public GitHub issue for this so you can subscribe to it and view it fully.

In the private issue a note has been posted in it saying:

What’s happening is that the endpoint used to update the templates (api/Spaces-1/actiontemplates/ActionTemplates-41/actionsUpdate ) doesn’t have a project ID or git ref, so the action filter won’t automatically add anything to the IProjectScope . When it’s time to commit the git transactions, we no longer have anything in the IProjectScope , hence the Stack empty. exception message.

This lines up with what the engineers are seeing in your HAR file so I do now think your issue is the same as the one in the forum post I linked. The private issue specifically mentions schema v4 that has the issue thats all so unfortunately that threw me off at the start.

Unfortunately the workaround will not work for you so I am going to go back to the engineers with the following:

Ask them to create a public GitHub issue for this so its visible to users.
Get them to update the issue to incorporate v3 schemas.
Get them to see if there is another workaround in place for customers who are unable to upgrade to v4.

It may be that the only way to get you unstuck here is to upgrade your octopus instance to a 2022.3 version so you are able to upgrade your schema and perform the workaround but I will see if our engineers have a better alternative first.

I am sorry this has gone on so long and we have ended up circling back around to the original response but hopefully I can get somewhere with the engineers over the next few days and have an answer for you.

Reach out in the meantime if you need anything else,

Kind Regards,

Clare

clare.martin · 3 October 2022 09:17

Hey @lbrody,

Another update for you, our engineers have created a Public GitHub Issue for this so please subscribe to that in order to track updates for when a fix is out. They have changed the title to incorporate all schema versions too.

The engineer working on the issue did note that:

In order to get the migration to complete properly the customer needs to make a change that is unrelated to step templates and commit it.

He suggested trying the workaround again as it should not matter what version of Octopus you are on. Are you able to do this but make a change to something unrelated to Step Templates. I am not sure exactly what you changed when you tried the workaround the first time but it looks like you deleted and added the step template in again. You did note you have made other changes to the project but did not mention what so I am unsure as to if that was also step template related. Changing the projects name (add a number onto the end of it or something) would be an acceptable change and will hopefully then get the workaround working.

I would follow the workaround in the Public GitHub Issue I posted. If you still do not see a commit with a message 'Octopus Schema Migration to vX ’ in your Git Repo and the schema version updated from version 3 to a higher version in your Schema version OCL let me know.

Let me know also if the workaround does work and it then fixes the step template issue.

Kind Regards,

Clare

system · 3 November 2022 09:18

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.