Users can't update tenant variables anymore

[yenti.verle]

Hey Octopus Team,

We’ve recently upgraded to version 2022.4.8344 but now we’re running into an issue where some users can’t update tenant variables anymore as the error “another user has made changes to one of these variables between when you started editing and when you saved your changes” always appears. The error persists even if the user logs out or changes to another browser.

I’m not sure why this is happening or how to fix it, one user ran into the issue the day after we upgraded to the new version of Octopus and another user has started running into it after somebody renamed a variable. When a user runs into this issue it seems to happen on every single tenant, even ones they have never interacted with before.

This is the full error our users are experiencing

Any idea why this is happening or how I can find out what is happening? We ran our previous version (2020.1.12) for multiple years without this ever happening.

Kind regards,
Yenti

[clare.martin]

Good afternoon @yenti.verle,

Thank you for contacting Octopus Support and welcome back to the forums!

I am sorry you are seeing that error when updating tenant project variable templates. Do you get the same error when trying to update those variables via the project page itself (so adding a variable to that project template)?

I took a deep dive into this and this is the code that we run which would produce that error:

if (resource.ConcurrencyToken != null
            && resource.ConcurrencyToken != TenantVariableProvider.GetConcurrencyKey(existingValues))
        {
            throw new DomainException("Changes to these tenant variables could not be saved, because another user has made changes to one of the variables between when you started editing and when you saved your changes. Please reload or open a new tab to make your changes.");
        }

During my testing if I take a HAR capture when I am on the tenant project variables page you can see I have a concurreny token shown if I click on the variables page in the dev tools capture and click on ‘Preview’

This is what Octopus is expecting, what do you see on your HAR file when loading that page and clicking on the variables name in dev tools and then clicking on ‘preview’?

Have you recently changed your process for updating the variables by any chance, have you started using the API as well as the UI at all?

The other thought I had was are you using a load balancer? We often see similar errors when customers upgrade as they are seeing a cached version of that UI page. You did mention you signed in and out and used a different browser but I am wondering if your LB is serving you up a cached version of that page (if you don’t use a LB ignore this comment).

I look forward to hearing from you,
Kind Regards,
Clare

[yenti.verle]

Hey Clare,

The affected users can still change a project variable template from the project screen. This issue only seems to appear when they try to set the value of a project variable template for a tenant. I forgot to post this in the original question but the affected users have this issue on every single tenant, even if the tenant is only linked to a single project that hasn’t had any changes in months.

I’ve asked them to create a HAR capture and I can see that there is a ConcurrencyToken in the request, both users have a different token, I’ve also checked my own token which is also different from the ones they have.

As far as I know nobody uses the API to change any variables, the only way we interact with the API is using the Octopus extension from Azure DevOps. We don’t have much information about how or when it started happening to the first user, but we know that the second user ran started running into the error after somebody changed the description of a project variable template.

We’re not running a load balancer and we moved Octopus to a new server and a new database during the upgrade. I don’t think it’s related to caching at all as it’s happening in multiple browsers and one of the users has been running into the issue for over a week now.

Kind regards,
Yenti

[clare.martin]

Good morning @yenti.verle,

Thank you for the update, good to know users can still change the project variable template from the project screen, at least thats an easy workaround for you for now so you can actually edit those variables.

Also good to note you do not use the API to make changes, that makes it easier for us to test.

One more question, you mentioned:

We moved Octopus to a new server and a new database during the upgrade

When you mean new database do you mean you are now hosting your Octopus DB (so the original upgraded from 2020.1.12) on a new DB server? Or you put a new version of Octopus on a new server, created a new DB and then imported your projects over from your old instance to your new one with something like our Octopus Migrator tool?

I am just in the process of running some tests on my on-prem instance to see if I can replicate this issue, though I am not sure I will be able to, the version you have was out last Wednesday and we have had no other customers contact us with this issue, so it may be specific to your instance.

I will get back to you with any findings,
Kind Regards,
Clare

[yenti.verle]

Hey Clare,

We had an old 2020.1.12 server that we couldn’t upgrade because our database was running on a very outdated MSSQL version. When we decided to upgrade we moved our database to a new server containing a newer MSSQL version, installed Octopus 2020.1.12 on a new server and then followed the Major Releases upgrade guide. So all of the data from our old Octopus instance is still available, the Octopus DB is just on a new MSSQL server and Octopus itself is running on a new VM with higher specs.

While the workaround allows us to edit the templates, the affected users still have to ask somebody else to set the value for one of the templates.

For testing purposes I’ve given one of the affected user the System Administrator role, when he had this role he was able to make the required changes. So for this user the timeline looks like this:
Octopus Upgrade → User can make changes to the values of the templates
Description of template changed → User can no longer make changes to the values of the templates
User gets administrator permissions → User can once again make changes to the values of the templates

Obviously we can’t make every single one of our users a System Administrator so this isn’t a good solution but it is strange that giving these permissions fixes the issue.

Kind regards,
Yenti

[clare.martin]

Hey @yenti.verle,

Oh! Bingo, thanks for that detailed description, I have been trying to get this to break all morning and I had not gotten round to permissions testing. Having just read your comment I created a test user who just has project deployer and tenant manager roles and tried to edit the tenant variable and I get the same error you see! That user can edit the project templates variable but I did realise that would not have been helpful anyway as you can only specify a default value there not the actual variable value so sorry for suggesting that would be a workaround.

I am just in the process of testing what permission you would need to get this to work temporarily (I agree with you, giving admin permissions to everyone is really a no-go so I will see what permission you need).

I will also get a GitHub issue raised for this so our engineers can take a look, thank you so much for providing that information and letting me know about the admin permissions, I will get back to you as soon as I know what permission you need to temporarily add.

I must admit I have not tried this on an older version (a 2022.3) so I will try that too and see if it is a new bug in 2022.4 or this was introduced a tad earlier.

Kind Regards,
Clare

[clare.martin]

Hey @yenti.verle,

Finally got there, this seems to be a permission scoping issue, I created a team which just has Tenant Manager permissions and I scoped that to my Dev enviroment. I then added a test user to that team:

I then have a tenant with a project attached to it (with the project variable template in), that project is scoped to both my test and dev environments:

Logging in as my test user I can see the tenant variable for the dev enviroment (not the variable for test) but when I go to edit the variable for dev I get the error:

If I then go into my team the test user is apart of and add the test environment:

You can see my test user can now see both test and dev variables and can edit them and save without the error:

I assume you will have similar scoping on your teams and one of the environments is missing from the team which is included on the tenants you have connected projects for.

Giving the user admin permissions takes precedence over any scoping, therefore meaning that user can access any environment and therefore are able to edit all the variables.

So the workaround here is to add the environment scope for the Tenant Manager role to the team your users are apart of so it matches what is scoped for the tenant projects shown on the tenant overview page (I hope that makes sense based on my screenshots above).

I will get this in front of our engineers tomorrow to see if this is how we should be doing it, if anything we need to change that error to make it clearer that user needs access to a certain environment in order to edit any tenant project variables. In my opinion if you have given the user access to one environment you should be able to see the variables for that environment (as you do at the moment) but you should also be able to edit those variables without needing access to ALL of the scoped tenant project environments!

Scoping can sometimes be a little tricky to workout though so it may be we have designed it this way. I will let you know what our engineers say, in the meantime you could test this by giving a test user Tenant Manager with access to all environments and see if you can edit those tenant variables.

Kind Regards,
Clare

[yenti.verle]

Hey Clare,

Thanks for the help, I’ve been trying to set this up but I’m still running into some issues.

For testing purposes I’ve created a new team and added the affected users to it, this team has Tenant Manager and Project Variable Manager permissions on all projects, environments and tenants. When set up this way they still run into the same issue.

I’ve already asked the users to sign out again and try a different browser to ensure that they aren’t getting a cached page, I’ve also confirmed they can see each environment and project that the tenant is linked to.

Kind regards,
Yenti

[clare.martin]

Hey @yenti.verle,

I am sorry the workaround is still not working as expected for you, I did some more testing surrounding this and added another team which had the same permissions (Tenant Manager) and was only scoped to my Dev environment (so the test user is part of one team not scoped to any environments, the other team scoped to dev) and I can still edit those variables so it doesn’t seem to be that you have a permission conflict between teams somewhere.

The other thing I noticed was, when I do limit the scoping to my Dev env I get this on the Project variables page for the tenant (if I click on the 1 more button it doesnt show me anything - it should show me the test environment):

Since my project is linked to two environments:

When I take all scoping off my test user I see this:

Are you able to check an affected user and see if you see all the environments scoped to that tenant project in the project variables screen. If not, that is your issue and you may have some permission mismatch somewhere, one I have not been able to replicate.

Are you able to create a brand new local octopus test user, link them to just that that new team you created and see if you get the same issue on login? If you do get the same issue there is something I have missed. If you do not get the same issue and your new test user can edit those tenant variables then your other users are in a team which is still blocking them from editing those tenant variables.

Kind Regards,
Clare

[clare.martin]

@yenti.verle

One other thing I need to mention - I didnt want this getting lost in my previous comment, the engineers got back to me last night this is what he mentioned:

The reason you are seeing this is due to the fact that we version tenant variables now. The engineer does not know the full details but we compare the new values to a hash of the existing values. Since a user with restricted permissions only has access to a subset of the values, when that user tries to save they also only send back those values in the save request.

So some values are missing and server says “these new values are missing something” and rejects the request. He said it sounds like something we overlooked when adding this feature, I am surprised this is the first time I have heard of this but it may be because you went from 2020.x which is quite an old version now to 2022.4. Whilst this is a supported upgrade path this probably wasnt a breaking change for most of our users when they upgraded.

The comparison of changes from your old version to this new 2022.4 are here so that github issue and the changes off the back of that were introduced in 2022.3.348.

Just a bit of background in case anyone else comes across this or you just wanted to know if you need to report on this incident after we have fixed it.

I will get a GitHub issue raised as this is a bug but I want to make sure we get a workaround in place for you first before I start writing the issue.

Let me know if the test user works,
Kind Regards,
Clare

[yenti.verle]

Hey Clare,

Thanks for all the help! It did indeed end up being a scoping issue which we fixed by giving the users access to the Environment Viewer and Project Viewer permissions without any scope. This does have an annoying side effect because all users are now able to see all projects, but at least we’re able to save our variables now.

Everything below is a step by step guide on what I did, changed and how I got to the conclusion I did. It’s mostly screenshots but I think it might help anybody who stumbles upon this post in the future.

I’ve created a new user and added it to the Variable Bug team, I’ve scoped all the permissions exclusively to the development environment.

I’ve then created a new tenant and a new project which is linked to the development and test environment (screenshot from my main account).

As you said this creates issue where I get a “1 more” box instead of the Test environment which results in the error (screenshot from the dummy account).

Interestingly the entire situation gets fixed for the dummy user if I simply unscope the Environment Viewer permissions.

I even went a step further by creating a second project, linking it and then scoping the Project Viewer permission to the first project. This resulted in me only being able to see the first project but I could still edit the variables for it and save it without any errors. That is until I used my main account to change the values of the second project.

So these permissions

Gave this result for the dummy user (can only see the 1 project)

While my main account can see the 2 projects

I added some dummy values using my main account

Which as expected resulted in me running into the error on my dummy account

So now with the knowledge that it breaks when I either can’t see all the projects or can’t see all the environments I decided to try and change the value of a real variable of a real tenant. I added my dummy user to one of the teams created to give full access to a project, so at this point I’m in both the access team (permissions in the screenshot below) and the Variable Bug team.

Going to the tenant shows me the 2 projects I’ve given myself access to but it also reveals that there are actually 11 projects that are linked to the tenant (so I can’t see all the projects).

As expected I can’t save at all because I’m missing the variables from the 9 projects I can’t see.

If I now simply remove the scope from the Project Viewer permissions I can see all projects

And most important I can save the variables

So for us the fix was giving everyone Environment Viewer and Project Viewer permissions without a scope. This has as an annoying side effect that all projects are visible for all users which isn’t ideal but at least it allows us to save our variables.

Kind regards,
Yenti

[clare.martin]

Hey @yenti.verle,

Wow, thank you SO much for posting all that up, that’s going to help a lot of users and has also helped me, since I only had one project connected to any of my tenants (did not think to connect more than one!) I did not see the issue where u have to have access to VIEW all projects. This kind of makes sense if you think about it. If the bug we initially found is that you need to be able to see all environments then it would also make sense that we did not account for seeing all the projects too.

That’s really helped, especially with those screenshots, I do agree giving users the ability to view all projects and environments associated with a tenant to get this to work is not ideal, whilst it is a workaround its definitely not a permanent fix we want to keep moving forward.

I will get a github issue raised and test the multiple project issue too so I know I can replicate it (I am sure I will be able to). This will be an interesting one for our engineers to look into fixing!

Again thank you for your patience whilst we worked though this together, its been a pleasure fault finding with you!

Once I get the GitHub Issue raised I will link it here, I will also post up to the engineers the fact you need to be able to view all environments and projects connected to a tenant for this to work!

Kind Regards,
Clare

[clare.martin]

Hey @yenti.verle,

As promised here is the GitHub Issue for this (that was a doosey to type up!)

On a side note - it explains this in the github issue but I managed to narrow down permissions to one custom user role with these permissions:

So you can see you do need EnvironmentView and ProjectView but those are the only permissions you need if you want a user to edit those tenant variables (alongside the team scoping of course which I go into more detail in the GitHub Issue).

Thanks again for your patience on this one, subscribe to that GitHub issue to get updated when the fix is out, I imagine this may be a head scratcher for our engineers to fix due to the complexity of permissions you need to account for.

Kind Regards,
Clare