we have a Octopus Deploy 2019.12.0 configuration with 2 nodes (paid subscription).
Our development teams have dedicated groups and access is synced from AD.
Our users are frequently reporting that they lost acces to projects or the entire group (ProjectView or GroupView permission missing error). The same happens if they are editing variables. Also for projects they created a few seconds ago (missing permission EditVariable error). If we check the permissions, everything looks good. After a while it will begin to work again without us taking any action.
Also if existing users are added to a AD group, sometimes they are granted access right away, sometimes they have to wait multiple hours unitl they get proper permissions assigned within Octodeploy.
Thanks for getting in touch! I can certainly see why this behavior would be frustrating and confusing!
if existing users are added to a AD group, sometimes they are granted access right away
This is by design, as we only check every hour (starting from when the server service starts). You can search for system tasks in the Tasks page and filter by the Sync External Security Groups task (as shown below). If you drill into this task, you can use the Rerun button in the overflow menu to run again immediately and get the groups updated.
The leader node is responsible for the scheduling of the task, but either node could pick up the task to do the sync. One explanation for the intermittent behavior could be that the user the nodes are running as is different, and have different permissions into AD. If one doesn’t get an error but gets back a constrained set of groups because of permissions, that’s what it’ll set the user’s groups to.
Could you confirm which user(s) the services are running as, and also whether there are multiple domains involved (sometimes crossing domain boundaries can be the source of permissions not being set up correctly)?
I hope that helps, and I look forward to hearing back and getting to the bottom of this one!
booth nodes run within the same user context. Servers are set up identical. Also, the group sync task does not seem to have any issues synchronizing groups.
We have a structure with multiple domains, but all relevant users and groups are members of the same domain and have read permission including the application account the octodeploy server instances use.
The strange thing is, that if the error occurs, and we run the sync task manually, it won’t update the missing permissions, and we have to wait. Usually after some time, it works again.
Your answer does not explain, why users randomly loose permissions they had previously without any changes introduced by us. If AD could not be synced correctly shouldn’t there be visible errors in the task log?
Our installation was created before the new permission system was in place. Could this be caused by side effects introduced by the upgrade? The legacy groups are still present but unused.
Upgrading shouldn’t have caused any issue here. It might be possible this is caused by AD replication. If there are multiple domain controllers and not all of them have the same picture at a point in time, then it depends on which one Octopus gets directed to as to the answer it will get. We get no say in which domain controller the Microsoft calls decide to connect to. A while back we saw an issue where one of the domain controllers wasn’t syncing for some reason which caused some problems - is it possible that’s happening in this case?
In the sync task log, if you open the verbose log you will see the sync details for every user, showing exactly which group SIDs AD gave back for each user. If you compare the run before a user had issues and a run after, comparing how the SID list differs might help point us in the right direction.
Let me know how you go, and I look forward to hearing back.
