My team is encountering a very strange JavaScript error that we cannot quite pin down. We have a runbook-only project (in these examples, it is named ‘Runtime operations’), and when clicking the name of any of the runbooks within that project (which would take you to the runbook overview screen for that runbook) we are getting a JavaScript error.
I have attached screenshots of:
The error as we see it in the Octopus front-end
The (same) error in the browser console
The JavaScript block where it is erroring
Based on the line of code where the error occurs, it seems to be erroring retrieving the runbook run history (which is displayed on the runbook overview screen), specifically relating to past runbook executions on tenants.
It is important to note that this error does not occur for administrators such as myself, but I created a local test account and gave myself the same permissions as other users who are encountering this issue and I am able to reproduce it. I have attached the exported permissions of my test account on which I am able to reproduce this error.
Thanks for reaching out to Octopus support! I appreciate the thoroughness of the information you’ve passed along as well.
I have encountered the error you are seeing one time in the past, however, the difference in the previous case was that the error occurred for all users. In the other instance, the error was caused by a Runbook run referencing a tenant that had been deleted. My hunch in your case is that there may be something similar where a previous Runbook run is referencing a tenant that the user does not have access to.
Having a look at your permission export it seems like it may be difficult to reproduce this issue in-house due to the number of different resources your users are scoped to. Would you be willing to share a copy of your database with us so that we can dig in more without excessive back and forth? We do not ask for your master key and therefore would not have access to any sensitive values in the database.
If you would be willing to share a backup of your database I can send you a secure upload link. If not we can continue working together to investigate. I look forward to hearing back and please let me know if you have any questions for me.
Glad to know this has been encountered before. I’m talking with my colleague that manages our Octopus instance about getting a copy of the database. I should have more information shortly. Thanks for the quick reply.
Hi @dan_close, can you send me that upload link? Looks like I should be able to send a copy of our database. I’m not too familiar with the Octopus database, is there a specific database you need?
I sent a DM to you with the upload link you can use. We’ll just need a backup of the database that is being used by the instance that is encountering this error. If you have access to the Octopus server you can open the Octopus Server Manager which will show you the SQL instance and database being used in the connection string.
That should be all I need for now. I’ll work on getting an instance spun up with your database and report back once I have more information. If you have any other questions for me in the meantime please let me know.
My hunch was correct. There were several tenants that had previous Runbook runs and were not scoped to the dev team. I’ve sent you a file that has the tenants from each Runbook that seemed to be causing issues.
I found that if I added those additional tenants to the team my test user was able to load the Runbooks without error. I have created a new GitHub ticket to see if we can handle this more gracefully. I also upgraded your database to the latest release to verify this hadn’t been fixed already.
For now, your workaround would be to update the permissions or delete the previous Runbook runs for those tenants in question. Hopefully this helps and if you have any other questions or need anything else from us please let me know.
Is the synopsis of the issue that the runbook overview page is trying to display runbook run history for tenants that the user does not have access to?
Does this have anything to do with granting a team access to a tenant and then later revoking that access?
Bonus 3rd question:
3. Can I workaround this by giving this team the RunbookRunView permission for all tenants?
Great questions! I’d probably need input from development, but my assumption is we see the error because the Runs tab is included on the main Runbooks screen. I don’t see any direct call made to the RunbookRuns endpoint which maybe explains why we get a UI error instead of the typical permission missing message.
As for how this happened, I took a look through the audit log to try to get an idea. There have been quite a few changes made to the scoped user roles, but I think the most likely scenario is that the team was at one point open to all tenants and then was later restricted to only the ones that are assigned now. I don’t see any record of the tenants I found being explicitly removed from the team.
You can also work around this by providing the RunbookRunView permission for all tenants. In my test instance, I applied the Runbook consumer role to the existing team without editing the tenants and my user was able to access all of the Runbooks. You may have to make additional adjustments however if that ends up granting more access than you would like.
Let me know if this helps or if you have any other questions I can answer.
This is great info, @dan_close, thanks again. It looks like we are able to work around this by granting them the RunbookRunView permission to all environments and tenants.
