Hi,
I upgraded our server yesterday to the latest release but we are now unable to start deployments via the following steps:
The "An unexpected error occurred in Octopus v2022.2.6729: “TypeError: Cannot read properties of undefined (reading ‘StartTime’)” error is displayed at step #5, i.e. clicking on the specific release.
Possibly related to:
Good morning @InstandaDeployments,
Thank you for contacting Octopus Support and sorry to hear you are having issues with one of your projects.
Good find there on the GitHub issue, this doesn’t seem to have been fixed yet, however, this is possibly due to a customer’s comment of ‘I restarted the server and the issue seems to have been fixed’. Our engineers would have sent that to the bottom of their pile to look at. Have you restarted your server at all, is it still giving the same error after restart? If not, I will add this ticket to that issue which means it will be looked into further.
If possible I would like to ask a few follow-up questions just to get an idea of your setup:
I look forward to hearing from you so we can see if we can fix this as soon as possible, reach out if you need anything else from us in the meantime.
Kind Regards,
Clare
Hey @InstandaDeployments,
I have done a bit more digging into this and it seems there is another GitHub issue which is similar in stack trace errors.
Do you have an auto-deploy to channels on the projects that are giving you the issue at all?
Are you able to check out that GitHub issue and see if it does relate to you? If it does I am wondering whether any of the workarounds would work. If you delete the release that is giving you the issue and then create a new release does that give you the error?
I look forward to hearing from you,
Kind Regards,
Clare
Hi Clare,
I have restarted the server and unfortunately with no change of outcome.
This does seem to only affect one project. We have a few other projects but they aren’t really used actively for production deployments.
We dont use Helm, so I guess we are just using the UI.
The error occurs when we click on any release when using the steps provided.
It does seem only to affect releases which are automatically deployed for one particular channel.
I don’t think I can provide a raw log because I believe you are referring to a deployment which has started and fails. This scenario is a pre-cursor to a deployment, i.e. I am trying to start a deployment, but cannot.
I am reviewing the Github issue but am apprehensive about deleting any database records or releases.
Thanks,
Brian
Hey @InstandaDeployments,
Thank you for that information and the screenshot, this has come up a few times and it seems the only thing that gets the project in a good state again is to delete ALL of the releases in that project and then create a new release.
Do you get any of the mentioned conditions from the GitHub issue (ie If you try to open the server task from the release details page, you get The resource 'Deployments-X' was not found.):
If so you are definitely running into that GitHub issue. This issue happens if a release cannot be deleted correctly, which causes a cascading effect on other releases.
One of my colleagues did find another workaround to export the project and then import it, effectively cloning the project. This allows for the historical data of the broken release and unblocking you from creating new releases.
I am wondering if you could try and export and then import the project and see if that allows you to open up the affected release. If not, you can then test deleting the releases from the imported project, essentially you would be deleting them on a ‘test’ project so it won’t affect the live project.
You can then replicate the behaviour on the live project (deleting that one release having the issue and then if that doesn’t work, deleting all releases). You would still have the export of that project if it does go awry so you can always re-import it again.
I know this is not the news you wanted but from previous tickets we have in for this, it seems like this is the only workaround available to get that project working again.
I look forward to hearing if the export/import works for you,
Kind Regards,
Clare
I suspect the tenants issue may be related. I did fairly recently change the automated deployment from untenanted to tenanted. The outcome was that tenanted deployments are successfully triggered but the system seems to try an untenanted deployment also, which results in a failed deployment being listed everytime in Tasks. It seems to me that the system is lacking something, for example, perhaps an option for controlling whether automated deployments should run for tenanted, untenanted or both.
I’m really am not keen on these drastic measures of remedy. This system is tightly coupled into our processes and doing so would lead to many other problems for us downstream. I am considerint rolling back to the previous version as a much more appealing option.
Regards,
Brian
Curiously, for the failed deployment task which I just mentioned, clicking on it in the previous version displayed nothing, yet since the new version I get the following error:
I found the ServerTask record in the database for this Task and the ErrorMessage column contains:
This deployment cannot continue because the variable snapshot associated with it has been deleted. This can happen if the deployment was imported from a different Octopus server.
I started looking at the Octopus database and found there were 5 deployment records with a null Task Id. I have deleted them and the releases now load after clicking on them.
Hopefully, future automatically generated deployments wont produce any more of these cases and need me to repeat this exercise.
I think I have a valid concern with the point I raised about how automatic deployment generation works tenanted and untenanted. It seems to be trying to deploy for untenanted and every tenant with no ability to control this. Perhaps the untenanted automatic deployment is the cause of this issue.
Hey @InstandaDeployments,
Thanks again for the extra information, ideally we shy away from DB manipulation as we also like to air on the side of caution, however, sometimes it is required depending on the nature of the fault.
The link is not set to a value error we do actually see this a lot after users upgrade their instances and most of the time a hard refresh of the browser works and it will load the page again, the browser can sometimes cache old pages from the previous upgrade so it may be that a clearing of cache and cookies for the browser does allow you to see the information on that page.
As I was typing I noticed your new comment, it’s good news that the deletion of those 5 deployment records with the null task in them has allowed those releases to load. It seems your issue was related to the second GitHub issue we linked then:
Thank you for investigating that and performing the deletions. Since the task ID for those releases was set to null and not a value this is why you were seeing the other errors too such as This deployment cannot continue because the variable snapshot associated with it has been deleted error you had in the DB.
As for your tenant concern, I don’t think it seems to be trying to deploy for untenanted and every tenant with no control over what tenant it’s using, although I can see why that GitHub issue would suggest this.
When you tell a project to deploy to tenanted and untenanted it will only deploy to whatever tenant you select for that project. In our documentation on tenanted and untenanted deployments it states:,
If you select an environment that’s when it uses untenanted deployments. So you do have control over what is deployed to where:
I believe this may be addressed in the GitHub issue itself?
Untenanted and tenanted
If the deployment is going to more than one tenant but deployment targets don’t exist for a certain tenant then this will fail.
Workaround
To avoid releases ending up in this state:
- Switch to a manual deployment phase lifecycle
- Check project tenanted deployment mode settings - if you only need tenanted deployments, don’t select both untenanted and tenanted
I believe this is more related to the environments rather than the tenanted or untenanted, I think the workaround of only selecting tenanted deployments rules out the fact you need to set an environment in the project (which you only do for untentanted).
In the PR for the issue fix our engineers wrote:
If a lifecycle is set to automatically promote a release to an environment, the deployment can fail during its creation. Usually, when this happens via the API we get a response back with the reason why it failed and the entire transaction gets rolled back. For automatic deployments these failed deployments were being partially persisted, meaning the deployment would be left in an invalid state. When opening the release to which this deployment belongs it could cause a React error on the release details page if there was more than one deployment made to the given environment.
So I do think this is more environment-related, it’s just having it set to only tenanted rules out the need to have environments selected, not actual tenants.
I hope that helps alleviate any concerns you may have had, sorry this was a long post but I wanted to try and get all the information to you so you could review it and it would make sense.
Are you happy your project is at a good state now and we can close off this ticket? I will put a link to this ticket on that GitHub issue though, as you are not on a 2022.3 version it would be nice to see if they would include a fix for 2022.2 so I can ask in the GitHub issue. If you subscribe to that you should get notified if our engineers respond.
Kind Regards,
Clare
The cause for me mentioning untenanted automated deployments was not due to the Github issue, but because for every automated deployment, in Tasks view, I see a deployment task for every tenant with the tenant name stated in its title. I also always see one failed deployment task without a tenant name in the title.
For example:
If I click on the failed deployment, a blank page is loaded, which also isn’t ideal.
This doesn’t necessarilty concern me, it’s more of an observation and thought it best to highlight since I thought it might be related.
I checked the deployment targets for each tenant for this automated deployment environment, and yes, there are some tenants without deployment targets, so I think that would explain the failure in this scenario.
Hey @InstandaDeployments,
Thanks for clarifying what you meant and also for the heads up on that, anything that helps us explain why issues occur we always welcome from users, it always helps and all our users are incredibly proactive with trying to help us which we really appreciate!
It does make sense now why you would have been concerned about the tenants from what you have said and seen from your instance, it also seems like the behaviours you are seeing (the untenanted without deployment targets) corresponds to the GitHub issue so I am really glad you found that as it’s just more evidence showing our engineers were on the right lines when they tried to work out why this happened to users.
Thank you for being really proactive with this issue, it seems like we are in a good spot now, I have asked our engineers if a fix will be out for the 2022.2 releases and will post it up in the GitHub issue if I find the answer so if you do subscribe to it you will be notified and can install the patch if you want to for 2022.2.
If there is anything else you need in future please let us know,
Kind Regards,
Clare
I’m just pleased we seem to have come to a resolution.
Thank you kindly for your prompt and proactive support.
Many thanks,
Brian
Hey @InstandaDeployments,
One last update for you, our engineer has gotten back to me with confirmation it’s the untenanted deployments with no deployment targets that are causing this issue.
They will not produce a patch fox for 2022.2 however, he has mentioned there is another way of deleting those releases with no task ID through the API. He has added this workaround to the GitHub issue. This will avoid users editing the DB in future so thank you for our conversation on that as it pushed us to provide other users with a better workaround than editing the DB.
If this happens again in future you can use the API to edit those releases which should fix this issue.
Thank you again for all your work on this and for allowing us to provide a better solution for other customers experiencing this issue.
Have a lovely Friday,
Clare
Hi,
I am seeing this issue reappear again today.
Hopefully you can fix this before too long because it will be wearing to need to delete the records numerous times every day 
Regards,
Brian
Hey Brian,
Sorry this has happened to you again, hopefully deleting the releases from the API is quicker than doing it via the DB, once you have a script setup too you should hopefully just be able to edit it with the relevant project info and run it, which should hopefully make it quicker.
As I mentioned a fix for 2022.2 won’t be implemented as our engineers have produced a valid workaround which doesn’t take too long to implement, I understand though if there are multiple projects that are having the issue it will become tedious to keep having to do the workaround.
A fix is in place for 2022.3 but since 2022.2 has only been released for GA for on-prem customers on Wed of this week it will be a long time before 2022.3 is released for our on-prem customers.
Hopefully, there are some projects you can edit to only make tenanted which should alleviate the issue, also, if you are able to check which tenants don’t have deployment targets you could delete them from that project, which should also alleviate the issue.
I do realise though this is an admin burden as you may want that project deployed to untenanted deployment targets in the future so this might not be a valid option for you.
Hopefully, once you create a script to delete the releases via the API you can manipulate that instead of the DB which should save you time in the future. Again I do apologise this is happening to you, it is frustrating but the workaround is all we have for our 2022.2 customers until we make 2022.3 GA for on-prem customers.
Kind Regards,
Clare
Hello @clare.martin,
as a on-prem customer, our Octopus instance is also affected. Some of our developers report issue with similar symptoms. If We remove deployments where TaskId is null, then back to normal.
It would be great if 2022.2 will also be fixed as a hotfix.
Best regards
Piotr
Hey @Piotr,
Sorry to hear this has also affected your instance, did you try the workarounds in the GitHub issue as that’s the fix that has worked for most of our customers so far. Unfortunately this wont be backported to 2022.2 as the developers have suggested the workarounds do fix the issue and allow you to deploy.
We have options in the GitHub issue to delete the null releases via the API, change a project to just run tenanted, switch to a manual deployment phase lifecycle or delete the null releases via the DB (the API would be preferred though and does work).
If you could try those options in the workaround, the scripts are provided in there too, that would fix your issue until we have a 2022.3 version out for you to upgrade to.
Kind Regards,
Clare Martin
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.
