Hello @clare.martin,
If you think it will be useful for you and if you are available maybe we can schedule a Teams meeting(or other software through which we can do a call&screen sharing) so we can show you the exact behaviour and like this you can collect some valuable information regarding our issue.
If you agree feel free to e-mail me so we can setup the meeting details.
Thank you.
Best Regards,
Mihai
Hey @mihai,
Thank you for offering up a screen share, I have asked the engineers if a call would be beneficial. I have also asked what they would like to go through on the call as this is more backend code related so the engineers have a better idea than we (Support) do on what specifically we would need to look at.
They have not managed to take a look at your build test project yet so that might show us something, I have given them a nudge to see if they can see something from that test build project you sent over and if a call would be beneficial for them.
I will get back to you as soon as I hear from them, they are based in Australia so there is a bit of a lag between when I message them and when they reply as they have all gone off shift for the night now.
Hopefully we can get to the bottom of this soon for you!
Kind Regards,
Clare
Hey @mihai,
Our engineers took a look at the build project you sent over but it doesn’t show them much so they would love to hop on a call with you to see if they can troubleshoot this live. Our engineers do work in Australia but can work around customer time zones.
To that affect, what time zone are you based in and what dates / times are you free to hop on the call, we usually aim for around 30 minutes per customer call just to give you an idea of timings and ideally we like to use Zoom for the calls but can accommodate teams and WEBeX if that is your preferred communication method.
Let me know what time zone you are in and what times you have available this week or next and I will liaise with our engineers to see if we can book you in for the call.
Kind Regards,
Clare
Hey @clare.martin,
Thank you for your availability!
Now, to answer your questions, I’ll try to find a free spot on my calendar for next week and I’ll get back to you. I am based in Bucharest (EEST) and as far as I can see Australia and our working hours are not overlaping at all, so we will have to find a solution for this. Unfortunately, except Teams, we ca not use any other communication method. Do you think the engineers can install Teams on their side and create an account?
Thank you.
Best Regards,
Mihai
Hey @mihai,
Thank you for getting back to us, we can certainly install Teams that is not a problem at all, as for the time zones, we do have engineers on call and so they may be able to accommodate EEST, we will get something set up don’t worry.
If you can share your availability for next week I can take the times back to our engineers and we can see if one of the slots suits us.
The engineers are potentially looking at getting you a .net 6.0 tentacle to test but I have enquired about that as this seems like an Octopus Server issue to me (the engineers do know more about the backend code though so I will trust their recommendations on this one).
If anything comes of that in the meantime I will let you know, I look forward to getting some times from you and we can arrange the teams meeting.
Kind Regards,
Clare
Hey @clare.martin,
I will be available for the meeting on Wednesday 21, from 11AM to 4PM. Please let me know if and who will be available so I can create and send the meeting invite.
Thank you.
Best Regards,
Mihai
Hey @mihai,
Thank you for getting back to me with those times, that is 18:00 to 23:00 Australian time (the engineers time zone) and 09:00 to 14:00 UK time (my time zone) so I have asked the engineers if they can make Wednesday and if so what time would they like to hop on the call with you.
I can make Wednesday so can be available for the call but since this is more backend code issues ideally we would like one of the engineers to come on the call with us so they can run through some things with you on the call.
I will get back to you with if they are available on Wednesday and, if so, what time.
Kind Regards,
Clare
Hey @mihai,
The engineers have confirmed they would love to attend a Teams call with you for their 18:00 (which would be your 11:00). I will also attend with the engineer at my 9:00 (don’t you just love time zone changes!).
So to confirm, your 11:00am tomorrow (Wednesday 21st) would be great if you could set that up please.
I will DM you now with the email addresses of who will be attending so you can send us an invite by email, we look forward to trying to get to the bottom of this with you tomorrow.
Kind Regards,
Clare
Hey @clare.martin,
Yes, indeed :).
Plese let me know if you received it.
Thank you.
Best Regards,
Mihai
Hey @mihai,
I can confirm I have received the email invite and have confirmed attendance, I think Shanes would have auto declined it because its out of hours for him but I will forward him my email just to make sure he has the link for it and will let him know to accept the invite if he can.
They have confirmed he will be attending so it looks like its just declined it based on office hours.
I will re-confirm tomorrow around 08:30 my time (your 10:30) we have got everything and both of us are set and we can join the meeting and go through some things with you.
Kind Regards,
Clare
Hey @mihai,
Just to confirm we are all set for the meeting in around 15 mins time, Shane has accepted the invite so we look forward to hopefully getting somewhere with this issue.
Kind Regards,
Clare
Hey @mihai,
It was great meeting with you today, we just had a thought surrounding this that I wondered if you could try for us before we do a big deep dive into troubleshooting our end.
Since you upgraded your tentacle from Windows 2012R2 to 2019 we are wondering if there are some stale Calamari tools hanging around from a previous install.
Would you be able to try completely deleting everything out of the C:\Octopus\Tools folder (this includes all calamari folders in there) on the tentacle 2019 box for me and re-trying a deployment. When you re-try the deployment it will detect there is no calamari and will pull the latest version.
Let me know how you get on there, if that does not work to resolve the NotSpecified issue I will continue with my investigations.
Kind Regards,
Clare
Hey @clare.martin,
Nice to meet you too! 
I just did the test you’ve mentioned but the error is the same. Nothing has changed.
Thank you.
Best Regards,
Mihai
Hey @mihai,
I have been running tests through out the day and first I do want to aploigise as on the call we had this morning when I mentioned I could replicate this, the error I was seeing was not the error you were!
I was getting the first error below but we needed the bootstrap error in the picture below or the second error:
curl: no URL specified!
NotSpecified: curl: no URL specified!
So I do want to hold my hands up there and say sorry for giving you false hope on the call, those errors look very similar, I realised after we took another look after the call and tried the calamari deletion!
I have been testing this all day, here is what I have so far:
I have the same versions as you regarding the Server 2019 build numbers and .net versions, the only difference is the ProductID for the OS but that’s for support and licensing purposes and the UBR which apparently relates to updates on the OS.
Shane hopefully will do some more digging into this tonight but once more thing before I leave for the day just to rule out some old registry tentacle things.
Can you please perform a full manual uninstall of the tentacle and reinstall it using this guide here if you have not done so already and then re-test the deployment. This will remove all traces of the tentacle from that box so you have a full fresh install. We can then rule out any stale references from old tentacles.
I assume your OS is also fully up to date windows update wise - I just want to rule that out!
Let me know how you get on and sorry again for the mistake I made with thinking I managed to replicate this issue on the call!
Kind Regards,
Clare
Hey @clare.martin,
We had a talk and we decided first to try configuring a new machine and to keep the one that we currently use as it is. It will take some time but I will get back to you on this thread as soon as I have more updates.
We took into consideration what you said above but we will keep it as one as the last options that we have.
Thank you.
Best Regards,
Mihai
Hey @mihai,
Thank you for the update, I was actually just reading the engineers comments as we have now created an incident channel for this in our slack application and there have been a few ideas thrown out there last night but nothing solid at the moment that would be easy to try without messing around with the box too much.
Your machine has been the only one we have seen so far with this issue where installing .NET 4.8 has not fixed it so I was going to ask if there was any way you could possibly try an in place upgrade to Server 2022 or, as you have done, spin up another brand new box and try it on that just as proof of concept to see if that works.
There must be some underlying setting somewhere that is affecting the way PowerShell and Calamari are interacting with each other and this is a very odd fault so we are struggling to get to the bottom of it.
If you spin up a new Windows Server 2019 box as proof of concept and this does work it may be that you may need to do that for your other tentacle and then copy files across you need from both boxes, which I realise would be time consuming but we have nothing solid at the moment as to why this is happening.
If you do have licences or the capability to try Windows Server 2022 (an in place upgrade from the 2019 box) that might also be an avenue you could explore?
Let me know how the testing goes as we are keen to help you as much as possible to get this sorted.
Kind Regards,
Clare
Hey @clare.martin,
Here’s what I tried until now and the resolution:
Maybe this will help you investigating further. I’ll do the same on my side.
I’ll try these days to do a new test on a windows server 2022 machine.
Thank you.
Best Regards,
Mihai
Hey @mihai,
Thank you for the detailed update, that is very interesting! To my knowledge the tentacle and Calamari just leverages PowerShell it doesn’t update or install anything in the background other than what you remove when you manually run through the documentation for removal.
I will get this in front of the engineers, its good to note that installing 6.1.1271 was fine, it means there is nothing untoward with a new box or some program you have on your infra that is interfering with the tenacle, though that would have given us other red flags in the logs.
Let me know how the 2022 install goes, if you can install that and this works then it is a bonus as you would be future proofing your servers OS update wise for a lot longer but that does depend on your network policy and licensing.
Ideally we want to get this working on 2019 for you and not just suggest an upgrade for the sake of it so I will get back to you with the engineers thoughts on what you have tested.
Kind Regards,
Clare
Hey @clare.martin,
More info incomming :).
I did more tests yesterday evening and today on our test server:
(as you know last tests I told you about yesterday ended with same thing, our issue, “NotSpecified” error) → NOT OK
I’m not that sure, for the moment, which of the above actions fixed the issue that we encountered…
I’ll move now to the production tentacle and see if I can fix it somehow without uninstalling and reinstalling the tentacle.
I’ll get back to you with more info soon.
Thanl you.
Best Regards,
Mihai
Hey @mihai,
This is wonderful news to see on a Friday!
Thank you so much for running all of those tests and detailing exactly what you have tested, the engineers did not get back to me last night regarding what I updated them with (your test findings yesterday) but I did ask them if we could check the manual uninstall documentation and see if it is all still correct for the new Tentacles.
I did think along the same lines as you as to if the new tentacles now install new registry files we have not accounted for in the documentation because they now use .NET 4.8 and not 4.6.
Its odd its installed updates to .NET after you removed those registry files, unless you accidentally removed some it needed and its prompted the server to install some updates to put them back in again.
Id be interested to know how this goes on production and if you can somehow force .NET updates on those servers and it fixes this issue, one of the engineers did mention trying to upgrade PowerShell but I didnt think that would make a difference.
I then mentioned trying to get you to re-install .NET but for the life of me I dont know where Windows actually installs it once it is installed as I cant find it in program files or features. It does live in the .NET folders of Windows but there does not seem to be a GUI option to see what versions you have like it used to pre-Windows 2016!
Let me know how the testing on Production goes and I will see if the engineers have any thoughts based on your new information.
Kind Regards,
Clare
