Notes and findings upgrading 3.3.27 to 2018.8.8

I’d like to share our notes and findings on upgradering Octopus from 3.3.27 to 2018.8.8.
We are running Octopus on Server2012r2 and SQL2014 (Server2012r2) in VMWare6.5 on SSD based FC-SAN. Our environment is forcibly behind http-proxy (no direct internet access).
Our setup and a mix of Listening and Polling agents, across 4 datacenters.
We have 32 environments, spanning 278 deployment targets in 15 roles. There is about 265 Projects and some 70+ users.
We heavily use step-templates, variable-libraries and script modules, and we have +6000 lines of custom powershell.

Overall status: Success

We followed the very simple procedure on: Redirecting to https://octopus.com/docs/administration/upgrading/guide

We ran the installer as a Domain Admin user, different from the Service-Account running the Octopus Application. Post install, the upgrader could not stop the Octopus Service.

Waiting for service to become Stopped. Current status: Running
Waiting for service to become Stopped. Current status: Running
Error: The service could not be stopped
Error: ===============================================================================
Error: Exceeded max retries
Error: System.Exception
Error: at Octopus.Shared.Startup.ConfigureServiceHelper.Retry(Func`1 func, Int32 maxRetries)
and following that, the Upgrader claimed the database-upgrade was being done a another process…

which is was as the Octo-server has started up again under its regular service-account.
Preparing to upgrade the database ‘Octopus’ on SQL Server at ‘’ to the expected schema running as ‘ourueraccount’…
Making sure it’s safe to upgrade the database schema…
The database upgrade is already running elsewhere. Please wait…
The database upgrade is already running elsewhere. Please wait…
The database upgrade is already running elsewhere. Please wait…

But by following the server logs, we could see that the upgrade was actually being performed.
Once the upgrade was finished and the Octopus Manager started. The Octopus Service was stopped, and we just started it manually.

Logging into the WebUI for the first time, the UI look very much like the old version in Chrome.
We got all kinds of CSRF errors, and we eventually realized that there was some caching issues and the new version looked very different in other browsers. Clearing the Chrome cache made the application work.

We went ahead and upgraded tentacles on one evironment, which was working on 9/10 nodes. The last one crashed and had to be started manually. (We have Puppet here, it might play a factor).
The remaining environments will be upgraded in the comming days. (Updates may follow).

We found a bad-practice index in changeset 91 on the IX_ServerTask_Common index. We are creating a actually support-topic on that one (Support case). The index ‘IX_ServerTask_Common’ has maximum length of 1000 bytes.

Last but not least, or dev-teams are not Octopus administrators. We have created a custom security-role for them, allowing them to deploy. With the new version, the did not yet have permissions to Tenant objects, and as such they could not release software. Granting them TenantView permissions solved that one.

So all-in-all, this was a very easy upgrade. Well done Octo-team!

Hi @kbrandenburg,

Glad to hear that the upgrade went well!

We’ve passed on you indexing report to the appropriate team who will investigate, thanks for sending that through. Let us know if there is anything else we can assist with,

Regards,
Alex

Small update: We’ve seen lots of Tentacle communication timeouts after the update.
Seems to be solved by upgrading the Tentacles to newest version.

The step failed: Activity on failed with error ‘Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it has already been disposed. Server exception: System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it has already been disposed. at Autofac.Core.Lifetime.LifetimeScope.CheckNotDisposed() at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable1 parameters) at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance) at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable1 parameters) at Autofac.ResolutionExtensions.ResolveNamed[TService](IComponentContext context, String serviceName, IEnumerable1 parameters) at Octopus.Shared.Communications.AutofacServiceFactory.CreateService(String serviceName) in Y:\Work\refs\tags\3.3.27\source\Octopus.Shared\Communications\AutofacServiceFactory.cs:line 18 at Halibut.ServiceModel.ServiceInvoker.Invoke(RequestMessage requestMessage) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\ServiceModel\ServiceInvoker.cs:line 21 at Halibut.Transport.Protocol.MessageExchangeProtocol.InvokeAndWrapAnyExceptions(RequestMessage request, Func`2 incomingRequestProcessor) in Z:\BuildAgent\work\e1bda71fea4b4831\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 153’.

Hi @kbrandenburg,

Thanks for the update. Can I assume that your Tentacles were still on 3.3.27 before you performed the upgrade?

Regards,
Alex

Hi Alex,
Yes, our tentacles where 3.3.27. I haven’t seen the issue after tentacles got upgraded.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.