We have upgraded our production Octopus Server to 2020.3.9 and afterwards it is experience some extreme slowness in the UI and API calls to it.
The scale of the server is :
- Approximately 8000 Polling tentacles (some 1500ish offline, the rest healthy(ish))
- Approximately 3000 Tenants for these tentacles
- Approximately 3000 Tenant Tag Sets (the original implementation created a tagset per tenant for simple association automatically during unattented installations of the tentacle)
The specifications for the VM server is :
- 12 CPU (usage remains consistently low)
- 24 GB RAM (usage remains consistently low)
There are occasional network spikes due to Octopus.Server.exe, most of which is connecting to the DB server. Our DBAs have investigated around the DB server and found no long queries or other issues they found to be concerning as far as return time for calls.
Some UI windows and API calls are more problematic than others, thus far the Infrastructure > Overview window is nearly unusable, as well as the Infrastructure > Deployment Targets. Currently I have the Overview page loading with a developer console up in chrome. I am seeing api/serverstatus/health calls taking up to almost 3 minutes now, the time for these calls was steadily increasing as the page was loading.
We have enabled metrics logging, and the highest I have seen the active requests is 7. All logging is only enabled at INFO level, I can get a change to Trace or Debug and provide those files if necessary. The OctopusServer log is riddled with various connection errors, which is a previous topic I had going for tentacles performing a denial of service attack on the server. This upgrade to 2020.3.9 was in part to help alleviate that potential concern, however slowness on the server now is remarkably similar to when tentacles would attempt to connect with incorrect server thumbprint.
Most of the connection errors listed are of ‘System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host’. There is the occasional ‘A client at  connected, and attempted a message exchange, but it presented a client certificate with the thumbprint ‘’ which is not in the list of thumbprints that we trust’, and I understand this error, it is also not very common.
I am working to do a faster turnaround to getting 2020.4.11 installed to see if that may alleviate some of the slowness we are experiencing. Please let me know what additional information I can provide and I will get that taken care of.