Dashboard performance

Hi, is there a way to disable automatic refreshing of the dashboard? Or to lower the refresh frequency or set the dasboard cache timeout manually?

We are hosting our build server on an Amazon T2.small instance, and it is important for us not to waste any CPU resources. Due to the Dashboard refreshing, the SQL Server uses typically 2-3 times more CPU time than our actual build server process (TeamCity),

Using SQL Server profiler, I can see that the duration of the Dashboard queries typically is between 400 - 8000 ms.

When running the dasboard query with io/time statistics I get the following result:

SELECT * FROM dbo.[Dashboard] ORDER BY Id

(972 row(s) affected)

Table ‘ServerTask’. Scan count 2, logical reads 78206, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table ‘Release’. Scan count 1, logical reads 940, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table ‘Workfile’. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table ‘Worktable’. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table ‘Deployment’. Scan count 3, logical reads 3399, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

SQL Server Execution Times:
CPU time = 531 ms, elapsed time = 1254 ms.
SQL Server parse and compile time:
CPU time = 0 ms, elapsed time = 0 ms.

Also, I can see from the browser network log that there is a dashboard XHR request running every 60 seconds even when I’m not viewing the dasboard page.

Our Octopus version is 3.2.13.

We don’t need the Dashboard to be refreshed automatically. I hope this is something you can look into.

Regards,
Sverre Hundeide

Hi Shu,
Thanks for getting in touch. With regards to the dashboard query you se coming up in your profiler this comes as a result of some of the polling done from the front end to gather details like the projects list for the main navigational bar drop down. If projects are added or renamed we want to make sure this data is kept up to date and to simplify the back end architecture we have used polling, with minimal caching (to avoid “dirty reads” in multi node environments which would may otherwise require the separation of the caching layer). The dashboard dataset is needed for some of this data so that we can keep the available projects synchronised with what the user would expect to see on the dashboard page.

Although there are a couple of optimizations that we could do around this that we hope to revisit at some point, we unfortunately have no immediate plans to prioritize them as this doesn’t seem to have been an issue for most of our other users. Do you have several different services sitting on the AWS instance that this is actually seriously impacting or has this largely come from the result of a server health check?

This probably isn’t the best response you were hoping for but hopefully you are still able to perform your deployments with minimal issue for the time being.
Cheers,
Robert

Hi, thanks for replying.

We are hosting our TeamCity service, Octopus Deploy services and Octopus Deploy database on an AWS T2.small instance. The impact of the problem is that all builds and deployment will be very slow because maximum 20% CPU can be used on the instance when there are no more CPU credits left.

The AWS T2 instances are normally a perfect choice for a build server because they give maximum performance when needed and CPU credits can be built when the server is idle (which it is most of the time). We currently have a three years prepaid plan for the server, so upgrading it is not an option at the moment.

We have about 40 projects, and several of them are configured as multi tenant projects with one environments per customers. With more than 900 build versions displayed on the dashboard, the page is mostly useless for us because it contains too much information and we usually navigate directly to the project we want to work with by using the toolbar menu.

We currently mitigate the problem by closing the Octopus Deploy browser tab when not using it. This is not an ideal solution because all developers frequently deploy to various environments. We are also considering configuring IIS as a reverse proxy server on the build server so that we can control the caching of the dashboard query.

Regards,
Sverre Hundeide

Hi Shu,
I understand your concern. When we address multi-tenancy in the next big release we will probably have to make some changes to the dashboard so I’ll make sure these concerns are heard.
Thanks again for your input.
Robert

Hi,

we experiencing same issue with 1,4k (Environment, Project) combinations and growing. I rewrote dbo.Dashbord view and added one index to support new view definition. The result is 10 times less time/cpu spent on frequent query “SELECT * FROM dbo.[Dashboard] ORDER BY Id”.
This workaround worked for us on version “Octopus 3.3.4” and maybe something to consider for “the next big release” in addition of fixing Dashboard refreshing feature.

Below is the change script (review it and use it at your own risk):

BEGIN TRAN

DROP VIEW [dbo].[Dashboard]
GO

CREATE VIEW [dbo].[Dashboard_orig]
–WITH SCHEMABINDING
AS
SELECT
d.Id as Id,
d.Created as Created,
d.ProjectId as ProjectId,
d.EnvironmentId as EnvironmentId,
d.ReleaseId as ReleaseId,
d.TaskId as TaskId,
CurrentOrPrevious,
t.[State] as [State],
t.HasPendingInterruptions as HasPendingInterruptions,
t.HasWarningsOrErrors as HasWarningsOrErrors,
t.ErrorMessage as ErrorMessage,
t.QueueTime as QueueTime,
t.CompletedTime as CompletedTime,
r.[Version] as [Version]
FROM (
SELECT
’C’ AS CurrentOrPrevious,
d.Id as Id,
d.Created as Created,
d.ProjectId as ProjectId,
d.EnvironmentId as EnvironmentId,
d.ReleaseId as ReleaseId,
d.TaskId as TaskId,
ROW_NUMBER() OVER (PARTITION BY d.EnvironmentId, d.ProjectId ORDER BY Created DESC) as [Rank]
FROM dbo.[Deployment] d
INNER JOIN
dbo.[ServerTask] t ON t.Id = d .TaskId
WHERE NOT ((t.State = ‘Canceled’ OR t.State = ‘Cancelling’) AND t.StartTime IS NULL)
UNION
SELECT
’P’ AS CurrentOrPrevious,
d.Id as Id,
d.Created as Created,
d.ProjectId as ProjectId,
d.EnvironmentId as EnvironmentId,
d.ReleaseId as ReleaseId,
d.TaskId as TaskId,
ROW_NUMBER() OVER (PARTITION BY d.EnvironmentId, d.ProjectId ORDER BY Created DESC) as [Rank]
FROM dbo.[Deployment] d
INNER JOIN dbo.[ServerTask] t on t.Id = d.TaskId
WHERE
t.State = ‘Success’ AND
d.Id NOT IN (
SELECT Id
FROM (
SELECT
d.Id as Id,
ROW_NUMBER() OVER (PARTITION BY d.EnvironmentId, d.ProjectId ORDER BY Created DESC) as [Rank]
FROM dbo.[Deployment] d
INNER JOIN
dbo.[ServerTask] t ON t.Id = d .TaskId
WHERE NOT ((t.State = ‘Canceled’ OR t.State = ‘Cancelling’) AND t.StartTime IS NULL)
) LatestDeployment
WHERE [Rank]=1)
) d
INNER JOIN dbo.[ServerTask] t on t.Id = d.TaskId
INNER JOIN dbo.[Release] r on r.Id = d.ReleaseId
WHERE ([Rank]=1 AND CurrentOrPrevious=‘P’) OR ([Rank]=1 AND CurrentOrPrevious=‘C’)

GO

CREATE VIEW [dbo].[Dashboard]
WITH SCHEMABINDING
AS

WITH ep
AS (
	SELECT DISTINCT
		d.EnvironmentId,
		d.ProjectId
	FROM dbo.Deployment AS d
),
CurrentDeployment
AS (
	SELECT
		'C' AS CurrentOrPrevious,
		ca.Id,
		ca.Created,
		ca.TaskId,
		ep.EnvironmentId,
		ep.ProjectId,
		ca.ReleaseId,
		ca.[State],
		ca.HasPendingInterruptions,
		ca.HasWarningsOrErrors,
		ca.ErrorMessage,
		ca.QueueTime,
		ca.CompletedTime
	FROM ep
	CROSS APPLY (
		SELECT TOP 1
			d.Id,
			d.Created,
			d.TaskId,
			d.ReleaseId,
			t.[State] AS [State],
			t.HasPendingInterruptions AS HasPendingInterruptions,
			t.HasWarningsOrErrors AS HasWarningsOrErrors,
			t.ErrorMessage AS ErrorMessage,
			t.QueueTime AS QueueTime,
			t.CompletedTime AS CompletedTime
		FROM dbo.Deployment AS d
		INNER JOIN dbo.[ServerTask] t
			ON t.Id = d.TaskId
		WHERE NOT ((t.State = 'Canceled'
		OR t.State = 'Cancelling')
		AND t.StartTime IS NULL)
		AND ep.EnvironmentId = d.EnvironmentId
		AND ep.ProjectId = d.ProjectId
		ORDER BY d.Created DESC
	) ca
),
PreviousDeployment
AS (
	SELECT
		'P' AS CurrentOrPrevious,
		d.Id,
		d.Created,
		d.TaskId,
		d.EnvironmentId,
		d.ProjectId,
		d.ReleaseId,
		d.[State],
		d.HasPendingInterruptions,
		d.HasWarningsOrErrors,
		d.ErrorMessage,
		d.QueueTime,
		d.CompletedTime
	FROM CurrentDeployment l
	CROSS APPLY (
		SELECT TOP 1
			dt.Id,
			dt.Created,
			dt.TaskId,
			dt.EnvironmentId,
			dt.ProjectId,
			dt.ReleaseId,
			t.[State] AS [State],
			t.HasPendingInterruptions AS HasPendingInterruptions,
			t.HasWarningsOrErrors AS HasWarningsOrErrors,
			t.ErrorMessage AS ErrorMessage,
			t.QueueTime AS QueueTime,
			t.CompletedTime AS CompletedTime
		FROM dbo.Deployment dt
		INNER JOIN dbo.[ServerTask] t
			ON t.Id = dt.TaskId
		WHERE dt.EnvironmentId = l.EnvironmentId
		AND dt.ProjectId = l.ProjectId
		AND dt.Created < l.Created
		AND t.State = 'Success'
		ORDER BY dt.Created DESC
	) d
)


SELECT
	d.Id,
	d.Created,
	d.ProjectId,
	d.EnvironmentId,
	d.ReleaseId,
	d.TaskId,
	d.CurrentOrPrevious,
	d.[State],
	d.HasPendingInterruptions,
	d.HasWarningsOrErrors,
	d.ErrorMessage,
	d.QueueTime,
	d.CompletedTime,
	r.[Version] as [Version]

FROM (
	SELECT
		Id,
		Created,
		ProjectId,
		EnvironmentId,
		ReleaseId,
		TaskId,
		CurrentOrPrevious,
		[State],
		HasPendingInterruptions,
		HasWarningsOrErrors,
		ErrorMessage,
		QueueTime,
		CompletedTime
	FROM CurrentDeployment
	UNION ALL
	SELECT
		Id,
		Created,
		ProjectId,
		EnvironmentId,
		ReleaseId,
		TaskId,
		CurrentOrPrevious,
		[State],
		HasPendingInterruptions,
		HasWarningsOrErrors,
		ErrorMessage,
		QueueTime,
		CompletedTime
	FROM PreviousDeployment
) d
INNER JOIN dbo.[Release] r ON r.Id = d.ReleaseId

GO

CREATE UNIQUE NONCLUSTERED INDEX [IX_Env_Pr_Cre_INCLUDE] ON [dbo].[Deployment]
(
[EnvironmentId] ASC,
[ProjectId] ASC,
[Created] DESC
)
INCLUDE ( [Id],
[TaskId],
[ReleaseId])
GO

COMMIT