Tentacle uploads degrade over time

I’ve been struggling with Getting 2.0 tentacles to work consistently… It seems after some time the tentacle service becomes more and latent in transferring nuget packages to the tentacle. Currently running 2.0.10.1066.

We upload some large (1.5 GB) nuget packages to our tentacles. When things are running good, the upload takes about 12 minutes… After time passes however, the same upload can still be running 4 hours later. and really never finish.

It appears restarting the tentacle service resolves the problem. Could this be a memory leak or something else?

The images attached show the detail:

Healthy_Tentacle_Upload - This shows what my network utilization looks like shortly after a reboot of the tentacle.
Unheathly_tentacle - This shows what my network utilization looks like after the tentacle has been running for a day.
Tentacle_ServiceRestart - This shows that immediately after restarting the tenctacle service, the upload speed improves dramatically.

I also had the need to download large data sets as part of a deployment, but decided that the octopus package itself was not the best solution for delivering the data. Instead I use the PostDeploy.ps1 powershell script convention to download the large data set from a secure server. Another advantage to this approach is that the script could contain logic to only download updated data.

I created a scheduled task that restarts the tentacle service every hour… Works very well in fixing this problem, but is obviously not a long term solution.

Hi Gavin,

I’ve recorded an issue here to investigate this:

Questions:

  • Is your Tentacle configured in Listening mode, or in Polling mode?
  • What does the memory usage look like when the uploads are deteriorating?

Thanks,

Paul

Hi,

I have the same problem’S … any other solution or fix issue ?

thanks

Same issue with this version 2.1.3.1223

:frowning:

Did you fix soon ? I’ts a blocker to demonstrate this product for my team !

Thanks, we’ll try to reproduce and fix this as soon as we can.

Paul

Hello,

When a machine is uploading slowly, could anyone go to the machine details in Octopus web portal, and go to the Connectivity tab, and send me a screenshot?

Paul

Hi Paul,

This is a screenshot for my server.

Notes : Octopus Server = Windows 2012 Standard
Tentacle : Windows 2008 R2

Thanks

Hello,

This is proving hard to reproduce. Could you add me to Skype (paulstovell) and contact me when the problem occurs, so that we can do a screen sharing session (or send me an invite to any other screen sharing tool you prefer to use) to try and debug this?

Paul

Thanks, the cause for this has been found and will be fixed in the 2.3 release!

Paul

Hi Paul,

With the version 2.3, i have the same problem but different.

The first, second, third deployment is Great (no latency).

If I wait more than x minutes for the next deployment (more ten minutes?) the large package upload have the same problem (uploads degrade over time). :frowning:

Do you have a Workaround ? I try do restart service in the process of the release but it’s totaly impossible.

Sebastien

Hi Sebastien,

Is the Tentacle in your setup a listening, or polling one?

We have some suspicions that in fixing one, we may have broken the other - please let us know :slight_smile:

Regards,
Nick

Hi,

listening mode

Thanks

Thanks - we’ll look into it. Also, the fix in 2.3 requires a Tentacle upgrade, let me know if you haven’t moved the Tentacles forward.

Regards,
Nick

Hi Nick,

Hi, this is a new installation of 2.3.3.

Look the screenshot.

Thanks

Hi Nick,

Check the graph when the transfert is slow …

Thanks

Hi Sebastien,

When the upload is happening, can you go to the Environments tab, click the machine, and go to the Connectivity tab?

Every few seconds that page should refresh and tell you when it next expects to contact the remote machine. During an upload, it should always say “Connected. The connection is functioning normally. The next scheduled poll is due in about a second.”

Can you tell me if it ever says anything else? E.g., if it says: “The next scheduled poll is due in about 5 seconds”.

Paul

Hi Paul,

This the Connectivity Tab :

Connectivity

Secure TCP connection to https://10.10.8.66:10933/mx/v1

Connected. The connection is functioning normally. The next scheduled poll is due in 5 seconds.

Last connected a few seconds ago

It’s a good news ?

thanks

Sebastien

Thanks Sebastien, that’s consistent with a slow upload.

I can see from an earlier screenshot that your Tentacle is running 2.3.3. Just to confirm, the Octopus server is also running 2.3.3?

Paul