Packages are not removed from workers

Hi,
our deployments on each environments are in behalf of linux workers. Packages are stored on machine. We are configured retention policies, but packages are not removed from worker. They don’t respect release cleaning policies. Each time admin need to login to VM and remove it. It can be reproduced. Configure cloud region, configure deployment process and make some deployments.
Maybe this is miss configuration and only I need is your help :smiley:

Best regards
Piotr

Hi @Piotr

Could you share with us how your retention policies are set for your project? We might be able to guide you on what might be happening. Would you also be able to let us know what version of Octopus you are currently using?

Packages are cleared after every deployment if they are outside of the retention policy that is set for that project (i.e. you have 4 days of packages, but only keep 3 days. When you deploy that project again it will clear up anything outside of the 3 day policy).

The retention policies also only apply for each individual project. The policies do not have awareness of other projects packages on the tentacles, so if you haven’t deployed a project in a while those packages will remain until the next deployment.

In addition to the above, if any project is still on your dashboard, those packages related to these will be kept and so will the previous one for rollback purposes.

Looking forward to hearing from you!
Kind Regards
Sean

Hi @sean.stanway,

thank you for clarification.
Current version of Octopus is 2022.4 (Build 8394)
This is out retention polices configuration:


Packages are stored on built-in repository for 3 days (but this is not a problem).

We have multiple projects in single space, so maybe all is correct and only what I need to increase disk space. Can you provide any details where I can find logs about retention ? Is second options from my screen is applied also for linux workers ?

Best
Piotr

Hi @Piotr

You can read more on our retention policies here. It details how they are applied and when, but if you have any questions on it let me know.

When I originally messaged you my brain got accidentally engaged into “tentacle” mode. The second number on the retention policies that you set does not apply to any workers. The worker cleanup for the package cache is a separate function within our Calamari library (which runs workers/tentacles/etc.).

From my understanding of the code, it currently works on a desired free space percentage on the drive that your package cache is located on. It will take the total space taken up by the packages and multiply that by the desired percentage then divide that by 100.

If the total space on the disk that the cache is located on is below that calculated number from the above equation then a list will be made based on the oldest package to delete, essentially freeing up enough space until it hits your desired percentage. The packages to be removed is weighted on; how many times the package has been used, what version it is, and the size of the package.

By default, the desired percentage of free space is set to 20%. There is an system environment variable that is checked when the clean up is run to check the desired percentage of disk space, so you should be able to set this and set it to a high integer and this will it clean more aggressively.

The environment variable is called OctopusPackageRetentionPercentFreeDiskSpace and you can then set that to the desired number you would like to have free on the disk (there is no need to have a percentage sign after it). I’d recommend trying for around 40-50 and see how that goes. From what I’ve seen this does need to be set on the Octopus Server, but if that doesn’t seem to work then it may be a unique one to the workers itself. This isn’t typical for our environment variables, so the server is where I’d recommend trying this first.

There is definitely a lot to take in here. Let me know how the environment variable works and if this does indeed solve your issue with the worker clean up. If you have any more questions, please let me know.

Kind Regards
Sean

Hi @sean.stanway,

I try to apply your solution.
Is there any mention on OctopusPackageRetentionPercentFreeDiskSpace in documentation ?
Currently We are applied custom retention policy script, that set warning status on machine and send us a email from Octopus. Alert is send, when there is only 10% of free space, so it’s very odd to me that default value is 20%. Actually is this only cache for package on other machine. But, I will try to look further on our side.

For clarification, environment variable that you mentioned need to be set in project variable in Octopus or on linux worker ?

Best
Piotr

Hi @Piotr

There is no current documentation on using this environment variable. We usually have system environment variables for parts of the system that a customer may want to customise beyond the standard way we offer. I think we have something in the pipeline to help customise worker retention, but I’ll double-check.

The alerts are fully customisable and are not just for the package cache, but the full machine.

It is not a project variable, it needs to be set on the machine. Most system environment variables are based on the server, so I would add in onto the Octopus Server itself. I’ve linked an example Oracle document that just highlights what I mean.

Let me know how it goes and if you have any questions please let me know.
Kind Regards
Sean

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.