@Dominik, maybe going through the steps would help make this clear.
When you deploy a package, it transfers the nuget package over to the target machine. Once the nuget package is there, it unzips the files to the directory you specify, and then performs any other necessary actions based on what you want it to do (register a windows service to point to an executable, or setup an IIS site pointing to the directory, etc.)
The retention policy you set only determines when the nuget packages will be deleted. We prefer to keep a few nuget packages from old releases on our CI machines. This helps so that if we have to roll back to an older release, the packages already exist, and all it has to do is unzip them instead of having to transfer the packages over the network.
Retention policies won’t delete the files and such that are in use. I use Custom Installation Directories (supported by Octopus) so that IIS sites and windows services aren’t running from the Octopus directory, but something much more apparent, like C:\Test\Service.
That being said, you never have to worry about having multiple versions of a site or service. If you install version 1.0 of a windows service to C:\Test\Service and 1.0 of a site to C:\Test\Site, it will handle setting the service and IIS configurations. When you decide to install 1.1 the next day, it will transfer the nuget packages, unzip them, turn off the Site and Windows service, place the unzipped files in C:\Test\Service and C:\Test\Site respectively, and restart the service and site. That way, any links you have between them, or any users on the machine, won’t have to change a thing. They might not even know it’s been updated.
There are ways to have multiple deployments exist on the target machine at the same time, but you have to go out of your way to set something like that up. (Can be helpful, but is often the exception.)
Your concerns of having existing packages fill up the machine are handled by the retention policy you set, and your concern of multiple versions existing simultaneously are handled the same way .MSI packages are, or any program that updates.
If you have any other questions, I’d be happy to help.