Cannot find package on deployment target; Deploy package step not running all commands

Hello!

First of all, thank you for making such a wonderful tool! We’ve been users/customers for a couple of years and Octopus has been a fantastic help with the deployment of our ASP.NET web application and other components that go along with it (installing prerequisites, scheduling Windows tasks, configuring Windows Services, etc.)! You guys are awesome!

We are experiencing a serious issue with one of our production environments. Could you please help us?

Our Octopus Server version is 3.12.4 . I’ve provided a few deployment logs (and can provide more; I’m trying to strike the right balance with the level of detail I’m giving). I’ve sanitized some names that would identify our organization or clients with replacement values.

In summary, the main issues we’ve observed are that:

  • The deployment log will sometimes say in a specific Deploy a package step (named Deploy Web Server Files) that it cannot locate the package (i.e., [CompanyName].[ProductName].WebApplication.1.13.9206-hotfix.nupkg). The error is: Could not find package file: C:\Octopus\Files\[CompanyName].[ProductName].WebApplication.1.13.9206-hotfix.nupkg-a0b4986b-f17b-4b4a-b14b-5310ac6aa363. Sometimes during package acquisition it will say, however, that it found the package on the deployment target and doesn’t need to redownload it. We tried deleting it to force redownload and break something loose but it would still sometimes say it was found cached on the deployment target even though the file wasn’t physically on disk at that location. After deleting it and checking the Re-download packages from the feed server setting we could get it to force a download, but it would still error on the Deploy Web Server files step saying it couldn’t find the package at that location even though it was there on disk.
  • Even on deployments where the Deploy Web Servers files step was successful, it did not log the config transformations, copying of files, etc. that it was supposed to, so we question whether it actually did those things properly. A following step to delete config transforms after their use (Web Server - File System - Clean Configuration Transforms) wouldn’t find the files indicating that the package may not have been deployed properly, but the config transform files at least one time were actually present on disk. Our IIS web application would also end up giving us a HTTP Error 503. The service is unavailable. error indicating that in the end it was not configured properly.

We just began having these issues on a specific deployment target/machine in production (in the deployment logs as [HostPrefix]WEB03); no other test or production deployment targets have been experiencing these issues as far as we can tell. :frowning: The environments we deploy to consist of 2 deployment targets (each target has 1 tentacle), and the other deployment target that’s part of the same environment has no problem. The deployment target having issues was seemingly working fine when we deployed a previous release to it several days earlier. I’ve included a deployment log for when it was working several days ago. It was a different environment but same the deployment targets. See the log: 1 - Several days before, successful (1.13.9204 , ServerTasks-30238).log.txt.

Now, onto the details of what happened when we experienced our issues… :slight_smile:

We scheduled 6 deployments at a time per deployment target (there were additional deployments running at the same time on other deployment targets but they didn’t exhibit these issues).

One deployment among the first of them created a package delta from a previous release. It and other deployments then tried using this delta of the package. In the case of the deployment that created the package delta, it experienced the issue where Deploy Web Server files didn’t log that it did much of anything but was still considered successful. We question whether this step actually ran properly despite the successful outcome. The Web Server - File System - Clean Configuration Transforms step that followed couldn’t find the config transforms to delete that were supposed to have been deployed as part of the package (the step was marked as a warning). In the end there was a 503 error indicating something with our IIS app had a problem. See the log: 2 - Created delta, didn't deploy files (ServerTasks-30458).log.txt.

In the case of another deployment running at the same time that also tried to use this delta, it found the package during acquisition but errored on the Deploy Web Server files step with Could not find package file: C:\Octopus\Files\[CompanyName].[ProductName].WebApplication.1.13.9206-hotfix.nupkg-a0b4986b-f17b-4b4a-b14b-5310ac6aa363. See the log: 3 - Could not find package (ServerTasks-30457).log.txt.

We continued with additional deployments and the types of errors mentioned above occurred multiple times.

Continuing in chronological order, here are a few more deployments that had similar issues with some differences (I can provide logs but don’t want to overwhelm with too much info right now):

  • Although the delta mentioned above was created, 5 minutes after that a deployment created a second delta. Subsequent deployments/re-deployments used either one or the other.
  • This case only occurred once, and it may be a separate issue, but the Deploy Web Server files step errored with not being able to find the package yet the step itself was flagged as a warning and not an error. Thus, it continued on and couldn’t find config transforms to delete.
  • A deployment seemed to deploy the package successfully but the clean config transforms step–despite being successful without warnings–didn’t log what files were deleted (it usually does). I know that’s a Community Library script, but it’s another instance of questioning if a step actually ran properly despite the successful outcome.
  • A deployment said it successfully ran Deploy Web Server files and extracted the files but it didn’t run config transforms. Then the clean config transforms step that followed couldn’t find the files.
  • A deployment seems to have successfully ran config transforms during Deploy Web Server files but then the clean config transforms step that followed couldn’t find those files.
  • A deployment said it successfully ran Deploy Web Server files but didn’t log that it did config transforms, etc. like it should have. While in other cases the clean config transforms step would’ve not found the files, this time it did.

By this time we had rebooted the deployment target machine, rebooted the Octopus Server, and were running one deployment at a time on the deployment target. We also deleted the package deltas on the deployment target and used Re-download packages from the feed server setting. The package redownloaded but the Deploy Web Server files step errored not being able to find the package.

We then reinstalled the tentacle on the troublesome deployment target and upgraded it/checked health in effort to give it a fresh start. However, we wonder if there’s some existing configuration state that it picked up that could have something to do with the issues we’re experiencing.

With the tentacle reinstalled, the next deployment found the package in the cache on acquisition, seemed to successfully run package deployment and config transform, but ultimately gave us a 503 at runtime. I would have liked to investigate this one more but we had to recover quickly given that it’s a production environment.

In a subsequent deployment it redownloaded the package (perhaps we tried the “re-download” option again) but then just kept resulting in the Deploy Web Server files step with Could not find package file: C:\Octopus\Files\[CompanyName].[ProductName].WebApplication.1.13.9206-hotfix.nupkg... We even tried the previous release that worked fine several days ago and it had the same issue with not being able to find the package. We have not been able to find a rhyme/reason/pattern for what’s going on yet, but something seems to be very amiss with deploying packages.

Since the issue seems specific to a certain production deployment target, we’ll try to get a clone of the machine so that we can further troubleshoot and try reproducing the issue. In the meantime, what are your thoughts and recommendations? I’m happy to provide additional information.

Thank you so very much for your time!!!

Kevin

1_-Several_days_before__successful(1.13.9204___ServerTasks-30238).log.txt (234 KB)

2_-Created_delta__didn’t_deploy_files(ServerTasks-30458).log.txt (241 KB)

3_-Could_not_find_package(ServerTasks-30457).log.txt (161 KB)

Hi Kevin,

I’m sorry to hear that you are having issues with one of the Tentacles.
Thank you so much for the amount of detail you have given us.

Just wondering, do you have antivirus installed on that machine, could it be quarantining some files ?
The other question is, do you have multiple Tentacles running and pointing to the same working directory ?

Cheers
John

Hi John!

Thank you so much for responding so promptly and for your willingness to help us!

While there is antivirus installed on the machine, we verified that it has not quarantined any files. Also, there is only one Tentacle running on the machine. :slight_smile:

As far as trying to make progress on our end, we have cloned the production server having these issues in hope of reproducing them. We’re trying to see if we can reconfigure the polling Tentacle to create a new deployment target in Octopus (trying to change as little as possible on the cloned machine). Any tips for doing this? If we can’t figure that out we will escalate to another Tentacle reinstall like we did previously. Update 1: We were able to turn off the Tentacle service on the production server and just reuse the same deployment target in Octopus and add a new test environment to it to deploy to. From the viewpoint of the Octopus Server the cloned machine is the same deployment target with the same thumbprint. No help needed on creating a new deployment target!

Update 2: So far we have not been able to reproduce any of the issues on the cloned machine nor with a newly-created test environment on the actual production machine. Not sure if something shook loose and everything is fine now nor how likely it is to return.

Thanks!

Kevin

Hi Kevin,

Thanks heaps for the update.
At this point I am not sure what is going on.
I guess we could try to figure out what is deleting the files with FileMon ?

Cheers
John

Hi John,

We’re not sure either. :slight_smile:

Thanks for the suggestion of using FileMon. However, we have not observed the NuGet package on the deployment target being deleted out from under us. Rather, even though the package was actually on disk, the Deploy Web Server files step claimed it couldn’t find the file at that location. That’s when we tried deleting the package manually ourselves in effort to “shake something loose” by forcing a redownload and thus detection that the package was present. It still resulted in the Could not find package file... error even though it was on disk.

We’re going to try deploying to production environments one at a time to see how it goes. Again, any thoughts/recommendations are appreciated.

Thanks!

Kevin

Hi Kevin,

If something else comes to mind we will update this thread.

Cheers
John

Hi John,

To follow-up, we’ve been successfully deploying to production environments on the server that was experiencing these issues. This includes multiple, scheduled deployments running at the same time. Therefore, in pretty much the same context as when the issues presented themselves, we still have not been able to reproduce them.

This is great! At the same time lack of explanation/understanding can be a little concerning. We’ll carry on for now hoping it’s unlikely to recur. :slight_smile:

Thanks for your help!

Kevin

Hi Kevin,

Thanks a lot for the update.
Yes a bit strange, fingers crossed it won’t come back. Just make sure you keep Octopus reasonably up to date, and hopefully it will be all smooth deployments from now on :wink:

Cheers
John