First of all, thank you for making such a wonderful tool! We’ve been users/customers for a couple of years and Octopus has been a fantastic help with the deployment of our ASP.NET web application and other components that go along with it (installing prerequisites, scheduling Windows tasks, configuring Windows Services, etc.)! You guys are awesome!
We are experiencing a serious issue with one of our production environments. Could you please help us?
Our Octopus Server version is 3.12.4 . I’ve provided a few deployment logs (and can provide more; I’m trying to strike the right balance with the level of detail I’m giving). I’ve sanitized some names that would identify our organization or clients with replacement values.
In summary, the main issues we’ve observed are that:
- The deployment log will sometimes say in a specific Deploy a package step (named
Deploy Web Server Files) that it cannot locate the package (i.e.,
[CompanyName].[ProductName].WebApplication.1.13.9206-hotfix.nupkg). The error is:
Could not find package file: C:\Octopus\Files\[CompanyName].[ProductName].WebApplication.1.13.9206-hotfix.nupkg-a0b4986b-f17b-4b4a-b14b-5310ac6aa363. Sometimes during package acquisition it will say, however, that it found the package on the deployment target and doesn’t need to redownload it. We tried deleting it to force redownload and break something loose but it would still sometimes say it was found cached on the deployment target even though the file wasn’t physically on disk at that location. After deleting it and checking the
Re-download packages from the feed serversetting we could get it to force a download, but it would still error on the
Deploy Web Server filesstep saying it couldn’t find the package at that location even though it was there on disk.
- Even on deployments where the
Deploy Web Servers filesstep was successful, it did not log the config transformations, copying of files, etc. that it was supposed to, so we question whether it actually did those things properly. A following step to delete config transforms after their use (
Web Server - File System - Clean Configuration Transforms) wouldn’t find the files indicating that the package may not have been deployed properly, but the config transform files at least one time were actually present on disk. Our IIS web application would also end up giving us a
HTTP Error 503. The service is unavailable.error indicating that in the end it was not configured properly.
We just began having these issues on a specific deployment target/machine in production (in the deployment logs as
[HostPrefix]WEB03); no other test or production deployment targets have been experiencing these issues as far as we can tell. The environments we deploy to consist of 2 deployment targets (each target has 1 tentacle), and the other deployment target that’s part of the same environment has no problem. The deployment target having issues was seemingly working fine when we deployed a previous release to it several days earlier. I’ve included a deployment log for when it was working several days ago. It was a different environment but same the deployment targets. See the log:
1 - Several days before, successful (1.13.9204 , ServerTasks-30238).log.txt.
Now, onto the details of what happened when we experienced our issues…
We scheduled 6 deployments at a time per deployment target (there were additional deployments running at the same time on other deployment targets but they didn’t exhibit these issues).
One deployment among the first of them created a package delta from a previous release. It and other deployments then tried using this delta of the package. In the case of the deployment that created the package delta, it experienced the issue where
Deploy Web Server files didn’t log that it did much of anything but was still considered successful. We question whether this step actually ran properly despite the successful outcome. The
Web Server - File System - Clean Configuration Transforms step that followed couldn’t find the config transforms to delete that were supposed to have been deployed as part of the package (the step was marked as a warning). In the end there was a 503 error indicating something with our IIS app had a problem. See the log:
2 - Created delta, didn't deploy files (ServerTasks-30458).log.txt.
In the case of another deployment running at the same time that also tried to use this delta, it found the package during acquisition but errored on the
Deploy Web Server files step with
Could not find package file: C:\Octopus\Files\[CompanyName].[ProductName].WebApplication.1.13.9206-hotfix.nupkg-a0b4986b-f17b-4b4a-b14b-5310ac6aa363. See the log:
3 - Could not find package (ServerTasks-30457).log.txt.
We continued with additional deployments and the types of errors mentioned above occurred multiple times.
Continuing in chronological order, here are a few more deployments that had similar issues with some differences (I can provide logs but don’t want to overwhelm with too much info right now):
- Although the delta mentioned above was created, 5 minutes after that a deployment created a second delta. Subsequent deployments/re-deployments used either one or the other.
- This case only occurred once, and it may be a separate issue, but the
Deploy Web Server filesstep errored with not being able to find the package yet the step itself was flagged as a warning and not an error. Thus, it continued on and couldn’t find config transforms to delete.
- A deployment seemed to deploy the package successfully but the clean config transforms step–despite being successful without warnings–didn’t log what files were deleted (it usually does). I know that’s a Community Library script, but it’s another instance of questioning if a step actually ran properly despite the successful outcome.
- A deployment said it successfully ran
Deploy Web Server filesand extracted the files but it didn’t run config transforms. Then the clean config transforms step that followed couldn’t find the files.
- A deployment seems to have successfully ran config transforms during
Deploy Web Server filesbut then the clean config transforms step that followed couldn’t find those files.
- A deployment said it successfully ran
Deploy Web Server filesbut didn’t log that it did config transforms, etc. like it should have. While in other cases the clean config transforms step would’ve not found the files, this time it did.
By this time we had rebooted the deployment target machine, rebooted the Octopus Server, and were running one deployment at a time on the deployment target. We also deleted the package deltas on the deployment target and used
Re-download packages from the feed server setting. The package redownloaded but the
Deploy Web Server files step errored not being able to find the package.
We then reinstalled the tentacle on the troublesome deployment target and upgraded it/checked health in effort to give it a fresh start. However, we wonder if there’s some existing configuration state that it picked up that could have something to do with the issues we’re experiencing.
With the tentacle reinstalled, the next deployment found the package in the cache on acquisition, seemed to successfully run package deployment and config transform, but ultimately gave us a 503 at runtime. I would have liked to investigate this one more but we had to recover quickly given that it’s a production environment.
In a subsequent deployment it redownloaded the package (perhaps we tried the “re-download” option again) but then just kept resulting in the
Deploy Web Server files step with
Could not find package file: C:\Octopus\Files\[CompanyName].[ProductName].WebApplication.1.13.9206-hotfix.nupkg... We even tried the previous release that worked fine several days ago and it had the same issue with not being able to find the package. We have not been able to find a rhyme/reason/pattern for what’s going on yet, but something seems to be very amiss with deploying packages.
Since the issue seems specific to a certain production deployment target, we’ll try to get a clone of the machine so that we can further troubleshoot and try reproducing the issue. In the meantime, what are your thoughts and recommendations? I’m happy to provide additional information.
Thank you so very much for your time!!!