I’ve recently setup deployments to Azure websites using FTPS. This is using Octopus 2.0.6.950
I previously had the deployment setup on Octopus 1.6, but was using a website deployment to IIS, not FTP deployment.
What’s happening is it runs through the FTP deploy, and log file shows “Beginning synchronization…”.
Logs show it runs through sync’ing the files to FTP. Problem is it doesn’t always finish, but it never shows any errors.
The deployment does a few steps, starting with FTP deploy, then Windows Service deployment, then email alert.
Sometimes it finishes the FTP deploy, and the rest of the steps finish fine.
When the FTP deployment hangs, usually I can check via another FTP client and see that the files have actually been uploaded fine.
Are you able to send a Raw task log (link on the upper right of the task output page) for each of a) a deployment that failed to finish, and b) a deployment that succeeded?
Also, have you cancelled a failed FTP deployment, then had the cancellation time out? This can cause problems if the FTP connection continues running, i.e. we don’t hard-terminate the FTP process during cancellation. Rebooting the tentacle can fix this.
I tried to cancel the deployment, and it timed out. I’ve rebooted the tentacle and tried the deployment again. This time it worked fine, completing in 2 mins.
I’ve attached two log files. Both deployments are for the same release to the same environment (just hit ‘try again’).
I’ve been having a few issues where tasks are hanging, specifically the FTP deployment.
I initially had everything on one Azure VM – Octopus server (and TeamCity), as well as the tentacle which was used for FTP deployments and windows service deployment.
I tried reinstalling Octopus, but this time split across 3 Azure VMs. One for Octopus server (and TeamCity), one VM running a tentacle purely for the FTP deployments, and a third for my Windows service. This is the setup currently running which is also giving me hanging deployments.
I’m hoping the log files can shed some light on why this is happening, and I’m hoping it’s a simple config or setup issue
Thanks for sending the logs; I’ve deleted them from the discussion just to keep public info minimised. By the looks of these lines in the failing task:
It seems like the hang is definitely somewhere in the works of the FTP connection; the same file’s transferred in the successful version and further file transfers follow.
In addition to client and server software issues, various network issues can cause FTP hangs - troubleshooting this probably comes down to making repeated file transfers:
From a different file transfer client on the Tentacle machine (e.g. FileZilla) with the same settings, to the same (Azure) server
From the Octopus Tentacle to a different (non-Azure) server, again with similar settings
Doing this should help to narrow down the problem, if it is reproducible or other errors pop up.
On the Octopus side, there are two things I think we can do to improve handling:
I’m running into this issue with 2.0.11. I can try to cancel the deployment, and it will sit indefinitely. If I try to restart the service, it hangs on stopping. I have to restart my computer to try again.
I’m currently still on 2.0.9.1020 and have had this issue a few times again.
Often it works fine, but then the same app that worked will just hang.
I usually reboot the server running the tentacle used for the FTP deployments and re-deploy. Often this seems to resolve the issue, for a while anyway.
To provide a little more detail, the tentacle doing the FTP deployment is an extra small Azure VM, and the only thing this VM does is the FTP deployments.
Also, it deploys to Azure Websites via FTP. The Websites and the VM are both in West Europe zone, so I would presume connectivity between the tentacle and website would be pretty reliable, all being on the Azure backend…
When it hangs, the verbose output doesn’t show any new activity. It’s always during the file transfer process. It will show a list of files being processed and then seemingly just stops processing files. No errors or warnings either.
I’ve recently switched from msdeploy to a command line sync ftp utility. (No, this isn’t Octopus, but I started with Octopus until I ran into this issue). With msdeploy, I would get “Warning: Retrying the sync because a socket error (10054) occurred.” and then it would fail to retry properly and the DLL was corrupted. 99% of the time it was a DLL larger than 5mb.
With winscp, the error is “The specified network name is no longer available”. No amount of keepalive tweaking in either msdeploy or winscp helps – Azure simply seems to close the connection randomly. With large syncs, I’ve seen it bomb on a very small file, and I’ve seen large syncs that take 15 minutes go through without a hitch. I’ve had this issue on an always-on web site so it’s not the site shutting down. Even FileZilla gets “550 The specified network name is no longer available
” in a time as short as 8 seconds. It’s not you, it’s bloody Azure.
Old school ftp to azure isn’t easy, as the filestamps won’t match because ftp doesn’t support setting the timestamp and uploaded files simply get the current local time.
The only thing keeping me sane at present is that winscp somehow manages to not leave a corrupted file on the other end, possibly by doing a transfer and then a rename, not sure. But any ftp sync utility targeting azure is going to need to be very robust.
