Azure - FTP - Unable to redeploy

Legacy Uncategorised
1

My first deployment to Azure Web Site via passive FTP(S) went through perfectly. However, since then I’ve been unable to redeploy. Every time I try, I get 78 of 113 files failed. It’s just too oddly specific that the same number of files fail on sync every time.

What I’ve tried: Reboot. Re-download nuget packages. Delete unrecognised files. Redeploy the release that worked before. Stop the web site in the management portal. Failed deployment retry thing. Try again at home.

What I haven’t tried: Fiddling with anything, as it worked before :slight_smile:

I think I’m mostly after something more specific than a sync failed. Is there a way to dig deeper into the failed ftp sync? Thanks for your help.

2

Additional troubleshooting steps and information:
Tried Active FTP
Able to deploy to Azure both using Web Deploy and FTP. FTP is very slow, it wasn’t this slow yesterday.
Event Viewer says Control Channel Closed in one of the error events.

3

Additional information:
I swapped the order of the two web sites I’m trying to deploy, and the one moved to the top deployed very quickly. However, when I try to redeploy, it now fails on 4 out of 4 files.

4

I’ve attached a raw log. I’ll be darned if I see any actual errors in there.

octopus_log.txt (12 KB)

5

Switching from NuPack to OctoPack worked for my small WCF project. For my larger MVC project 560 of 610 files fail.

6

Hi Jeff,

We’ve heard of network issues hitting FTP deployments to Azure in the past (possibly on the Azure side) - can you please try making the same transfers (from the same source machine) using an FTP client like FileZilla? It may succeed, but may also produce some log output pointing to any issues or files that it needed to retry.

Thanks for all the detailed info.

Nick

7

Azure is fine. FileZilla uploads changed files and ignores existing
files and completes without error. Incidentally, FZ is substantially
faster than Octopus.

I think the log snip below is where the trouble is. It gets to a point
where the connection closes, and then only one file is retried. I think
the switch from NuPack to OctoPack corrected the main issue, and a
secondary issue is that an Azure deploy can only get so far before the
connection has to be re-opened. Also, I think the error message may be
wrong, and the real result is that 234 of 517 succeeded, as in
successive deployments the count went up, until now it consistently
seems to fail at 412 of 517. The timeout seems to occur at precisely 5.5
minutes into the sync.

—> STOR /site/wwwroot/Scripts/i18n/angular-locale_nl-be.js

13:09:22 Verbose | —> REST 0

13:09:22 Verbose | —> PASV

13:09:22 Verbose | Connection closed

13:09:27 Verbose | Connecting…

13:09:28 Verbose | —> AUTH TLS

13:09:28 Verbose | —> PBSZ 0

13:09:28 Verbose | —> PROT P

13:09:28 Verbose | —> USER eve-epm-beta$eve-epm-beta

13:09:28 Verbose | —> PASS ********

13:09:29 Verbose | Connected

13:09:29 Verbose | —> TYPE I

13:09:29 Verbose | —> PWD

13:09:29 Verbose | —> CWD /

13:09:30 Verbose | —> PWD

13:09:30 Verbose | —> SIZE /site/wwwroot/Scripts/i18n/angular-locale_nl-cw.js

13:09:30 Verbose | —> PASV

13:09:30 Verbose | —> STOR /site/wwwroot/Scripts/i18n/angular-locale_nl-cw.js

13:09:31 Verbose | Synchronize complete

13:09:31 Verbose | Total operations: 517

13:09:31 Fatal | 234 out of 517 transfers failed

8

Finally found a workaround. FTPS has the timeout issue as I described.
FTP works. To summarise: I think your error message is incorrect, and I
think there is an issue where reconnecting doesn’t work properly.

9

Thanks for the detailed info Jeff, I’ve made a note here to investigate next release:

Paul

10

Unfortunately I have to conclude that the current release of Octopus Deploy is not robust enough to deal with what may be going on with Azure web app ftp deployments. While I can see that upload speed at times can slow (from watching how FileZilla handles itself), it’s not okay that publish software falls over when using a transport where one should expect errors to occur. Here’s my latest log, which has been stuck for over 30 minutes – again; the problem appears to be a slow upload of a 5mb DLL. That 5mb can still cause trouble sucks. I’ll have to wait until Web Deploy to Azure web apps is properly supported. For now I’ll have to stick with TeamCity and Visual Studio Publish targets for my deployments.

—> STOR /site/wwwroot/bin/EntityFramework.dll
21:01:06 Verbose | —> PWD
21:01:25 Verbose | Connection closed
21:01:25 Verbose | Connecting…
21:01:26 Verbose | —> USER eve-epm-services-ci$eve-epm-services-ci
21:01:26 Verbose | —> PASS ********
21:01:27 Verbose | Connected
21:01:27 Verbose | —> TYPE I
21:01:27 Verbose | —> PWD
21:01:27 Verbose | —> CWD /
21:01:27 Verbose | —> PWD
21:01:27 Verbose | —> SIZE /site/wwwroot/bin/EntityFramework.dll
21:01:27 Verbose | —> PASV
21:01:28 Verbose | —> STOR /site/wwwroot/bin/EntityFramework.dll

11

Hello,

This discussion is interesting to me as I’ve also been attempting to deploy to an Azure website, 2 virtual applications via FTPS. I have managed to get the deploy to work, but it takes 20 minutes to copy up, ~40MB of files. The FTP synchronisation seems to be very slow.

Is there a setting that can be tweaked possibly to improve the FTP transfer speeds?

Thanks very much,

Martyn.

12

I should have mentioned that Visual Studio was also unable to deploy via FTP, and it timed out (but didn’t hang) on the same file. And then I switched back to Web Deploy and it went right through. I may take a crack at using NuPack + command line Web Deploy to get this working for now, but it’s disappointing to have to nearly start over as I had just sorted out a nuspec that would integrate properly with NuPack + Octopus.

13

Just taking another angle - is it possible that earlier FTP deployments were cancelled, and that “cancellation timed out”? It might be possible that an earlier deployment attempt is still running, however “control channel closed” and matching behaviour from Visual Studio does probably indicate a failure on the Azure side. Will keep an eye out for any more clues.

Regards,
Nick

14

No, I was pretty careful to be on a stable baseline, and I went from a
successful deployment directly to a hung deployment without making any
other changes. I believed the issue to be sorted, returned to work, and
later tried to deploy to CI. When it hung again, I then did the
compare/contrast to Azure FTP and Web deployments as described.

15

OK, thanks for the info.

16

An update - the Azure team have acknowledged a problem and will address it in a future release:

http://social.msdn.microsoft.com/Forums/windowsazure/en-US/96e27c8b-54cf-4629-9e1c-fe1acde94c5f/timeouts-and-connection-problems-when-using-ftps-upload-to-azure?forum=windowsazurewebsitespreview

Paul

17

Cool, but the problem could be mitigated by doing a sync rather than a
full deploy. I’m currently using msdeploy in TeamCity and it has never
failed.

18

Hi Paul,

I’ve just installed v2 this weekend and imported our 1.6 config. I ran straight into this problem and it locked up consistently 3 times in a row attempting a deployment via SFTP - which is interesting given I’ve only very occasionally had a SFTP session fail under 1.6.

What was worse was that it also locked up the Tentacle completely and as per this discussion - http://help.octopusdeploy.com/discussions/problems/15530-everything-got-queued

The only way I could get things going again was by deleting the contents of the Tentacle/Actors directory manually. The database repair process did not help (because the Tentacle was not on the same machine as the Octopus Manager?)

I switched to vanilla FTP and all worked well again.

Let me know if there is anything useful I can pass on.

19

Hi Phil,

Thanks for the info. I think we might be using a newer version of our FTP library, I wonder if it introduced a bug? I’ll investigate.

The issue of failed deployments causing problems or hanging is now fixed - clicking Cancel in Octopus 2.3 (today’s release) will now forcibly cancel.

Paul