Failed upgrade from 1.6 - System.Net.WebException: The request was aborted: The request was canceled

Hi there,

I’m trying to upgrade an existing production Octopus server by installing version 2.3.6 on a new server and then using the upgrade wizard on a backup (from version 1.6)

I’m getting the error below, any help would be greatly appreciated!

Cheers,
Rob


Importing attachment attachments/tasks-147745/output-log
Done with reading attachments, total: 93634
Begin reading transformers
Done with reading transformers, total: 0
Imported 99,603 documents and 93,634 attachments in 21,776,601 ms
Finished writing all results to server

A fatal exception occurred
System.AggregateException: One or more errors occurred. —> System.Net.WebException: The request was aborted: The request was canceled.

Server stack trace:
at System.Net.HttpWebRequest.BeginGetResponse(AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory1.FromAsyncImpl(Func3 beginMethod, Func2 endFunction, Action1 endAction, Object state, TaskCreationOptions creationOptions)
at AsyncCompatLibExtensions.<>c__DisplayClass30.b__2f()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()

Exception rethrown at [0]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Raven.Client.Connection.HttpJsonRequest.d__4b.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line 933

Exception rethrown at [1]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Raven.Client.Document.RemoteBulkInsertOperation.d__18.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\RemoteBulkInsertOperation.cs:line 289

Exception rethrown at [2]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Raven.Smuggler.SmugglerApi.d__6.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Smuggler\SmugglerApi.cs:line 96
— End of inner exception stack trace —
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Octopus.Server.Legacy.LegacyOctopusImporter.ImportTheRavenBackup(String ravenDbBackup) in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\Legacy\LegacyOctopusImporter.cs:line 1067
at Octopus.Server.Legacy.LegacyOctopusImporter.ImportFromBackup(String ravenDbBackupFilePath, Nullable1 tentaclePort, String upnSuffix) in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\Legacy\LegacyOctopusImporter.cs:line 89 at Octopus.Server.Commands.MigrateCommand.Start() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\Commands\MigrateCommand.cs:line 116 at Octopus.Shared.Startup.ConsoleHost.Run(Action1 start, Action shutdown) in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Shared\Startup\ConsoleHost.cs:line 36
—> (Inner Exception #0) System.Net.WebException: The request was aborted: The request was canceled.

Server stack trace:
at System.Net.HttpWebRequest.BeginGetResponse(AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory1.FromAsyncImpl(Func3 beginMethod, Func2 endFunction, Action1 endAction, Object state, TaskCreationOptions creationOptions)
at AsyncCompatLibExtensions.<>c__DisplayClass30.b__2f()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()

Exception rethrown at [0]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Raven.Client.Connection.HttpJsonRequest.d__4b.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Connection\HttpJsonRequest.cs:line 933

Exception rethrown at [1]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Raven.Client.Document.RemoteBulkInsertOperation.d__18.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\RemoteBulkInsertOperation.cs:line 289

Exception rethrown at [2]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Raven.Smuggler.SmugglerApi.d__6.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Smuggler\SmugglerApi.cs:line 96<—


Error: The previous command returned a non-zero exit code of: 100
Error: The command that failed was: “C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe” migrate --instance=“OctopusServer” --no-prompt --backup=“E:\BackupFromPulseBuild\20140418-222717.octobak” --tentacle-port=“10934”

Hi Robert,

We’ve seen timeouts like this under some circumstances that are tricky to track down; it would seem in this case that the import has taken an extremely long time.

Is it possible to temporarily install the 2.3.6 server on a fast machine (my 16 GB Macbook Pro with SSD eats them for breakfast) and once the backup is imported, create a 2.x backup that you can restore on the “real” production server?

Hope this kicks off some ideas, let me know what you think.

Cheers,
Nick

if it is of any help I had the same issue with my 1.6 instance and once I bumped cpu and ram up on the instance I was importing on the import worked good.

In addition I changed the priority of the import process to the highest setting Realtime… I did this using Sysinternals process explorer.

Thanks for the tips! It’s a VM, so I’ll speak to my ops team and see if they can up the speed and see if this helps.

It’s a fairly long running Octopus server which is heavily used for continuous integration, so it has lots of history. I noticed that it is importing a lot (90k) tasks. When I looked directly in raven I noticed these all seem to be closed off jobs, mainly checking tenticle health. This seems to take the majority of time to complete. Is there any way to remove old records as I can’t imaging these are really needed.

Cheers,
Rob

We’ve upgraded the VM to add a lot more processing power and results are noticable in the output window. However, unfortunately I’m still getting an error.

The error message seems to have changed. I’ve copied the final part of the log below.

Has anyone seen this before, any other suggestions would be welcome!

Also does anyone know if it is possible to clear down some of the historic data as i mentioned in my previous comment?

Cheers,
Rob

Importing attachment attachments/tasks-49549/output-log
Importing attachment attachments/tasks-49550/output-log
Importing attachment attachments/tasks-49551/output-log
Finished writing all results to server

A fatal exception occurred
System.AggregateException: One or more errors occurred. —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted 127.0.0.1:10936
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
— End of inner exception stack trace —

Server stack trace:
at System.Net.HttpWebRequest.GetResponse()
at Raven.Abstractions.Connection.HttpRavenRequest.SendRequestToServer(Action`1 action) in c:\Builds\RavenDB-Stable\Raven.Abstractions\Connection\HttpRavenRequest.cs:line 190
at Raven.Smuggler.SmugglerApi.PutAttachment(AttachmentExportInfo attachmentExportInfo) in c:\Builds\RavenDB-Stable\Raven.Smuggler\SmugglerApi.cs:line 305
at Raven.Abstractions.Smuggler.SmugglerApiBase.d__41.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Abstractions\Smuggler\SmugglerApiBase.cs:line 499

Exception rethrown at [0]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Raven.Abstractions.Smuggler.SmugglerApiBase.d__2f.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Abstractions\Smuggler\SmugglerApiBase.cs:line 424

Exception rethrown at [1]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Raven.Smuggler.SmugglerApi.d__6.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Smuggler\SmugglerApi.cs:line 87
— End of inner exception stack trace —
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Octopus.Server.Legacy.LegacyOctopusImporter.ImportTheRavenBackup(String ravenDbBackup) in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\Legacy\LegacyOctopusImporter.cs:line 1067
at Octopus.Server.Legacy.LegacyOctopusImporter.ImportFromBackup(String ravenDbBackupFilePath, Nullable1 tentaclePort, String upnSuffix) in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\Legacy\LegacyOctopusImporter.cs:line 89 at Octopus.Server.Commands.MigrateCommand.Start() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\Commands\MigrateCommand.cs:line 116 at Octopus.Shared.Startup.ConsoleHost.Run(Action1 start, Action shutdown) in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Shared\Startup\ConsoleHost.cs:line 36
—> (Inner Exception #0) System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted 127.0.0.1:10936
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
— End of inner exception stack trace —

Server stack trace:
at System.Net.HttpWebRequest.GetResponse()
at Raven.Abstractions.Connection.HttpRavenRequest.SendRequestToServer(Action`1 action) in c:\Builds\RavenDB-Stable\Raven.Abstractions\Connection\HttpRavenRequest.cs:line 190
at Raven.Smuggler.SmugglerApi.PutAttachment(AttachmentExportInfo attachmentExportInfo) in c:\Builds\RavenDB-Stable\Raven.Smuggler\SmugglerApi.cs:line 305
at Raven.Abstractions.Smuggler.SmugglerApiBase.d__41.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Abstractions\Smuggler\SmugglerApiBase.cs:line 499

Exception rethrown at [0]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Raven.Abstractions.Smuggler.SmugglerApiBase.d__2f.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Abstractions\Smuggler\SmugglerApiBase.cs:line 424

Exception rethrown at [1]:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Raven.Smuggler.SmugglerApi.d__6.MoveNext() in c:\Builds\RavenDB-Stable\Raven.Smuggler\SmugglerApi.cs:line 87<—


Error: The previous command returned a non-zero exit code of: 100
Error: The command that failed was: “C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe” migrate --instance=“OctopusServer” --no-prompt --backup=“E:\BackupFromPulseBuild\20140418-222717.octobak” --tentacle-port=“10934”

Hi Robert,

Sorry this is so much trouble. It looks like during the import we tried to use a port that was already in use. Can you try running the import again to see if you get the same error?

If you like, you could send a backup to us (we can give you an FTP repository) and we can convert it for you and send you a 2.0 backup ready to use. I totally understand if this isn’t possible of course.

Paul

Hi Paul,

Thanks for getting back to me! I ran the import several times, each with the same error, but each time at a slightly different attachment number.

Your offer to send you a backup would be greatly appreciated, can you email me the FTP details?

Thanks again for your help,
Rob

Thanks Robert, I’ve emailed you some FTP details.

Paul