Octopus server hang after network hardware event

Hi Support,
Had an issue today where something happened to our azure vm hosting octopus causing the network connection to ‘go away’ (i think azure was updating the nic drivers for us). When the NIC went away octopus threw an exception, because clearly it couldnt do its job without one, but then just hung in that state until we came along to restart the service. I would be great if octopus could try to rebind after such an event but barring that i would rather it just exit so that windows can try to restart the service and get things back in to a good state when the network is happy again.
Here is the exception.

2019-07-25 09:45:19.0923 8636 619 ERROR Failed to receive incoming request.
System.Net.HttpListenerException (0x80004005): An operation was attempted on a nonexistent network connection
at System.Net.HttpListener.EndGetContext(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Octopus.Server.Web.OctopusNancyHost.d__24.MoveNext()

Hi Tim,

Thanks for getting in touch! We had a discussion with the developers about this and believe that we currently handle this how we would expect. Generally, when the NIC becomes unavailable, Octopus will sit and wait for the connection to be restored. However, in your case the connection was changed, which Octopus does not really have a way to handle, so the service sat there waiting for the previous NIC connection to be restored.

After discussing with the developers, I do not believe this is a behaviour we are going to change as any changes to how we currently handle this would make the primary case worse.

If I am missing something here, or you have further thoughts on this, please don’t hesitate to let me know.

Best regards,
Daniel

Thanks Daniel,
I think I understand your perspective and see where your coming from. Is there a way for you guys to detect network unavailable (someone unplugged the cable and it could come back) vs someone removed the whole interface (its not coming back). Not sure how much visibility you have as an app. If there is nothing that can be done I understand and please close this out. Kind of shitty when you have to try to handle someone removing hardware in the middle of running :slight_smile:
thanks
tim

Hi Tim,

Thanks for getting back here. I agree that it could be beneficial for us to be able to identify the difference in events here and react accordingly. It could potentially be possible to build something up to handle this, however, I don’t think our developers have any plans to support this kind of feature at this time.

I think the best place to take a feature like this could be UserVoice. We user UserVoice to gauge community support fore new features and ideas, you can vote and comment on suggestions or create your own. If a suggestion gathers enough votes, we can look into its implementation into Octopus.

https://octopusdeploy.uservoice.com/

Let me know if you have any further questions here.

Best regards,
Daniel

Thanks Daniel, Voice posted, have a great week!

tim

1 Like

Hi Tim,

Thanks, you too! Feel free to get in touch at any time. :slight_smile:

Best regards,
Daniel

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.