MessageSecurityException

Legacy Uncategorised
1

Using 1.0.13.1206

I added a new tentacle and it is getting an error during health checks. I can browse to the tentacle url from the Octopus server over port 10933, so I know it is connecting. I’ve double checked the certificates and even tried regenerating the fingerprint on the tentacle side. FYI the tentacle server is in a fairly locked down networked environment. Could this be network related? I would think the fact that I can browse the wcf service using IE would mean that everything is talking correctly. The only logs on the server are messages in the Application event log mentioning that each service is listening. I have 6 other tentacles working, so I know it should just be a problem with this new one.

2012-08-07 15:02:18 ERROR System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. —> System.ServiceModel.FaultException: An error occurred when verifying security for the message.
— End of inner exception stack trace —

Server stack trace:`
at System.ServiceModel.Channels.SecurityChannelFactory1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionClientSettings1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ReliableChannelBinder1.ChannelSynchronizer.SyncWaiter.TryGetChannel()
at System.ServiceModel.Channels.ReliableChannelBinder1.ChannelSynchronizer.SyncWaiter.TryWait(TChannel& channel)
at System.ServiceModel.Channels.ReliableChannelBinder1.ChannelSynchronizer.TryGetChannel(Boolean canGetChannel, Boolean canCauseFault, TimeSpan timeout, MaskingMode maskingMode, TChannel& channel)
at System.ServiceModel.Channels.ClientReliableChannelBinder1.Request(Message message, TimeSpan timeout, MaskingMode maskingMode)
at System.ServiceModel.Channels.RequestReliableRequestor.OnRequest(Message request, TimeSpan timeout, Boolean last)
at System.ServiceModel.Channels.ReliableRequestor.Request(TimeSpan timeout)
at System.ServiceModel.Channels.ClientReliableSession.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ReliableRequestSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Octopus.Shared.Contracts.IHealthService.CheckHealth()
at Octopus.Server.Proxies.ClientBroker1.AssignableClosure1.Execute(TService service) in c:\BuildAgent\work\7bf5272a44079f5\source\Octopus.Server\Proxies\ClientBroker.cs:line 88
at Octopus.Server.Proxies.ClientBroker1.CallOneWay(MachineEndpoint endpoint, Action1 callback) in c:\BuildAgent\work\7bf5272a44079f5\source\Octopus.Server\Proxies\ClientBroker.cs:line 70
at Octopus.Server.Proxies.ClientBroker1.Call[TResult](MachineEndpoint endpoint, Func2 callback) in c:\BuildAgent\work\7bf5272a44079f5\source\Octopus.Server\Proxies\ClientBroker.cs:line 30
at Octopus.Server.Tasks.Health.CheckTentacleHealthActivity.b__0() in c:\BuildAgent\work\7bf5272a44079f5\source\Octopus.Server\Tasks\Health\CheckTentacleHealthActivity.cs:line 29`

2

Hi,

Sorry for the trouble you are having getting the connection to work.

On the Tentacle, what kind of user account is the service running under - the default Local System account or a custom user account?

Can you make sure that the user account has read/write access to the C:\Windows\Temp folder?

Paul

3

Local System. I expanded the write permissions to that folder to everyone and restarted tentacle. No luck.

4

Thanks for the update. Can you:

  1. Stop the Octopus Tentacle windows service
  2. Using an elevated admin command prompt, run Tentacle.exe manually

Hopefully this will give us more error information.

Paul

5
  • Tentacle

IHealthService listening on: http://localhost:10933/
IJobService listening on: http://localhost:10933/Jobs/
IPackageService listening on: http://localhost:10933/Packages/
Running. Press to shut down…

Then I did a health check, no output. I see that there is log4net, is there any further logging I can turn on?

To be sure that I really am talking to the right server, I stopped the service and did another check. This time the check gave a TimeoutException. That shows that octopus is talking to the correct server. Though it is strange that it didn’t give a more specific “connection refused” style error.

To further confuse things, I stopped one of the working tentacles and did a health check. It gives an EndpointNotFoundException instead of a TimeoutException.

Maybe this is network related. I don’t know enough about how this secure negotiation is happening. Is this still over soap, does it go binary or tls? If I know more, maybe I could take it to the network admins and possibly inspect the traffic.

6

Hi,

The communication happens using HTTP and SOAP over TCP port 10933 (by default) if that helps. It sounds like there is some communication happening, but that the messages aren’t being encrypted properly - a wireshark trace might help us to figure it out.

It’s common to get a mix of timeout exceptions and endpoint not found exceptions when contacting a remote machine which has shut down - it seems to depend on whether the host aborts the connection before the timeout is reached or not. The latest version of Octopus should at least print a message indicating that the remote host is probably offline.

Are you in a US timezone? Perhaps we can set up a Skype session tomorrow and I can try and debug it with you? My skype name is paulstovell.

Paul

7

I managed to get a capture of the traffic and emailed it to support@octopusdeploy.com. We can go from there.
Thanks

8

Thanks, I will check it out shortly and get back to you.

Paul

9

Think I just solved this one for one of our servers.

The time on the server hadn’t been updated for daylight savings, so the Octopus server was an hour ahead of the tentacle. Fixing the time on the tentacle seemed to solve it!

10

I just had this same issue, I’m running a VM with a tentacle on top of a host with a tentacle, both passed health checks initially & were working fine, however after rebooting the VM, the time zone changed to another region and broke the health check. After putting this back to the correct zone & rebooting, I found the clock out -3 hrs, after amending the clock, it is now working again.

11

Just got this error. Did a time-sync on both Octopus Server and Tentacle, and problem was gone. Timezones can be different, but the UTC time on both servers must be in sync (to some degree).