We are getting a A child activity failed: SOAP security negotiation with<agent/packages url> for target failed. See inner exception for more details.
This last worked for this server 2/24/2017, it failed on 3/10/2017 and subsequent retries.
I found one discussion that indicated a time sync issue so I checked that and the tentacle was 45 seconds fast. I forced a resync of the time and they are now within a second of each other, but still no go.
I thought I saw a message before about the thumbprint but can’t see that message anywhere.
For this particular environment the four web servers have bad health checks but the other 7 servers are fine.
This is the health status:
2017-03-15 15:44:30 ERROR System.ServiceModel.EndpointNotFoundException: Unable to communicate with the remote tentacle ‘http://server01:10933/’ . This happens when either the tentacle is offline, or when firewalls are preventing communication to the service. Please verify that TCP port 10933 is open on both the Windows Firewall and any other hardware/software firewalls between the machines, and that the Tentacle Windows Service is running on the remote machine. You can test the connection by browsing to ‘http://server01:10933/’ in a web browser from the Octopus server. The error message given was: There was no endpoint listening at http://server01:10933/ that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.
at Octopus.Server.Proxies.ClientBroker1.CallOneWay(MachineEndpoint endpoint, Action1 callback) in c:\w\e6923628be6eaf72\source\Octopus.Server\Proxies\ClientBroker.cs:line 118
at Octopus.Server.Proxies.ClientBroker1.Call[TResult](MachineEndpoint endpoint, Func2 callback) in c:\w\e6923628be6eaf72\source\Octopus.Server\Proxies\ClientBroker.cs:line 30
at Octopus.Server.Tasks.Health.CheckTentacleHealthActivity.<>c__DisplayClass2.b__0() in c:\w\e6923628be6eaf72\source\Octopus.Server\Tasks\Health\CheckTentacleHealthActivity.cs:line 29
Obviously I tried browsing to that url; it was successful.
Here’s the server stack trace from the deployment.
Upload to machine http://server01:10933/
2017-03-11 02:05:47 DEBUG Uploading package Service.Web.Host 2.0.1041-Dev to tentacle http:/server01:10933/
2017-03-11 02:05:47 ERROR System.ServiceModel.Security.SecurityNegotiationException: SOAP security negotiation with ‘http://server01:10933/Packages/’ for target ‘http://server01:10933/Packages/’ failed. See inner exception for more details. —> System.ComponentModel.Win32Exception: One or more of the parameters passed to the function was invalid
at System.ServiceModel.Security.TlsSspiNegotiation.GetOutgoingBlob(Byte[] incomingBlob, ChannelBinding channelbinding, ExtendedProtectionPolicy protectionPolicy)
at System.ServiceModel.Security.SspiNegotiationTokenProvider.GetNextOutgoingMessageBody(Message incomingMessage, SspiNegotiationTokenProviderState sspiState)
at System.ServiceModel.Security.IssuanceTokenProviderBase1.GetNextOutgoingMessage(Message incomingMessage, T negotiationState) at System.ServiceModel.Security.IssuanceTokenProviderBase1.DoNegotiation(TimeSpan timeout)
— End of inner exception stack trace —
Server stack trace:
at System.ServiceModel.Security.IssuanceTokenProviderBase1.DoNegotiation(TimeSpan timeout) at System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan timeout) at System.ServiceModel.Security.TlsnegoTokenProvider.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Security.SymmetricSecurityProtocol.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Channels.SecurityChannelFactory1.ClientSecurityChannel1.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout) at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout) at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout) at System.ServiceModel.Security.SecuritySessionClientSettings1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ReliableChannelBinder1.ChannelSynchronizer.SyncWaiter.TryGetChannel() at System.ServiceModel.Channels.ReliableChannelBinder1.ChannelSynchronizer.SyncWaiter.TryWait(TChannel& channel)
at System.ServiceModel.Channels.ReliableChannelBinder1.ChannelSynchronizer.TryGetChannel(Boolean canGetChannel, Boolean canCauseFault, TimeSpan timeout, MaskingMode maskingMode, TChannel& channel) at System.ServiceModel.Channels.ClientReliableChannelBinder1.Request(Message message, TimeSpan timeout, MaskingMode maskingMode)
at System.ServiceModel.Channels.RequestReliableRequestor.OnRequest(Message request, TimeSpan timeout, Boolean last)
at System.ServiceModel.Channels.ReliableRequestor.Request(TimeSpan timeout)
at System.ServiceModel.Channels.ClientReliableSession.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ReliableRequestSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Octopus.Shared.Contracts.IPackageService.BeginUpload(PackageMetadata metadata)
at Octopus.Server.Tasks.Deploy.PackagePusher.<>c__DisplayClass1.b__0(IPackageService service) in c:\w\e6923628be6eaf72\source\Octopus.Server\Tasks\Deploy\PackagePusher.cs:line 33
at Octopus.Server.Proxies.ClientBroker1.CallOneWay(MachineEndpoint endpoint, Action1 callback) in c:\w\e6923628be6eaf72\source\Octopus.Server\Proxies\ClientBroker.cs:line 118
at Octopus.Server.Tasks.Deploy.UploadPackageActivity.<>c__DisplayClass1.b__0() in c:\w\e6923628be6eaf72\source\Octopus.Server\Tasks\Deploy\UploadPackageActivity.cs:line 26
at Octopus.Shared.Activities.Activity`1.StartNewThread.Execute() in c:\w\e6923628be6eaf72\source\Octopus.Shared\Activities\Activity.cs:line 43
