We use Octo with RedGate DLM Suite to deploy MSSQL database changes. Our tentacles are configured to use a service account. The account has permissions to the target database on each instance. It also has access to a network file share where packages are extracted and compare filters are stored. All of this broke when we upgraded to 3.5.1.
The file shares and the SQL Servers began reporting authentication attempts from the computer accounts instead of the service accounts.
From SQL Server
Login failed for user ‘NT AUTHORITY\SYSTEM’. Reason: Failed to open the explicitly specified database ‘%%%%%’. [CLIENT: <local machine>]
From Tentacle
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.NetworkStream.Read(Byte buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte buffer, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadLine()
at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine()
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd()
at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func’2 incomingRequestProcessor)
at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsServer(Func’2 incomingRequestProcessor, Func’2 pendingRequests)
at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client)
Granting permissions on the file share to the computer account(s) and granting permissions on the database to the local system account on each DB server serves as a work-around. We would rather use a service account for this. Please let me know if there is any additional information I can provide that will help.
Thanks for getting in touch. After the upgrade your Tentacles are now running as SYSTEM? Or are they still running as the correct service accounts but you are just seeing these connection attempts from SYSTEM when you try to run a Database deployment?
Also, did you upgrade your Tentacles to 3.5.1 through Octopus Server? If so could you attach the raw task log of the Octopus upgrade task, and also the task log for one of the deployments that is using the wrong accounts? You can clean up any private information in it, and/or set this discussion to private with the MAKE PRIVATE button on the right. Or send the logs through to support@octopus.com and they’ll get to me.
Thanks for the reply. The tentacles are running under the same service account from before the upgrade. Both the file share and SQL Server are showing that the connection attempts as SYSTEM. The tentacle is installed on the SQL Server host.
The upgrade was done via Octo Server. I’ll email the task logs. Note that I added the service account to the local admin group to facilitate the install and service restart. The service account was not a local admin previously nor is it one now.
Taking a look at your logs, the Tentacle upgrade threw an error, but the continued and claimed it succeeded, which is unusual. We sometimes see strange behavior on upgrades when Tentacle is running as a different user, but adding that user to local admin as you did should have been enough.
Then in the deployment logs you can see the Redgate DLM module logs out the current user:
13:52:55 Info | CurrentUser: NT AUTHORITY\SYSTEM
So that’s our problem. This should always just be the user the Tentacle service is running as, possibly something was corrupted as a result of that upgrade error. Would you be able to run the 3.5.1 Tentacle installer by hand on one of these machines, then confirm in Task Manager/Procmon that it comes back up as the right user and see if that clears things up? You can find the installer here: https://download.octopusdeploy.com/octopus/Octopus.Tentacle.3.5.1-x64.msi
If you don’t want to run a full release for testing you can make a test project with a single script step containing something like:
$name = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name
write-host "tentacle running scripts as $name"
to check the user. You could also add something to access your network file share.
I’m really sorry you’re seeing this problems with the upgrade, hopefully we can clear this up for you soon.
I uninstalled the Tentacle. The service remained, even after a restart. I removed the service and Octopus directory manually followed by another restart. I then installed the Tentacle using the installer you provided. It appeared successful. I changed the service to use the previous service account. Lastly, I re-ran the previously failed deployment. Unfortunately the same error occurred. Is there any additional information I can provide?
I’m really sorry you are still seeing this issue. Would you be able to go to the Octopus Script Console (instructions on the Script Console can be found here: http://docs.octopusdeploy.com/display/OD/Script+Console) and use it to run the following script on one of the Machines where you are seeing this issue and send me the raw task log? The link to the Raw log display is on the right side at the top of the screen when you are viewing the task log.
$name = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name
write-host "tentacle running scripts as $name"
Get-WmiObject Win32_Process -Filter "Name like '%powershell%'" | select-Object CommandLine | format-list
foreach ($var in $OctopusParameters.Keys){
Write-Host $var $OctopusParameters[$var]
}