Octopus v2.0.8.907 not authorising anyone using "Domain" authorisation mode

Hello,

We are struggling to make Octopus authenticate our users via Active Directory (“Domain”), but without any success. Here’s the message from Windows audit log (please note that the box where Octopus sits is in DOMAIN1 whereas our users are in DOMAIN2, and there are trust relationships between the two):

@@@
An account failed to log on.

Subject:
Security ID: SYSTEM
Account Name: ASPDEV01$
Account Domain: DOMAIN1
Logon ID: 0x3e7

Logon Type: 3

Account For Which Logon Failed:
Security ID: NULL SID
Account Name: DOMAIN2\user
Account Domain:

Failure Information:
Failure Reason: Unknown user name or bad password.
Status: 0xc000006d
Sub Status: 0xc0000064

Process Information:
Caller Process ID: 0x10e0
Caller Process Name: D:\Octopus2\Server\Octopus.Server.exe

Network Information:
Workstation Name: ASPDEV01
Source Network Address: -
Source Port: -

Detailed Authentication Information:
Logon Process: Advapi
Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Transited Services: -
Package Name (NTLM only): -
Key Length: 0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
@@@

Octopus v1.6 on the very same machine in Windows authentication mode authorises our users absolutely fine. I understand that v1.6 runs under IIS whereas 2.0 uses some kind of embedded HTTP server and this is probably the cause of the problem. Maybe there’s a setting somewhere which will force Octopus to use DOMAIN2 instead of DOMAIN1 when authenticating?
Two instances run on the same box under Windows 2008R2 SP1 with that patch which is needed to install v2 applied.

Any ideas how to solve this?

Thank you very much
Igor

Thank you for the detailed information. I suspect but can’t confirm that the different domains for the machine and users will be the problem, as you suggest.

I’ve created an issue here to track it: https://github.com/OctopusDeploy/Issues/issues/462

Regards,
Nick

Hi Nick,

I tried logging with an account from a domain Octopus’ box belongs to, and this is what I got in Audit Success entry in Windows Security log:

New Logon:
Security ID: DOMAIN1\user1
Account Name: user1
Account Domain: DOMAIN1

However, when I try logging with a user from another domain, I’ve got this Audit Failure:

Account For Which Logon Failed:
Security ID: NULL SID
Account Name: DOMAIN2\user2
Account Domain:

In the latter case Account Domain is blank, event if I try entering my account in DOMAIN\account form (I also tried account@domain.com form with the same result).

All of the above makes me think that whatever authentication module you use in Octopus doesn’t parse account names properly and treats them literally assuming current domain, hence the logon failures.

Hope this helps,
Igor

Hi Igor - we just published 2.0.9.1020 to address these issues, though there’s been limited testing/different AD configurations to verify it in (more to do next week). If you happen to be able to give the 2.0.9.1020 build a run in your environment I’d appreciate any feedback on how the situation is improved.

Regards,
Nick

Hi Nick

I tried 2.0.9, but unfortunately with no success. I tried registering an admin user using

  • plain username (no domain part) - registration was ok, could not log in afterwards
  • UPN - the same result, I even tried to be case-sensitive when logging in, i.e. I copied and pasted my username from Raven
  • DOMAIN\USER - got this during registration:

@@@
A fatal exception occurred
System.ArgumentException: Couldn’t find a user named ‘DOMAIN\user’ in the current domain: XYZP.corp.com; if the domain being searched is not the expected one, please make sure the machine has a connection to the domain controller
at Octopus.Server.Commands.AdminCommand.Start() in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Server\Commands\AdminCommand.cs:line 91
at Octopus.Shared.Startup.ConsoleHost.Run(Action`1 start, Action shutdown) in c:\TeamCity\buildAgent\work\1116bd9da9e239fd\source\Octopus.Shared\Startup\ConsoleHost.cs:line 36
@@@

I tried logging in from the server itself as well as from remote machines, the same result. I also tried doing something like runas /user:user@domain.corp calc.exe to verify that the server recognises my login and everything was fine.

I forgot to mention one strange thing: I don’t see any Audit Failure entries related to Octopus in Security log any more.

Thanks for the info - disappointing to hear this one isn’t closed, it seems like we’re close.

In Octopus 2.0, when authenticating with AD, we store the UPN in RavenDB as the username.

When we create user accounts and at login, we search using whatever name format is provided, then use the UPN of whatever we find to log the user in.

It appears from your results for 1. and 2. that we’ve got the first part right. Am I correct to understand that the username of the user in RavenDB is a correct UPN, e.g. user@DOMAIN2?

If so, then the problem is likely to be at login time; if you log in using the same username format as you provided to the admin command this should work fine. Logging in with the UPN as username should also work.

The changes in 2.0.9.1020 reorder the search for the user and the LogonUser() call, so if we don’t find a user to log in, you won’t see a security audit event any more.

The next step for us here is to set up a more representative test server, I’ll let you know how progress goes on that.

Regards,
Nick

Igor, one other quick question - is Octopus running as Local System on the server machine? If not, is it a local administrator? Thanks in advance, Nick

Hi Nick

Am I correct to understand that the username of the user in RavenDB is a correct UPN, e.g. user@DOMAIN2?

Yes, you are. The username is a correct UPN and the email is also correct email. The very presence of emails in Raven proves that Octopus manages to find users in AD as far as I understand.

if you log in using the same username format as you provided to the admin command this should work fine

This is exactly what I tried but unfortunately it doesn’t work

is Octopus running as Local System on the server machine

Yes, it is

Hi again,

I’ve done two things to hopefully sort this scenario out. First, I’ve added more logging including the error information returned by the failing LogonUser call. Second, I’ve enabled authentication using an LDAP bind rather than going through Windows.

Before we release another public build I was wondering if you’d be able to help verify that we’ve fixed this? The new release is 2.0.9.1039:

http://download.octopusdeploy.com/octopus/Octopus.2.0.9.1039.msi

It would be great if you could use this version to:

  1. Run the admin command with a plain username
  2. Try logging in with the plain username and password; this will record the reason for the login failure to the OctopusServer.txt log file - hopefully clearing up the reasons for our trouble so far
  3. If the information produced from 2. doesn’t solve the problem, run:
Octopus.Server.exe service --stop
Octopus.Server.exe configure --webAuthenticationDomain=YOURDOMAIN
Octopus.Server.exe service --start
  1. (4) Then, try logging in again with your plain username and password

If 4. still does not work, there may be information in the log. We don’t use any specific flags (require SSL, etc.) when connecting to the directory, so if there are any special constraints on your directory let me know.

Thanks for all the help, I think we’re close!

Cheers,
Nick

Hello Nick,

I installed 2.0.9.1039 and this is what I got.

  1. Run the admin command with a plain username

Completed successfully. My username is ikorkhov, and this is what I have in Raven after having finished step 1:

@@@
{
“Username”: “IKorkhov@DOMAIN2”,
“DisplayName”: “Igor Korkhov”,
“EmailAddress”: "IKorkhov@DOMAIN2.co.uk",
“Notes”: null,
“PasswordHashed”: “3E8$K0duRp+U/y+z6LTN7Xulxg==$NvgJDYVcrQVZT3DmiL6ui22GFBe4Sz+3”,
“IsActive”: true,
“ApiKey”: “API-KE9LVELJW83IH1RR3NZ74EJOPPU”,
“IdentificationToken”: “d14015b4-e00a-4a8f-bcdd-2f74ad1258fd”
}
@@@

As you can see, Octopus managed to fetch my proper full username in UPN form and my email from AD.

  1. Try logging in with the plain username and password

This step failed. This is the relevant part from OctopusServer.log

@@@
2014-01-14 10:47:17.3090 INFO Adding certificate to store
2014-01-14 10:47:17.3246 INFO Distribution service listening on: 10943
2014-01-14 10:47:17.3870 INFO Web server is ready to process requests
2014-01-14 10:47:29.9012 INFO A principal identifiable by ‘ikorkhov’ was not found in 'System.DirectoryServices.AccountManagement.PrincipalContext’
2014-01-14 10:47:55.5791 INFO Starting health check for all environments.
2014-01-14 10:47:55.6103 INFO There are no active machines to check.
2014-01-14 10:49:13.2203 INFO A principal identifiable by ‘DOMAIN2\ikorkhov’ was not found in 'System.DirectoryServices.AccountManagement.PrincipalContext’
2014-01-14 10:49:32.4440 INFO A principal identifiable by ‘IKorkhov@DOMAIN2’ was not found in ‘System.DirectoryServices.AccountManagement.PrincipalContext’
@@@

You can see from the log above that I tried plain username as well as UPN and DOMAIN\USER forms, the result was the same.

  1. If the information produced from 2. doesn’t solve the problem, run:
    @@@
    Octopus.Server.exe service --stop
    Octopus.Server.exe configure --webAuthenticationDomain=DOMAIN2
    Octopus.Server.exe service --start
    @@@

This solved the problem, now everyone from DOMAIN2 (and from this domain only!) can log into Octopus!

Now I think we’re really, really close. The missing bit IMO is understand what prevents users from logging in when they specify their domains (DOMAIN\user or user@domain). --webAuthenticationDomain is nice but it doesn’t cover scenario when people from several different domains need to use Octopus.

Anyway, thank you very much for all your help, we’re almost there!

Excellent! Thanks for that. I think you’re right, one last piece in the puzzle to go. I’ll post here with progress.

Regards,
Nick

By the way - just so I can avoid drawing incorrect conclusions; do user@domain1 style logins definitely not work, with --webAuthenticationDomain enabled? Thanks!

For users belonging to a domain specified with --webAuthenticationDomain both user and user@domain formats work, but not DOMAIN\user. For users from other domains nothing works.

When people from “foreign” domains try to log in, the following messages appear in OctopusServer.txt:

  • user@domain and user:

@@@
2014-01-15 11:04:02.0208 INFO A principal identifiable by ‘user@domain’ was not found in 'System.DirectoryServices.AccountManagement.PrincipalContext’
2014-01-15 11:08:27.5447 INFO A principal identifiable by ‘user’ was not found in ‘System.DirectoryServices.AccountManagement.PrincipalContext’
@@@

  • DOMAIN\user (Please note that the log file contains my proper UPN and the message is different. Looks like Octopus managed to fetch users’ info from AD but could not authenticate them).

@@@
2014-01-15 11:03:08.6669 WARN Principal ‘Igor.Korkhov@domain’ (‘DOMAIN\ikorkhov’) credentials could not be verified
@@@

Hi Igor,

Is there a single domain controller that can authenticate users from all of these domains?

Regards,
Nick

Hi Nick,

Well, there are different domains, each with its own controller, but there are trust relationships between them. Normally we don’t care which box belongs to which domain, we simply enter our logins in UPN or DOMAIN\user form and AD does the rest. It works for Octopus v1.x without any problem, too.

Alrighty, thanks for bearing with us.

I’ve gone back over our success and failures so far, and decided on a slightly different approach. There’s a new test build at:

http://download.octopusdeploy.com/octopus/Octopus.2.0.10.1052.msi

--webAuthenticationDomain is gone from this revision; instead, for users that can’t be logged on via their username or UPN, we’ll split DOMAIN\user style usernames and explicitly tell Windows the domain we’re logging in to.

We now also do a couple of fallbacks; if a high-performance WIN32 network logon doesn’t work using the UPN or split DOMAIN\user combo, we’ll fall back to an LDAP bind on the directory that we found the user’s account in. So far we’ve successfully located accounts in the default domain, so verifying credentials there (in user/password form, not using UPNs) should be a safe and we’d hope reliable way to do it.

If neither works we’ll attempt the style of WIN32 login typically used for network-only impersonation, which has been reported to succeed when other options fail.

After installing this build and re-starting the server, you won’t need to run any new configuration commands. Please try simple, UPN and DOMAIN\user logins - I’d hope that they all work, but chances are good that at least one of them will.

If any of the other bug fixes in this build cause a regression for you (still testing here!), falling back to 2.0.9 will be safe.

Crossing my fingers that this is the last one :slight_smile:

Nick

Hi Nick,

Now, after installing 2.0.10.1052, I can confirm it works! I installed it from scratch, provided my plain username as an administrator login and then managed to log in using DOMAIN\user combo.

To sum up:

  • Plain logins from a “foreign” domains don’t work (I expected this)
  • UPNs from the default domain work
  • UPNs from “foreign” domain don’t work (I am not sure if this is what you expect or not).

Being not able to login using UPNs is absolutely not a problem for us, but I think it might be a bit confusing for some people as Octopus displays UPNs of users under Users section in Configuration even though they used DOMAIN\password to register themselves (possibly after trying UPNs which didn’t work).

Of course, it’s up to you to decide whether it needs improving or not, we are quite happy we can start using v2.

Thank you for your hard work solving this issue!

Igor

Wonderful! This is the best news I’ve heard in a long time :slight_smile:

The UPN behaviour hinges on some choices we have to make between treating UPNs as “user@DOMAIN” vs supporting the AD facility for custom UPN suffixes (“user@ARBITRARY”) - we might be able to improve that in the future.

Thanks so much for your persistence helping us to get to the bottom of this, your encouragement made a big difference!

Regards,
Nick

Hi Nick,

Glad it works now and thank you for making such a great product!

Best of luck,
Igor