Troubleshoot and solve logon problems

This page explains how to troubleshoot and solve logon problems.

Logon problems from Windows

To troubleshoot a problem with a user who cannot log on a to Linux or Unix computer, perform the following series of diagnostic tests sequentially.

• On a Windows computer, log off and then log on again with the problem user's AD credentials to verify that the password is correct and that the account is not locked or disabled.
• Try to SSH to the target Linux or Unix computer again with the user's full NT4-style credentials and password, not just the user's alias. In your SSH command, make sure to use a slash character to escape the slash.
• If you are using AD Bridge, make sure that the user's computer is in the correct AD Bridge Cell.
• Make sure that the user is enabled to log on the computer, either by being enabled in the cell (with AD Bridge) or by being in a group allowed to access the computer. Then try to log on the target computer again.
• Ensure that the AD Bridge client can communicate with the Active Directory domain controller.
• Make sure that the shell specified for the user account in Active Directory is available on the target computer. Specifying a shell that is unavailable will block the user account from logging on.
• Verify that the home directory is set and can be created. A home directory that cannot be created because the path is incorrect or the permissions are insufficient can block an attempt to log on.
• Make sure there are no logon restrictions in place, for example, the Group Policy setting that restricts logon to certain users or groups, that prevents the user account from logging on the computer.
• Log on the computer with a different user account, and that is enabled for access to the computer.

Logon problems on Linux or Unix

To troubleshoot problems logging on a Linux computer with Active Directory credentials after you joined the computer to a domain, perform the following series of diagnostic tests sequentially with a root account.

The tests can also be used to troubleshoot logon problems on a Unix computer. However, the syntax of the commands on Unix might be slightly different.

Ensure you are joined to the domain

Execute the following command:

/opt/pbis/bin/domainjoin-cli query

Check whether you are using a valid logon form

When troubleshooting a logon problem, use your full domain credentials: DOMAIN\username.

Example: example.com\hoenstiv

When logging on from the command line, you must escape the slash character with a slash character, making the logon form DOMAIN\username.

Example: example.com\hoenstiv

Clear the cache

You may need to clear the cache to ensure that the client computer recognizes the user's ID.

ℹ️

Note

For more information, see Install AD Bridge.

Destroy the Kerberos cache

Clear the AD Bridge Kerberos cache to make sure there is not an issue with a user's Kerberos tickets. Execute the following command with the user account that you are troubleshooting:

/opt/pbis/bin/kdestroy

Check the status of the AD Bridge authentication service

Check the status of the authentication service on a Unix or Linux computer running the AD Bridge Agent by executing the following command as the root user:

/opt/pbis/bin/lwsm status lsass

Check communication between the AD Bridge service and AD

Verify that the AD Bridge service can exchange data with AD by executing this command:

/opt/pbis/bin/get-dc-name FullDomainName

✅

Example

/opt/pbis/bin/get-dc-name example.com

If the result does not show the name and IP address of your domain controller:

1. Make sure the domain controller is online and operational.
2. Check network connectivity between the client and the domain controller.
3. Join the domain again.
4. View log files.

If the result shows the correct domain controller name and IP address, proceed to the next test.

Verify that AD Bridge can find a user in Active Directory

Verify that the AD Bridge agent can find your user by executing the following command, substituting the name of a valid AD domain for domainName and a valid user for ADuserName:

/opt/pbis/bin/find-user-by-name domainName \\ ADuserName

✅

Example

/opt/pbis/bin/find-user-by-name example\\hab

If the command fails to find the user:

1. Check whether the computer is joined to the domain by executing the following command as root:

domainjoin-cli query

This displays the hostname, current domain, and distinguished name, which includes the OU to which the computer belongs. Make sure the OU is correct. If the computer is not joined to a domain, it displays only the hostname.

2. Check Active Directory to make sure the user has an account. If you are using AD Bridge, also ensure that the user is associated with the correct cell.
3. Check whether the same user is in the /etc/passwd file. If necessary, migrate the user to Active Directory.
4. Make sure the AD authentication provider is running by proceeding to the next test.

If the user is found, proceed to the PAM test later in this topic.

Ensure the AD authentication provider is running

AD Bridge includes two authentication providers:

• the local provider
• the Active Directory provider

If the AD provider is not online, users are unable to log on with their AD credentials. To check the status of the authentication providers, execute the following command as root:

/opt/pbis/bin/get-status

A healthy result should look like this:

LSA Server Status:

Compiled daemon version: 10.1.561.63589
Packaged product version: 10.1.725.63590
Uptime:        6 days 23 hours 36 minutes 29 seconds

[Authentication provider: lsa-activedirectory-provider]

    Status:        Online
    Mode:          Default Cell
    Domain:        EXAMPLE.COM
    Domain SID:
    Forest:        example.com
    Site:          Default-First-Site-Name

An unhealthy result will not include the AD authentication provider or will indicate that it is offline. If the AD authentication provider is not listed in the results, restart the authentication service.

If the result looks like the line below, check the status of the AD Bridge services to make sure they are running.

Failed to query status from LSA service.
The LSASS server is not responding.

ℹ️

Note

For more information, see AD Bridge services and status.

Run the id command to check the user

Run the following id command to check whether nsswitch is properly configured to handle AD user account information:

id DOMAIN\\username

✅

Example

id example\\kathy

If the command does not show information for the user, check whether the /etc/nsswitch.conf file is properly configured for passwd and group: Both entries should include the lsass parameter.

If /etc/nsswitch.conf is properly configured, the AD Bridge name service libraries might be missing or misplaced. It is also possible that the LD_PRELOAD or LD_LIBRARY_PATH variables are defined without including the AD Bridge libraries.

Switch user to check PAM

Verify that a user's password can be validated through PAM by using the switch user service. Either switch from a non-root user to a domain user or from root to a domain user. If you switch from root to a domain user, run the command below twice so that you are prompted for the domain user's password:

su DOMAIN\\username

✅

Example

su example\\hoenstiv

If the switch user command fails to validate the user:

• Generate a PAM debug log.

ℹ️

Note

For more information, see Pluggable Authentication Modules (PAM).

• Also, check the following log files for error messages (the location of the log files varies by operating system):
  • /var/log/messages
  • /var/log/secure

Test SSH

Check whether you can log on with SSH by executing the following command:

sh DOMAIN\\username@localhost

✅

Example

ssh example.com\\hoenstiv@localhost

ℹ️

Note

If you believe the issue might be specific to SSH, see SSH SSO login problems.

Run the authentication service in debug mode

To troubleshoot the lookup of a user or group ID, you can set the AD Bridge authentication service to run in debug mode and show the log in the console by executing this command:

/opt/pbis/bin/lwsm set-log-level lsass - debug

Check Nsswitch.Conf

Make sure /etc/nsswitch.conf is configured correctly to work with AD Bridge.

ℹ️

Note

For more information, see Configuring Clients Before Agent Installation in Install AD Bridge.

Additional diagnostic tools

There are additional command-line utilities that you can use to troubleshoot logon problems in the  /opt/pbis/bin directory:

ℹ️

Note

For more information, see Accounts and attributes.

Red Hat enterprise Linux 9 Fips systems

If AD authentication fails, run the following command and then reboot the machine.

update-crypto-policies --set FIPS:AD-SUPPORT

This will allow AD authentication through the encryption types required by Active Directory.

SSH SSO login problems

Solve problems logging on with SSH to Linux and Unix computers running AD Bridge.

ℹ️

Note

Make sure you are joined to the domain by executing the following command as root:

/opt/pbis/bin/domainjoin-cli query

ℹ️

Note

If you are not joined, see Join Active Directory with the Command Line in Install AD Bridge.

You can use the following steps to troubleshoot problems logging on to Linux and Unix computers with ssh. It is assumed that the computer is connected to Microsoft Active Directory with AD Bridge and that you are trying to log on with an Active Directory account.

Use NT4-style credentials and escape the slash character

Try to SSH to the target Linux or Unix computer again with the user's full NT4-style credentials, not the user's alias. In your ssh command, make sure to use a slash character to escape the slash.

✅

Example

ssh example.com\\kathy@localhost

Perform general logon troubleshooting

ℹ️

Note

If you cannot logon after you escape the slash character in your full NT4-style credentials and use your password, execute the general logon troubleshooting steps in Troubleshoot and solve logon problems and Solve logon problems on Linux or Unix . If those steps do not help solve the problem, return to this page and perform the following AD Bridge-specific ssh troubleshooting steps in the order listed.

Get an SSH log

You should obtain debug logs for the AD Bridge authentication service (lsass), PAM, and sshd.

ℹ️

Note

For more information, see the following:

• Generate debug logs for AD Bridge services
• Generate a PAM debug log for AD Bridge

To get an ssh log, locate sshd and then start it in a separate terminal window with the following options:

`which sshd` -vvv -p 9999 >/tmp/sshd.log 2>&1

The command starts an instance of sshd listening on Port 9999 and routes logging information to a log file in /tmp/sshd.log.

Now try to ssh to the localhost at that port:

ssh -ddd -p 9999 yourADuserName@localhost

When the logon fails, kill ssh; the sshd session will stop as well.

Finally, check the log file at /tmp/sshd.log for information that might help you resolve the issue. In addition, check the log files for lsass and PAM.

After an upgrade, reconfigure SSH for AD Bridge

If ssh was recently upgraded, run the following command as root to make sure that the sshd_config file is set up properly to work with AD Bridge:

domainjoin-cli configure --enable ssh

Verify that port 22 is open

A common problem is that a firewall is blocking the port used by SSH. Take a moment to verify that Port 22, which SSH typically connects to, is available by telneting to it. Failure looks like this:

root@example:~# telnet 10.0.0.17 22
Trying 10.0.0.18...
telnet: Unable to connect to remote host: Connection refused

Success looks like this:

root@example:~# telnet 10.0.0.17 22
Trying 10.0.0.17...
Connected to 10.0.0.17.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.1p1 Debian-5

Ensure PAM is enabled for SSH

If your Active Directory account is not working with SSH, make sure that UsePAM is enabled in sshd_config and make sure that your sshd application is linked to the PAM libraries.

1. Determine which sshd is running by executing the following command:

bash-3.2# ps -ef | grep sshd
    root  8199     1  0  Feb  6  ?         0:00 /opt/ssh/sbin/sshd
    root  2987  8199  0  Mar  3  ?         0:04 sshd: root@notty
    root 24864  8199  0 12:16:25 ?         0:00 sshd: root@pts/0
    root  2998  8199  0  Mar  3  ?         0:05 sshd: root@notty
    root 24882 24880  0 12:16:54 pts/0     0:00 grep sshd

1. Either use lsof to find out which configuration file it is reading or start it up with debugging to figure out the default path.

✅

Example

username@computer:~$ /usr/sbin/sshd -dd -t
   debug2: load_server_config: filename /etc/ssh/sshd_config
   debug2: load_server_config: done config len = 664
   debug2: parse_server_config: config /etc/ssh/sshd_config len 664
   debug1: sshd version OpenSSH_5.1p1 Debian-3ubuntu1
   Could not load host key: /etc/ssh/ssh_host_rsa_key
   Could not load host key: /etc/ssh/ssh_host_dsa_key

1. Verify that UsePAM is enabled in the config file. As a best practice, make a backup copy of the configuration file before you change it.
2. Run ldd on sshd to make sure it links with libpam.

Ensure GSSAPI is configured for SSH

Logging onto a system with keys does not provide that system with the means of getting a PAC from the domain controller. Without a PAC there is no group membership information for the user. Automated Kerberos ticket renewal will also be unavailable. So, when the ssh login hits the login restrictions in the account phase as it tests for the group memberships, it will not find the user's group information, causing an ssh error like this:

Not in an Allowed Group!

A workaround is to have each user log in once with a password. Subsequent logins with keys should work until the AD cache is flushed, after which the user will have to log in again.

Check the configuration of SSH for SSO

Although AD Bridge automatically configures OpenSSH to support SSO through Kerberos using GSSAPI, it is worthwhile to review how AD Bridge does. Since you might need to configure or troubleshoot other applications for SSO, understanding the process will make it easier to apply the technique to other applications.

ℹ️

Note

Not all versions of OpenSSH support Kerberos. Versions older than 4.2p1 might not work or might work improperly.

SSH service principal name

The first thing that needs to be considered is the Kerberos service principal name (SPN) used by ssh and sshd. The SPN is a string that identifies the service for which an authentication ticket is to be generated. In the case of ssh, the SPN has the form:

host/<server name>@<REALMNAME>

For example, when a user uses ssh to connect to a computer named fozzie.mycorp.com, the ssh program requests a service ticket for the SPN:

host/[email protected]

The Kerberos realm is the computer's domain name in uppercase letters.

System keytab generation

In order for Microsoft Active Directory to generate a Kerberos ticket for this SPN, a service account must exist for it. Additionally, a keytab must be created for the service account and placed on the sshd server. AD Bridge completely automates this operation. When a Linux or Unix computer is joined to AD, a machine account is created for the computer. If the computer is called fozzie, a machine account called fozzie$ is created in AD. AD Bridge then automatically creates a keytab for the SPN and places it in the standard system location (typically, /etc/krb5.keytab).

User keytab generation

When the user runs the ssh program and OpenSSH determines that it will use Kerberos authentication, it will need to access a keytab for the user so that it can obtain a service ticket for the service or computer to which it is trying to connect. This keytab must be created using the user's account name and password. Manually, this can be performed by using the kinit utility. AD Bridge, however, does it automatically when the user logs on the computer. On most systems, the user keytab is placed in the /tmp directory and named krb5cc_UID where UID is the numeric user ID assigned by the system.

Configure OpenSSH

AD Bridge automatically configures OpenSSH at both the client and server computer. On the client, the ssh_config file (typically in /etc/ssh/ssh_config) is modified. On the server, ssh_config (typically in /etc/ssh/ssh_config) is modified. AD Bridge adds the following lines of code to the right files if they are not already present and if they are required by the system's version of sshd:

In the server, the following lines must be present in sshd_config. if you are troubleshooting, make sure these lines are there:

GSSAPIAuthentication yes
GSSAPICleanupCredentials yes

On the client, the following line must be present in ssh_config:

GSSAPIAuthentication yes

On the client, GSSAPIDelegateCredentials yes is an optional setting that instructs the ssh client to delegate the krb5 TGT to the destination machine when SSH single sign-on is used.

In addition, if any of the following options are valid for the system's version of sshd, they are required and configured by AD Bridge:

ChallengeResponseAuthentication yes
UsePAM yes
PAMAuthenticationViaKBDInt yes
KbdInteractiveAuthentication yes

Setting these options to yes instructs ssh to use the kbdinteractive ssh authentication mechanism and allows that mechanism to use PAM, settings that are required for AD Bridge to function properly.

ℹ️

Note

For more information, see the man pages for ssh, sshd, and the comments in the ssh and sshd configuration files.

Test SSO

With OpenSSH properly configured, demonstrating SSO support is simple: Log on a Linux or Unix machine running AD Bridge by using your Active Directory credentials and then use ssh to connect to another machine that is also running AD Bridge. OpenSSH should establish a connection without prompting for a username or password.

Platform-specific issues

If you are using Red Hat, CentOS, Fedora, or AIX operating systems, review any of the following sections that are relevant for your operating system.

Red Hat and CentOS: solve the SSO problem

There is a known bug with some versions of Red Hat and CentOS that prevents SSO from working with SSH, SSHD, and PuTTY. The following versions are known to be affected:

• CentOS 5
• Red Hat Enterprise Linux 5

The system incorrectly concatenates the Kerberos ticket's service principal name on the target Linux computer. For example, in the final entry of the results of the klist command below, the full name of the service principal is cut off after the @ symbol:

[EXAMPLE\fanthony@centos52 ~]$ /opt/pbis/bin/klist
Ticket cache: FILE:/tmp/krb5cc_1689257039
Default principal: [email protected]
Valid starting     Expires            Service principal
07/31/08 09:25:13  07/31/08 19:25:31  krbtgt/[email protected]
    renew until 08/07/08 09:25:13
07/31/08 09:25:31  07/31/08 19:25:31  [email protected]
    renew until 08/07/08 09:25:13
07/31/08 09:30:04  07/31/08 19:25:31  host/centos52.example.com@
    renew until 08/07/08 09:25:13

To determine whether you need to implement the solution below on your Red Hat or CentOS computer, execute the following series of tests:

1. Connect to your target machine with SSH by using PuTTY and a valid Active Directory user. Be sure to use the FQDN of the host.
2. Execute the following command:

/opt/pbis/bin/klist

The results should look like this:

EXAMPLE\fanthony@centos52 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_1689257039
Default principal: [email protected]
Valid starting     Expires            Service principal
07/31/08 09:25:13  07/31/08 19:25:31  krbtgt/[email protected]
    renew until 08/07/08 09:25:13
07/31/08 09:25:31  07/31/08 19:25:31  [email protected]
    renew until 08/07/08 09:25:13

3. SSH again to the same host and when prompted for the password, type CTRL+C.
4. Execute the klist command again:

/opt/pbis/bin/klist

5. Check the results to determine whether there is an incorrectly concatenated service principal, as there is in the following output:

[EXAMPLE\fanthony@centos52 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_1689257039
Default principal: [email protected]
Valid starting     Expires            Service principal
07/31/08 09:25:13  07/31/08 19:25:31  krbtgt/[email protected]
    renew until 08/07/08 09:25:13
07/31/08 09:25:31  07/31/08 19:25:31  [email protected]
    renew until 08/07/08 09:25:13
07/31/08 09:30:04  07/31/08 19:25:31  host/centos52.example.com@
    renew until 08/07/08 09:25:13

If the tests confirm that the problem exists, implement the following solution:

1. On Red Hat Enterprise Linux 5, make sure that the reverse PTR host definitions are defined in DNS.
2. On the target Linux computer, add the following line to /etc/krb5.conf under the [domain_realm] entry of the file:

.yourdomainname.com = YOURDOMAINNAME.COM

✅

Example

[domain_realm]
.example.com = EXAMPLE.COM

1. Restart SSHD by running the following command at the shell prompt:

/sbin/service sshd restart

Red Hat and Fedora: solve SSH config problem

On Fedora 14 and Red Hat 5, there is an issue with the configuration of the platform that blocks SSH SSO. You must either use a workaround to connect to the client or modify the sshd_config file on the server side. This section illustrates the problem and shows you how to connect to the client or fix the server.

After you join a domain with AD Bridge, Network Manager restarts and leaves the /etc/hosts file looking like this:

[root@nile-fedora14 etc]# cat /etc/hosts
10.100.0.26 nile-fedora14.nile-domain.example.com nile-fedora14 # Added by NetworkManager
127.0.0.1 localhost.localdomain localhost localhost4
::1 nile-fedora14.nile-domain.example.com nile-fedora14 localhost6 nile-fedora14.ramp.example.com

It should, however, look like this, but Network Manager keeps resetting it:

10.100.0.26  nile-fedora14.nile-domain.example.com  nile-fedora14   # Added by NetworkManager
127.0.0.1  nile-fedoar14.nile-domain.example.com nile-fedora14 localhost.localdomain  localhost  localhost4
::1  nile-fedora14.nile-domain.example.com  nile-fedora14   localhost6.localdomain6 localhost6

The configuration set by Network Manager blocks SSO because it ends up restricting reverse name lookups to ipv4 only.

When using the client, you can work around the problem by connecting by the external IP address. In other words, instead of using ssh -l user nile-fedora14.nile-domain.example.com to connect, use the following form:

ssh -l user 10.100.0.26

Alternatively, to fix the problem, you can turn off GSSAPIStrictAcceptorCheck in sshd_config on the server, but such a resolution might be unavailable when you do not have administrative access to the server or when doing so might cause intractable side effects or security holes.

Another way to fix the problem is to turn off reverse DNS lookups in Kerberos. However, such a solution might result in side effects that block other applications or operations.

AIX and Red Hat: set reverse PTR host definitions for SSO

For single sign-on with SSH to work on Red Hat Enterprise Linux 5 and AIX, reverse PTR host definitions must be set in DNS.

AIX: configure for outbound single sign-on

On AIX 5.3, client-side SSH is not set up by default. Here is how to configure it so that it will work with AD Bridge:

1. On your AIX 5.3 computer, make sure the network authentication service, version 1.4.0.8, is installed.

   ✅

   Example

   -bash-3.00$ lslpp -l | grep krb
   krb5.client.rte 1.4.0.8 COMMITTED Network Authentication Service
   

2. After joining an Active Directory domain with AD Bridge, append the following lines to the end of /etc/krb5/krb5.conf:

   [domain_realm]
   .demo.example.com = DEMO.EXAMPLE.COM
   demo.example.com = DEMO.EXAMPLE.COM
   

3. Make sure that /etc/krb5/krb5.conf links to /etc/krb5.conf.

4. Also make sure that /etc/krb5/krb5.keytab links to /etc/krb5.keytab.

5. Make a backup of the credentials directory by executing the following command as root:

mv /var/krb5/security/creds /var/krb5/security/creds_old

6. As root, make a symbolic link to the /tmp directory so that the AIX Kerberos libraries can access the directory in which AD Bridge stores its credential caches:

ln -s /tmp /var/krb5/security/creds

7. Open /etc/environment, which contains the list of environmental variables that are set when a user logs on, and add the following line to the end of it:

KRB5_CONFIG=/var/lib/pbis/krb5-affinity.conf:/etc/krb5.conf

8. If you are logged on the machine whose environmental variable you changed, you must log off and log on again for the change to take effect.