Troubleshoot the AD Bridge database

If the information in your reports or the events displayed in the Operations Dashboard seem incomplete, perform the following series of diagnostic tests sequentially:

• Check the endpoints
• Check the AD Bridge BTCollector
• Check events in the AD Bridge database
• Troubleshoot checklists for reporting components
• Switch between databases in AD Bridge

Check the endpoints

To troubleshoot potential endpoint problems:

1. Log on to a computer that you suspect might have a problematic endpoint and confirm that events are logged in the local event database. Run the following command as root or as an AD user with administrator privileges:

/opt/pbis/bin/eventlog-cli –s – localhost

2. Note the ID of the last event. If you run the following command, the last ID in this database should match the ID if the events are getting to the collector properly. If the IDs do not match, there is a configuration issue with one of the endpoints.

cat /var/lib/pbis/db/eventfwd-next-record.db

3. If no recent events are displayed or if the command returns errors, make sure that the eventlog service is running:

/opt/pbis/bin/lwsm status eventlog

4. If it is not running, check /var/log/messages to find out why and report the information to BeyondTrust Technical Support. Then, restart the service:

/opt/pbis/bin/lwsm start eventlog

5. If recent events are present but are not being forwarded, make sure that the event forwarding service is running:

/opt/pbis/bin/lwsm status eventfwd

6. If it is not running, check /var/log/messages to try to identify the cause and report the information to BeyondTrust Technical Support. Then, restart the service:

/opt/pbis/bin/lwsm start eventfwd

7. Check the event forwarding service's configuration in the AD Bridge registry to make sure that it properly identifies a collector server and, if the collector server is identified by its IP address, its collector-principal. If you modify the settings of the eventfwd service, you must restart the service for the changes to take effect.

Example of a configuration that uses the host name of its collector:

[HKEY_THIS_MACHINE\Services\eventfwd\Parameters]
 "Collector"="w2k3-r2.example.com"

8. Make sure the collector can be resolved:

[root@rhel5d bin]# nslookup w2k3-r2.example.com
Server:         192.168.1.20
Address:        192.168.1.20#53
Name:   w2k3-r2.example.com
Address: 192.168.1.20

9. Make sure the collector server can be reached:

[root@rhel5d bin]# ping w2k3-r2.example.com
PING w2k3-r2.example.com (192.168.1.20) 56(84) bytes of data.
64 bytes from 192.168.1.20: icmp_seq=1 ttl=128 time=1.40 ms

10. If the collector is identified by IP address, make sure the collector-principal is properly set. For example, if the collector server is at IP address 192.168.1.255 and has a Kerberos machine name of EventCollector in the AD domain example.com, the collector-principal parameter would be:

collector-principal = host/[email protected]

11. Check /var/log/messages for errors.
12. Stop the eventfwd service and then run it from the command line to display error information about the event forwarder's communication with the collector server:

/opt/pbis/bin/lwsm stop eventfwd
/opt/pbis/sbin/eventfwd --loglevel debug

After you run eventfwd from the command line, stop it with CTRL-C and then restart it:

/opt/pbis/bin/lwsm start eventfwd

After you verify that the endpoint is properly receiving events and forwarding them to a collector server, check the collector. If there are recent events, make a note of the last event's time stamp, event category, and event description.

ℹ️

Note

To check whether the collector received the event, see Check the AD Bridge BTCollector.

Troubleshoot checklists for reporting components

The checklists in this section can help you troubleshoot problems with the reporting components.

Endpoints

To check for endpoint problems, confirm the following:

• eventlog service running

• eventfwd service running

• reapsysl service running

• eventfwd service properly configured

  ✅

  Example

  /opt/pbis/bin/regshell
  HKEY_THIS_MACHINE\> ls Policy\Services\eventfwd\parameters\
  
  [HKEY_THIS_MACHINE\Policy\Services\eventfwd\parameters]
  +  "Collector" REG_SZ          "services.umon.com"
  

• Collector name resolvable and address reachable

✅

Example

ping services.umon.com
PING services.umon.com (10.100.1.1) 56(84) bytes of data.
64 bytes from services.umon.com (10.100.1.1): icmp_seq=1 ttl=128 time=0.867 ms

ℹ️

Note

For more information about the services, see AD Bridge services and status.

• Collector principal properly set

  ✅

  Example

  /opt/pbis/bin/regshell
  HKEY_THIS_MACHINE\> ls Policy\Services\eventfwd\parameters\
          
  [HKEY_THIS_MACHINE\Policy\Services\eventfwd\parameters]
  +  "CollectorPrincipal" REG_SZ          "10.100.1.1"
  

• /etc/syslog.conf properly configured

• events present in local event log (test with eventlog-cli)

• eventfwd service seems to forward messages properly (run from command-line to test)

• firewall not blocking RPC access of collector server

Collector servers

To check for problems with the collector servers, confirm the following:

• BTCollector service running
• BTEventDBReaper service running
• events present in local collector database (test with BTCollector-cli)
• BTEventDBReaper properly configured (test with BTEventDBReaper /s)
• database provider and connection string properly set
• collector ACL allows endpoints to write to it (set with Event Management Console)
• collector machine account has sufficient privileges to write to database
• no unusual errors in Windows event log (run eventvwr.exe)
• firewall not blocking incoming RPC connections or outgoing database connections

Database

To check for problems with the database, confirm the following:

• can connect to it with SQL Server Management Studio
• Events table contains events
• EventsWithOUName view contains events
• database security set to allow writing by collector servers, by ldbupdate user, and by administrators
• ldbupdate utility recently run to account for new endpoints joined to AD
• named-pipe client access enabled in SQL Server
• firewall not blocking incoming database connection

Windows reporting components

To check for problems with the Windows reporting components, confirm the following:

• database connection strings set properly
• user has sufficient privileges to access database
• firewall not blocking database connections

Check the AD Bridge BTCollector

1. Make sure BTCollector is running by executing the following command at the shell prompt of the Windows computer running the collector:

   C:\Program Files\BeyondTrust\PBIS\Enterprise\DBUtilities>sc query BTCollector
   
   SERVICE_NAME: BTCollector
           TYPE               : 10  WIN32_OWN_PROCESS
           STATE              : 4  RUNNING
   

2. If the process is stopped, use eventvwr.exe to check the Windows event log for information about why the service failed.

ℹ️

Note

The collector server must be running Windows 2003 or Windows 2008.

3. If the process is not running, start it by executing the following command:

C:\Program Files\BeyondTrust\Enterprise\DBUtilities>sc start BTCollector

4. Verify that the service is receiving forwarded events by viewing the contents of the collector's local SQLite database. To execute the following command, the BTCollector process must be running and you must have read privileges in the access control list:

C:\Program Files\BeyondTrust\Enterprise\DBUtilities>BTCollector-cli -s - localhost

ℹ️

Note

The command should return a list of the events collected from the endpoints. If there is no data, it is likely that your endpoints are improperly configured (see the previous section). If the event that you noted when you checked the event forwarder in the previous section is among the results, make sure the BTEventDBReaper service is functioning properly.

5. Verify that BTEventDBReaper is running:

C:>sc query BTEventDBReaper

6. If the process is stopped, use eventvwr.exe to check the Windows event log for information about why the service failed. Restart the service with:

C:>sc start BTEventDBReaper

7. Check the database connection string and the service's other execution parameters:

C:\Program Files\BeyondTrust\Enterprise\DBUtilities>BTEventDBReaper /s

The results should look something like this:

Database provider:     System.Data.SqlClient
Connection string:     Data Source=SomeCollector;Initial Catalog=LikewiseEnterprise;Integrated Security=yes
Record id last copied: 487
Records per period:    120
Seconds in a period:   10000

If the database server (Data Source= for SQL Server) is identified by name (as in the example), verify that the name can be resolved to an address by using nslookup and verify that the address is reachable from the collector server by using ping.

8. Use eventvwr.exe to check the Windows event log for messages. If BTEventDBReaper is failing to write to the central AD Bridge database and if you are using SQL Server with integrated security, make sure that the collector server’s machine account has sufficient privileges to write to the AD Bridge database.

Check events in the AD Bridge database

1. Check the AD Bridge database on the database server to check whether the table containing events is complete. If necessary, write a manual query to view recent events or to look for an event. For example, with SQL you can use the SQL command-line utility to open the LikewiseEnterprise database and run the following command to display all the events in the table named Events:

select * from Events;

2. If you cannot open or read the database, you might not have sufficient privileges to access it, which can result in problems when you run reports in the management console or use the Operations Dashboard.
3. If you use SQL Server and the Events table is empty, use the SQL Server Configuration Manager to make sure that the name-pipe client protocol is enabled. If it is not and you have to enable it, you must restart the SQL Server service for the changes to take effect.
4. If you find events in the Events table, check whether the events are also present in the EventsViewWithOUName view. If an event appears in the Events table but not in the EventsWithOUName view, it is because the database cannot associate your event with a computer in Active Directory. Run the ldbupdate.exe script and then check whether the event now appears in both views.

Collector is not displayed in the management console

1. Right-click the Enterprise Database Management node to check the Reporting database connection.
2. Run a test connection to ensure that it can connect.
3. Close the BeyondTrust management console.
4. On the services server, open the Reporting Database Connection.
5. Configure the reporting connection for the Reaper service.
6. Restart the Collector and Reaper services.
7. Open the management console and the collector is displayed under the Database Management node.

Switch between databases in AD Bridge

To send events to a different database, you must change the database connection string in at least two places:

• The reaper service for the database (BTEventDBReaper)
• The Enterprise Database Management page in the BeyondTrust Management Console.

ℹ️

Note

Changing the setting on the Enterprise Database Management page automatically changes the same setting on the console's Audit and Access Reporting page and the Operations Dashboard.

However, if you installed different plug-ins of the BeyondTrust Management Console on different computers - to run the Operations Dashboard on a separate computer, for example, then you must change the database connection string on each computer. You may also have to change it in the following additional locations, especially if the computer's AD Bridge Console does not include the Enterprise Database Management plug-in: the Audit and Access Reporting page and the Operations Dashboard page.

After making the changes, you must reset the reaper service so it begins sending events to the new database.

1. In the AD Bridge Console tree on your Windows administrative workstation, right-click the Enterprise Database Management node and then click Connect to database.

   • Click Change. Under Database Type, select Microsoft SQL Server, and then enter the name of the database server instance in the Server/Instance box.
   
   

   

   • Enter the credentials of the database definer account if required for the authentication type that you selected, and then click OK.

2. In the console tree, right-click the Operations Dashboard node and then click Connect to.

   • Click Change.
   • Change the database settings as needed, and then click OK.

3. In the console tree, right-click the Audit and Access Reporting node, and then click Advanced.

   • Click Change.
   • Change the database settings as needed, and then click OK.

4. Open a command prompt window as an administrator and then change directories to C:\Program Files\BeyondTrust\PBIS\Enterprise\DBUtilities, and then run the following command:

BTEventDBReaper /gui

Make the changes that you want, and then click OK.

5. Reset the BTEventDBReaper to 0 and then refresh its settings to prompt it to send the events to the new database. To do so, from the C:\Program Files\BeyondTrust\PBIS\Enterprise\DBUtilities folder, run the following commands as an administrator:

BTEventDBReaper /f 0
BTEventDBReaper /r