Monitoring Systems Using Event Triggers

Monitoring Systems Using Event Triggers

Now that you know how to view, filter, and create events, let’s look at a technique using event triggers that you can use to automate the event monitoring process. With event triggers, you can configure system tasks that monitor the event logs and then take a specific action if an event occurs. For example, you can create a trigger that monitors the event logs for low disk space events and if such events occur, you can run a script that removes any temporary or unnecessary files to resolve the low disk space condition. Thus, not only can event triggers help you automate the monitoring process, the actions triggers take can also help you resolve issues as they arise to maintain system performance, ensure system integrity, and more.

Creating event triggers isn’t something you should do casually, without careful forethought. You need to have a clear plan of action—a set of goals that you hope to achieve by using event triggers. Let’s take a look at the reasons you might want to use event triggers and then look at the tools you can use to manage them.

Why Use Event Triggers?

Maintaining application and system performance is a key reason for using event triggers. For example, if an application running on a server has known issues that you usually have to resolve manually, you may be able to configure event triggers that monitor the event logs for related errors and then run scripts that take the appropriate actions to resolve the problem. Here, you would want to track down the known issues for the application by searching the event logs, asking other administrators about issues, or searching for knowledge-base articles that describe the issues. Afterward, match issues to specific events or types of events for which you can configure event triggers to monitor, and then write a script that notifies administrators of the issue or takes appropriate actions to resolve the issue. This script is then used as the task that the event trigger runs.

Another common reason for using event triggers is to help you identify application and service outages quickly, and to possibly restore normal operations. When an application or service stops, users can no longer use the resource and this can cost the organization dearly in time, money, and wasted resources. Here, you would want to search for documentation on the types of errors that can occur if the application or service isn’t responding normally. Then, searching the event logs to see if you find similar or matching events in the logs, you would note sources, event IDs, and descriptions used so that you can create event triggers to watch for the related events. Finally, you could write a script that restarts the application or takes other appropriate actions to resolve the outage.

You may also want to use event triggers to help you maintain system security and integrity. When a system is under attack, events may be written to the log files that indicate the application, component or service that is under attack. With a brute force attack, a hacker may be trying various user name and password combinations in an attempt to gain access. If you are monitoring the system under attack, you would see failed logon attempts in the security logs as the hacker attempts to gain access. A hacker may also try to bring down the system, application, or service using a denial-of-service attack. Typically hackers deny service by sending continuous streams of malformed service requests. These attempts should show up in the related application, system, or service-specific logs as errors. To combat such attacks, you could configure event triggers that watch for related events, such as account lockouts due to a series of failed logon attempts.

Getting Ready to Use Event Triggers

Before you start creating event triggers, you should consider what you hope to achieve through automated monitoring, as well as any impact the monitoring might have on the affected systems and the network as a whole. You should

1. Identify the events you want to monitor and define the reasons for monitoring each event. Use the event logs on multiple systems and documentation of known issues and errors, such as knowledge-base articles, to help you pinpoint places to start.
2. Specify the actions you want to take when an event occurs. Initially, write this as a list. Be sure to consider the impact any corrective actions might have on the system or the network as a whole.
3. Write scripts or applications to handle the necessary corrective actions or user notifications. Don’t implement them as triggers yet. You should test the scripts first on an isolated network or development system to uncover any flaws in the planning.
4. Define the event triggers and the tasks to execute, and then implement the triggers. Make sure you monitor the affected systems closely for the next several days or weeks to ensure there are no adverse affects.
5. Maintain and remove triggers as necessary to ensure continuing operations.

Steps 1, 2, and 3 can be accomplished using the earlier discussions in this and other chapters of this book. Steps 4 and 5, however, involve the processes of defining, maintaining, and removing event triggers. These processes are handled with the following subcommands of the Eventtriggers utility:

• Eventtriggers /create  Creates a new event trigger and sets the action to take
• Eventtriggers /query  Displays the event triggers currently configured on a specified system
• Eventtriggers /delete  Removes an event trigger when it is no longer needed

  Note

  Unlike most other commands with subcommands, Eventtriggers subcommands use a forward slash (/).

The sections that follow discuss each of these subcommands and their usage.

Creating Event Triggers

Event triggers can be configured to run executable programs with the .exe extension and scripts with the .bat or .cmd extension when an event occurs. You create event triggers using

eventtriggers /create /tr Name /l LogName [Constraints] /d Description 

/tk Task

where

• Name  Sets the name of the trigger as a string of characters enclosed in quotation marks, such as “Connection Failure”.
• LogName  Sets the name of the log to monitor. Use quotation marks if the log name contains spaces, as in “DNS Server.” The default value is asterisk (*), which specifies that all logs should be monitored.
• Constraints  Sets the constraints that determine whether an event matches the trigger. Constraints limit the trigger’s scope according to event ID, event source, or event type, using the /Eid EventID, /So EventSource, or /T EventType parameters respectively.

  Tip

  You can use multiple constraints as well. If you do, the event must match each constraint in order to be triggered. Thus, additional constraints narrow the scope of the trigger.

• Task  Sets the program or script to execute. Be sure to type the full path to the program or scripts you want to run.

  Note

  Eventtriggers does not verify file paths and if you enter an invalid file path, you will not see a warning. To pass arguments to an executable or script, enclose the file path and the command arguments in a set of double quotation marks, such as “c:\scripts\trackerror.bat system y”.

• Description  Sets the description for the trigger and can be any string of characters. Be sure to enclose the description in double quotation marks.

Don’t be intimidated by the parade of parameters here. It’s a lot easier than it looks once you get started. Consider the following examples:

Create an event trigger that monitors all the event logs for events with the ID 9220 and then runs Record-prob.bat:

Eventtriggers /create /tr "Monitor 9220 Errors" /eid 9220 /tk 

\\Mailer1\scripts\record-prob.bat

Create an event trigger that monitors the DNS Server log for events with the source as DNS and the event ID 4004 and then runs Dns-adfix.bat:

Eventtriggers /create /tr "DNS AD Fix" /l "DNS Server" /so "DNS" 

/eid 4004 /tk c:\admin\scripts\dns-adfix.bat

Create an event trigger that monitors the security log for failure audit events with the source as Security:

Eventtriggers /create /tr "Failure Audit Checks" /l "Security" /so 

"Security" /t Failureaudit

Event triggers are created and their associated tasks are run by default on the local computer with the permissions of the user who is currently logged on. Because this command is used primarily for administration, you will be prompted for a password before the event trigger is added. If the triggered task needs to run with different or specific user permissions, provide the Run As permissions using /u [Domain\]User [/p Password], where Domain is the optional domain name in which the user account is located, User is the name of the user account whose permissions you want to use, and Password is the optional password for the user account, such as

Eventtriggers /create /u adatam\wrstanek /p R4Runner! /tr "Exchange 

Monitor" /l "Application" /so "MSExchangeMTA" /t warning /tk c:\ admin

\scripts\exe-errlog.bat 

As necessary, you can also specify the remote computer on which you want to create the event trigger using /S Computer, where Computer is the remote computer name or IP address, such as

Eventtriggers /create /s 192.168.1.150 /tr "Exchange Monitor" /l 

"Application" /so "MSExchangeMTA" /t warning /tk c:\admin\scripts

\exe-errlog.bat 

Displaying Currently Configured Event Triggers

You can obtain information about currently configured event triggers using Eventtriggers /query. Simply type the command at the prompt, such as

eventtriggers /query

The basic output of the query shows you the event trigger ID, event trigger name, and the task that is run, as shown in the following example:

Trigger ID     Event Trigger Name       Task

==========     ====================     ==================================

         4     Failure Audit Checks     c:\admin\scripts\auditing.bat

         2     Monitor 9220 Errors      \\Mailer1\scripts\record-prob.bat

         3     DNS AD Fix               c:\admin\scripts\dns-adfix.bat

         1     Disk Cleanup             d:\windows\system32\cleanmgr.exe

Note

You will use the trigger ID to delete the trigger. The output format, by default, is table (/Fo Table). You can use /Fo Csv to format the output as comma-separated values or /Fo List to format the output as a list. You can also use the /Nh parameter to turn off the display of headers, if either the Table or the Csv format option is specified.

To get more detailed information, use the /V (verbose) flag. With verbose output the additional columns of information are

• Hostname  The computer name or IP address of the computer on which the event trigger is configured.
• Query  The complete command text used to create the event trigger.
• Description  The description of the trigger, if provided when the trigger was created.
• Run As (User name)  The Run As user used to create the task and run the associated task for the event trigger.

As necessary, you can specify the remote computer whose triggers you want to query using /s Computer, where Computer is the remote computer name or IP address, such as

eventtriggers /query /s Mailer1

You can also specify the Run As user permissions using /U [Domain\]User [/P Password], where Domain is the optional domain name in which the user account is located, User is the name of the user account whose permissions you want to use, and Password is the optional password for the user account, such as

eventtriggers /query /s Mailer1 /u adatam\administrator /p dataset5

Deleting Event Triggers

When event triggers are no longer needed, you can delete them using the eventtriggers /delete command. The syntax is

eventtriggers /delete /tid ID

where ID is the trigger ID you want to delete. You can also use asterisk (*) as the trigger ID to delete all event triggers. Consider the following examples:

Delete event trigger 5:

eventtriggers /delete /tid 5

Delete all event triggers:

eventtriggers /delete /tid *

As necessary, you can specify the remote computer whose triggers you want to delete using /S Computer, where Computer is the remote computer name or IP address, and the Run As permission using /U [Domain\]User [/P Password], where Domain is the optional domain name in which the user account is located, User is the name of the user account whose permissions you want to use, and Password is the optional password for the user account, such as

eventtriggers /delete /tid 3 /s Mailer1 /u adatam\wrstanek /p outreef7

Caution

You can’t restore event triggers once you’ve deleted them. If you think you may use an event trigger again, write the output produced from Eventtriggers /query /v to a file and then save the file for future reference. With the /V parameter, the file will contain the complete command text used to create the trigger.