Release : LogIDS 2.2 and LogAgent 5.2

["SecurIT Informatique Inc." <securit () iquebec com>]

Hello list members.  This is to announce the release of the following items 
in the softwares I developped and distribute on my website 
http://securit.iquebec.com/ :

- LogAgent 5.1 Open Source
- LogAgent 5.2 Pro
- LogIDS 2.2 Open Source
- LogIDS 2.2 Pro

LogAgent was originally designed as a log monitoring and centralizing agent 
for the Win32 platform, both for ASCII log files and for the Event 
Viewer.  LogAgent Pro eventually evolved into a log analysis agent, used 
for distributed analysis with LogIDS.

LogIDS is a universal log analysis intrusion detection console, allowing 
you to configure the software to specify how you want it to treat your 
logs, and eventually display it on a graphical representation of your 
network map, as the events occurs.  The Pro version is built for improved 
performance, and offers pop-up windows that allow for bigger log viewing 
windows.

Changes made to LogAgent 5.1 Open Source : Fixed a minor bug when reporting 
to the Event Viewer

Changes made to LogAgent 5.2 Pro : Improved ruleset from which you have a 
wider selection of actions to apply when your rules checks true: organize 
log data in basic reports, assing custom priorities and user messages to 
your logs, send e-mails and launch external programs.  More details below.

Changes made to LogIDS 2.2 (Open Source and Pro) : Improved ruleset, 
similar to LogAgent 5.2 Pro additions, from which you have a wider 
selection of actions to apply when your rules checks true: organize log 
data in basic reports, assing custom priorities and user messages to your 
logs, send e-mails and launch external programs.  More details below.

Here is a description of each of the ruleset elements for LogAgent 
5.2/LogIDS 2.2.  New additions for this release are preceded by a *.  For 
more details about the ruleset or these softwares in general, please refer 
to these software's documentations (http://securit.iquebec.com/).

Legend:
(1) : applies to LogAgent 5.2 Pro only
(2) : applies to LogIDS 2.2 only (Open Source and Pro)
(3) : applies to both LogAgent Pro and LogIDS

LOG= (1) : the first and most basic action implemented in LogAgent's rule 
system, LOG= allows you to actually log the log data that checked true on 
the condition. This means that the data will be forwarded to the 
destinations specified in the file config.txt, as previous versions 
neutrally did. You can simply put the list of fields you want to be printed 
and by separating them with '#', in the order of appearance that you wish 
to dispose them. You can also use this feature to dispose of redundant 
fields that you don't want to unnecessary pass to the console, or remove 
fields that you see little value into. If you want to display the whole 
line in its original field order, simply leave the list empty.

BACKUP= (1) : BACKUP= works the same way as LOG=, but will forward the logs 
in the \backup directory within the destination folder(s). This is a trick 
implemented mostly for LogIDS, to still have all your log data in the same 
place, even if it gets separated at the console. After LogIDS have analysed 
data, it also moves the data to the \backup directory, so this way you get 
all your logs at one place even if you separated them at pre-analysis 
(LogAgent) stage. These logs can then be reviewed with other programs if 
you like. This can also be useful when storing logs on storage servers, the 
important data ending up in the primary folder, and "informational" data 
kept in separate files in the \backup folder. A way to separate the wheat 
from the chaff, but still keeping the chaff for future reference. As with 
LOG, you can specify fields to be included separated by #, or leave the 
line empty to obtain the whole data. It is recommended to use the same 
fields specification as you do with LOG= fro a particular file if you want 
to maintain some sort of data integrity within your stored log files. Note 
that backed-up lines will not get displayed on the LogAgent console, even 
with SHOWCONSOLE=Y in config.txt.

IGNORE (1) : which can be used to discard irrelevant or unnecessary data, 
to avoid log pollution with noise (the log line does not get LOGged or 
BACKedUP, but still ends up in the local backed-up log file 
filename.ext.blg (filename gets a extra '.blg' extension after being 
processed by LogAgent)).

WARN (2) : The simplest one of them is WARN. This will cause the LogIDS 
system to emit a beep sound each time this rule checks OK (to actually have 
a PC-speaker beep, you need to go in the Control Panel, go in the Sounds 
applet, and put the "Exclamation" and "Default Beep" to None. Playing 
Windows default wav files reduces drastically LogIDS performance).

ALERT (2) : A bit more drastic, ALERT will behave in the same way, but with 
three beeps instead of one. If you have an application that reports large 
volume of data in a short amount of time, and you want sound alerts on 
these items, maybe you'll find the rule WARNING is more appropriate than 
ALERT, because the system can make a lot of noise if too many ALERT items 
are reported simultaneously. Using reports from LogAgent 5.x Pro or within 
LogIDS can also help in reducing the amount of events that will trigger 
sounds.

ICON= (2) : Another easy one is ICON=, where you can specify a 40 X 40 
pixels icon to depict graphically the event the rule just checked true. For 
example, it could be the picture of a virus icon when the antivirus reports 
a virus, or a "virus cleaned" icon with the antivirus reports that the 
virus has been cleaned. I have made several icons depicting various kind of 
events that could be reported with the programs I was playing with when 
developing LogIDS, feel free to make your own depicting events that are 
related to your own environment. The bitmaps have to be located in the 
\logids\bmp\ folder. Feel free to send them to me so I can share them with 
the rest of the community on my website.

DISPLAY= (2) : Another important action statement is DISPLAY=, where you 
get to choose which fields from your log record you find relevant to print 
in the text window associated with the reporting machine on the LogIDS GUI. 
You simply put the list of fields you want to be printed and by separating 
them with #, in the order of appearance that you wish to dispose them. If 
you want to display the whole record with the fields in their original 
order, simply leave the list empty. DROP is used for the cases where you 
make a rule to determine non-important data that you don't necessarily want 
to be displayed in the graphical interface, but you still want to keep an 
eye on it in the case it could be useful. DROP will print the whole record 
on the LogIDS console, the DOS-looking window behind the graphical console. 
REJECT can be used in the same conditions, but with REJECT the record will 
not get printed to the console. Only a notification of a REJECTed record 
will be displayed on the console.

PRIORITY= (*3) : This action item allows you to specify a priority value to 
the log line being analyzed, in case such a value is not already present by 
the generating application. This allows you to give some level of custom 
definition over what this data actually means to you. Note that the 
PRIORITY= action actually appends a new field at the end of the existing 
ones, and you can find this data in your final log files in 
\logids\log\backup\. Since LogAgent/LogIDS tests each rule one after 
another for all possible matches, this also enable you to use this value 
from this point on in your rulebase just like any other value. If there is 
already a Priority field in the log format, but also want to add your own 
priority token to the log without overwriting the original one, simply give 
the application's Priority field a somewhat different name in filter.txt, 
like "apppriority" for example, to avoid overlapping. You can unse the same 
trick to assign a Priority value from LogAgent, and another different one 
from LogIDS, each Priority token potentially having different meanings.

USERMESSAGE= (*3) : This action item allows you to specify a custom message 
to the log line being analyzed. This allows you to give some level of 
custom definition over what this data actually means to you. Note that the 
USERMESSAGE= action actually appends a new field at the end of the existing 
ones, and you can find this data in your final log files in 
\logids\log\backup\. Since LogIDS tests each rule one after another for all 
possible matches, this also enable you to use this value from this point on 
in your rulebase just like any other value. Note that if you plan to pass a 
custom message value as a parameter to an external program with EXEC, you 
should put your message between double-quotes "", or else the receiving 
application may not receive the complete information if the value contains 
empty spaces. Similarly, if you plan to put commas ',' in your message, 
make sure that this field is the last one appended to your logs by 
LogAgent/LogIDS, so that LogIDS can reconstruct the logs normally despite 
the use of comma-delimited fields. If you have defined a USERMESSAGE value 
with LogAgent and still wish to add another one from within LogIDS, simply 
use the same trick as with PRIORITY=.

EXEC= (*3) : This action allows you to launch external programs when the 
condition is met. You simply put the command line with arguments after the 
'=' sign. If you wish to pass values from your log to the external program 
as arguments, simply precede the field name with a $, as it is the 
convention in Perl for variable identifiers. If you need to pass an 
argument that really begins with the character $, then simply double-it 
like this $$. This can be useful to launch a counter-measure program (like 
nmap, for example, to gather more info from a suspect machine), hook 
LogAgent/LogIDS with the Phone Dialer to reach you on your pager, or even 
to simply launch a batch file that will automate all these tasks at once.

CUMULxx= (*3) : This is one of the most interesting and powerful action 
item in terms of analysis depth and flexibility over the previous versions. 
CUMULxx is actually a set of 100 accumulator variables, or buffers if you 
prefer, where xx is the index number of the accumulator variable. So, 
CUMULxx buffers are accessible by calling them with their index number like 
this: CUMUL00, CUMUL01, ..., CUMUL99. What makes CUMULxx so interesting is 
that it sports several properties that can be easily accessed from within 
the rulebase, mostly by simply refering to the index of the variable. The 
first use of CUMULxx is to store log lines into a buffer for further usage. 
Just like with DISPLAY= or LOG=, you can choose to specify or not which 
fields you are most interested into, separated by #.

Now, what makes CUMULxx so interesting, it's the fact that it is not only 
an action item, but it is also a variable as well, which means that some 
manipulations can be done to it too. To access the data within a CUMULxx 
variable, you simply need to use it as a regular field value in your 
condition statement ("expr1" in "expr1 cond expr2"). There are 2 types of 
condition checking you can do on a CUMULxx buffer variable : the number of 
log lines stored in the buffer, and the time since the first record was 
stored. With these 2 types of checks, you can call a variety of actions 
that will work specifically for the CUMULxx variable. To check a CUMULxx 
variable against a threshold value, as a condition you simply type "CUMULxx 
eq X", where "xx" is the index of the variable, and "X" the threshold value 
(i.e. when the variable CUMULxx contains X lines in its buffer, the rule 
will check true). To test against a certain amount of time since the first 
record was stored, as a condition you simply type "TIMERxx > X", where "xx" 
is the index of the variable, and "X" the amount of time after which a 
certain action is to be taken. The use of the word TIMER instead of CUMUL 
tells LogAgent that we are interested in the time aspect on the CUMULxx 
variable, but the xx actually indicates which variable we are referring to. 
It is recommended to always use '>' for TIMERxx as a condition check, for 
the following reasons. First, specifying '<' will always check true, and 
you won't amount much data in your CUMULxx variable before it triggers. 
Second, since LogIDS performs no regular check on the files (no polling), 
rules are checked only when a log file actually receives some data. So, if 
you use '=' as a check, in the case an attack produces some logs, but not 
enough to trigger the CUMULxx threshold, it will take a log item being 
produced at the exact same second that you are performing an '=' check 
onto, which is statistically almost impossible. By specifying '>', the 
first line submitted to the analysis engine after the time period has 
expired will trigger the TIMER true. Note that you can combine a CUMULxx 
and a TIMERxx together on the same condition with a OR mega-condition, at 
the condition that 'xx' is the same for both. Also note that if you want to 
mix a CUMULxx or TIMERxx variable with another condition check, then 
CUMULxx or TIMERxx have to be the first item mentionned in the condition. 
It is not recommended mixing checks for multiple CUMULxx variable on the 
same conditions, since only the index of the first item in the line is 
considered. This may be subject to further development if I see it as 
somehow being useful.

The following action items actually applies to CUMULxx variables only, both 
from LogAgent Pro and LogIDS. They are called by having a CUMULxx or 
TIMERxx variable in a condition check. Note that rules BACKUP, IGNORE, 
WARN, ALERT, ICON, DROP and REJECT can also be applied to CUMULxx 
variables. LOG and DISPLAY RE replaced by the three report-producing 
functions for each program we will see shortly.

PRIORITY= (*3) : This works the same as the one described above, but when 
applied to a CUMULxx variable, it is the whole variable that gets assigned 
a priority, or if you prefer the reports based on this CUMULxx variable.

USERMESSAGE= (*3) : This works the same as the one described above, but 
when applied to a CUMULxx variable, it is the whole variable that gets 
assigned a custom message, or if you prefer the reports based on this 
CUMULxx variable.

EXEC= (*3) : This works the same as above, except that you can't pass 
individual log values as arguments to the external program. Instead, you 
can use some pre-defined fields contained in the CUMULxx variable used to 
produce the reports. These pre-defined values are : ip, host, priority, 
usermessage, rules, fields and buffer. These elements are used by LogIDS to 
produce reports, and can be sent out to external programs as arguments if 
you wish. Launching external programs with CUMULxx variables may be more 
efficient than with single log lines in some scenarios, so the flexibility 
is provided here.

LONG (*3) : This will prepare a report, long format, and display it on the 
console in the appropriate window (LogIDS), or send it to the file 
logsummary.log (LogAgent Pro). The output is also sent in the file 
logsummary.log in the \logids\log\backup folder. The long report format 
means that the whole content of the buffer is in the report. This can be 
useful when using low thresholds or low timer values.

MEDIUM (*3) : This will prepare a report, medium format, and display it on 
the console in the appropriate window (LogIDS), or send it to the file 
logsummary.log (LogAgent Pro). The output is also sent in the file 
logsummary.log in the \logids\log\backup folder. The medium report format 
means that roughly one third of the content of the buffer (more or less one 
line) is in the report. This can be useful when using relatively high 
thresholds.

SHORT (*3) : This will prepare a report, short format, and display it on 
the console in the appropriate window (LogIDS), or send it to the file 
logsummary.log (LogAgent Pro). The output is also sent in the file 
logsummary.log in the \logids\log\backup folder. The short report format 
means that only the first and last line of the buffer are in the report. 
This can be useful when using very high thresholds (such as for gathering 
items related to a network port scan, for example).

MAILLONG= (*3) : This will prepare a report, long format, and send it by 
e-mail to the specified destination(s) after the '='. You specify e-mail 
destinations by putting a list of couples e-mail addresses and SMTP 
servers, separated by commas. Each e-mail address must be followed by a 
SMTP server address before specifying another e-mail address, even if they 
are using the same SMTP server. The long report format means that the whole 
content of the buffer is in the report. This can be useful when using low 
thresholds or low timer values.

MAILMEDIUM= (*3) : This will prepare a report, medium format, and send it 
by e-mail to the specified destination(s) after the '='. You specify e-mail 
destinations by putting a list of couples e-mail addresses and SMTP 
servers, separated by commas. Each e-mail address must be followed by a 
SMTP server address before specifying another e-mail address, even if they 
are using the same SMTP server. The medium report format means that roughly 
one third of the content of the buffer (more or less one line) is in the 
report. This can be useful when using relatively high thresholds.

MAILSHORT= (*3) : This will prepare a report, short format, and send it by 
e-mail to the specified destination(s) after the '='. You specify e-mail 
destinations by putting a list of couples e-mail addresses and SMTP 
servers, separated by commas. Each e-mail address must be followed by a 
SMTP server address before specifying another e-mail address, even if they 
are using the same SMTP server. The short report format means that only the 
first and last line of the buffer are in the report. This can be useful 
when using very high thresholds (such as for gathering items related to a 
network port scan, for example).

FLUSH (*3) : This is used to flush the content of the CUMULxx variable, and 
reset its associated counters. This is because the rule system is 
sequential, and each rule that matches true is applied. This his how you 
can append data with PRIORITY and USERMESSAGE. This way, you can also send 
multiple reports with the same CUMULxx content before dumping it from the 
system memory. Once FLUSH is met in the rulebase and it matched true, any 
further reference to this CUMULxx variable in the rulebase will be done on 
NULL values. For this reason, FLUSH should be one of the last rule you put 
for each application you define rules for.

Thank you for your time, and don't hesitate to send your feedback towards 
these tools and the other on my website at securit () iquebec com.

Adam Richard, aka Floydman
SécurIT Informatique Inc. 

---------------------------------------------------------------------------
---------------------------------------------------------------------------