Re: Operational question: formatting Splunk alerts

[Garrett Hildebrand <gdh () UCI EDU>]

Emily,

> The intended audience for this alert is someone in payroll.  It needs to be really clear - it should say person A, 
> located in geographical
> location B, changed their bank information on this date, at this time, using that type of device.

That should not be too hard to do. I have an example I created that might serve to guide you.

Background: I take logs from Thychotic's Secret Server. I wanted
to be able to detect when someone locked themselves out by crossing
the threshold of allowed failed login attempts -- where the account
is auto-locked.

So, I created a search for that which returns various pieces of
information from the logs in the form of a table. The email generated
contains the table, but it also contains this message:

$result.username$, Splunk has a locked-out condition in Secret Server for user $result.UCInetID$, which means your account is locked. You can try 
using the "Forgot your password?" link to obtain a password reset link. If this does not work, you'll need to submit a Service Now 
Secret Server Request asking for password reset. You'll find this in Service Catalogs -> Catalog -> Information Security -> Secret
Server Request. If you do not believe you are responsible for the login failures leading to this lockout, please 
contact security () uci edu right away, referencing this alert by name.

Each of these $result.somefieldname$ meta references refer to a
field in the search results. Taking the first one as an example,
one of the columns is called 'username', which is the first and last
(and sometimes middle) name of the user that got locked out. By
wrapping it with $result.$ the email sent by the alert opens the
message with the recipient's name.

Even more cool is that this alert is dynamically sent to the user
in question. How? This same technique can be used in the "To"
box of the alert.

In the search one of the fields we mine out of the log entry is
the UCInetiD, which is a unique ID that everyone has at UCI. Even
if a user has a vanity email address, combining their UCInetID
with @uci.edu creates a deliverable email address. So, in the
search:

| eval recipient = UCInetID . "@uci.edu"

Then, in the "To" box of the alert we simple put $result.recipient$
as the email recipient.

Then, in the cc box, we have the names of the security people who
are likely to be one of the responders to the Service Now request
when it comes in later.

Garrett