Event Enrichment

This LogZilla App allows users to enrich incoming events matching a specified criteria (such as a string or a pattern). The typical use for this feature is to add metadata from external sources of information such as a Configuration Management Database (CMDB)

Once the events are enriched, the metadata will be available as User Tags.

App Installation and Configuration

Step 1 - Install

Click Install to install this application

Step 2 - Configure

SSH to the server and edit /etc/logzilla/apps/event_enrichment/config/config.yaml with the samples below.

NOTE: You must have root permission in order to edit these files.

config.yaml

yaml
---
- name: Simple Host Lookup
  description: |
    Used to add device type and location information
    Referenced by host
  metadata_file: metaData
  lookup_field: host

The description element is not required; it can be provided for purposes of documenting the event enrichment process, for your own reference.

There are four constituent elements of the config.yaml file: filter, lookup_field, lookup_re, and apply_to_message. They are defined in a separate section as explained below.

Step 3 - Add Metadata

Create /etc/logzilla/apps/event_enrichment/config/metaData.yaml with the following sample data (or modify as needed to suit your environment).

In the example below, each IP address is the start of the meta match from the rule specified above (lookup_field: host).

metaData.yaml

yaml
---
"246.219.157.165":
  DeviceID: "1001590"
  Wan-Interface: GigabitEthernet1/0/1
  Device-Role: "DC"
"107.122.210.185":
  DeviceID: "3001590"
  Wan-Interface: GigabitEthernet1/0/3
  Device-Role: "Core"
"CE5-G":
  DeviceID: "3301590"
  Wan-Interface: Ethernet0/1
  Device-Role: "Customer Edge"
"PE4":
  DeviceID: "2401590"
  Wan-Interface: Ethernet0/3
  Device-Role: "Provider Edge"

Matching by network (CIDR)

If you're using CIDR matching, then you have to use CIDR specifications as keys in your metadata. All matching networks will be merged, so for example for this metadata if you match IP "192.168.5.10" then you'll get tags {"foo": "f1", "bar": "b5"} - as it will first match more generic 192.168.0.0/16, and then match for 192.168.5.0/24 will override tag foo.

yaml
---
"192.168.0.0/16":
  foo: f0
  bar: b5
"192.168.5.0/24":
  foo: f1

If you want to match single IP, then use mask /32.

Step 4 - Enable New Config

Reload the new rules by running logzilla rules reload (or sudo logzilla rules reload if you are not logged in as root)

Elements of `config.yaml`

There are four constituent elements of the config.yaml file: filter, lookup_field, lookup_re, and apply_to_message.

`filter`

Filters provide the option to limit data enrichment to matched criteria.

Sample: match on incoming hosts, with an optional message filter

As an example, the following filter would allow setting event enrichment for hosts listed in the MetaData file (see below), but the filter adds additional criteria indicating that enrichment will only be applied if the event matches the specified host and contains foo anywhere in the message.

yaml
---
- name: Host Lookup with pre-filter
  metadata_file: metaData
  lookup_field: host
  filter:
  - field: message
    op: "=*"
    value: foo

Note that the eq field allows for different types of filter operators. Operators control the way the filter's match condition is applied. If no op is supplied, the default operator eq is assumed.

Operator	Match Type	Description
eq	String or Integer	Matches entire incoming message against the string/integer specified in the `match` condition
ne	String or Integer	Does not match anything in the incoming message `match` field.
gt	Integer Only	Given integer is greater than the incoming integer value
lt	Integer Only	Given integer is less than the incoming integer value
ge	Integer Only	Given integer is greater than or equal to the incoming integer value
le	Integer Only	Given integer is less than or equal to the incoming integer value
=~	RegEx	Match based on RegEx pattern
!~	RegEx	Does not match based on RegEx pattern
=*	RegEx	RegEx appears anywhere in the incoming message

For the field value (in the filter) typically you would use either host for the originating host of the log message, as per the incoming syslog message, or you would use message for the message text of the log message.

You can see more about the allowed operators and fields in help section 10. Data Transformation / 1. Rewrite Rules.

`lookup_field`

lookup_field works the same way as the filter / field parameter we just explained, in that it corresponds to either the host of the log message or the message text of the log message. However adding lookup_ to the field specifier is indicating that for our event enrichment operation, we want to use that particular field (either host or message) to reference in our metadata file which particular record or bit of information should match this log message.

For example, if our config.yaml uses host for the lookup_field, as follows:


---
- name: Simple Host Lookup
  metadata_file: metaData
  lookup_field: host

that means that our metaData.yaml file will contain a particular individual record for each log message host.

For example, let's say that we are receiving events such as the following: Sample Events Display with Hosts

Since we are using lookup_field: host in our config.yaml, our metaData.yaml would have data such as the following:


---
"plqzobph.lzdemo":
  Description: web server
  Location: building A
"160.197.58.38":
  Description: Infoblox device
  Location: building B
"LZU-INTERNALFW-ASACL01":
  Description: Cisco ASA
  Location: building C

If you want to use part of the log message text as the reference to your event enrichment data, you will likely want to use the process outlined in the next section about lookup_re.

`lookup_re`

lookup_re stands for "lookup regular expression", and it allows you to pull data from the log message text and use that as the reference into the event enrichment data. The way this works is that you specify a regular expression capture expression to extract information from the log message, and then that extracted information is used as the lookup into the metaData.yaml data. Note that a lookup_re applies to the lookup_field (not a filter)

For example, let's say that our log messages contain IP addresses in the message text body. We want to lookup additional information pertaining to those IP addresses. So we need to create a regular expression to extract the IP address from the message body, as follows:


---
- name: Host Lookup with message IP pattern matching
  metadata_file: metaData
  lookup_field: message
  lookup_re: "(\\d+\\.\\d+\\.\\d+\\.\\d+)"

As the above IP matching is very common, you can use special tag instead: %IP_REGEX%, so these two are equivalent:

  lookup_re: "(\\d+\\.\\d+\\.\\d+\\.\\d+)"

  lookup_re: "(%IP_REGEX%)"

Please note that either of that regexes doesn't check for the actual range of the octet values, so it will also match strings like 275.341.5.800 or even 01234567.8.9999.33333.

This example will search the message text body for text that matches the given regular expression, in this case an IP address. That IP address will then be used as the lookup field into the metadata:


---
"192.168.0.1":
  User: John Doe
  Department: Administration
"192.168.0.2":
  User: Jane Foo
  Department: HR
"192.168.0.3":
  User: Michael Bar
  Department: IT

so if the log message were something like:

text
HTTP request from 192.168.0.2 to http://banned.com

our regular expression would extract "192.168.0.2" as the IP address for use as the lookup into the metadata, and then set user tags User = Jane Foo and Department = HR.

Sample: Using network (CIDR) matching

If you have a lot of data for your network IPs and they can be assigned to the whole networks - then you can use CIDR matching and often simplify your metadata a lot.

For this first you have to enable this with the use_cidr_match: true option:

text
---
- name: CIDR matching
  metadata_file: metaData
  lookup_field: host
  use_cidr_match: true

Then you have to provide only CIDR in your metadata file - see matching-by-network-cidr for example. In this example all values of the host field will be checked if then are contained in any of the provided networks - and if so, appropriate tags will be added to the event. All matches will be applied, in the order from most generic to most specific ones. See matching-by-network-cidr for example.

Sample: Using network (CIDR) with regex

You can combine both methods and use cidr matching with the regex on e.g. message field - for this use the config.yaml like the above:

text
---
- name: CIDR matching with regex
  metadata_file: metaData
  use_cidr_match: true
  lookup_field: message
  lookup_re: "src_ip=(%IP_REGEX%)"

With such configuration only IPs occurring in the message field after src_ip= will be looked up for the matching CIDRs.

Sample: Using multiple metadata files

text
---
- name: Simple Host Lookup
  metadata_file: metaData1
  lookup_field: host
- name: Simple Program Lookup
  metadata_file: metaData2
  lookup_field: program

`apply_to_message`

apply_to_message specifies whether the message text of the event should be modified to add the looked-up metadata to the message text itself. If this is false, the message is left unmodified. However if this is set to true, then each of the user tags (that is specified in the matching record in the metadata file) is added to the message text as a key-value pair.

In our example from above, let's say we expect messages such as:

text
HTTP request from 192.168.0.2 to http://banned.com

so we set our config.yaml as follows:

text
---
- name: Host Lookup with message IP pattern matching
  metadata_file: metaData
  lookup_field: message
  lookup_re: "(\\d+\\.\\d+\\.\\d+\\.\\d+)"
  apply_to_message: true

```text

and our `metaData.yaml` as follows:

```text
---
"192.168.0.1":
  User: John Doe
  Department: Administration
"192.168.0.2":
  User: Jane Foo
  Department: HR
"192.168.0.3":
  User: Michael Bar
  Department: IT

Then our event enrichment process will modify the message text body to be

text
HTTP request from 192.168.0.2 to http://banned.com "User"="Jane Foo" "Department"="HR"

so that when you see the event in the event list in the LogZilla UI, you can see at a glance the extra information that has been added to the message itself.