BlogZilla - Preduplication, Not Deduplication

Preduplication, Not Deduplication

Stop storing data you don't need.

How to deal with 10TB/day of Network analytics.

Share this:

LogZilla’s ability to forward events after they’ve been deduplicated is a vital component for large network data management. This module allows customers to specify a “hold timer” (and optional match of specific events) which will forward a single event to a downstream receiver along with metadata about the number of times it happened in a given time period.

LogZilla Dedup

LogZilla Dedup Module

In large networks, this capability saves companies millions due to the fact that when things go wrong, Network Devices, Firewalls, Servers, and Applications all generate more events more often. Because the original design for syslog was UDP based, it also means they will send the same event repeatedly in the hopes that “something” is listening.

Take the following real-world “event storm” example:

Event Storm

Duplex Storm

As seen, almost 1 Billion events were generated in a very short time.

On the right side, LogZilla shows 70k to 90k of the same events being generated every minute.

By using the LogZilla forwarding module, companies can still generate the information needed by downstream receivers without fear of degrading performance of those systems.

In the case of the event storm above, the downstream receivers would have received 4 events instead of 308,642, but those 4 events would have had a count of the number of times the event was generated.

For example, if this were set up to forward as an SNMP Trap to Dell/EMC Smarts, then Smarts would only need to process 4 events similar to the one below:

SNMPv2-MIB::snmpTrapOID.0 = UCD-SNMP-MIB::ucdavis.991	SNMPv2-SMI::enterprises.9.9.41.1.2.3.1.2.0 = "10"	SNMPv2-SMI::enterprises.9.9.41.1.2.3.1.3.0 = 5	SNMPv2-SMI::enterprises.9.9.41.1.2.3.1.4.0 = ""	SNMPv2-SMI::enterprises.9.9.41.1.2.3.1.5.0 = "%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on GigabitEthernet1/0/26 (not half duplex), with CE10-M.lab Ethernet1/0 (half
 duplex)."	SNMPv2-SMI::enterprises.9.9.41.1.2.3.1.99.0 = 70822

Note: The OIDs used in this example may be set in the configuration of the forwarder so that customers are not limited in which OIDs must be used.