How to deal with 36TB/day of Network analytics.

Stop storing data you don't need

Clayton Dukes, CEO

How to deal with 36TB/day of Network analytics.

Thursday, August 09, 2018

LogZilla’s preduplication™ engine enables events to be forwarded and is a vital component for large network data management. This module allows customers to specify a “hold timer” (and optional match of specific events) which will forward a single event to a downstream receiver along with metadata about the number of times it happened in a given time period.

LogZilla Pre-Dup™ Algorithm and Process

LogZilla Predup Module

In large networks, this capability saves companies millions due to the fact that when things go wrong, Network Devices, Firewalls, Servers, and Applications all generate more events more often. Because the original design for syslog was UDP based, it also means they will send the same event repeatedly in the hopes that “something” is listening.

Take the following real-world “event storm” example:

Event Storm

As seen, almost 1 Billion events were generated in a very short time.

On the right side, LogZilla shows 70k to 90k of the same events being generated every minute.

By using the LogZilla forwarding module, companies can still generate the information needed by downstream receivers without fear of degrading performance of those systems.

In the case of the event storm above, the downstream receivers would have received 4 events instead of 308,642, but those 4 events would have had a count of the number of times the event was generated.

For example, if this were set up to forward as an SNMP Trap to Dell/EMC Smarts, then Smarts would only need to process 4 events similar to the one below:

SNMPv2-MIB::snmpTrapOID.0 = UCD-SNMP-MIB::ucdavis.991	SNMPv2-SMI::enterprises. = "10"	SNMPv2-SMI::enterprises. = 5	SNMPv2-SMI::enterprises. = ""	SNMPv2-SMI::enterprises. = "%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on GigabitEthernet1/0/26 (not half duplex), with CE10-M.lab Ethernet1/0 (half
 duplex)."	SNMPv2-SMI::enterprises. = 70822

Note: The OIDs used in this example may be set in the configuration of the forwarder so that customers are not limited in which OIDs must be used.

In large networks, LogZilla’s preduplication™ engine enables events to be forwarded and is a vital component for large network data management and saves companies millions of dollars per year.

Clayton Dukes

Clayton Dukes


4819 Emperor Boulevard Suite 400
Raleigh, NC,27703

About Clayton

Clayton Dukes leverages over two decades of experience in network systems design, implementation, and management. Early years included designing an open source solution to solve network event management challenges as a Datacenter Lead Engineer at Cisco, which and ultimately led to a later-creation of the LogZilla Network Event Orchestrator platform. Dukes has co-authored the CCIE SP OPS certification and resides in North Carolina
Tags: Smarts , Scalability , Predup , Dedup