LogZilla

Overview

NGINX is open-source web server software for web serving, reverse proxying, caching, load balancing, media streaming, and more. NGINX is widely used for high-performance web applications and microservices architectures.

App Function

The NGINX app processes web server logs and extracts user tags for web traffic analysis, performance monitoring, and security analysis.

Vendor Documentation

Incoming Log Format

NGINX uses space-separated values in its default log format. To use the LogZilla NGINX app, the log format must be customized to use key-value pairs as detailed in the Configuration section below.

The customized format provides structured data that enables comprehensive web traffic analysis and monitoring.

Configuration

NGINX requires configuration for both log format and destination. Ensure that include /etc/nginx/conf.d/*.conf; exists in the http {} section of /etc/nginx/nginx.conf.

Create /etc/nginx/conf.d/logging.conf with the following content:

nginx
# LogZilla Custom Log Format
# Requires Nginx >= v1.7.1

log_format logzilla 'Site="$server_name" Server="$host" DstPort="$server_port" '
               'DstIP="$server_addr" Src="$remote_addr" SrcIP="$realip_remote_addr" '
               'User="$remote_user" Time_Local="$time_local" Protocol="$server_protocol" '
               'Status="$status" Bytes_Out="$bytes_sent" '
               'Bytes_In="$upstream_bytes_received" HTTP_Referer="$http_referer" '
               'User_Agent="$http_user_agent" Nginx_Version="$nginx_version" '
               'HTTP_X_Forwarded_For="$http_x_forwarded_for" '
               'HTTP_X_Header="$http_x_header" URI_Query="$query_string" URI="$uri" '
               'HTTP_Method="$request_method" Response_Time="$upstream_response_time" '
               'Cookie="$http_cookie" Request_Time="$request_time" ';

  # Send logs to LogZilla Server
  access_log syslog:server=logzilla.abcd.com:514,tag=nginx_access logzilla;
  error_log syslog:server=logzilla.abcd.com:514,tag=nginx_error notice;

After making configuration changes, restart NGINX and verify log reception:

bash
sudo systemctl restart nginx
# or
sudo service nginx restart

Parsed Metadata Fields

User tag names match the field names defined in the configuration above. Underscores (_) in field names are converted to spaces ( ) in user tag names.

Tagged	Field Name	Example	Description
☑	`Site`	`logs.logzilla`	Site being accessed
☑	`Server`	`logs.logzilla`	Server hosting the site
	`DstPort`	`443`	Destination port for request
☑	`DstIP`	`18.68.11.35`	Destination IP for request
	`Src`	`16.56.15.199`	Source of request
☑	`SrcIP`	`16.56.15.199`	Source IP of request
	`User`	`johnd`	Username (if available)
	`Time_Local`	`03/Jan/2018:14:07:58 -0500`	Date and time of request
	`Protocol`	`HTTP/1.1`	Transport protocol
☑	`Status`	`200`	HTTP status code
	`Bytes_Out`	`3995`	Bytes sent for request
	`Bytes_In`	`-`	Bytes received for request
	`HTTP_Referer`	`-`	Referring URI
☑	`User_Agent`	`Mozilla/5.0`	User agent string
	`Nginx_Version`	`1.12.2`	NGINX version
	`HTTP_X_Forwarded_For`	`-`	Originating IP address
	`HTTP_X_Header`	`-`	Custom HTTP header
	`URI_Query`	`-`	Query string parameters
☑	`URI`	`/index.html`	Request URI
	`HTTP_Method`	`GET`	HTTP method
	`Response_Time`	`-`	Response time
	`Cookie`	`-`	Client cookies
	`Request_Time`	`0.000`	Request processing time

High-Cardinality (HC) Tags

SrcIP
DstIP
URI

Log Examples

Typical Page GET Request (200)

text
Site="localhost" Server="192.168.250.112” DstPort="80" DstIP="192.168.250.112" 
Src="192.168.250.2" SrcIP="192.168.250.2" User="-" 
Time_Local="17/Nov/2021:17:45:07 +0000" Protocol="HTTP/1.1" Status="304" 
Bytes_Out="189" Bytes_In="-" HTTP_Referer="-" User_Agent="Mozilla/5.0 (X11; 
Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0" Nginx_Version="1.18.0" 
HTTP_X_Forwarded_For="-" HTTP_X_Header="-" URI_Query="-" URI="/main.html" 
HTTP_Method="GET" Response_Time="-" Cookie="-" Request_Time="0.000"

Page POST Request (200)

text
Site="localhost" Server="192.168.250.112” DstPprt="80" DstIP="192.168.250.112" 
Src="192.168.250.2" SrcIP="192.168.250.2" User="-" 
Time_Local="17/Nov/2021:17:46:13 +0000" Protocol="HTTP/1.1" Status="200" 
Bytes_Out="331" Bytes_In="-" HTTP_Referer="http://192.168.250.112/poster.html" 
User_Agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 
Firefox/94.0" Nginx_Version="1.18.0" HTTP_X_Forwarded_For="-" HTTP_X_Header="-" 
URI_Query="" URI="/posted.html" HTTP_Method="GET" Response_Time="-" Cookie="-" 
Request_Time="0.000"

Page Not Found (404)

text
Site="localhost" Server="192.168.250.112” DstPort="80" DstIP="192.168.250.112" 
Src="192.168.250.2" SrcIP="192.168.250.2" User="-" 
Time_Local="17/Nov/2021:17:46:21 +0000" Protocol="HTTP/1.1" Status="404" 
Bytes_Out="326" Bytes_In="-" HTTP_Referer="-" User_Agent="Mozilla/5.0 (X11; 
Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0" Nginx_Version="1.18.0" 
HTTP_X_Forwarded_For="-" HTTP_X_Header="-" URI_Query="-" URI="/nothere.html" 
HTTP_Method="GET" Response_Time="-" Cookie="-" Request_Time="0.000"