Aws Cloudwatch Kinesis Setup

LogZilla documentation for Aws Cloudwatch Kinesis Setup

AWS CloudWatch and Kinesis Data Firehose

AWS CloudWatch logs can be forwarded to LogZilla using Kinesis Data Firehose over HTTP/HTTPS. Firehose delivers logs to LogZilla's /firehose endpoint with authentication via access tokens.

Prerequisites

  • A LogZilla access token for Firehose authentication (see Getting Started).
  • AWS account with CloudWatch logs and Kinesis Data Firehose access.
  • Network access from AWS to the LogZilla server on HTTP/HTTPS ports.

Configure Kinesis Data Firehose

Create delivery stream

  1. In the AWS Console, navigate to Kinesis Data Firehose.
  2. Select Create delivery stream.
  3. Set:
    • Source: Direct PUT or Kinesis Data Streams
    • Destination: HTTP Endpoint
  4. Enter a delivery stream name (for example, logzilla-firehose).

Configure destination settings

Set the HTTP endpoint details:

  • HTTP endpoint URL: https://LOGZILLA_HOST/firehose
  • Access key: the LogZilla access token
  • Content encoding: GZIP (recommended)
  • Retry duration: 3600 seconds (or as required)

Configure backup settings

Select an S3 bucket for failed delivery backup. This is required by Firehose for error handling and replay.

Create the stream

Review settings and create the delivery stream. AWS will validate the endpoint and begin processing.

Connect CloudWatch logs

Create subscription filter

  1. In CloudWatch, select the log group to forward.
  2. Create a subscription filter:
    • Destination: the Kinesis Data Firehose delivery stream
    • Filter pattern: leave empty to forward all logs, or specify a pattern
    • Role: create or select an IAM role with Firehose write permissions

Verify delivery

Check LogZilla reception

Search LogZilla for events with:

  • Program: kinesis (default program for Firehose events)
  • Recent timestamps matching the CloudWatch log entries

Test with curl

Send a test event to verify the /firehose endpoint:

bash
url="https://LOGZILLA_HOST/firehose"
apikey="YOUR_TOKEN_HERE"

# Create test payload (base64-encoded gzipped data)
test_data=$(echo "Test CloudWatch log entry" | gzip | base64 -w 0)

curl -X POST "$url" \
  -H "Content-Type: application/json" \
  -H "X-Amz-Firehose-Access-Key: $apikey" \
  -d "{\"requestId\": \"test-123\", \"records\": [{\"data\": \"$test_data\"}]}"

The test event should appear in LogZilla with program kinesis.

Troubleshooting

Common issues

  • 403 Forbidden: Check the access token and ensure it matches the Firehose configuration.
  • 404 Not Found: Verify the endpoint URL uses /firehose (not /incoming).
  • No events in LogZilla: Check CloudWatch subscription filter and Firehose delivery metrics in AWS.

Firehose delivery metrics

Monitor Firehose delivery in the AWS Console:

  • Delivery success/failure rates
  • Error logs for failed deliveries
  • S3 backup bucket for failed events

Network troubleshooting

For packet-level checks, see Syslog Troubleshooting.

AWS VPC Flow Logs (optional)

For AWS VPC Flow Logs specifically, install the AWS CloudWatch VPC Flow app from the LogZilla appstore (Settings → App store). This app provides specialized parsing rules and dashboards for VPC flow log data.

Related topics

Aws Cloudwatch Kinesis Setup | LogZilla Documentation