Ask or search…
K

Log Optimization

Overview

Nimbus analyzes all your logs and finds high volume log patterns based on incoming data. From these patterns, Nimbus generates transformations that aggregate related logs into a single event.
We refer to this style of transformation as lossless aggregation. You can see an example of how this works below.
Lossless Aggregation

Optimization Triggers

Nimbus can automatically optimize logs when it detects the following situations:
  1. 1.
    Logs with common message patterns
  2. 2.
    Logs with common identifiers
  3. 3.
    Multi-line Logs
For before and after examples of these triggers, see examples.

Logs with common message patterns

These are high volume log events that repeat most of their content. For most applications most of the time, this will be the primary driver of log volume. Examples include health checks and heart beat notifications.

Logs with common identifiers

These are logs that describe a sequence of related events. These sequences usually have some sort of common identifier like a transactionId or a jobId. Examples include a background job and business specific user flows.

Multi-line Logs

These are logs where the message body can be spread across multiple new lines. Unless you add special logic on the agent side, the default behavior is to emit each newline delimited message as a separate log event.

Optimization Dimensions

Nimbus optimizes logs across the following dimensions:
  1. 1.
    Volume: Optimize to reduce the number of events logged
  2. 2.
    Size: Optimize to reduce the size of events logged
For before and after examples of optimizations along these dimensions, see examples.

Volume

When optimizing for volume, Nimbus aggregates as many logs as it can given the constraints of the destination.
For example, datadog has specific limits around total array size as well as log size. Nimbus makes sure to aggregate underneath this limit to maximize volume reduction.

Size

When optimizing for size, Nimbus deduplicates and removes redundant metadata as it aggregates logs.
For example, when aggregating logs with common message patterns, its often the case that 40% or more of the metadata (tags and attributes) are the same.

Optimization Fidelity

Nimbus generated optimizations can be tuned via fidelity levels to indicate how much of the original log message to preserve.

High

Nimbus optimizes for preserving original log data with perfect fidelity. This means there is no reduction in ingest size and aggregated logs contain all fields of the original log entires with only identical fields deduplicated.

Medium

Nimbus preserves most of the data. Individual timestamps in aggregated logs are discarded.

Low

Nimbus optimizes for ingest size. Low value fields are nominated for removal. All nimbus attributes except nimsize are removed from the resulting log.
Last modified 7d ago