Reduce Optimizations
Reduce optimizations reduce the volume of your logs, either along number of events or raw ingested bytes.
Nimbus analyzes all your logs and finds high volume log patterns based on incoming data. From these patterns, Nimbus generates transformations that aggregate related logs into a single event.
We refer to this style of transformation as lossless aggregation. You can see an example of how this works below.
Optimization Triggers
Nimbus can automatically optimize logs when it detects the following situations:
Logs with common message patterns
Logs with common identifiers
Multi-line Logs
For before and after examples of these triggers, see examples.
Logs with common message patterns
These are high volume log events that repeat most of their content. For most applications most of the time, this will be the primary driver of log volume. Examples include health checks and heart beat notifications.
Logs with common identifiers
These are logs that describe a sequence of related events. These sequences usually have some sort of common identifier like a transactionId or a jobId. Examples include a background job and business specific user flows.
Multi-line Logs
These are logs where the message body can be spread across multiple new lines. Unless you add special logic on the agent side, the default behavior is to emit each newline delimited message as a separate log event.
Optimization Dimensions
Nimbus optimizes logs across the following dimensions:
Volume: Optimize to reduce the number of events logged
Size: Optimize to reduce the size of events logged
For before and after examples of optimizations along these dimensions, see examples.
Volume
When optimizing for volume, Nimbus aggregates as many logs as it can given the constraints of the destination.
For example, datadog has specific limits around total array size as well as log size. Nimbus makes sure to aggregate underneath this limit to maximize volume reduction.
Size
When optimizing for size, Nimbus deduplicates and removes redundant metadata as it aggregates logs.
For example, when aggregating logs with common message patterns, its often the case that 40% or more of the metadata (tags and attributes) are the same.
Optimization Fidelity
Nimbus generated optimizations can be tuned via fidelity levels to indicate how much of the original log message to preserve.
High
Nimbus optimizes for preserving original log data with perfect fidelity. This means there is no reduction in ingest size and aggregated logs contain all fields of the original log entires with only identical fields deduplicated.
Medium
Nimbus preserves most of the data. Individual timestamps in aggregated logs are discarded.
Low
Nimbus optimizes for ingest size. Low value fields are nominated for removal. All nimbus attributes except nimsize are removed from the resulting log.
Last updated
