Reduce Optimizations
Last updated
Was this helpful?
Last updated
Was this helpful?
Reduce optimizations reduce the volume of your logs, either along number of events or raw ingested bytes.
Nimbus analyzes all your logs and finds high volume log patterns based on incoming data. From these patterns, Nimbus generates transformations that aggregate related logs into a single event.
We refer to this style of transformation as lossless aggregation. You can see an example of how this works below.
Nimbus can automatically optimize logs when it detects the following situations:
Logs with common message patterns
Logs with common identifiers
Multi-line Logs
These are high volume log events that repeat most of their content. For most applications most of the time, this will be the primary driver of log volume. Examples include health checks and heart beat notifications.
These are logs that describe a sequence of related events. These sequences usually have some sort of common identifier like a transactionId or a jobId. Examples include a background job and business specific user flows.
These are logs where the message body can be spread across multiple new lines. Unless you add special logic on the agent side, the default behavior is to emit each newline delimited message as a separate log event.
Nimbus optimizes logs across the following dimensions:
Volume: Optimize to reduce the number of events logged
Size: Optimize to reduce the size of events logged
When optimizing for volume, Nimbus aggregates as many logs as it can given the constraints of the destination.
When optimizing for size, Nimbus deduplicates and removes redundant metadata as it aggregates logs.
Nimbus generated optimizations can be tuned via fidelity levels to indicate how much of the original log message to preserve.
Nimbus optimizes for preserving original log data with perfect fidelity. This means there is no reduction in ingest size and aggregated logs contain all fields of the original log entires with only identical fields deduplicated.
Nimbus preserves most of the data. Individual timestamps in aggregated logs are discarded.
For before and after examples of these triggers, see .
For before and after examples of optimizations along these dimensions, see .
For example, has specific limits around total array size as well as log size. Nimbus makes sure to aggregate underneath this limit to maximize volume reduction.
For example, when aggregating , its often the case that 40% or more of the metadata (tags and attributes) are the same.
Nimbus optimizes for ingest size. Low value fields are nominated for removal. All except nimsize are removed from the resulting log.
