Working with Aggregated Logs

Aggregated logs are just regular logs with specific nimbus attributes.

The individual payload of the pre-aggregated logs can be found in the nimdata field which is an array of the underlying log events.

The message field is an array of the original log bodies

Search

When searching for values within a JSON array, use the same syntax as when searching a regular property.

{
    message: ["item 123 refreshed", "item 345 refreshed", "item 567 error"],
    jobId: 1,
    nimdata: [
        {
            jobId: 1,
            message: "item 123 refreshed",
            category: "luxury"
        },
        {
            jobId: 1,
            message: "item 345 refreshed",
            category: "toys"
        },
        {
            jobId: 1,
            message: "item 567 refreshed",
            category: "luxury"
        }
    ]
    ...
}

For example, to find log messages with "error", you can use the following search

"error"

Searching for values within a JSON array of objects, you can use the following search

@nimdata.category:"luxury"

Monitors

Nimbus is compatible with existing log monitoring setups. We'll walkthrough three common scenarios below and how monitors would behave after Nimbus:

Error Monitors

These are monitors that alert based on logs with errors. Error logs are automatically detected by Nimbus and go through a separate pipeline that bypasses aggregation. This means any monitors on error logs will be unaffected.

Count based monitors

These are monitors that measure the number of logs during a set interval. You can retrieve the original size of of pre-aggregated logs by using Sum of @nimsize instead of Count of All Logs.

Original Monitor based on Count

Original Monitor based on Count

Aggregated Monitor based on @nimsize

Aggregated Monitor based on @nimsize

Attribute based monitors

These are monitors that depend on a specific attribute within the aggregated log. You can either modify the monitor to alarm based on the nested attribute or use the pull_up directive to keep attributes that you alarm on at the top level.

Dashboards

All instructions for monitors also apply to dashboards.