LogoLogo
  • Overview
    • Introduction
    • Getting Started
    • Nimbus Hub
    • Log Optimization
      • Reduce Optimizations
      • Lint Optimizations
    • Working with NTL
      • Working with Transforms
    • Working with Aggregated Logs
    • Examples
    • Concepts
    • FAQ
    • Architecture
  • Additional Features
    • Configuration Overrides
    • Pause All
    • Error Detection
    • Private Link
  • Integrations
    • Datadog
      • AWS Lambda Extension
      • AWS Lambda Forwarder
      • DD Agent
      • DD Log Forwarding Destination
      • Heroku
      • Journald
      • OpenTelemetry
  • Resources
    • Nimbus Attributes
    • Changelog
    • SLA
    • Bug Bounty
    • Support
  • Preview Features
    • Metric Optimization
    • Datadog CLI
Powered by GitBook
On this page
  • Search
  • Monitors
  • Error Monitors
  • Count based monitors
  • Attribute based monitors
  • Dashboards

Was this helpful?

Export as PDF
  1. Overview

Working with Aggregated Logs

Last updated 1 year ago

Was this helpful?

Aggregated logs are just regular logs with specific .

The individual payload of the pre-aggregated logs can be found in the nimdata field which is an array of the underlying log events.

The message field is an array of the original

Search

When searching for values within a JSON array, use the same syntax as when searching a regular property.

{
    message: ["item 123 refreshed", "item 345 refreshed", "item 567 error"],
    jobId: 1,
    nimdata: [
        {
            jobId: 1,
            message: "item 123 refreshed",
            category: "luxury"
        },
        {
            jobId: 1,
            message: "item 345 refreshed",
            category: "toys"
        },
        {
            jobId: 1,
            message: "item 567 refreshed",
            category: "luxury"
        }
    ]
    ...
}

For example, to find log messages with "error", you can use the following search

"error"

Searching for values within a JSON array of objects, you can use the following search

@nimdata.category:"luxury"

Monitors

Nimbus is compatible with existing log monitoring setups. We'll walkthrough three common scenarios below and how monitors would behave after Nimbus:

Error Monitors

Count based monitors

These are monitors that measure the number of logs during a set interval. You can retrieve the original size of of pre-aggregated logs by using Sum of @nimsize instead of Count of All Logs.

Original Monitor based on Count

Aggregated Monitor based on @nimsize

Attribute based monitors

Dashboards

All instructions for monitors also apply to dashboards.

These are monitors that alert based on logs with errors. Error logs are automatically detected by Nimbus and go through a separate pipeline that . This means any monitors on error logs will be unaffected.

Original Monitor based on Count
Aggregated Monitor based on @nimsize

These are monitors that depend on a specific attribute within the aggregated log. You can either modify the monitor to alarm based on the nested attribute or use the directive to keep attributes that you alarm on at the top level.

nimbus attributes
log bodies
bypasses aggregation
pull_up