Steps

1. Go to cloudformation, and search “datadog”, you should either see one of the following

   • datadog-forwarder as its own entry

   • DatadogIntegration-ForwarderStack-*** as a nested stack

2. Click on the forwarder stack and click update

3. Select use current template

4. Set the following 2 parameters:

   • DdUrl: $YOUR_NIMBUS_ENDPOINT

   • DdPort: 443

5. Update the stack

By default, Nimbus automatically detects errors and routes error logs through a separate pipeline that does pass through aggregation. This means that error logs get routed through in near real time without any transformations.

The Nimbus Hub (or Hub for short) acts as your command center for all optimizations.

Nimbus automatically identifies high traffic log patterns and displays them on the console as a table.

Table Properties

• Name: Autogenerated name for the traffic pattern

• Volume: The total number of logs analyzed for the given pattern (for an 1h period)

• Percentage: The percentage of logs this pattern represents (for an 1h period) compared to all logs

• Updated: When this log pattern was last updated

Applying a Transform

To apply a transform, click on Details link on the log pattern you wish to update.

This will open up the transform modal with two panels.

The left panel shows the transform that Nimbus generated for the given pattern. The transform is constructed using the , a domain specific language optimized for expressing telemetry optimizations.

The right panel shows a sample of raw logs that the transform would act over.

To apply a transform, click on the Apply button in the left panel. This will immediately deploy the transform.

Transformation Previews

Nimbus lets you preview of how logs will be shaped post transformation. Raw Preview shows the log in pure JSON whereas Rich Preview shows you how those logs would show up in Datadog.

Aggregated logs are just regular logs with specific nimbus attributes.

The individual payload of the pre-aggregated logs can be found in the nimdata field which is an array of the underlying log events.

The message field is an array of the original log bodies

Search

When searching for values within a JSON array, use the same syntax as when searching a regular property.

For example, to find log messages with "error", you can use the following search

Searching for values within a JSON array of objects, you can use the following search

Monitors

Nimbus is compatible with existing log monitoring setups. We'll walkthrough three common scenarios below and how monitors would behave after Nimbus:

Error Monitors

These are monitors that alert based on logs with errors. Error logs are automatically detected by Nimbus and go through a separate pipeline that . This means any monitors on error logs will be unaffected.

Count based monitors

These are monitors that measure the number of logs during a set interval. You can retrieve the original size of of pre-aggregated logs by using Sum of @nimsize instead of Count of All Logs.

Original Monitor based on Count

Aggregated Monitor based on @nimsize

Attribute based monitors

These are monitors that depend on a specific attribute within the aggregated log. You can either modify the monitor to alarm based on the nested attribute or use the directive to keep attributes that you alarm on at the top level.

Dashboards

All instructions for monitors also apply to dashboards.

This guide walks you through using Nimbus to forward and optimize your log traffic from datadog. This assumes you're sending logs using the datadog agent. If that's not the case, see integrations for documentation for other sources.

1. Connect your Datadog Account

1. Click on Sinks in the left navbar and click Add Sink and select Datadog

1. Enter your datadog site and a valid API key:

• you can find what site you're on by matching your datadog URL to the following table

• you can either use an existing api key or create a new one in Organizational Settings > API Keys

2. Connect your Logs

This sections walks through adding Nimbus via the datadog agent. If you are using a different integration, see for integration specific instructions.

Start by adding the following configuration to your datadog config

NOTE: YOUR_NIMBUS_ENDPOINT is an URL that is generated for you when you first create an account

Optionally, you can configure the endpoint using the following environmental variables. This is useful when you're running the datadog agent in kubernetes like environments and don't have easy access to the raw configuration.

Update your datadog agent to run with the new configuration. Congratulations - you're now forwarding log traffic with Nimbus!

At this point, Nimbus will start analyzing your traffic. It can take up to 24h for initial results to show up if this is your first time integrating. So go grab some coffee and go on with your day. We'll send you an email when the findings are ready for you to review in the .

1. Add Nimbus as a log endpoint

Add the following configuration to your datadog config

NOTE: YOUR_NIMBUS_ENDPOINT is an URL that is generated for you when you first create an account

Optionally, you can configure the endpoint using the following environmental variables. This is useful when you're running the datadog agent in kubernetes like environments and don't have easy access to the raw configuration.

Update your datadog agent to run with the new configuration. Congratulations - you're now forwarding log traffic with Nimbus!

The following is a list of currently documented integration sources with datadog

Prerequisites

This guide goes over integrating Nimbus with the

Steps

1. Go to the lambda that you want to forward logs from

2. Add the following environmental variables to lambdas you want to add logs to

Nimbus helps companies reduce datadog costs by 60% or more. Our data optimization pipeline analyzes telemetry and aggregates it in flight to reduce your volume without dropping data.

How it works

Nimbus analyzes all your logs and finds high volume log patterns based on incoming data. From these patterns, Nimbus generates optimizations that aggregate related logs into a single event.

We refer to this style of transformation as lossless aggregation. You can see an example of how this works below.

This guide walks you through forwarding your logs to Nimbus.

1. Setup a custom log forwarding destination in datadog

In datadog's configuration, select a custom destination.

Add the following details:

Nimbus optimizations are automatically surfaced by our traffic analysis engine and take up to 24 hours to surface when you first connect to Nimbus.

Nimbus currently supports the following optimization types:

• Reduce Optimization: Reduce log volume

• Lint Optimization: Improve log hygiene

You can reach out to [email protected] for any queries regarding Nimbus. If you're on any paid plan, you also have priority support via a dedicated slack channel.

Overview

The Nimbus Transformation Language (NTL) is a high level language for reshaping telemetry data.

When Nimbus makes an optimization, it generates NTL to describe when and how an optimization should be made.

You can edit generated transforms to tailor it for your specific business requirements (this is not necessary in the majority of cases).

Nimbus Predicates

Nimbus predicates evaluate a series of expressions and return a boolean. They have the following syntax

• key: the dot delimited path

• op: any NTL valid operator

• val: the expected value

Note that key can be omitted in which case the value is expected to be a list of Nimbus predicates.

Comparison Operators

EQUAL

Checks whether two elements are exactly equal in value

EXISTS

Checks whether a particular path exists in an object

MATCH

Regex match

MATCH_ANY

Regex match against an array of values

Logical Operators

AND

NOT

OR

Product

How long does it take to integrate Nimbus?

This depends on your specific vendor and your application integration. On average, for customers using the regular datadog agent to forward data, initial setup takes under 5 minutes.

Is Nimbus an observability pipeline? How is it different from other services in this space?

1. Other services put the burden on you to understand your traffic patterns. Nimbus automatically analyzes your traffic and creates a top N list of high traffic patterns.

2. Other services make you manually create the rules and filters to create a pipeline. Nimbus automatically generates transforms based on its traffic analysis.

3. Other services have you sample and drop data to reduce volume. Nimbus applies lossless aggregation which means that you reduce volume without losing visibility.

Do you support other vendors besides Datadog?

Yes. Please reach out to [email protected] to get details on vendor specific integrations

Logs

Does Nimbus introduce extra latency to my logs?

Nimbus processes logs in near real time - the average message is received and forwarded in under 100ms.

There is a caveat for aggregated logs. These are held in memory (and buffered on disk) until a is met (eg. max_events, expire_after_ms, etc). Aggregations can be disabled at any time. Nimbus also has a button that lets you disable all aggregations at once if needed.

Note that Nimbus has built-in rules to not aggregate error logs which means that they will still come through in near real time.

Can I still query for the same data after my logs have been aggregated?

Yes. Because Nimbus uses to optimize your log volume, you don't end up losing any data. You can find out more in

Will this impact my log based monitors?

Short answer - no. Any log based monitor you currently have can be replicated post-aggregation, either with no changes or some small tweaks. When you onboard, a dedicated Nimbus engineer will work with you to ensure that none of your existing monitors will be impacted.

You can find out more in .

Will this impact my log based dashboards?

No. See answer above about log based monitors.

Does data get lost when I aggregate?

By default, Nimbus deduplicates common metadata and merges unique values from logs when aggregation. The only field that gets discarded is the timestamp field - Nimbus preserves the start time and adds a timestamp_end field to designate the time interval for the aggregation. You can see examples of what this looks like in the

Can I pause transformations during an incident?

Yes. See the section for details.

What if Datadog changes its pricing model to not be event based?

Nimbus is extremely effective at reducing number of events (100X) and very effective at reducing the size of events (40%). So regardless of the type of pricing model, we will be able to deliver significant savings.

Security and Availability

What is your data retention policy?

Nimbus keeps observability data for a period of up to 7 days in order to analyze traffic patterns. It does not store or retain data beyond the observability window.

What is your availability?

Nimbus offers a 99.9% SLA on uptime. See more details .

Etc

What if I've already committed to an annual commitment?

Even if you've made a commitment, it's likely that you'll exceed the committed usage and have on demand spend (overage) on top of the committed usage. Nimbus can drop the on demand portion to 0 and make sure you don't exceed it.

We can also help you negotiate with Datadog for alternative contracts with your account executive.

Lint optimizations scan logs for common hygiene issues like sensitive (eg. api tokens) and redundant data (eg. timestamp appearing the log).

Optimization Triggers

Nimbus can automatically optimize logs when it detects the following situations:

1. Logs with timestamp appearing in message body

2. Common kind of secrets (AWS tokens, github and gitlab, etc)

Example

Take the following log

There are two issues:

• the timestamp is emitted with the json log and prevents datadog from properly parsing the log as json

• datadog adds its own timestamp at the time of ingestion (when the log was processed by datadog) which is not the same as the time of emission (when the log was originally emitted)

Nimbus can now recognize this class of issues and apply a lint optimization to fix it. In this case, Nimbus would come up with the following optimization

The log post lint optimization would look like the following

This applies the correct timestamp and lets datadog properly parse the json log as a structured log. This also makes it possible to do queries like @retry_count > 0 which previously would not have been possible over the string based log data

Interaction with existing Optimizations

In rare cases, lint optimizations can interfere with existing .

For example, if a current reduce optimization relies on a timestamp to be present in the log body and the lint optimization pulls it out as a log attribute, it means that those logs will no longer be aggregated.

For example, say you have the following log.

You also have the current reduce optimization

You might get a lint optimization that pulls out the current timestamp into a separate attribute

This means that your previous reduce optimization would no longer work because the it was using the date as an activation filter.

Today, you can either manual adjust the process_when clause and change the to fix it yourself or wait for Nimbus to re-analyze your logs and provide updated recommendations.

You can think of Nimbus as a data pipeline for your telemetry. We provide an out of the box opinionated framework to process your telemetry according to industry best practices.

Flow Diagram

Components

Global Ingress Preprocessor

• parse logs according to source format

• meter and derive analytics from ingress

Global Router

• routes telemetry depending on condition

  • if message is identified as an error, forward to error route

  • if message matches a optimization predicate, forward it to filter route

  • all messages not processed by a transform or matched as an error go to the default route

Error Route

• applies error specific attributes and optimizations

Filter Routes

• applies optimization specific attributes and optimizations

Default Route

• applies default attributes and optimizations

• currently, this applies nimkind: raw to the log

Global Egress Processor

• meter and derive analytics from ingress

Reduce optimizations reduce the volume of your logs, either along number of events or raw ingested bytes.

Nimbus analyzes all your logs and finds high volume log patterns based on incoming data. From these patterns, Nimbus generates transformations that aggregate related logs into a single event.

We refer to this style of transformation as lossless aggregation. You can see an example of how this works below.

Optimization Triggers

Nimbus can automatically optimize logs when it detects the following situations:

1. Logs with common message patterns

2. Logs with common identifiers

3. Multi-line Logs

For before and after examples of these triggers, see .

Logs with common message patterns

These are high volume log events that repeat most of their content. For most applications most of the time, this will be the primary driver of log volume. Examples include health checks and heart beat notifications.

Logs with common identifiers

These are logs that describe a sequence of related events. These sequences usually have some sort of common identifier like a transactionId or a jobId. Examples include a background job and business specific user flows.

Multi-line Logs

These are logs where the message body can be spread across multiple new lines. Unless you add special logic on the agent side, the default behavior is to emit each newline delimited message as a separate log event.

Optimization Dimensions

Nimbus optimizes logs across the following dimensions:

1. Volume: Optimize to reduce the number of events logged

2. Size: Optimize to reduce the size of events logged

For before and after examples of optimizations along these dimensions, see .

Volume

When optimizing for volume, Nimbus aggregates as many logs as it can given the constraints of the destination.

For example, has specific limits around total array size as well as log size. Nimbus makes sure to aggregate underneath this limit to maximize volume reduction.

Size

When optimizing for size, Nimbus deduplicates and removes redundant metadata as it aggregates logs.

For example, when aggregating , its often the case that 40% or more of the metadata (tags and attributes) are the same.

Optimization Fidelity

Nimbus generated optimizations can be tuned via fidelity levels to indicate how much of the original log message to preserve.

High

Nimbus optimizes for preserving original log data with perfect fidelity. This means there is no reduction in ingest size and aggregated logs contain all fields of the original log entires with only identical fields deduplicated.

Medium

Nimbus preserves most of the data. Individual timestamps in aggregated logs are discarded.

Low

Nimbus optimizes for ingest size. Low value fields are nominated for removal. All except nimsize are removed from the resulting log.

Last Updated: April 4th, 2024

This Nimbus Service Level Agreement (“SLA”) is a policy governing the use of Nimbus and applies separately to each account using Nimbus.

Capitalized terms used herein but not defined herein shall have the meanings set forth in the Agreement.

Service Commitment

Nimbus commits to use commercially reasonable efforts to make the Observability Pipeline, specifically focusing on the data ingestion component, available with the Monthly Uptime Percentages set forth in the table below. In the event the Observability Pipeline does not meet the Service Commitment, you will be eligible to receive a Service Credit as described below.

Definitions

An “Observability Pipeline” refers to the infrastructure and services provided by Nimbus for the collection, normalization, transformation, and routing of observability data (e.g., metrics, logs, traces) for a specific domain.

“Monthly Uptime Percentage” for the Observability Pipeline is calculated by subtracting from 99.999% the percentage of minutes during the month in which the data ingestion component of the Observability Pipeline was Unavailable. Monthly Uptime Percentage measurements exclude Unavailability resulting directly or indirectly from any Nimbus SLA Exclusions.

A “Service Credit” is a dollar credit, calculated as set forth above, that we may credit back to an eligible account.

The Observability Pipeline is “Unavailable” during a given minute if the data ingestion component fails to receive and process data for all attempts made to the pipeline throughout the minute.

Service Credits

Service Credits are calculated as a percentage of the total charges paid by you for the affected component of the Observability Pipeline for the monthly billing cycle in which the Service Commitment was not met, in accordance with the schedule below:

We will apply any Service Credits only against future Nimbus payments otherwise due from you. At our discretion, we may issue the Service Credit to the credit card you used to pay for the billing cycle in which the Unavailability occurred. Service Credits will not entitle you to any refund or other payment from Nimbus. A Service Credit will be applicable and issued only if the credit amount for the applicable monthly billing cycle is greater than one dollar ($1 USD). Service Credits may not be transferred or applied to any other account. Unless otherwise provided in the Agreement, your sole and exclusive remedy for any unavailability, non-performance, or other failure by us to provide the Observability Pipeline is the receipt of a Service Credit (if eligible) in accordance with the terms of this SLA.

Credit Request and Payment Procedures

To receive a Service Credit, you must submit a claim by contacting Nimbus support team. To be eligible, the credit request must be received by us by the end of the second billing cycle after which the incident occurred and must include:

• i. the words “SLA Credit Request” in the subject line;

• ii. the dates, times, and descriptions of each Unavailability incident that you are claiming;

• iii. evidence that corroborates the claimed Unavailability, such as logs or monitoring alerts (any confidential or sensitive information in these documents should be removed or replaced with asterisks).

If the Monthly Uptime Percentage of such request is confirmed by us and is less than the Service Commitment, then we will issue the Service Credit to you within one billing cycle following the month in which the request occurred. Your failure to provide the request and other information as required above will disqualify you from receiving a Service Credit.

Nimbus SLA Exclusions

The Service Commitment does not apply to any unavailability, suspension, or termination of the Observability Pipeline, or any other Nimbus performance issues: (i) caused by factors outside of our reasonable control, including any force majeure event or Internet access or related problems beyond the demarcation point of Nimbus; (ii) that result from any actions or inactions by you or any third party; (iii) that result from your equipment, software, or other technology and/or third party equipment, software, or other technology (other than third party equipment within our direct control); (iv) arising from our suspension or termination of your right to use the Observability Pipeline in accordance with the Agreement; or (v) that result from your failure to follow the guidelines and best practices described in Nimbus documentation, including exceeding usage limits. If availability is impacted by factors other than those used in our Monthly Uptime Percentage calculation, then we may issue a Service Credit considering such factors at our discretion.

This guide goes over integrating Nimbus with the OpenTelemetry Collector.

Steps

1. In your OTEL collector, add an otlphttp exporter - replace $API_KEY with your Nimbus API key

exporters:
  otlphttp/nimbus:
    endpoint: https://$API_KEY-otlp-intake.logs.us1.nimbus.dev:443

1. Add the otlphttp exporter to any existing pipeline that processes logs

1. Reload existing collectors with the new configuration.

2. That's it - you're done. Nimbus is now optimizing your logs!

Nimbus lets you set up private connectivity between your cloud provider and Nimbus.

How it Works

With Nimbus Private Link, you can directly connect your VPC with Nimbus using AWS VPC Endpoints. Note that this is currently only supported for AWS accounts in region us-east-1.

Benefits

• cost reduction: with private link, your egress cost go down by 90% (regular egress on AWS is $0.09/GB. With private link, this becomes $0.01/GB)

• compliance and security: prevent sensitive data from traversing the public internet

Setup

1. Create a VPCEndpoint using our cloudformation template

Ensure Cloudformation stack is in status CREATE_COMPLETE and VPC Endpoint is Available with has Private DNS names enabled before proceeding

2. Verify the connection

You can test the endpoint by sending data to $API_KEY-http-intake.privatelink.logs.us1.nimbus.dev in a connected subnet

NOTE: Sending the request outside of the connected VPC will result in 403 response

3. Update your Nimbus Endpoints

To switch over to private link, update your Nimbus endpoint. to the new schema by adding privatelink to your Nimbus endpoint.

See specific docs for your integration endpoints.

The Datadog CLI lets you query logs from your terminal. If you're querying aggregated logs, this also gives you the option to disaggregate them into individual log lines.

setup

git clone [email protected]:nimbushq/dd-cli.git
cd dd-cli
yarn && yarn build
npm link

logs

nimbus logs

Search across dd logs. The following environmental variables need to be set in
order to run this command: DD_SITE, DD_API_KEY, DD_APP_KEY

Options:
      --version                             Show version number        [boolean]
      --help                                Show help                  [boolean]
  -q, --query                               dd log query     [string] [required]
  -f, --from                                time in the following format:
                                            2024-02-14T11:35:00-08:00
                                                             [string] [required]
      --to, --t, desc: "time in the
      following format:
      2024-02-14T11:35:00-08:00"                             [string] [required]
  -i, --indexes                             log indexes to search
                                                     [array] [default: ["main"]]
  -d, --disaggregate                        disaggregate aggregated logs
                                                                       [boolean]

Usage

Replace env variables with your org specific values

Nimbus is architected around observability pipeline best practices and usually requires no manual configuration. That said, we understand that real life systems are complex and more flexibility is needed.

To that end, configuration overrides let you override any part of the Nimbus pipeline with your custom VRL code.

1. Click on the Configuration Tab

2. Edit a configuration

You can use any valid VRL to edit the configuration.

You can use the Override dropdown to change what part of the pipeline you wish to edit. The current options are:

• nim/in/global_remap: controls ingress. all data will pass by this transform

• nim/out/global_remap: controlls egress. all data that is sent upstream will pass by this transform

For a full list of configuration options, visit the

3. Save your configuration

Hit Save to apply your changes.

Nimbus supports pausing all transforms in times of distress.

1. Click on the Pause All button

Go to the transforms tab and then click Pause All

2. Confirm and Save

Clicking will open a modal with a dialogue box asking you to type CONFIRM to continue. Type the letters and hit confirm to un

3. Resume

When you are ready to resume transforms, click the Resume button to enable existing transforms

Nimbus adds custom attributes to logs that it processes

nimdata

• type: list

Contains the raw values of reduced logs

Lossless Aggregation

A method of aggregating observability data that preserves 100% the fidelity of the original data.

Log Body

The body of the log entry. This value is displayed when browsing the log and usually indexed for full text search.

Nimbus Metric Hub enables you to pre-aggregate your host metrics before sending them to datadog. This means we can reduce your billable infrastructure host count by an order of magnitude while still preserving the individual metrics for each host.

Nimbus Metric Optimization is currently in private preview. To get early access, please reach out to [email protected]

Introduction

Nimbus is committed to maintaining the security and integrity of our services. We understand that no technology is perfect, and we believe in working collaboratively with the security community to find and resolve vulnerabilities. Our bug bounty program encourages this collaboration by rewarding security researchers who provide us with high-quality security information.

Scope

Steps

1. Install vector in your target environment.

curl --proto '=https' --tlsv1.2 -sSfL https://sh.vector.dev | bash -s -- -y

1. Install the nimbus configuration

   NOTE: you'll need to replace $YOUR_NIMBUS_ENDPOINT with your specific endpoint

1. Create the systemd script

1. Execute

1. Verify

0.23

Release Date: 2024/04/26

Features

Want to try Nimbus without speaking to a sales rep? Nimbus now supports self serve onboarding!

All new accounts get a free 14 day trial and can send any amount of data to Nimbus without any caps! You can get started by signing up from the and setup your in under 10 minutes!

0.22

Release Date: 2024/04/12

Features

You can now reduce you data egress fees by 90% using Nimbus Private Link 🎉

Data egress fees are the "hidden cost" of observability. They are hard to detect because they don't show up in your observability vendor bill but rather in the data transfer fees from your cloud provider and can double your ingest costs for observability.

As an example, AWS charges $0.09/GB for egress (for comparison, datadog charges $0.10/GB for data ingress). Private link sets up a private connection between your VPC and Nimbus and reduces the cost of data transfer to $0.01/GB.

Private link is available for free to all Nimbus customers in us-east-1. Instructions for getting started

0.21

Release Date: 2024/03/28

Features

• engine: Nimbus now supports . Lint optimizations scan logs for common hygiene issues like sensitive (eg. api tokens) and redundant data (eg. timestamp appearing the log)

Enhancements

• pipeline: improvements pipeline p99 latency by using provisioned throughput for block storage

• pipeline: reduce impact of az failover by buffering telemetry data on disk across multiple azs

0.20

Release Date: 2024/03/14

Enhancements

• ntl: support directive

• ntl: support directive

Fixes

• api: sink updates are now immediately applied

0.19

Release Date: 2024/02/29

Features

We launched a . this lets you query logs from your terminal. If you're querying aggregated logs, this also gives you the option to disaggregate them into individual log lines.

Enhancements

• ui: handle various text overflow issues in the ui

0.18

Release Date: 2024/02/15

| Heads up that we will be switching to a bi-weekly release model moving forward due to the growing scope of what the team is currently taking on.

Enhancements

• ui: you can now update your observability sinks in the UI

• ntl: Nimbus now support the NOT operator

Fixes

• UI now shows all modals in full screen regardless of content size

0.17

Release Date: 2024/02/08

Enhancements

• self serve onboarding: you can now provision and manage your Nimbus destinations without human contact (that said, we're still here if you need us)

• UI improvements: snappier page loads and consisting alignment of tables and elements

0.16

Release Date: 2024/02/01

Features

Nimbus SLA - Nimbus now has a of 99.999% uptime. You can follow our public status page to be notified of incidents.

0.15

Release Date: 2024/01/25

Enhancements

• Nimbus now shows ingest bytes reduction in addition to event reduction on optimizations

0.14

Release Date: 2024/01/18

Features

• optimization fidelity: you can now customize log fidelity during optimization, choosing between preserving 100% of the original input and minimizing data ingest size

0.13

Release Date: 2024/01/11

Enhancements

• support max, min, retain, flat_unique, and longest_array

Fixes

• bad validation rule when updating a transform causes update to fail on certain merge strategies

0.12

Release Date: 2024/01/04

New year, new look.

We launched the Nimbus website. You should have received new credentials in your email. The new website offers a much snappier and light weight version of our previous retool application and will enable us to ship much more ambitious features later this year.

Features

• new frontend at https://hub.nimbus.dev

0.11

Release Date: 2023/12/28

Features

• support optimization_mode to tune Nimbus optimization

• traffic analysis now shows volume reduction by either events or size

Enhancements

• support which accepts a list of paths to be deleted from the target payload

0.10

Release Date: 2023/12/21

Enhancements

• support additional_sinks in

• support additional_sources in

• support MATCH_ANY

0.9

Release Date: 2023/12/14

Features

• : support aggregating metrics across hosts (private preview)

Enhancements

• better error messages when there is a syntax error when manually creating a transform

0.8

Release Date: 2023/12/07

Features

• directive allows you to pull properties inside of nested objects to the top level

• support for the inside of compound statements

0.7

Release Date: 2023/11/23

Features

• Pause All: Support manual override to

Enhancements

• clicking on the log pattern from analysis will now take you directly to events in datadog

• transforms now show usage stats

• revamped documentation with demo video

• reduce egress bytes by removing nimraw

0.6

Release Date: 2023/11/09

Features

• analysis now shows log output preview for generated transformation

• support modifying transforms generated by analysis

0.5

Release Date: 2023/10/26

Features

• analysis now shows log samples for findings

• support ability to delete a transform

Enhancements

• analysis now auto generates names for findings

• analysis now highlights new findings

0.4

Release Date: 2023/10/12

Features

• Configuration Overrides: You can now add custom VRL to control all aspects of the ingress stage of the nimbus pipeline

Enhancements

• support directive

• support directive

• support directive

• support directive

0.3

Release Date: 2023/09/28

Features

• Usage dashboards: you can now access your usage dashboard graphs

Enhancements

• Smarter error detection - we now autodetect errors based on

0.2

Release Date: 2023/09/14

Features

• Heroku Datadog Integration: Support Datadog

• Heroku Log Forwarding Integration: Support Datadog

Enhancemehts

0.1

Release Date: 2023/08/31

Hello world!

Features

• Nimbus Transformation Language (NTL): A high level language for working with telemetry data.

• Nimbus Traffic Analysis: Automatically identify high traffic log patterns

• Nimbus Transform Recommendations: Auto generated transforms using the based on traffic analysis results

Overview

Nimbus Transforms are high level NTL functions that have specialized logic for specific optimizations.

Once an optimization is applied, you can find its corresponding transformation in the transforms section of the console.

You can click on Edit to either update or delete an existing transform.

Global Options

The following properties are available on all transforms

process_when

• status: required

• type:

Determines when a transform should be applied. Takes one or more predicates as input.

Example:

include_errors

• status: optional

• type: boolean

• default: false

When set to true, designate that the current transform can apply to error logs. By default, error logs are not transformed but immediately proxied downstream for immediate processing.

msg_field

• status: optional

• type: string

• default: message

The key where the is located

Example:

pull_up

• status: optional

• type: string[]

When specified, a list of paths that should be made into

Example:

Before:

After:

remove

• status: optional

• type: string[]

When specified, a list of paths that should be removed

Example:

remove_from_nimdata

• status: optional

• type: string[]

If set, removes the selected paths from

Example:

remove_nimdata

• status: optional

• type: boolean

If set, removes the . Helps with significantly removing dataisze

Example:

Reduce Transform

The Nimbus reduce transform is a superset of the transform.

Options

merge_strategies

• status: optional

• type: enum

The default behavior is as follows:

• The first value of a string field is kept and subsequent values are discarded.

• For timestamp fields the first is kept and a new field [field-name]_end is added with the last received timestamp value.

• Numeric values are summed.

Strategies:

starts_when

• status: optional

• type: NTL

A condition used to distinguish the first event of a transaction. If this condition resolves to true for an event, the previous transaction is flushed (without this event) and a new transaction is started.

Example:

max_events

• status: optional

• type: integer

The maximum number of events to group together.

Example:

expire_after_ms

• status: optional

• type: integer

• default: 30000

The maximum period of time to wait after the last event is received, in milliseconds, before a combined event should be considered complete.

Example

Suppose you have the following logs:

And you have the following reduce transform

Your processed logs would look like the following

Examples of log patterns identified and optimized by Nimbus.

Logs with common message patterns

These are high volume log events that repeat most of their content. For most applications most of the time, this will be the primary driver of log volume. Examples include health checks and heart beat notifications.

[
  {
    "ddsource": "nodejs",
    "host": "itemrefresh-0",
    "message": "refresh item catalogue for itemId: ITEM470",
    "path": "/",
    "service"

Logs with common identifiers

These are logs that describe a sequence of related events. These sequences usually have some sort of common identifier like a transactionId or a jobId. Examples include a background job and business specific user flows.

Multi-Line Logs

Many times, an application will emit a single log across multiple lines such as the case with a JSON log. Unless you specifically account for this, most logging agents will consume each newline as a separate log event. Nimbus can identify when this happens and stitch these logs back together.