# Introduction

Nimbus helps companies reduce datadog costs by 60% or more. Our data optimization pipeline analyzes telemetry and aggregates it in flight to reduce your volume without dropping data.

## How it works

Nimbus analyzes all your logs and finds high volume log patterns based on incoming data. From these patterns, Nimbus generates optimizations that aggregate related logs into a single event.

We refer to this style of transformation as **lossless aggregation**. You can see an example of how this works below.

![Lossless Aggregation](https://ik.imagekit.io/fpjzhqpv1/Cursor_and_Excalidraw_l96xP4vXM.png?updatedAt=1701279438503)

In this case, we've reduced log event volume by 4X by grouping all logs that refer to the same transaction into one event.

To see what this looks like in practice, watch our demo!

{% embed url="<https://www.loom.com/share/938474913fd84c7fa4261d893d9cf652?sid=5915a1ae-b683-465c-8980-a777e0cd0fc6>" %}

To get an better understanding of supported optimizations, see [understanding log optimizations](/overview/optimization.log) for details.


# Getting Started

This guide walks you through using Nimbus to forward and optimize your log traffic from datadog. This assumes you're sending logs using the datadog agent. If that's not the case, see [integrations](/integrations/integrations.datadog) for documentation for other sources.

## 1. Connect your Datadog Account

1. Click on `Sinks` in the left navbar and click `Add Sink` and select `Datadog`

![Add Datadog Sink](https://ik.imagekit.io/fpjzhqpv1/Fullscreen_2_5_24__08_27_Dw4pec_4Z.png?updatedAt=1707151305056)

2. Enter your datadog site and a valid API key:

* you can find what site you're on by matching your datadog URL to the following table

![Datadog Site Table](https://ik.imagekit.io/fpjzhqpv1/Cursor_and_Getting_Started_with_Datadog_Sites_cuVhfxlme.png?updatedAt=1707151304437)

* you can either use an existing api key or create a new one in `Organizational Settings > API Keys`

{% embed url="<https://www.loom.com/share/8e9934edd3c94cbf868640a9b7b724be?sid=11f0b402-830f-4d3f-a14e-239ad75541bc>" %}

## 2. Connect your Logs

> This sections walks through adding Nimbus via the datadog agent. If you are using a different integration, see [datadog integrations](https://github.com/nimbushq/public-docs/blob/main/datadog.md) for integration specific instructions.

Start by adding the following configuration to your datadog config

```yaml
observability_pipelines_worker:
  logs:
    enabled: true
    url: "https://YOUR_NIMBUS_ENDPOINT"
```

> NOTE: YOUR\_NIMBUS\_ENDPOINT is an URL that is generated for you when you first create an account

Optionally, you can configure the endpoint using the following environmental variables. This is useful when you're running the datadog agent in kubernetes like environments and don't have easy access to the raw configuration.

```bash
DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED=true 
DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL="https://YOUR_NIMBUS_ENDPOINT"
```

Update your datadog agent to run with the new configuration. Congratulations - you're now forwarding log traffic with Nimbus!

At this point, Nimbus will start analyzing your traffic. It can take up to 24h for initial results to show up if this is your first time integrating. So go grab some coffee and go on with your day. We'll send you an email when the findings are ready for you to review in the [hub](/overview/hub).


# Nimbus Hub

The Nimbus Hub (or Hub for short) acts as your command center for all optimizations.

{% embed url="<https://www.loom.com/share/938474913fd84c7fa4261d893d9cf652?sid=5915a1ae-b683-465c-8980-a777e0cd0fc6>" %}

Nimbus automatically identifies high traffic log patterns and displays them on the console as a table.

{% hint style="info" %}
It can take up to 24 hours for initial results to show up. Grab a coffee and go on with your day - we'll send you an email when findings are ready!
{% endhint %}

![Console Table](https://ik.imagekit.io/fpjzhqpv1/Cursor_and_Nimbus_Log_Optimization_Demo\(1\)_HDMZsUIk4.png?updatedAt=1706560044072)

## Table Properties

* Name: Autogenerated name for the traffic pattern
* Volume: The total number of logs analyzed for the given pattern (for an 1h period)
* Percentage: The percentage of logs this pattern represents (for an 1h period) compared to all logs
* Updated: When this log pattern was last updated
* Optimized: Whether this log pattern is already optimized

## Applying a Transform

To apply a transform, click on `Details` link on the log pattern you wish to update.

![Apply a transform](https://ik.imagekit.io/fpjzhqpv1/Cursor_and_Nimbus_Log_Optimization_Demo_5BYTKykGC.png?updatedAt=1706560793215)

This will open up the transform modal with two panels.

The left panel shows the transform that Nimbus generated for the given pattern. The transform is constructed using the [Nimbus Transformation Language](/overview/ntl), a domain specific language optimized for expressing telemetry optimizations.

The right panel shows a sample of raw logs that the transform would act over.

![Transform Modal](https://ik.imagekit.io/fpjzhqpv1/Cursor_and_Nimbus_Log_Optimization_Demo-2_STPNZebfi.png?updatedAt=1706560793167)

To apply a transform, click on the `Apply` button in the left panel. This will immediately deploy the transform.

## Transformation Previews

Nimbus lets you preview of how logs will be shaped post transformation. `Raw Preview` shows the log in pure JSON whereas `Rich Preview` shows you how those logs would show up in Datadog.

{% tabs %}
{% tab title="Raw Preview" %}
![Raw Preview](https://ik.imagekit.io/fpjzhqpv1/Cursor_and_Nimbus_Log_Optimization_Demo\(2\)_SX-e3KzNh.png?updatedAt=1706560176589)
{% endtab %}

{% tab title="Rich Preview" %}
![Rich Preview](https://ik.imagekit.io/fpjzhqpv1/Nimbus_Log_Optimization_Demo_Zj4MLXt6u.png?updatedAt=1706560213005)
{% endtab %}
{% endtabs %}


# Log Optimization

Nimbus optimizations are automatically surfaced by our traffic analysis engine and take up to 24 hours to surface when you first connect to Nimbus.

Nimbus currently supports the following optimization types:

* [Reduce Optimization](/overview/optimization.log/optimization.log.reduce): Reduce log volume
* [Lint Optimization](/overview/optimization.log/optimization.log.lint): Improve log hygiene


# Reduce Optimizations

Reduce optimizations reduce the volume of your logs, either along number of events or raw ingested bytes.

Nimbus analyzes all your logs and finds high volume log patterns based on incoming data. From these patterns, Nimbus generates transformations that aggregate related logs into a single event.

We refer to this style of transformation as **lossless aggregation**. You can see an example of how this works below.

![Lossless Aggregation](https://ik.imagekit.io/fpjzhqpv1/Cursor_and_Excalidraw_l96xP4vXM.png?updatedAt=1701279438503)

## Optimization Triggers

Nimbus can automatically optimize logs when it detects the following situations:

1. Logs with common message patterns
2. Logs with common identifiers
3. Multi-line Logs

For before and after examples of these triggers, see [examples](https://github.com/nimbushq/public-docs/blob/main/examples.md).

### Logs with common message patterns

These are high volume log events that repeat most of their content. For most applications most of the time, this will be the primary driver of log volume. Examples include health checks and heart beat notifications.

### Logs with common identifiers

These are logs that describe a sequence of related events. These sequences usually have some sort of common identifier like a `transactionId` or a `jobId`. Examples include a background job and business specific user flows.

### Multi-line Logs

These are logs where the message body can be spread across multiple new lines. Unless you add special logic on the agent side, the default behavior is to emit each newline delimited message as a separate log event.

## Optimization Dimensions

Nimbus optimizes logs across the following dimensions:

1. Volume: Optimize to reduce the number of events logged
2. Size: Optimize to reduce the size of events logged

For before and after examples of optimizations along these dimensions, see [examples](https://github.com/nimbushq/public-docs/blob/main/examples.md).

### Volume

When optimizing for volume, Nimbus aggregates as many logs as it can given the constraints of the destination.

For example, [datadog](https://docs.datadoghq.com/api/latest/logs/#send-logs) has specific limits around total array size as well as log size. Nimbus makes sure to aggregate underneath this limit to maximize volume reduction.

### Size

When optimizing for size, Nimbus deduplicates and removes redundant metadata as it aggregates logs.

For example, when aggregating [logs with common message patterns](https://github.com/nimbushq/public-docs/blob/main/optimizations.log.md#logs-with-common-message-patterns), its often the case that 40% or more of the metadata (tags and attributes) are the same.

## Optimization Fidelity

Nimbus generated optimizations can be tuned via fidelity levels to indicate how much of the original log message to preserve.

### High

Nimbus optimizes for preserving original log data with perfect fidelity. This means there is no reduction in ingest size and aggregated logs contain all fields of the original log entires with only identical fields deduplicated.

### Medium

Nimbus preserves most of the data. Individual timestamps in aggregated logs are discarded.

### Low

Nimbus optimizes for ingest size. Low value fields are nominated for removal. All [nimbus attributes](/resources/ref.attributes) except `nimsize` are removed from the resulting log.


# Lint Optimizations

Lint optimizations scan logs for common hygiene issues like sensitive (eg. api tokens) and redundant data (eg. timestamp appearing the log).

## Optimization Triggers

Nimbus can automatically optimize logs when it detects the following situations:

1. Logs with timestamp appearing in message body
2. Common kind of secrets (AWS tokens, github and gitlab, etc)

## Example

Take the following log

```yaml
message: '2024/01/23 01:33:122 {"method": "process_checkout", "retry_count": 3}'
timestamp: 2024/01/23 01:33:126
service: checkout
...
```

There are two issues:

* the timestamp is emitted with the json log and prevents datadog from properly parsing the log as `json`
* datadog adds its own timestamp at the **time of ingestion** (when the log was processed by datadog) which is not the same as the **time of emission** (when the log was originally emitted)

Nimbus can now recognize this class of issues and apply a **lint optimization** to fix it. In this case, Nimbus would come up with the following optimization

```yaml
process_when:
- key: message
  op: EQUAL
  value: 'checkout'
- key: message
  op: MATCH
  value: '^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} .+'
vrl: |
  groups = parse_regex!(.message, r'^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) (?<data>.+)') 
  .message = groups.data
  .timestamp = parse_timestamp!(groups.time + "+00:00", format:"%Y/%m/%d %H:%M:%S%:z")
```

The log post lint optimization would look like the following

```yaml
method: "process_checkout"
retry_count: 3
timestamp: 2024/01/23 01:33:122
service: checkout
...
```

This applies the correct timestamp and lets datadog properly parse the json log as a structured log. This also makes it possible to do queries like `@retry_count > 0` which previously would not have been possible over the string based log data

## Interaction with existing Optimizations

In rare cases, lint optimizations can interfere with existing [reduce optimizations](/overview/optimization.log/optimization.log.reduce).

For example, if a current reduce optimization relies on a timestamp to be present in the log body and the lint optimization pulls it out as a log attribute, it means that those logs will no longer be aggregated.

For example, say you have the following log.

```yaml
message: 2024/01/23 01:33:12 foo did bar
...
```

You also have the current reduce optimization

```yaml
process_when:
- key: message
  op: MATCH
  value: '^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} foo.+'
...
```

You might get a lint optimization that pulls out the current timestamp into a separate attribute

```yaml
message: foo did bar
timestamp: 2024/01/23 01:33:12
```

This means that your previous reduce optimization would no longer work because the it was using the date as an activation filter.

Today, you can either manual adjust the `process_when` clause and change the [predicate](https://docs.nimbus.dev/overview/ntl/ntl.transforms) to fix it yourself or wait for Nimbus to re-analyze your logs and provide updated recommendations.


# Working with NTL

## Overview

The Nimbus Transformation Language (NTL) is a high level language for reshaping telemetry data.

When Nimbus makes an optimization, it generates NTL to describe when and how an optimization should be made.

You can edit generated transforms to tailor it for your specific business requirements (this is not necessary in the majority of cases).

## Nimbus Predicates

Nimbus predicates evaluate a series of expressions and return a boolean. They have the following syntax

```yml
- key: 
  op:
  val:
```

* key: the dot delimited path
* op: any NTL valid operator
* val: the expected value

Note that `key` can be omitted in which case the value is expected to be a list of Nimbus predicates.

## Comparison Operators

### EQUAL

Checks whether two elements are exactly equal in value

```yml
key: foo
op: equal
val: 42
```

### EXISTS

Checks whether a particular path exists in an object

```yml
key: foo
op: exists
val: true
```

### MATCH

Regex match

```yml
key: foo
op: match
val: "foo"
```

### MATCH\_ANY

Regex match against an array of values

```yml
key: foo
op: match_any
val: ["foo", "foobar"]
```

## Logical Operators

### AND

```yml
op: AND
val: 
  - key: foo
    op: exists
    val: true
  - key: foo
    op: equal
    val: 42
```

### NOT

```yml
op: NOT
val: 
  key: foo
  op: equal
  val: 42
```

### OR

```yml
op: OR
val: 
  - key: foo
    op: exists
    val: true
  - key: foo
    op: equal
    val: 42
```


# Working with Transforms

## Overview

**Nimbus Transforms** are high level [NTL](/overview/ntl) functions that have specialized logic for specific optimizations.

Once an optimization is applied, you can find its corresponding transformation in the [transforms](https://hub.nimbus.dev/transforms) section of the console.

![Transforms Console](https://ik.imagekit.io/fpjzhqpv1/Cursor_and_Nimbus_EqzhLWZ3tX.png?updatedAt=1707939313920)

You can click on `Edit` to either update or delete an existing transform.

## Global Options

The following properties are available on all transforms

### process\_when

* status: required
* type: [NTL](/overview/ntl#evaluate)

Determines when a transform should be applied. Takes one or more [NTL](/overview/ntl) predicates as input.

Example:

```yml
process_when: 
  - {key: service, op: EQUAL, val: foo}
```

### include\_errors

* status: optional
* type: boolean
* default: `false`

When set to true, designate that the current transform can apply to error logs. By default, error logs are not transformed but immediately proxied downstream for immediate processing.

### msg\_field

* status: optional
* type: string
* default: `message`

The key where the [log body](https://github.com/nimbushq/public-docs/blob/main/concepts.md#log-body) is located

Example:

```yaml
# logs sent by dd lambda extension have the message field nested inside the message key
# eg:
# {message: { message: "START ...", lambda: {arn: arn:aws:lambda:us-east-1:33333333:function:test-lambda, ...}}}
msg_field:
  - message.message
```

### pull\_up

* status: optional
* type: string\[]

When specified, a list of paths that should be made into [top level keys](https://github.com/nimbushq/public-docs/blob/main/concepts.md#top-level-keys)

Example:

```yaml
pull_up:
  - message.transactionId
```

Before:

```json
{
  "message": {
    "transactionId": 1,
    ...
  }
}
```

After:

```json
{
  "transactionId": 1,
  "message": {
    ...
  }
}
```

### remove

* status: optional
* type: string\[]

When specified, a list of paths that should be removed

Example:

```yaml
remove:
  - message.id
  - message.source
  - message.timeout
```

### remove\_from\_nimdata

* status: optional
* type: string\[]

If set, removes the selected paths from [nimdata](/resources/ref.attributes#nimdata)

Example:

```yaml
remove_from_nimdata: 
  - status
  - hostname
  - ...
```

### remove\_nimdata

* status: optional
* type: boolean

If set, removes the [nimdata attribute](/resources/ref.attributes#nimdata). Helps with significantly removing dataisze

Example:

```yaml
remove_nimdata: true
```

## Reduce Transform

The Nimbus reduce transform is a superset of the [vector reduce](https://vector.dev/docs/reference/configuration/transforms/reduce/) transform.

{% hint style="warning" %}
When using reduce, remember that `group_by` only works on [top level keys](https://github.com/nimbushq/public-docs/blob/main/concepts.md#top-level-keys)

If the key you need is nested, make sure to pull it up using the `pull_up` directive.
{% endhint %}

### Options

#### merge\_strategies

* status: optional
* type: enum

The default behavior is as follows:

* The first value of a string field is kept and subsequent values are discarded.
* For timestamp fields the first is kept and a new field **`[field-name]_end`** is added with the last received timestamp value.
* Numeric values are summed.

Strategies:

| Option          | Description                                              |
| --------------- | -------------------------------------------------------- |
| array           | Append each value to an array.                           |
| concat          | Concatenate each string value, delimited with a space.   |
| concat\_newline | Concatenate each string value, delimited with a newline. |
| concat\_raw     | Concatenate each string, without a delimiter.            |
| discard         | Discard all but the first value found.                   |
| flat\_unique    | Create a flattened array of all unique values.           |
| longest\_array  | Keep the longest array seen.                             |
| max             | Keep the maximum numeric value seen.                     |
| min             | Keep the minimum numeric value seen.                     |
| retain          | Discard all but the last value found.                    |

#### starts\_when

* status: optional
* type: NTL

A condition used to distinguish the first event of a transaction. If this condition resolves to true for an event, the previous transaction is flushed (without this event) and a new transaction is started.

Example:

```yml
starts_when:
  - {key: message, op: MATCH, val: "\n\{"}
```

#### max\_events

* status: optional
* type: integer

The maximum number of events to group together.

Example:

```yml
max_events: 200
```

#### expire\_after\_ms

* status: optional
* type: integer
* default: 30000

The maximum period of time to wait after the last event is received, in milliseconds, before a combined event should be considered complete.

### Example

Suppose you have the following logs:

```json
[
  {
    "host": "host1",
    "fooatt": "one",
    "baratt": "alpha"
  },
  {
    "host": "host2",
    "fooatt": "two",
    "baratt": "beta"
  },
  {
    "host": "host1",
    "fooatt": "three",
    "baratt": "gamma"
  },
  {
    "host": "host1",
    "baratt": "gamma"
  }
]

```

And you have the following reduce transform

```yaml
name: hostreducer
# only apply this reducer when the log event has both a `host` and `fooatt` keys
process_when: 
  - {key: host, op: exists, val: true}
  - {key: fooatt, op: exists, val: true}
group_by:
  - host

```

Your processed logs would look like the following

```json
[
  // this log was processed and grouped correctly
  {
    "host": "host1",
    "nimdata": [
      {
        "host": "host1",
        "fooatt": "one",
        "baratt": "alpha"
      },
      {
        "host": "host1",
        "fooatt": "three",
        "baratt": "gamma"
      }
    ],
    "nimsize": 2,
    "nimkind": "opt",
    "nimmatch": "hostreducer"
  },
  {
    "host": "host2",
    "nimdata": [
      {
        "host": "host2",
        "fooatt": "two",
        "baratt": "beta"
      }
    ],
    "nimsize": 1,
    "nimkind": "opt",
    "nimmatch": "hostreducer"
  },
  // this log did not get processed as it did not have a `fooatt` key
  {
    "host": "host1",
    "baratt": "gamma",
    "nimkind": "noopt"
  }
]

```


# Working with Aggregated Logs

Aggregated logs are just regular logs with specific [nimbus attributes](/resources/ref.attributes).

The individual payload of the pre-aggregated logs can be found in the `nimdata` field which is an array of the underlying log events.

The message field is an array of the original [log bodies](/overview/concepts#log-body)

## Search

When searching for values within a JSON array, use the same syntax as when searching a regular property.

```yaml
{
    message: ["item 123 refreshed", "item 345 refreshed", "item 567 error"],
    jobId: 1,
    nimdata: [
        {
            jobId: 1,
            message: "item 123 refreshed",
            category: "luxury"
        },
        {
            jobId: 1,
            message: "item 345 refreshed",
            category: "toys"
        },
        {
            jobId: 1,
            message: "item 567 refreshed",
            category: "luxury"
        }
    ]
    ...
}
```

For example, to find log messages with "error", you can use the following search

```
"error"
```

Searching for values within a JSON array of objects, you can use the following search

```
@nimdata.category:"luxury"
```

## Monitors

Nimbus is compatible with existing log monitoring setups. We'll walkthrough three common scenarios below and how monitors would behave after Nimbus:

### Error Monitors

These are monitors that alert based on logs with errors. Error logs are automatically detected by Nimbus and go through a separate pipeline that [bypasses aggregation](/additional-features/advanced.error-detection). This means any monitors on error logs will be unaffected.

### Count based monitors

These are monitors that measure the number of logs during a set interval. You can retrieve the original size of of pre-aggregated logs by using `Sum of @nimsize` instead of `Count of All Logs`.

> Original Monitor based on Count

![Original Monitor based on Count](https://ik.imagekit.io/fpjzhqpv1/t1_GJ0Y_FcA8i.jpg?updatedAt=1707333218591)

> Aggregated Monitor based on @nimsize

![Aggregated Monitor based on @nimsize](https://ik.imagekit.io/fpjzhqpv1/t2_8ZXmygOvX.jpg?updatedAt=1707333218799)

### Attribute based monitors

These are monitors that depend on a specific attribute within the aggregated log. You can either modify the monitor to alarm based on the nested attribute or use the [pull\_up](/overview/ntl/ntl.transforms#pull_up) directive to keep attributes that you alarm on at the top level.

## Dashboards

All instructions for monitors also apply to dashboards.


# Examples

Examples of log patterns identified and optimized by Nimbus.

### Logs with common message patterns

These are high volume log events that repeat most of their content. For most applications most of the time, this will be the primary driver of log volume. Examples include health checks and heart beat notifications.

{% tabs %}
{% tab title="Before" %}

```json
[
  {
    "ddsource": "nodejs",
    "host": "itemrefresh-0",
    "message": "refresh item catalogue for itemId: ITEM470",
    "path": "/",
    "service": "itemrefresh",
    "status": "info",
    "timestamp": "2023-11-23T00:16:09.970Z"
  },
  {
    "ddsource": "nodejs",
    "host": "itemrefresh-0",
    "message": "refresh item catalogue for itemId: ITEM8185",
    "path": "/",
    "service": "itemrefresh",
    "status": "info",
    "timestamp": "2023-11-23T00:16:09.997Z"
  },
  {
    "ddsource": "nodejs",
    "host": "itemrefresh-0",
    "message": "refresh item catalogue for itemId: ITEM7594",
    "path": "/",
    "service": "itemrefresh",
    "status": "info",
    "timestamp": "2023-11-23T00:16:10.010Z"
  },
  // 37 more messages
  ...
]
```

{% endtab %}

{% tab title="After" %}

* 97.5% event volume reduction
* 79% ingest volume reduction

```json
{
  "ddsource": "nodejs",
  "host": "itemrefresh-0",
  "path": "/",
  "service": "itemrefresh",
  "status": "info",
  "message": [
    "refresh item catalogue for itemId: ITEM470",
    "refresh item catalogue for itemId: ITEM8185",
    "refresh item catalogue for itemId: ITEM7594",
    // 37 more messages
    ...
  ],
  "nimsize": 40,
  "timestamp": "2023-11-23T00:16:09.970Z",
  "timestamp_end": "2023-11-23T00:16:11.322Z"
} 
```

{% endtab %}

{% tab title="Transform" %}

```yml
- name: itemrefresh
  type: reduce
  rules:
    process_when:
      - key: service
        op: EQUAL
        val: itemrefresh
    group_by:
      - host
      - path
    pull_up:
      - ddsource
      - path
      - status
      - service
    msg_field: message
```

{% endtab %}
{% endtabs %}

### Logs with common identifiers

These are logs that describe a sequence of related events. These sequences usually have some sort of common identifier like a `transactionId` or a `jobId`. Examples include a background job and business specific user flows.

{% tabs %}
{% tab title="Before" %}

```json
[
  {
    "ddsource": "nodejs",
    "host": "checkout-0",
    "message": {
      "customerId": "CU26940939",
      "itemId": "ITEM1417",
      "itemName": "Product 7",
      "itemPrice": 2.612748019396105,
      "msg": "adding ITEM9798 to cart",
      "quantity": 2,
      "transactionId": "TX79924095"
    },
    "service": "checkout",
    "status": "info",
    "timestamp": "2024-04-26T15:45:14.000000138Z"
  },
  {
    "ddsource": "nodejs",
    "host": "checkout-0",
    "message": {
      "customerId": "CU26940939",
      "discountAmount": 13.837782236831986,
      "msg": "applying discount DISC16",
      "transactionId": "TX79924095"
    },
    "service": "checkout",
    "status": "info",
    "timestamp": "2024-04-26T15:45:14.000000831Z"
  },
  {
    "ddsource": "nodejs",
    "host": "checkout-0",
    "message": {
      "customerId": "CU26940939",
      "estimatedDelivery": "2023-11-24",
      "msg": "calculating shipping info for ITEM9798",
      "shippingAddress": "902 Main St, Anytown, AN 68387",
      "shippingMethod": "Standard",
      "transactionId": "TX79924095"
    },
    "service": "checkout",
    "status": "info",
    "timestamp": "2024-04-26T15:45:15.000000523Z"
  },
  {
    "ddsource": "nodejs",
    "host": "checkout-0",
    "message": {
      "customerId": "CU26940939",
      "msg": "payment for ITEM9798 succeeded",
      "paymentMethod": "PayPal",
      "totalAmount": 56.645267111988474,
      "transactionId": "TX79924095"
    },
    "service": "checkout",
    "status": "info",
    "timestamp": "2024-04-26T15:45:16.000000214Z"
  }
]
```

{% endtab %}

{% tab title="After" %}

* 75% reduction in event volume
* 4% reduction in ingest volume

```json
{
  "customerId": "CU26940939",
  "ddsource": "nodejs",
  "host": "checkout-0",
  "message": [
    "adding ITEM9798 to cart",
    "applying discount DISC16",
    "calculating shipping info for ITEM9798",
    "payment for ITEM9798 succeeded",
  ],
  "nimdata": [
    {
      "message": {
        "itemId": "ITEM1417",
        "itemName": "Product 7",
        "itemPrice": 2.612748019396105,
        "msg": "adding ITEM9798 to cart",
        "quantity": 2
      },
      "timestamp": "2024-04-26T15:45:14.000000138Z"
    },
    {
      "message": {
        "discountAmount": 13.837782236831986,
        "msg": "applying discount DISC16"
      },
      "timestamp": "2024-04-26T15:45:14.000000831Z"
    },
    {
      "message": {
        "estimatedDelivery": "2023-11-24",
        "msg": "calculating shipping info for ITEM9798",
        "shippingAddress": "902 Main St, Anytown, AN 68387",
        "shippingMethod": "Standard"
      },
      "timestamp": "2024-04-26T15:45:15.000000523Z"
    },
    {
      "message": {
        "msg": "payment for ITEM9798 succeeded",
        "paymentMethod": "PayPal",
        "totalAmount": 56.645267111988474
      },
      "timestamp": "2024-04-26T15:45:16.000000214Z"
    }
  ],
  "service": "checkout",
  "status": "info",
  "timestamp": "2024-04-26T15:45:14.000000138Z",
  "timestamp_end": "2024-04-26T15:45:16.000000905Z",
  "transactionId": "TX79924095"
}
```

{% endtab %}

{% tab title="Transform" %}

```yaml
- name: checkout
  type: reduce
  rules:
    # process when service is exactly equal to "checkout"
    process_when:
      - key: service
        op: EQUAL
        val: checkout
    # make sure these fields are still available at the "top level" instead of being nested
    pull_up:
      - message.transactionId
      - message.customerId
    # group all logs by the following top level keys
    group_by:
      - customerId
      - transactionId
    # specify the message field, the highlighted body of the log
    msg_field: message.msg
    # remove unecessary timestamp fields
    remove:
      - timestamp
      - message.timestamp
    # details how custom top level keys will be merged
    merge_strategies:
      transactionId: discard
      customerId: discard
```

{% endtab %}
{% endtabs %}

### Multi-Line Logs

Many times, an application will emit a single log across multiple lines such as the case with a JSON log. Unless you specifically account for this, most logging agents will consume each newline as a separate log event. Nimbus can identify when this happens and stitch these logs back together.

{% tabs %}
{% tab title="Before" %}

```json
[
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "{",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.108Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  \"id\": \"2460\",",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.134Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  \"method\": \"GET\",",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.147Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  \"url\": \"/health\",",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.160Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  \"query\": {},",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.174Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  \"params\": {},",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.187Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  \"headers\": {",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.199Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "    \"host\": \"100.119.27.217:8080\",",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.210Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "    \"user-agent\": \"kube-probe/1.18\",",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.221Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "    \"accept-encoding\": \"gzip\",",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.233Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "    \"connection\": \"close\"",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.245Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  },",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.256Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  \"remoteAddress\": \"::ffff:172.20.65.189\",",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.269Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  \"remotePort\": 60444",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.280Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "}",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.292Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "{",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.304Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  \"statusCode\": 200,",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.316Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  \"headers\": {",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.327Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "    \"x-powered-by\": \"Express\"",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.338Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "  }",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.350Z"
  },
  {
    "ddsource": "nimbus",
    "host": "some-host",
    "message": "}",
    "path": "/",
    "service": "healthcheck",
    "source_type": "http_server",
    "status": "info",
    "timestamp": "2023-11-23T00:05:58.361Z"
  }
]
```

{% endtab %}

{% tab title="After" %}

* 90% in event volume reduction
* 87% in ingest volume reduction

```json
{
  "ddsource": "nimbus",
  "host": "some-host",
  "message": "{  \"id\": \"2460\",  \"method\": \"GET\",  \"url\": \"/health\",  \"query\": {},  \"params\": {},  \"headers\": {    \"host\": \"100.119.27.217:8080\",    \"user-agent\": \"kube-probe/1.18\",    \"accept-encoding\": \"gzip\",    \"connection\": \"close\"  },  \"remoteAddress\": \"::ffff:172.20.65.189\",  \"remotePort\": 60444}{  \"statusCode\": 200,  \"headers\": {    \"x-powered-by\": \"Express\"  }}",
  "nimkind": "opt",
  "nimmatch": "healthcheck",
  "nimsize": 21,
  "path": "/",
  "service": "healthcheck",
  "source_type": "http_server",
  "status": "info",
  "timestamp": "2023-11-23T00:05:58.108Z"
}
```

{% endtab %}

{% tab title="Transform" %}

```yaml
- name: healthcheck
  type: reduce
  rules:
    process_when:
      - key: service
        op: EQUAL
        val: healthcheck
    group_by:
      - host
    msg_field: message
    starts_when:
      - key: message
        op: MATCH
        val: \n\{
    merge_strategies:
      msg_source: concat_newline
```

{% endtab %}
{% endtabs %}


# Concepts

### Lossless Aggregation

A method of aggregating observability data that preserves 100% the fidelity of the original data.

### Log Body

The body of the log entry. This value is displayed when browsing the log and usually indexed for full text search.

For JSON logs, the log body is usually represented by the value of the `message` key.

### Top Level Keys

Top level keys are the first level of keys in a json logs.

For example, take the following log:

```json
{
    "top": 1,
    "nested": {
        "inner": 2
    },
    "bottom": 3
}
```

In this case, `top`, `nested`, and `bottom`, would be top level keys

### Vector Remap Language (VRL)

[Vector Remap Language](https://vector.dev/docs/reference/vrl/) is a domain-specific language developed by [Vector](https://vector.dev/) for modifying your observability data.


# FAQ

## Product

### How long does it take to integrate Nimbus?

This depends on your specific vendor and your application integration. On average, for customers using the regular datadog agent to forward data, initial setup takes under 5 minutes.

### Is Nimbus an observability pipeline? How is it different from other services in this space?

1. Other services put the burden on you to understand your traffic patterns. Nimbus **automatically analyzes your traffic** and creates a top N list of high traffic patterns.
2. Other services make you manually create the rules and filters to create a pipeline. Nimbus **automatically generates transforms** based on its traffic analysis.
3. Other services have you sample and drop data to reduce volume. Nimbus **applies lossless aggregation** which means that you reduce volume without losing visibility.

### Do you support other vendors besides Datadog?

Yes. Please reach out to `support@nimbus.dev` to get details on vendor specific integrations

## Logs

### Does Nimbus introduce extra latency to my logs?

Nimbus processes logs in near real time - the average message is received and forwarded in under 100ms.

There is a caveat for aggregated logs. These are held in memory (and buffered on disk) until a [termination criteria](/overview/ntl/ntl.transforms#reduce-transform) is met (eg. `max_events`, `expire_after_ms`, etc). Aggregations can be disabled at any time. Nimbus also has a [pause all](/additional-features/advanced.pause-all) button that lets you disable all aggregations at once if needed.

Note that Nimbus has built-in rules to not aggregate error logs which means that they will still come through in near real time.

### Can I still query for the same data after my logs have been aggregated?

Yes. Because Nimbus uses [lossless aggregations](/overview/concepts#lossless-aggregation) to optimize your log volume, you don't end up losing any data. You can find out more in [working with aggregated logs](/overview/logs)

### Will this impact my log based monitors?

Short answer - no. Any log based monitor you currently have can be replicated post-aggregation, either with no changes or some small tweaks. When you onboard, a dedicated Nimbus engineer will work with you to ensure that none of your existing monitors will be impacted.

You can find out more in [working with aggregated logs](/overview/logs).

### Will this impact my log based dashboards?

No. See answer above about log based monitors.

### Does data get lost when I aggregate?

By default, Nimbus deduplicates common metadata and merges unique values from logs when aggregation. The only field that gets discarded is the `timestamp` field - Nimbus preserves the start time and adds a `timestamp_end` field to designate the time interval for the aggregation. You can see examples of what this looks like in the [examples](/overview/examples)

### Can I pause transformations during an incident?

Yes. See the [pause all](/additional-features/advanced.pause-all) section for details.

### What if Datadog changes its pricing model to not be event based?

Nimbus is extremely effective at reducing number of events (100X) and very effective at reducing the size of events (40%). So regardless of the type of pricing model, we will be able to deliver significant savings.

## Security and Availability

### What is your data retention policy?

Nimbus keeps observability data for a period of up to 7 days in order to analyze traffic patterns. It does not store or retain data beyond the observability window.

### What is your availability?

Nimbus offers a **99.9% SLA** on uptime. See more details [here](/resources/res.sla).

## Etc

### What if I've already committed to an annual commitment?

Even if you've made a commitment, it's likely that you'll exceed the committed usage and have on demand spend (overage) on top of the committed usage. Nimbus can drop the on demand portion to 0 and make sure you don't exceed it.

We can also help you negotiate with Datadog for alternative contracts with your account executive.


# Architecture

You can think of Nimbus as a data pipeline for your telemetry. We provide an out of the box opinionated framework to process your telemetry according to industry best practices.

## Flow Diagram

![Nimbus Pipeline](https://ik.imagekit.io/fpjzhqpv1/Nimbus%20Filter%20Pipeline_jDB_TNCmp.png?updatedAt=1701103529900)

## Components

### Global Ingress Preprocessor

* parse logs according to source format
* meter and derive analytics from ingress

### Global Router

* routes telemetry depending on condition
  * if message is identified as an error, forward to error route
  * if message matches a optimization predicate, forward it to filter route
  * all messages not processed by a transform or matched as an error go to the default route

### Error Route

* applies error specific attributes and optimizations

### Filter Routes

* applies optimization specific attributes and optimizations

### Default Route

* applies default attributes and optimizations
* currently, this applies `nimkind: raw` to the log

### Global Egress Processor

* meter and derive analytics from ingress


# Configuration Overrides

Nimbus is architected around observability pipeline best practices and usually requires no manual configuration. That said, we understand that real life systems are complex and more flexibility is needed.

To that end, configuration overrides let you override any part of the [Nimbus pipeline](broken://pages/Zu3Ksowuusm7qXXO3zlc) with your custom [VRL code](broken://pages/tgA3NyNSJruDUU0GKhT6#vector-remap-language-vrl).

{% hint style="warning" %}
Configuration Overrides is currently in **Limited Access**. Please contact `support@nimbus.dev` if you want to use it
{% endhint %}

## 1. Click on the Configuration Tab

![Configuration Override](https://ik.imagekit.io/fpjzhqpv1/home___Editor___Nimbus_JlhgzGTmC.png?updatedAt=1701708907491)

## 2. Edit a configuration

You can use any valid VRL to edit the configuration.

You can use the **Override** dropdown to change what part of the pipeline you wish to edit. The current options are:

* `nim/in/global_remap`: controls ingress. all data will pass by this transform
* `nim/out/global_remap`: controlls egress. all data that is sent upstream will pass by this transform

For a full list of configuration options, visit the [configuration reference](https://github.com/nimbushq/public-docs/blob/main/ref.config.md)

## 3. Save your configuration

Hit **Save** to apply your changes.


# Pause All

Nimbus supports pausing all transforms in times of distress.

## 1. Click on the `Pause All` button

Go to the **transforms** tab and then click **Pause All**

![Pause All](https://ik.imagekit.io/fpjzhqpv1/Cursor_and_home___Editor___Nimbus_SPF6LTf3K.png?updatedAt=1701712428401)

## 2. Confirm and Save

Clicking will open a modal with a dialogue box asking you to type `CONFIRM` to continue. Type the letters and hit confirm to un

## 3. Resume

When you are ready to resume transforms, click the **Resume** button to enable existing transforms


# Error Detection

By default, Nimbus automatically detects errors and routes error logs through a separate pipeline that does pass through aggregation. This means that error logs get routed through in near real time without any transformations.


# Private Link

Nimbus lets you set up private connectivity between your cloud provider and Nimbus.

## How it Works

![Datadog to Nimbus](https://ik.imagekit.io/fpjzhqpv1/Excalidraw_8CVOZRM7R.png?updatedAt=1713226588813)

With Nimbus Private Link, you can directly connect your VPC with Nimbus using AWS VPC Endpoints. Note that this is currently only supported for `AWS` accounts in region `us-east-1`.

## Benefits

* cost reduction: with private link, your egress cost go down by 90% (regular egress on AWS is $0.09/GB. With private link, this becomes $0.01/GB)
* compliance and security: prevent sensitive data from traversing the public internet

## Setup

### 1. Create a VPCEndpoint using our cloudformation template

[![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=nimbus-privatelink\&templateURL=https://nimbuscommonstack-nimbuspublicassetsde3f333d-vidman8dlq9v.s3.amazonaws.com/docs/private-link-prod.yaml)

Ensure Cloudformation stack is in status `CREATE_COMPLETE` and VPC Endpoint is Available with has Private DNS names enabled before proceeding

![Nimbus Private Link Setup](https://ik.imagekit.io/fpjzhqpv1/Cursor_and_cust_enable-privatelink_DLjvdf5Bw.png?updatedAt=1713228305706)

### 2. Verify the connection

You can test the endpoint by sending data to `$API_KEY-http-intake.privatelink.logs.us1.nimbus.dev` in a connected subnet

```sh
curl -v -d "{msg: ping}" https://$API_KEY-http-intake.privatelink.logs.us1.nimbus.dev 
```

> NOTE: Sending the request outside of the connected VPC will result in `403` response

### 3. Update your Nimbus Endpoints

To switch over to private link, update your Nimbus endpoint. to the new schema by adding `privatelink` to your Nimbus endpoint.

See [integrations](/integrations/integrations.datadog) specific docs for your integration endpoints.

```diff
- https://$API_KEY-$INTEGRATION-intake.logs.us1.nimbus.dev
+ https://$API_KEY-$INTEGRATION-intake.privatelink.logs.us1.nimbus.dev
```


# Datadog

The following is a list of currently documented integration sources with datadog

* [AWS Lambda Extension](/integrations/integrations.datadog/datadog.lambda-extension)
* [AWS Lambda Forwarder](/integrations/integrations.datadog/datadog.lambda-log-forwarder)
* [DD Agent](/integrations/integrations.datadog/datadog.agent)
* [DD Log Forwarding Destination](/integrations/integrations.datadog/datadog.log-forwarding)
* [Heroku](/integrations/integrations.datadog/datadog.heroku)
* [Journald](/integrations/integrations.datadog/datadog.journald)
* [OpenTelemetry](/integrations/integrations.datadog/datadog.otel)


# AWS Lambda Extension

This guide goes over integrating Nimbus with the [Datadog Lambda Extension](https://docs.datadoghq.com/serverless/libraries_integrations/extension/.)

## Steps

1. Go to the lambda that you want to forward logs from
2. Add the following environmental variables to lambdas you want to add logs to

   ```yaml
   DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED: true,
   DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL: $YOUR_NIMBUS_ENDPOINT
   ```
3. Update your lambda function
4. That's it - you're done. Nimbus is now optimizing your logs!


# AWS Lambda Forwarder

## Steps

1. Go to cloudformation, and search “datadog”, you should either see one of the following
   * `datadog-forwarder` as its own entry
   * `DatadogIntegration-ForwarderStack-***` as a nested stack
2. Click on the forwarder stack and click **update**
3. Select `use current template`
4. Set the following 2 parameters:
   * `DdUrl: $YOUR_NIMBUS_ENDPOINT`
   * `DdPort: 443`
5. Update the stack


# DD Agent

### 1. Add Nimbus as a log endpoint

Add the following configuration to your datadog config

```yaml
observability_pipelines_worker:
  logs:
    enabled: true
    url: "https://YOUR_NIMBUS_ENDPOINT"
```

> NOTE: YOUR\_NIMBUS\_ENDPOINT is an URL that is generated for you when you first create an account

Optionally, you can configure the endpoint using the following environmental variables. This is useful when you're running the datadog agent in kubernetes like environments and don't have easy access to the raw configuration.

```bash
DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED=true 
DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL="https://YOUR_NIMBUS_ENDPOINT"
```

Update your datadog agent to run with the new configuration. Congratulations - you're now forwarding log traffic with Nimbus!


# DD Log Forwarding Destination

This guide walks you through forwarding your logs to Nimbus.

## 1. Setup a custom log forwarding destination in datadog

In datadog's [log forwarding](https://docs.datadoghq.com/logs/log_configuration/forwarding_custom_destinations/?tab=http) configuration, select a custom destination.

Add the following details:

* Name: `nimbus`
* Endpoint: `$YOUR_NIMBUS_ENDPOINT` (this will be provided by nimbus)
* Authentication: Basic Auth
* Username and Password: (this will be provided by nimbus)

Click Save when done

Congratulations, you're now forwarding logs to Nimbus!


# Heroku

## Prerequisites

* [connect your datadog account to Nimbus](/overview/quickstart#1-connect-your-datadog-account)

## Steps

### 1. Add your Nimbus Endpoint as a Log Drain

```sh
heroku drains:add "https://$YOUR_NIMBUS_ENDPOINT?dd-api-key=<DD_API_KEY>&ddsource=heroku&env=<ENV>&service=<SERVICE>&host=<HOST>" -a <APPLICATION_NAME>
```

### 2. Add custom attributes (optional)

You can add custom attributes by appending it to the end of your drain query string

```sh
heroku drains:add "https://$YOUR_NIMBUS_ENDPOINT?dd-api-key=<DD_API_KEY>&ddsource=heroku&env=<ENV>&service=<SERVICE>&host=<HOST>&<attKey>=<attValue>" -a <APPLICATION_NAME>
```

### 3. Confirm receiving logs

Verify via the datadog log console that logs are coming in with the expected attributes

### 4. Remove your old endpoint

```sh
heroku drains:remove <id-of-dd-drain>
```


# Journald

## Steps

1. Install vector in your target environment.

```sh
curl --proto '=https' --tlsv1.2 -sSfL https://sh.vector.dev | bash -s -- -y
```

2. Install the nimbus configuration

   > NOTE: you'll need to replace `$YOUR_NIMBUS_ENDPOINT` with your specific endpoint

```sh
cat << EOF > /usr/local/etc/
data_dir: /tmp/
api:
address: 0.0.0.0:8686
enabled: true
playground: false
sources:
source/journald:
    type: journald
    current_boot_only: true
sinks:
sink/nimbus:
    type: http
    encoding: 
    codec: json
    compression: gzip
    inputs:
    - source/*
    uri: $YOUR_NIMBUS_ENDPOINT
EOF
```

3. Create the systemd script

```sh
cat << EOF > /tmp/nimsetup.sh
#!/bin/bash

# Define the binary and service names
BINARY_PATH=$1
BINARY_NAME="vector"
SERVICE_NAME="vector.service"

# Copy the binary to /usr/local/bin
echo "Copying $BINARY_PATH to /usr/local/bin..."
cp "$BINARY_PATH" "/usr/local/bin/$BINARY_NAME"
chmod +x "/usr/local/bin/$BINARY_NAME"

# updating config file
chmod a+r /usr/local/etc/vector-config.yaml

# Create a systemd service file
SERVICE_FILE_PATH="/etc/systemd/system/$SERVICE_NAME"
echo "Creating $SERVICE_FILE_PATH..."

cat <<EOF1 > "$SERVICE_FILE_PATH"
[Unit]
Description=Nimbus Collector
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/$BINARY_NAME -c /usr/local/etc/vector-config.yaml
Restart=on-abort

[Install]
WantedBy=multi-user.target
EOF1

# Reload systemd to recognize the new service
echo "Reloading systemd manager configuration..."
systemctl daemon-reload

# Enable the service to start on boot
echo "Enabling $SERVICE_NAME..."
systemctl enable "$SERVICE_NAME"

# Start the service
echo "Starting $SERVICE_NAME..."
systemctl start "$SERVICE_NAME"

echo "$SERVICE_NAME is now running."
EOF
```

4. Execute

```sh
. ~/.zprofile
chmod +x /tmp/nimsetup.sh
sudo /tmp/nimsetup.sh `which vector`
```

5. Verify

```sh
systemctl status vector

vector.service - Nimbus Collector
    Loaded: loaded (/etc/systemd/system/vector.service; enabled; vendor preset: enabled)
    Active: active (running) since Thu 2024-02-29 22:41:32 UTC; 6min ago
Main PID: 3505 (vector)
    Tasks: 6 (limit: 18945)
    Memory: 16.5M
```


# OpenTelemetry

This guide goes over integrating Nimbus with the [OpenTelemetry](https://opentelemetry.io/) Collector.

## Steps

1. In your OTEL collector, add an [otlphttp exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter) - replace `$API_KEY` with your Nimbus API key

```yaml
exporters:
  otlphttp/nimbus:
    endpoint: https://$API_KEY-otlp-intake.logs.us1.nimbus.dev:443
```

2. Add the `otlphttp` exporter to any existing pipeline that processes logs

```yaml
service:
  pipelines:
    ...
    $YOUR_PIPELINE:
      ...
      exporters: [..., otlphttp/nimbus]
```

3. Reload existing collectors with the new configuration.
4. That's it - you're done. Nimbus is now optimizing your logs!


# Nimbus Attributes

Nimbus adds custom attributes to logs that it processes

### nimdata

* type: list

Contains the raw values of reduced logs

### nimkind

* type: enum
* values:
  * `opt`: processed by a nimbus transform
  * `noopt`: not processed by nimbus transform
  * `error`: detected as an error by nimbus

### nimsize

* optional (only present if logs have been reduced)
* type: int

Number of items in `data` field

### nimmatch

* optional (only present if logs match a transform)

Name of the transform that has processed the log


# Changelog

## 0.23

Release Date: 2024/04/26

### Features

Want to try Nimbus without speaking to a sales rep? Nimbus now supports self serve onboarding!

All new accounts get a free 14 day trial and can send any amount of data to Nimbus without any caps! You can get started by signing up from the [hub](https://hub.nimbus.dev/) and setup your [first integration](/overview/quickstart) in under 10 minutes!

## 0.22

Release Date: 2024/04/12

### Features

You can now reduce you data egress fees by 90% using Nimbus Private Link 🎉

Data egress fees are the "hidden cost" of observability. They are hard to detect because they don't show up in your observability vendor bill but rather in the data transfer fees from your cloud provider and can double your ingest costs for observability.

As an example, AWS charges $0.09/GB for egress (for comparison, datadog charges $0.10/GB for data ingress). Private link sets up a private connection between your VPC and Nimbus and reduces the cost of data transfer to $0.01/GB.

Private link is available for free to all Nimbus customers in `us-east-1`. Instructions for getting started [here](/additional-features/advanced.private-link)

## 0.21

Release Date: 2024/03/28

### Features

* engine: Nimbus now supports [lint optimizations](/overview/optimization.log/optimization.log.lint). Lint optimizations scan logs for common hygiene issues like sensitive (eg. api tokens) and redundant data (eg. timestamp appearing the log)

### Enhancements

* pipeline: improvements pipeline p99 latency by using provisioned throughput for block storage
* pipeline: reduce impact of az failover by buffering telemetry data on disk across multiple azs

## 0.20

Release Date: 2024/03/14

### Enhancements

* ntl: support [remove\_from\_nimdata](/overview/ntl/ntl.transforms#remove_from_nimdata) directive
* ntl: support [remove\_nimdata](/overview/ntl/ntl.transforms#remove_nimdata) directive

### Fixes

* api: sink updates are now immediately applied

## 0.19

Release Date: 2024/02/29

### Features

We launched a [datadog cli](https://github.com/nimbushq/dd-cli). this lets you query logs from your terminal. If you're querying aggregated logs, this also gives you the option to disaggregate them into individual log lines.

### Enhancements

* ui: handle various text overflow issues in the ui

## 0.18

Release Date: 2024/02/15

\| Heads up that we will be switching to a bi-weekly release model moving forward due to the growing scope of what the team is currently taking on.

### Enhancements

* ui: you can now update your observability sinks in the UI
* ntl: Nimbus now support the `NOT` operator

### Fixes

* UI now shows all modals in full screen regardless of content size

## 0.17

Release Date: 2024/02/08

### Enhancements

* self serve onboarding: you can now provision and manage your Nimbus destinations without human contact (that said, we're still here if you need us)
* UI improvements: snappier page loads and consisting alignment of tables and elements

## 0.16

Release Date: 2024/02/01

### Features

Nimbus SLA - Nimbus now has a [public SLA](/resources/res.sla) of 99.999% uptime. You can follow our public status page [here](https://status.nimbus.dev/) to be notified of incidents.

## 0.15

Release Date: 2024/01/25

### Enhancements

* Nimbus now shows ingest bytes reduction in addition to event reduction on optimizations

## 0.14

Release Date: 2024/01/18

### Features

* optimization fidelity: you can now customize log fidelity during optimization, choosing between preserving 100% of the original input and minimizing data ingest size

## 0.13

Release Date: 2024/01/11

### Enhancements

* support `max`, `min`, `retain`, `flat_unique`, and `longest_array` [merge\_strategies](/overview/ntl/ntl.transforms#merge_strategies)

### Fixes

* bad validation rule when updating a transform causes update to fail on certain merge strategies

## 0.12

Release Date: 2024/01/04

New year, new look.

We launched the Nimbus website. You should have received new credentials in your email. The new website offers a much snappier and light weight version of our previous retool application and will enable us to ship much more ambitious features later this year.

### Features

* new frontend at <https://hub.nimbus.dev>

## 0.11

Release Date: 2023/12/28

### Features

* support `optimization_mode` to tune Nimbus optimization
* traffic analysis now shows volume reduction by either events or size

### Enhancements

* support [`remove` directive](/overview/ntl/ntl.transforms#remove) which accepts a list of paths to be deleted from the target payload

## 0.10

Release Date: 2023/12/21

### Enhancements

* support `additional_sinks` in [config overrides](https://github.com/nimbushq/public-docs/blob/main/ref.config.md)
* support `additional_sources` in [config overrides](https://github.com/nimbushq/public-docs/blob/main/ref.config.md)
* support `MATCH_ANY` [operation](/overview/ntl#match_any)

## 0.9

Release Date: 2023/12/14

### Features

* [Metrics Hub](/preview-features/metrics): support aggregating metrics across hosts (private preview)

### Enhancements

* better error messages when there is a syntax error when manually creating a transform

## 0.8

Release Date: 2023/12/07

### Features

* [pull\_up](/overview/ntl/ntl.transforms#pull_up) directive allows you to pull properties inside of nested objects to the top level
* support for the [NOT operator](/overview/ntl#not) inside of compound statements

## 0.7

Release Date: 2023/11/23

### Features

* **Pause All**: Support manual override to [pause all transforms](/additional-features/advanced.pause-all)

### Enhancements

* clicking on the log pattern from analysis will now take you directly to events in datadog
* transforms now show usage stats
* revamped documentation with demo video
* reduce egress bytes by removing `nimraw` data from non-optimized logs

## 0.6

Release Date: 2023/11/09

### Features

* analysis now shows log output preview for generated transformation
* support modifying transforms generated by analysis

## 0.5

Release Date: 2023/10/26

### Features

* analysis now shows log samples for findings
* support ability to delete a transform

### Enhancements

* analysis now auto generates names for findings
* analysis now highlights new findings

## 0.4

Release Date: 2023/10/12

### Features

* **Configuration Overrides**: You can now add custom VRL to control all aspects of the ingress stage of the nimbus pipeline

### Enhancements

* support [starts\_when](/overview/ntl/ntl.transforms#starts_when) directive
* support [msg\_field](/overview/ntl/ntl.transforms#msg_field) directive
* support [max\_events](/overview/ntl/ntl.transforms#max_events) directive
* support [expire\_after\_ms](/overview/ntl/ntl.transforms#expire_after_ms) directive
* support [merge\_strategies](/overview/ntl/ntl.transforms#merge_strategies) directive

## 0.3

Release Date: 2023/09/28

### Features

* **Usage dashboards**: you can now access your usage dashboard graphs

### Enhancements

* Smarter error detection - we now autodetect errors based on [bunyan/pino levels format](https://github.com/trentm/node-bunyan#levels)

## 0.2

Release Date: 2023/09/14

### Features

* **Heroku Datadog Integration**: Support Datadog [Heroku Log Integration](/integrations/integrations.datadog/datadog.heroku)
* **Heroku Log Forwarding Integration**: Support Datadog [Log Forwarding Integration](/integrations/integrations.datadog/datadog.log-forwarding)

### Enhancemehts

## 0.1

Release Date: 2023/08/31

Hello world!

### Features

* **Nimbus Transformation Language (NTL)**: A high level language for working with telemetry data.
* **Nimbus Traffic Analysis**: Automatically identify high traffic log patterns
* **Nimbus Transform Recommendations**: Auto generated transforms using the [Nimbus Transformation Language (NTL)](/overview/ntl) based on traffic analysis results


# SLA

Last Updated: April 4th, 2024

This Nimbus Service Level Agreement (“SLA”) is a policy governing the use of Nimbus and applies separately to each account using Nimbus.

Capitalized terms used herein but not defined herein shall have the meanings set forth in the Agreement.

### Service Commitment

Nimbus commits to use commercially reasonable efforts to make the Observability Pipeline, specifically focusing on the data ingestion component, available with the Monthly Uptime Percentages set forth in the table below. In the event the Observability Pipeline does not meet the Service Commitment, you will be eligible to receive a Service Credit as described below.

### Definitions

An “**Observability Pipeline**” refers to the infrastructure and services provided by Nimbus for the collection, normalization, transformation, and routing of observability data (e.g., metrics, logs, traces) for a specific domain.

“**Monthly Uptime Percentage**” for the Observability Pipeline is calculated by subtracting from 99.999% the percentage of minutes during the month in which the data ingestion component of the Observability Pipeline was Unavailable. Monthly Uptime Percentage measurements exclude Unavailability resulting directly or indirectly from any Nimbus SLA Exclusions.

A “**Service Credit**” is a dollar credit, calculated as set forth above, that we may credit back to an eligible account.

The Observability Pipeline is “**Unavailable**” during a given minute if the data ingestion component fails to receive and process data for all attempts made to the pipeline throughout the minute.

### Service Credits

Service Credits are calculated as a percentage of the total charges paid by you for the affected component of the Observability Pipeline for the monthly billing cycle in which the Service Commitment was not met, in accordance with the schedule below:

| Monthly Uptime Percentage                              | Service Credit Percentage |
| ------------------------------------------------------ | ------------------------- |
| Less than 99.999% but greater than or equal to 99.995% | 10%                       |
| Less than 99.995% but greater than or equal to 99.99%  | 25%                       |
| Less than 99.99%                                       | 80%                       |

We will apply any Service Credits only against future Nimbus payments otherwise due from you. At our discretion, we may issue the Service Credit to the credit card you used to pay for the billing cycle in which the Unavailability occurred. Service Credits will not entitle you to any refund or other payment from Nimbus. A Service Credit will be applicable and issued only if the credit amount for the applicable monthly billing cycle is greater than one dollar ($1 USD). Service Credits may not be transferred or applied to any other account. Unless otherwise provided in the Agreement, your sole and exclusive remedy for any unavailability, non-performance, or other failure by us to provide the Observability Pipeline is the receipt of a Service Credit (if eligible) in accordance with the terms of this SLA.

### Credit Request and Payment Procedures

To receive a Service Credit, you must submit a claim by contacting Nimbus support team. To be eligible, the credit request must be received by us by the end of the second billing cycle after which the incident occurred and must include:

* i. the words “SLA Credit Request” in the subject line;
* ii. the dates, times, and descriptions of each Unavailability incident that you are claiming;
* iii. evidence that corroborates the claimed Unavailability, such as logs or monitoring alerts (any confidential or sensitive information in these documents should be removed or replaced with asterisks).

If the Monthly Uptime Percentage of such request is confirmed by us and is less than the Service Commitment, then we will issue the Service Credit to you within one billing cycle following the month in which the request occurred. Your failure to provide the request and other information as required above will disqualify you from receiving a Service Credit.

### Nimbus SLA Exclusions

The Service Commitment does not apply to any unavailability, suspension, or termination of the Observability Pipeline, or any other Nimbus performance issues: (i) caused by factors outside of our reasonable control, including any force majeure event or Internet access or related problems beyond the demarcation point of Nimbus; (ii) that result from any actions or inactions by you or any third party; (iii) that result from your equipment, software, or other technology and/or third party equipment, software, or other technology (other than third party equipment within our direct control); (iv) arising from our suspension or termination of your right to use the Observability Pipeline in accordance with the Agreement; or (v) that result from your failure to follow the guidelines and best practices described in Nimbus documentation, including exceeding usage limits. If availability is impacted by factors other than those used in our Monthly Uptime Percentage calculation, then we may issue a Service Credit considering such factors at our discretion.


# Bug Bounty

## Introduction

Nimbus is committed to maintaining the security and integrity of our services. We understand that no technology is perfect, and we believe in working collaboratively with the security community to find and resolve vulnerabilities. Our bug bounty program encourages this collaboration by rewarding security researchers who provide us with high-quality security information.

## Scope

This program covers the following application(s) and services:

* Nimbus Website: <https://hub.nimbus.dev>
* Nimbus API: <https://api.nimbus.dev>
* Nimbus Data Pipeline

The following are explicitly out of scope:

* Third-party services and dependencies
* Denial of Service (DoS) attacks
* Spam or social engineering techniques

## Eligibility

Participants must:

* Not be a former or current employee of Nimbus or its affiliates.
* Not violate any laws or breach any agreements in order to discover vulnerabilities.
* Adhere to the guidelines and scope of this program.

## Rewards

Nimbus provides rewards as follows:

* Critical vulnerabilities: Up to $1000
* High severity vulnerabilities: Up to $500
* Medium severity vulnerabilities: Up to $200
* Low severity vulnerabilities: Recognition in our Hall of Fame

Reward amounts are determined by the impact, ease of exploitation, and quality of the report. Decisions on reward eligibility and amounts are made by Nimbus and are final.

## Submission Guidelines

To submit a vulnerability, please follow these guidelines:

* Provide detailed steps to reproduce the vulnerability, including proof of concept (PoC) code if applicable.
* Include your contact information for further communication.
* Do not disclose the vulnerability publicly or to any third parties without explicit permission from Nimbus.

Submissions should be sent to `security(at)nimbus.dev`

## Legal

Participants agree to:

* Handle any confidential information obtained through this program responsibly.
* Refrain from exploiting any vulnerabilities beyond what is necessary for demonstration purposes.
* Comply with all applicable laws and regulations.

Nimbus commits to:

* Respond promptly to submissions.
* Not pursue legal action against researchers who adhere to this policy.
* Work with researchers to understand and remediate reported vulnerabilities.

## Contact

For questions or more information about the bug bounty program, please contact `security(at)nimbus.dev`.


# Support

You can reach out to `support@nimbus.dev` for any queries regarding Nimbus. If you're on any paid plan, you also have priority support via a dedicated slack channel.


# Metric Optimization

Nimbus Metric Hub enables you to pre-aggregate your host metrics before sending them to datadog. This means we can reduce your billable infrastructure host count by an order of magnitude while still preserving the individual metrics for each host.

{% embed url="<https://www.loom.com/share/3f107f7310e245bd98b8ee19af13be6f>" %}

Nimbus Metric Optimization is currently in private preview. To get early access, please reach out to `support@nimbus.dev`


# Datadog CLI

The Datadog CLI lets you query logs from your terminal. If you're querying aggregated logs, this also gives you the option to disaggregate them into individual log lines.

## setup

```sh
git clone git@github.com:nimbushq/dd-cli.git
cd dd-cli
yarn && yarn build
npm link
```

## logs

```sh
nimbus logs

Search across dd logs. The following environmental variables need to be set in
order to run this command: DD_SITE, DD_API_KEY, DD_APP_KEY

Options:
      --version                             Show version number        [boolean]
      --help                                Show help                  [boolean]
  -q, --query                               dd log query     [string] [required]
  -f, --from                                time in the following format:
                                            2024-02-14T11:35:00-08:00
                                                             [string] [required]
      --to, --t, desc: "time in the
      following format:
      2024-02-14T11:35:00-08:00"                             [string] [required]
  -i, --indexes                             log indexes to search
                                                     [array] [default: ["main"]]
  -d, --disaggregate                        disaggregate aggregated logs
                                                                       [boolean]
```

## Usage

Replace env variables with your org specific values

```sh
# regular query
env DD_SITE="***" DD_API_KEY="***" DD_APP_KEY="***" dd-cli logs -q "service:elb " -f "2024-02-14T11:35:00-08:00" -t "2024-02-14T11:38:00-08:00"

# query with log disaggregation
env DD_SITE="***" DD_API_KEY="***" DD_APP_KEY="***" dd-cli logs -q "service:elb " -f "2024-02-14T11:35:00-08:00" -t "2024-02-14T11:38:00-08:00" -d
```