Working with Transforms

Overview

Nimbus Transforms are high level NTL functions that have specialized logic for specific optimizations.

Once an optimization is applied, you can find its corresponding transformation in the transforms section of the console.

Transforms Console

You can click on Edit to either update or delete an existing transform.

Global Options

The following properties are available on all transforms

process_when

• status: required

• type: NTL

Determines when a transform should be applied. Takes one or more NTL predicates as input.

Example:

process_when: 
  - {key: service, op: EQUAL, val: foo}

include_errors

• status: optional

• type: boolean

• default: false

When set to true, designate that the current transform can apply to error logs. By default, error logs are not transformed but immediately proxied downstream for immediate processing.

msg_field

• status: optional

• type: string

• default: message

The key where the log body is located

Example:

# logs sent by dd lambda extension have the message field nested inside the message key
# eg:
# {message: { message: "START ...", lambda: {arn: arn:aws:lambda:us-east-1:33333333:function:test-lambda, ...}}}
msg_field:
  - message.message

pull_up

• status: optional

• type: string[]

When specified, a list of paths that should be made into top level keys

Example:

pull_up:
  - message.transactionId

Before:

{
  "message": {
    "transactionId": 1,
    ...
  }
}

After:

{
  "transactionId": 1,
  "message": {
    ...
  }
}

remove

• status: optional

• type: string[]

When specified, a list of paths that should be removed

Example:

remove:
  - message.id
  - message.source
  - message.timeout

remove_from_nimdata

• status: optional

• type: string[]

If set, removes the selected paths from nimdata

Example:

remove_from_nimdata: 
  - status
  - hostname
  - ...

remove_nimdata

• status: optional

• type: boolean

If set, removes the nimdata attribute. Helps with significantly removing dataisze

Example:

remove_nimdata: true

Reduce Transform

The Nimbus reduce transform is a superset of the vector reduce transform.

When using reduce, remember that group_by only works on top level keys

If the key you need is nested, make sure to pull it up using the pull_up directive.

Options

merge_strategies

• status: optional

• type: enum

The default behavior is as follows:

• The first value of a string field is kept and subsequent values are discarded.

• For timestamp fields the first is kept and a new field [field-name]_end is added with the last received timestamp value.

• Numeric values are summed.

Strategies:

Option	Description
array	Append each value to an array.
concat	Concatenate each string value, delimited with a space.
concat_newline	Concatenate each string value, delimited with a newline.
concat_raw	Concatenate each string, without a delimiter.
discard	Discard all but the first value found.
flat_unique	Create a flattened array of all unique values.
longest_array	Keep the longest array seen.
max	Keep the maximum numeric value seen.
min	Keep the minimum numeric value seen.
retain	Discard all but the last value found.

starts_when

• status: optional

• type: NTL

A condition used to distinguish the first event of a transaction. If this condition resolves to true for an event, the previous transaction is flushed (without this event) and a new transaction is started.

Example:

starts_when:
  - {key: message, op: MATCH, val: "\n\{"}

max_events

• status: optional

• type: integer

The maximum number of events to group together.

Example:

max_events: 200

expire_after_ms

• status: optional

• type: integer

• default: 30000

The maximum period of time to wait after the last event is received, in milliseconds, before a combined event should be considered complete.

Example

Suppose you have the following logs:

[
  {
    "host": "host1",
    "fooatt": "one",
    "baratt": "alpha"
  },
  {
    "host": "host2",
    "fooatt": "two",
    "baratt": "beta"
  },
  {
    "host": "host1",
    "fooatt": "three",
    "baratt": "gamma"
  },
  {
    "host": "host1",
    "baratt": "gamma"
  }
]

And you have the following reduce transform

name: hostreducer
# only apply this reducer when the log event has both a `host` and `fooatt` keys
process_when: 
  - {key: host, op: exists, val: true}
  - {key: fooatt, op: exists, val: true}
group_by:
  - host

Your processed logs would look like the following

[
  // this log was processed and grouped correctly
  {
    "host": "host1",
    "nimdata": [
      {
        "host": "host1",
        "fooatt": "one",
        "baratt": "alpha"
      },
      {
        "host": "host1",
        "fooatt": "three",
        "baratt": "gamma"
      }
    ],
    "nimsize": 2,
    "nimkind": "opt",
    "nimmatch": "hostreducer"
  },
  {
    "host": "host2",
    "nimdata": [
      {
        "host": "host2",
        "fooatt": "two",
        "baratt": "beta"
      }
    ],
    "nimsize": 1,
    "nimkind": "opt",
    "nimmatch": "hostreducer"
  },
  // this log did not get processed as it did not have a `fooatt` key
  {
    "host": "host1",
    "baratt": "gamma",
    "nimkind": "noopt"
  }
]