etl

Event fields used to define/normalize specific metadata about the event during the processing of an ETL (Extract, Transform, Load) pipeline.

Attributes

Name

Type

Description

Sample Value

etl_format_applied

string

Formatting or encoding applied during the ETL processing. Also referred to as CODEC in some use cases. Can be an array if multiple formats were applied/determined

[ "sylog", "json" ]

etl_format_is_cef

boolean

During ETL processing, event is determined to be CEF (format)

false

etl_format_is_json

boolean

During ETL processing, event is determined to be JSON (format)

true

etl_format_is_syslog

boolean

During ETL processing, event is determined to be Syslog (format). Technically you could send data encoded in different format over syslog (ie: CEF or JSON), therefore an event/log can have this tag/field as well as other format fields

true

etl_format_is_xml

boolean

During ETL processing, event is determined to be XML (format)

true

etl_host_agent_type

string

Type of forwarder from the client (i.e. winlogbeat, nxlog, rsyslog, etc)

nxlog

etl_host_agent_uid

string

UID for the host’s software/agent a part of the event

fe4fb818-088f-4529-a343-b94baf057a53

etl_info_tags

string

Use for any additional information about an event/log during ETL/processing pipeline. Commonly, you would use this for things that are rare but happen (i.e. parsing error for non conforming RFC). Use this field to alert or give context to a user/analyst when looking at the data.

inferred network_protocol as udp

etl_input_application_name

string

Application name used to receive or gather the log for the very first portion of the ETL processing (i.e. kafka, beats, syslog)

kafka

etl_input_application_protocol

string

Application protocol used to receive or gather the log for the very first portion of the ETL processing (ex: syslog, http, sftp)

kafka

etl_input_port

integer

Port (network) used to receive or gather the log for the very first portion of the ETL processing

9092

etl_input_protocol

string

Protocol (network) used to receive or gather the log for the very first portion of the ETL processing (ie: tcp, udp, icmp)

tcp

etl_input_src_port

integer

The Port (network) the client/source used to send the log for the very first portion of the ETL processing. Mostly used in syslog

48231

etl_kafka_consumer_group

string

Consumer group that the etl was apart of from consuming from a Kafka topic

helk_logstash

etl_kafka_key

string

Record key, if any

``````

etl_kafka_offset

long

Kafka partition for the event

204802842

etl_kafka_partition

integer

Kafka partition for the event

1

etl_kafka_time

date

Depending on your Kafka broker configuration, this can be either when the record was created (default) or when it was received by the broker

4/11/2018 5:49:25

etl_kafka_topic

string

Kafka topic name

winevent

etl_pipeline

string

Used to keep track of tags related to transforms, enrichment, or modifications made in an ETL pipeline

all-add_processed_timestamp

etl_processed_time

date

The first time the event gets processed by the ETL (processing pipeline)

4/11/2018 5:49:25

etl_version

string

The schema or transform versioning that is being applied

v1.0.1