etl
Contents
etl¶
Event fields used to define/normalize specific metadata about the event during the processing of an ETL (Extract, Transform, Load) pipeline.
Attributes¶
Name |
Type |
Description |
Sample Value |
|---|---|---|---|
etl_format_applied |
string |
Formatting or encoding applied during the ETL processing. Also referred to as CODEC in some use cases. Can be an array if multiple formats were applied/determined |
|
etl_format_is_cef |
boolean |
During ETL processing, event is determined to be CEF (format) |
|
etl_format_is_json |
boolean |
During ETL processing, event is determined to be JSON (format) |
|
etl_format_is_syslog |
boolean |
During ETL processing, event is determined to be Syslog (format). Technically you could send data encoded in different format over syslog (ie: CEF or JSON), therefore an event/log can have this tag/field as well as other format fields |
|
etl_format_is_xml |
boolean |
During ETL processing, event is determined to be XML (format) |
|
etl_host_agent_type |
string |
Type of forwarder from the client (i.e. winlogbeat, nxlog, rsyslog, etc) |
|
etl_host_agent_uid |
string |
UID for the host’s software/agent a part of the event |
|
etl_info_tags |
string |
Use for any additional information about an event/log during ETL/processing pipeline. Commonly, you would use this for things that are rare but happen (i.e. parsing error for non conforming RFC). Use this field to alert or give context to a user/analyst when looking at the data. |
|
etl_input_application_name |
string |
Application name used to receive or gather the log for the very first portion of the ETL processing (i.e. kafka, beats, syslog) |
|
etl_input_application_protocol |
string |
Application protocol used to receive or gather the log for the very first portion of the ETL processing (ex: syslog, http, sftp) |
|
etl_input_port |
integer |
Port (network) used to receive or gather the log for the very first portion of the ETL processing |
|
etl_input_protocol |
string |
Protocol (network) used to receive or gather the log for the very first portion of the ETL processing (ie: tcp, udp, icmp) |
|
etl_input_src_port |
integer |
The Port (network) the client/source used to send the log for the very first portion of the ETL processing. Mostly used in syslog |
|
etl_kafka_consumer_group |
string |
Consumer group that the etl was apart of from consuming from a Kafka topic |
|
etl_kafka_key |
string |
Record key, if any |
`````` |
etl_kafka_offset |
long |
Kafka partition for the event |
|
etl_kafka_partition |
integer |
Kafka partition for the event |
|
etl_kafka_time |
date |
Depending on your Kafka broker configuration, this can be either when the record was created (default) or when it was received by the broker |
|
etl_kafka_topic |
string |
Kafka topic name |
|
etl_pipeline |
string |
Used to keep track of tags related to transforms, enrichment, or modifications made in an ETL pipeline |
|
etl_processed_time |
date |
The first time the event gets processed by the ETL (processing pipeline) |
|
etl_version |
string |
The schema or transform versioning that is being applied |
|