Pipe Plugins

The pipe plugin configurations contain a series of pipe configuration. Each pipe configuration has 5 parts, which are common_config, gatherer, processor and the sender.

common_config

Config Description
pipe_name The unique collect space name.

Gatherer

The gatherer has 2 roles, which are the receiver and fetcher.

Receiver Role

Config Description
server_name The server name in the sharing pipe, which would be used in the receiver plugin.
receiver The receiver configuration. Please read the doc to find all receiver plugins.
queue The queue buffers the input telemetry data. Please read the doc to find all queue plugins.

Fetcher Role

Config Description
fetch_interval The time interval between two fetch operations. The time unit is millisecond.
fetcher The fetcher configuration. Please read the doc to find all fetcher plugins.
queue The queue buffers the input telemetry data. Please read the doc to find all queue plugins.

processor

The filter configuration. Please read the doc to find all filter plugins.

sender

Config Description
flush_time The time interval between two flush operations. And the time unit is millisecond.
max_buffer_size The maximum buffer elements.
min_flush_events The minimum flush elements.
client_name The client name used in the forwarders of the sharing pipe.
forwarders The forwarder plugin list. Please read the doc to find all forwarders plugins.
fallbacker The fallbacker plugin. Please read the doc to find all fallbacker plugins.

Example

pipes:
  - common_config:
      pipe_name: pipe1
    gatherer:
      server_name: "grpc-server"
      receiver:
        plugin_name: "grpc-native-log-receiver"
      queue:
        plugin_name: "mmap-queue"
        segment_size: ${SATELLITE_MMAP_QUEUE_SIZE:524288}
        max_in_mem_segments: ${SATELLITE_MMAP_QUEUE_MAX_IN_MEM_SEGMENTS:6}
        queue_dir: "pipe1-log-grpc-receiver-queue"
    processor:
      filters:
    sender:
      fallbacker:
        plugin_name: none-fallbacker
      flush_time: ${SATELLITE_PIPE1_SENDER_FLUSH_TIME:1000}
      max_buffer_size: ${SATELLITE_PIPE1_SENDER_MAX_BUFFER_SIZE:200}
      min_flush_events: ${SATELLITE_PIPE1_SENDER_MIN_FLUSH_EVENTS:100}
      client_name: kafka-client
      forwarders:
        - plugin_name: native-log-kafka-forwarder
          topic: ${SATELLITE_NATIVELOG-TOPIC:log-topic}