Traffic

The traffic is used to collecting the network access logs from services through the Service Discovery, and send access logs to the backend server for analyze.

Configuration

Name Default Environment Key Description
access_log.active false ROVER_ACCESS_LOG_ACTIVE Is active the access log monitoring.
access_log.exclude_namespaces istio-system,cert-manager,kube-system ROVER_ACCESS_LOG_EXCLUDE_NAMESPACES Exclude processes in the specified Kubernetes namespace. Multiple namespaces split by “,”
access_log.exclude_cluster ROVER_ACCESS_LOG_EXCLUDE_CLUSTER Exclude processes in the specified cluster which defined in the process module. Multiple clusters split by “,”
access_log.flush.max_count 2000 ROVER_ACCESS_LOG_FLUSH_MAX_COUNT The max count of the access log when flush to the backend.
access_log.flush.period 5s ROVER_ACCESS_LOG_FLUSH_PERIOD The period of flush access log to the backend.
access_log.flush.merge_kernel_logs false ROVER_ACCESS_LOG_FLUSH_MERGE_KERNEL_LOGS Report one kernel log per operation of a message (the consecutive reads or writes of one message merge into one record with the summed L4 sizes and counts) instead of one per socket syscall. The protocol logs stay the same.
access_log.protocol_analyze.per_cpu_buffer 4MB ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_PER_CPU_BUFFER The size of the socket data perf buffer on each CPU (allocated per CPU, so this times the CPU count). One event carries up to 20KB of data. The agent logs dropped samples when it is too small.
access_log.protocol_analyze.parse_parallels 1 ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_PARSE_PARALLELS The count of goroutines reading the socket data rings. One goroutine reads each ring, because the perf library serialises the reads of a ring (several only add a hand-off per event and deliver the events of a connection out of order), so values above 1 are ignored.
access_log.connection_analyze.analyze_parallels 1 ROVER_ACCESS_LOG_CONNECTION_ANALYZE_PARALLELS The count of parallel connection analyzer: a positive integer, or auto (or 0) for 2 x GOMAXPROCS bounded to [2, 10].
access_log.protocol_analyze.analyze_parallels 2 ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_PARALLELS The count of parallel protocol analyzer: a positive integer, or auto (or 0) for 2 x GOMAXPROCS bounded to [2, 10]. GOMAXPROCS follows the CPU limit of the container, so 1.5 cores give 4 and 5 or more cores give 10. The resolved count is logged at startup.
access_log.protocol_analyze.queue_size 5000 ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_QUEUE_SIZE The size of per paralleled analyze queue.
access_log.protocol_analyze.trigger_events 1000 ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_TRIGGER_EVENTS Analyze the buffered socket data of a partition as soon as this many data events are pending (each unparsed event holds a pooled buffer of the size of its data, up to 20KB) instead of only on the flush period, 0 disables it.
access_log.protocol_analyze.max_memory_percent 80 ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_MAX_MEMORY_PERCENT The socket data events held in memory (their buffers plus a per event overhead) may take at most this percent of the memory the agent can be allocated: the limit of its cgroup (from /proc/self/cgroup, else by looking up the PID under the kubepods hierarchy of the host cgroup tree, for a hostPID container that mounts the host /sys; the node allocatable the kubelet sets on kubepods and the QoS cgroups is ignored), else the memory of the host. While they hold it, a new event is dropped (counted) and the connection it belonged to restarts at its next message. The agent also sets the Go soft memory limit to 90% of that memory, unless GOMEMLIMIT is set to something else. 0 disables it.
access_log.sender.parallelism 8 ROVER_ACCESS_LOG_SENDER_PARALLELISM The count of streams sending access logs to the backend at the same time. The backend consumes a stream serially, so one stream bounds the access log rate; a batch that fits in one send still uses one stream. Every stream names the node on its first message and each connection on its first message of the connection.
access_log.sender.max_logs_per_send 2000 ROVER_ACCESS_LOG_SENDER_MAX_LOGS_PER_SEND The most logs (protocol logs and kernel logs) one stream carries.
access_log.sender.max_pending_logs 200000 ROVER_ACCESS_LOG_SENDER_MAX_PENDING_LOGS The most logs waiting to be sent. Past it the oldest batches are dropped (rover_access_log_send_dropped_logs_total) and warned about, instead of held in memory. It is a count of logs, not bytes. 0 does not bound them.
access_log.event_reader.wakeup_events 32 ROVER_ACCESS_LOG_EVENT_READER_WAKEUP_EVENTS Wake the reader of the socket data and connection event rings once a per CPU ring holds this many events instead of for every event; 0 wakes it for every event. The rings are flushed every flush_period, so no event waits longer.
access_log.event_reader.flush_period 1s ROVER_ACCESS_LOG_EVENT_READER_FLUSH_PERIOD How often the rings are flushed when wakeup_events or batch_size is set, and the longest a partial batch waits.
access_log.event_reader.batch_size 64 ROVER_ACCESS_LOG_EVENT_READER_BATCH_SIZE Hand the events to the analyzing partitions in batches of up to this many, collected until the ring is drained or flush_period has passed, instead of one by one; 0 hands them one by one. The queue_size of the partitions stays a count of events.
access_log.protocol_analyze.drop_h2c_unused_frame false ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_DROP_H2C_UNUSED_FRAME Do not upload the socket calls of a cleartext HTTP/2 (h2c) connection that carry nothing but WINDOW_UPDATE, PING, PRIORITY or SETTINGS ACK frames, which the HTTP/2 parser reads nothing from. The BPF side follows the frames of each direction of a connection traced from its connect or accept and drops a call only when it holds exactly such complete frames (or the 9 byte header of one followed by exactly its payload); a direction it cannot follow is never dropped from. The kernel logs of the calls are still sent. TLS connections are not affected.
access_log.protocol_analyze.skip_h2c_data_payload false ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_SKIP_H2C_DATA_PAYLOAD Upload a socket call of a cleartext HTTP/2 (h2c) connection that lies wholly inside the payload of a DATA frame without the PADDED flag as an event without content that carries the size of the call, instead of its bytes: the HTTP/2 parser only counts DATA payloads into the body size, so the access logs, the body sizes and the kernel logs stay the same. Every other call, including any call that holds a byte of a frame header, is uploaded in full. It follows the frames of a connection traced from its connect or accept with the same cursor as drop_h2c_unused_frame, and works with or without it. TLS connections are not affected.
access_log.protocol_analyze.skip_http1_body_payload false ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_SKIP_HTTP1_BODY_PAYLOAD Decode a socket call of an HTTP/1 connection that lies wholly inside a Content-Length body without its bytes: the event keeps the size and the kernel details of the call but takes no data buffer. The parser only counts body bytes, so the access logs and the kernel logs stay the same; a call it cannot prove to be body is copied as before. Works with or without skip_http1_body_payload_bpf.
access_log.protocol_analyze.skip_http1_body_payload_bpf false ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_SKIP_HTTP1_BODY_PAYLOAD_BPF Upload a socket call of a cleartext HTTP/1 connection that lies wholly inside a body framed by Content-Length, wholly inside the data of a chunk of a chunked body, or behind the head of a response delimited by the close (where the parser does not look for a message start in it) as an event without content that carries the size of the call, instead of its bytes: the HTTP/1 parser only counts such bytes, and parses the framing of a chunked body from the calls that carry it, which are uploaded in whole, so the access logs, the body sizes and the kernel logs stay the same. The BPF side follows the message heads of a connection traced from its connect or accept (Content-Length, Transfer-Encoding: chunked, the responses to HEAD, 1xx, 204 and 304 without a body) and stops following a direction for good at what it cannot follow (another transfer coding, a chunk size line or trailers it does not follow, more chunk boundaries in one call than it walks, Upgrade, CONNECT, a head longer than 1KB or split over calls, pipelining). A call that overlaps another call of its fd is never skipped. With access_log.flush.merge_kernel_logs on, the consecutive skipped calls of a body (of a chunk) that the kernel logs merge into one record are uploaded as one event. TLS connections are not affected.
access_log.protocol_analyze.batch_h2c_records false ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_BATCH_H2C_RECORDS With skip_h2c_data_payload and flush.merge_kernel_logs on, send the skipped and tail-cut socket calls of cleartext HTTP/2 connections as compact records in per-CPU batches (one perf event for many calls) instead of one event each. The records carry what the events would, so the access logs and the kernel logs stay the same.
access_log.protocol_analyze.summarize_h2c_payload false ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_SUMMARIZE_H2C_PAYLOAD With batch_h2c_records on, sum the socket calls of cleartext HTTP/2 connections that lie wholly inside a DATA payload up per stream in BPF instead of sending them; the parser counts their bytes from the positional events, so the access logs and the kernel logs of the messages stay the same.
access_log.protocol_analyze.analyze_trace_context true ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_ANALYZE_TRACE_CONTEXT The HTTP logs carry the tracing context of their requests (the SkyWalking sw8 and the Zipkin b3 headers). Off, no header is looked at for it.
access_log.bpf.attach_mode auto ROVER_ACCESS_LOG_BPF_ATTACH_MODE How the socket call programs attach: auto (from BPF trampolines or raw tracepoints where the kernel runs them, from the syscall tracepoints and kprobes elsewhere), trampoline (auto without the check that the kernel runs trampoline programs without preemption) or tracepoint (the syscall tracepoints and kprobes everywhere).
access_log.bpf.nat_hook auto ROVER_ACCESS_LOG_BPF_NAT_HOOK Where the remote address of a connect behind a DNAT is read: auto (fexit __nf_conntrack_confirm where it attaches, otherwise a kprobe on it), confirm (the fexit only), kprobe (the kprobe only) or legacy (the kprobes __nf_conntrack_hash_insert and nf_confirm).
access_log.kernel_metrics.l2_l3 true ROVER_ACCESS_LOG_KERNEL_METRICS_L2_L3 The kernel logs of the reads and writes carry the L2/L3 metrics of their socket calls (the interface, the time in the IP layer, the netfilter hooks, the neighbour resolution and the device queue) and the L4 duration, retransmits and the packages __tcp_transmit_skb sent, taken by kprobes and tracepoints on the network stack and sent in an event of their own per call. Off, none of those programs is attached (they are loaded as stubs) and the kernel logs carry the L4 package count and size of each call only. With access_log.flush.merge_kernel_logs on, the logs of a message are merged by their interface with every metric summed. A hook the kernel does not have, or a program it refuses, only leaves its metrics at 0.

Collectors

Socket Connect/Accept/Close

Monitor all socket connect, accept, and close events from monitored processes by attaching eBPF program to the respective trace points.

Socket traffic

Capture all socket traffic from monitored processes by attaching eBPF program to network syscalls.

Protocol

Data collection is followed by protocol analysis. Currently, the supported protocols include:

  1. HTTP/1.x
  2. HTTP/2

Note: As HTTP2 is a stateful protocol, it only supports monitoring processes that start after monitor. Processes already running at the time of monitoring may fail to provide complete data, leading to unsuccessful analysis.

TLS

When a process uses the TLS protocol for data transfer, Rover monitors libraries such as OpenSSL, BoringSSL, GoTLS, and NodeTLS to access the raw content. This feature is also applicable for protocol analysis.

Note: the parsing of TLS protocols in Java is currently not supported.

L2-L4

During data transmission, Rover records each packet’s through the network layers L2 to L4 using kprobes. This approach enhances the understanding of each packet’s transmission process, facilitating easier localization and troubleshooting of network issues.