Traffic
The traffic is used to collecting the network access logs from services through the Service Discovery, and send access logs to the backend server for analyze.
Configuration
| Name | Default | Environment Key | Description |
|---|---|---|---|
| access_log.active | false | ROVER_ACCESS_LOG_ACTIVE | Is active the access log monitoring. |
| access_log.exclude_namespaces | istio-system,cert-manager,kube-system | ROVER_ACCESS_LOG_EXCLUDE_NAMESPACES | Exclude processes in the specified Kubernetes namespace. Multiple namespaces split by “,” |
| access_log.exclude_cluster | ROVER_ACCESS_LOG_EXCLUDE_CLUSTER | Exclude processes in the specified cluster which defined in the process module. Multiple clusters split by “,” | |
| access_log.flush.max_count | 2000 | ROVER_ACCESS_LOG_FLUSH_MAX_COUNT | The max count of the access log when flush to the backend. |
| access_log.flush.period | 5s | ROVER_ACCESS_LOG_FLUSH_PERIOD | The period of flush access log to the backend. |
| access_log.flush.merge_kernel_logs | false | ROVER_ACCESS_LOG_FLUSH_MERGE_KERNEL_LOGS | Report one kernel log per operation of a message (the consecutive reads or writes of one message merge into one record with the summed L4 sizes and counts) instead of one per socket syscall. The protocol logs stay the same. |
| access_log.protocol_analyze.per_cpu_buffer | 4MB | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_PER_CPU_BUFFER | The size of the socket data perf buffer on each CPU (allocated per CPU, so this times the CPU count). One event carries up to 20KB of data. The agent logs dropped samples when it is too small. |
| access_log.protocol_analyze.parse_parallels | 1 | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_PARSE_PARALLELS | The count of goroutines reading the socket data rings. One goroutine reads each ring, because the perf library serialises the reads of a ring (several only add a hand-off per event and deliver the events of a connection out of order), so values above 1 are ignored. |
| access_log.connection_analyze.analyze_parallels | 1 | ROVER_ACCESS_LOG_CONNECTION_ANALYZE_PARALLELS | The count of parallel connection analyzer: a positive integer, or auto (or 0) for 2 x GOMAXPROCS bounded to [2, 10]. |
| access_log.protocol_analyze.analyze_parallels | 2 | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_PARALLELS | The count of parallel protocol analyzer: a positive integer, or auto (or 0) for 2 x GOMAXPROCS bounded to [2, 10]. GOMAXPROCS follows the CPU limit of the container, so 1.5 cores give 4 and 5 or more cores give 10. The resolved count is logged at startup. |
| access_log.protocol_analyze.queue_size | 5000 | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_QUEUE_SIZE | The size of per paralleled analyze queue. |
| access_log.protocol_analyze.trigger_events | 1000 | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_TRIGGER_EVENTS | Analyze the buffered socket data of a partition as soon as this many data events are pending (each unparsed event holds a pooled buffer of the size of its data, up to 20KB) instead of only on the flush period, 0 disables it. |
| access_log.protocol_analyze.max_memory_percent | 80 | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_MAX_MEMORY_PERCENT | The socket data events held in memory (their buffers plus a per event overhead) may take at most this percent of the memory the agent can be allocated: the limit of its cgroup (from /proc/self/cgroup, else by looking up the PID under the kubepods hierarchy of the host cgroup tree, for a hostPID container that mounts the host /sys; the node allocatable the kubelet sets on kubepods and the QoS cgroups is ignored), else the memory of the host. While they hold it, a new event is dropped (counted) and the connection it belonged to restarts at its next message. The agent also sets the Go soft memory limit to 90% of that memory, unless GOMEMLIMIT is set to something else. 0 disables it. |
| access_log.sender.parallelism | 8 | ROVER_ACCESS_LOG_SENDER_PARALLELISM | The count of streams sending access logs to the backend at the same time. The backend consumes a stream serially, so one stream bounds the access log rate; a batch that fits in one send still uses one stream. Every stream names the node on its first message and each connection on its first message of the connection. |
| access_log.sender.max_logs_per_send | 2000 | ROVER_ACCESS_LOG_SENDER_MAX_LOGS_PER_SEND | The most logs (protocol logs and kernel logs) one stream carries. |
| access_log.sender.max_pending_logs | 200000 | ROVER_ACCESS_LOG_SENDER_MAX_PENDING_LOGS | The most logs waiting to be sent. Past it the oldest batches are dropped (rover_access_log_send_dropped_logs_total) and warned about, instead of held in memory. It is a count of logs, not bytes. 0 does not bound them. |
| access_log.event_reader.wakeup_events | 32 | ROVER_ACCESS_LOG_EVENT_READER_WAKEUP_EVENTS | Wake the reader of the socket data and connection event rings once a per CPU ring holds this many events instead of for every event; 0 wakes it for every event. The rings are flushed every flush_period, so no event waits longer. |
| access_log.event_reader.flush_period | 1s | ROVER_ACCESS_LOG_EVENT_READER_FLUSH_PERIOD | How often the rings are flushed when wakeup_events or batch_size is set, and the longest a partial batch waits. |
| access_log.event_reader.batch_size | 64 | ROVER_ACCESS_LOG_EVENT_READER_BATCH_SIZE | Hand the events to the analyzing partitions in batches of up to this many, collected until the ring is drained or flush_period has passed, instead of one by one; 0 hands them one by one. The queue_size of the partitions stays a count of events. |
| access_log.protocol_analyze.drop_h2c_unused_frame | false | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_DROP_H2C_UNUSED_FRAME | Do not upload the socket calls of a cleartext HTTP/2 (h2c) connection that carry nothing but WINDOW_UPDATE, PING, PRIORITY or SETTINGS ACK frames, which the HTTP/2 parser reads nothing from. The BPF side follows the frames of each direction of a connection traced from its connect or accept and drops a call only when it holds exactly such complete frames (or the 9 byte header of one followed by exactly its payload); a direction it cannot follow is never dropped from. The kernel logs of the calls are still sent. TLS connections are not affected. |
| access_log.protocol_analyze.skip_h2c_data_payload | false | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_SKIP_H2C_DATA_PAYLOAD | Upload a socket call of a cleartext HTTP/2 (h2c) connection that lies wholly inside the payload of a DATA frame without the PADDED flag as an event without content that carries the size of the call, instead of its bytes: the HTTP/2 parser only counts DATA payloads into the body size, so the access logs, the body sizes and the kernel logs stay the same. Every other call, including any call that holds a byte of a frame header, is uploaded in full. It follows the frames of a connection traced from its connect or accept with the same cursor as drop_h2c_unused_frame, and works with or without it. TLS connections are not affected. |
| access_log.protocol_analyze.skip_http1_body_payload | false | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_SKIP_HTTP1_BODY_PAYLOAD | Decode a socket call of an HTTP/1 connection that lies wholly inside a Content-Length body without its bytes: the event keeps the size and the kernel details of the call but takes no data buffer. The parser only counts body bytes, so the access logs and the kernel logs stay the same; a call it cannot prove to be body is copied as before. Works with or without skip_http1_body_payload_bpf. |
| access_log.protocol_analyze.skip_http1_body_payload_bpf | false | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_SKIP_HTTP1_BODY_PAYLOAD_BPF | Upload a socket call of a cleartext HTTP/1 connection that lies wholly inside a body framed by Content-Length, wholly inside the data of a chunk of a chunked body, or behind the head of a response delimited by the close (where the parser does not look for a message start in it) as an event without content that carries the size of the call, instead of its bytes: the HTTP/1 parser only counts such bytes, and parses the framing of a chunked body from the calls that carry it, which are uploaded in whole, so the access logs, the body sizes and the kernel logs stay the same. The BPF side follows the message heads of a connection traced from its connect or accept (Content-Length, Transfer-Encoding: chunked, the responses to HEAD, 1xx, 204 and 304 without a body) and stops following a direction for good at what it cannot follow (another transfer coding, a chunk size line or trailers it does not follow, more chunk boundaries in one call than it walks, Upgrade, CONNECT, a head longer than 1KB or split over calls, pipelining). A call that overlaps another call of its fd is never skipped. With access_log.flush.merge_kernel_logs on, the consecutive skipped calls of a body (of a chunk) that the kernel logs merge into one record are uploaded as one event. TLS connections are not affected. |
| access_log.protocol_analyze.batch_h2c_records | false | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_BATCH_H2C_RECORDS | With skip_h2c_data_payload and flush.merge_kernel_logs on, send the skipped and tail-cut socket calls of cleartext HTTP/2 connections as compact records in per-CPU batches (one perf event for many calls) instead of one event each. The records carry what the events would, so the access logs and the kernel logs stay the same. |
| access_log.protocol_analyze.summarize_h2c_payload | false | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_SUMMARIZE_H2C_PAYLOAD | With batch_h2c_records on, sum the socket calls of cleartext HTTP/2 connections that lie wholly inside a DATA payload up per stream in BPF instead of sending them; the parser counts their bytes from the positional events, so the access logs and the kernel logs of the messages stay the same. |
| access_log.protocol_analyze.analyze_trace_context | true | ROVER_ACCESS_LOG_PROTOCOL_ANALYZE_ANALYZE_TRACE_CONTEXT | The HTTP logs carry the tracing context of their requests (the SkyWalking sw8 and the Zipkin b3 headers). Off, no header is looked at for it. |
| access_log.bpf.attach_mode | auto | ROVER_ACCESS_LOG_BPF_ATTACH_MODE | How the socket call programs attach: auto (from BPF trampolines or raw tracepoints where the kernel runs them, from the syscall tracepoints and kprobes elsewhere), trampoline (auto without the check that the kernel runs trampoline programs without preemption) or tracepoint (the syscall tracepoints and kprobes everywhere). |
| access_log.bpf.nat_hook | auto | ROVER_ACCESS_LOG_BPF_NAT_HOOK | Where the remote address of a connect behind a DNAT is read: auto (fexit __nf_conntrack_confirm where it attaches, otherwise a kprobe on it), confirm (the fexit only), kprobe (the kprobe only) or legacy (the kprobes __nf_conntrack_hash_insert and nf_confirm). |
| access_log.kernel_metrics.l2_l3 | true | ROVER_ACCESS_LOG_KERNEL_METRICS_L2_L3 | The kernel logs of the reads and writes carry the L2/L3 metrics of their socket calls (the interface, the time in the IP layer, the netfilter hooks, the neighbour resolution and the device queue) and the L4 duration, retransmits and the packages __tcp_transmit_skb sent, taken by kprobes and tracepoints on the network stack and sent in an event of their own per call. Off, none of those programs is attached (they are loaded as stubs) and the kernel logs carry the L4 package count and size of each call only. With access_log.flush.merge_kernel_logs on, the logs of a message are merged by their interface with every metric summed. A hook the kernel does not have, or a program it refuses, only leaves its metrics at 0. |
Collectors
Socket Connect/Accept/Close
Monitor all socket connect, accept, and close events from monitored processes by attaching eBPF program to the respective trace points.
Socket traffic
Capture all socket traffic from monitored processes by attaching eBPF program to network syscalls.
Protocol
Data collection is followed by protocol analysis. Currently, the supported protocols include:
- HTTP/1.x
- HTTP/2
Note: As HTTP2 is a stateful protocol, it only supports monitoring processes that start after monitor. Processes already running at the time of monitoring may fail to provide complete data, leading to unsuccessful analysis.
TLS
When a process uses the TLS protocol for data transfer, Rover monitors libraries such as OpenSSL, BoringSSL, GoTLS, and NodeTLS to access the raw content. This feature is also applicable for protocol analysis.
Note: the parsing of TLS protocols in Java is currently not supported.
L2-L4
During data transmission, Rover records each packet’s through the network layers L2 to L4 using kprobes. This approach enhances the understanding of each packet’s transmission process, facilitating easier localization and troubleshooting of network issues.