Linux Monitoring

SkyWalking leverages Prometheus node-exporter to collect metrics data from the VMs and leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System. VM entity as a Service in OAP and on the Layer: OS_LINUX.

SkyWalking also provides InfluxDB Telegraf to receive VMs' metrics data by Telegraf receiver. The telegraf receiver plugin receiver, process and convert the metrics, then it send converted metrics to Meter System. VM entity as a Service in OAP and on the Layer: OS_LINUX.

Data flow

For OpenTelemetry receiver:

  1. The Prometheus node-exporter collects metrics data from the VMs.
  2. The OpenTelemetry Collector fetches metrics from node-exporter via Prometheus Receiver and pushes metrics to the SkyWalking OAP Server via OpenTelemetry gRPC exporter.
  3. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.

For Telegraf receiver:

  1. The InfluxDB Telegraf input plugins collects various metrics data from the VMs.
  2. The cpu, mem, system, disk and diskio input plugins should be set in telegraf.conf file.
  3. The InfluxDB Telegraf send JSON format metrics by HTTP messages to Telegraf Receiver, then pushes converted metrics to the SkyWalking OAP Server Meter System.
  4. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate ad store the results.
  5. The meter_vm_cpu_average_used metrics indicates the average usage of each CPU core for telegraf receiver.

Setup

For OpenTelemetry receiver:

  1. Setup Prometheus node-exporter.
  2. Setup OpenTelemetry Collector. This is an example for OpenTelemetry Collector configuration otel-collector-config.yaml.
  3. Config SkyWalking OpenTelemetry receiver.

For Telegraf receiver:

  1. Setup InfluxDB Telegraf’s telegraf.conf file according to Telegraf office document.
  2. Setup InfluxDB Telegraf’s telegraf.conf file specific rules according to Telegraf receiver document.
  3. Config SkyWalking Telegraf receiver.

Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
CPU Usage % meter_vm_cpu_total_percentage The total percentage usage of the CPU core. If there are 2 cores, the maximum usage is 200%. Prometheus node-exporter
Telegraf input plugin
Memory RAM Usage MB meter_vm_memory_used The total RAM usage Prometheus node-exporter
Telegraf input plugin
Memory Swap Usage % meter_vm_memory_swap_percentage The percentage usage of swap memory Prometheus node-exporter
Telegraf input plugin
CPU Average Used % meter_vm_cpu_average_used The percentage usage of the CPU core in each mode Prometheus node-exporter
Telegraf input plugin
CPU Load meter_vm_cpu_load1
meter_vm_cpu_load5
meter_vm_cpu_load15		 The CPU 1m / 5m / 15m average load Prometheus node-exporter
Telegraf input plugin
Memory RAM MB meter_vm_memory_total
meter_vm_memory_available
meter_vm_memory_used
meter_vm_memory_buff_cache		 The RAM statistics, including Total / Available / Used / Buff-Cache Prometheus node-exporter
Telegraf input plugin
Memory Swap MB meter_vm_memory_swap_free
meter_vm_memory_swap_total		 Swap memory statistics, including Free / Total Prometheus node-exporter
Telegraf input plugin
File System Mountpoint Usage % meter_vm_filesystem_percentage The percentage usage of the file system at each mount point Prometheus node-exporter
Telegraf input plugin
Disk R/W KB/s meter_vm_disk_read
meter_vm_disk_written		 The disk read and written Prometheus node-exporter
Telegraf input plugin
Network Bandwidth Usage KB/s meter_vm_network_receive
meter_vm_network_transmit		 The network receive and transmit Prometheus node-exporter
Telegraf input plugin
Network Status meter_vm_tcp_curr_estab
meter_vm_tcp_tw
meter_vm_tcp_alloc
meter_vm_sockets_used
meter_vm_udp_inuse		 The number of TCPs established / TCP time wait / TCPs allocated / sockets in use / UDPs in use Prometheus node-exporter
Telegraf input plugin
Filefd Allocated meter_vm_filefd_allocated The number of file descriptors allocated Prometheus node-exporter

Customizing

You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in /config/otel-rules/vm.yaml and /config/telegraf-rules/vm.yaml. The dashboard panel confirmations are found in /config/ui-initialized-templates/os_linux.

Blog

For more details, see the blog article SkyWalking 8.4 provides infrastructure monitoring.