SkyWalking
Kubernetes (K8s) monitoring from kube-state-metrics and cAdvisor
SkyWalking leverages K8s kube-state-metrics (KSM) and cAdvisor for collecting metrics data from K8s. It leverages OpenTelemetry Collector to transfer the metrics to
OpenTelemetry receiver and into the Meter System. This feature requires authorizing the OAP Server to access K8s’s API Server.
Data flow
- K8s kube-state-metrics and cAdvisor collect metrics data from K8s.
- OpenTelemetry Collector fetches metrics from kube-state-metrics and cAdvisor via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
- The SkyWalking OAP Server access to K8s’s
API Servergets meta info and parses the expression with MAL to filter/calculate/aggregate and store the results.
Setup
- Setup kube-state-metric.
- cAdvisor is integrated into
kubeletby default. - Set up OpenTelemetry Collector . For details on Prometheus Receiver in OpenTelemetry Collector for K8s, refer to here. For a quick start, we have provided a complete example of configuration and recommended version; you can refer to showcase.
- Config SkyWalking OpenTelemetry receiver.
Kubernetes Cluster Monitoring
K8s cluster monitoring provides monitoring of the status and resources of the whole cluster and each node. K8s cluster as a Service in OAP, K8s node as an Instance in OAP, and land on the Layer: K8S.
Kubernetes Cluster Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| Node Total | k8s_cluster_node_total | The number of nodes | K8s kube-state-metrics | |
| Namespace Total | k8s_cluster_namespace_total | The number of namespaces | K8s kube-state-metrics | |
| Deployment Total | k8s_cluster_deployment_total | The number of deployments | K8s kube-state-metrics | |
| StatefulSet Total | k8s_cluster_statefulset_total | The number of statefulsets | K8s kube-state-metrics | |
| DaemonSet Total | k8s_cluster_daemonset_total | The number of daemonsets | K8s kube-state-metrics | |
| Service Total | k8s_cluster_service_total | The number of services | K8s kube-state-metrics | |
| Pod Total | k8s_cluster_pod_total | The number of pods | K8s kube-state-metrics | |
| Container Total | k8s_cluster_container_total | The number of containers | K8s kube-state-metrics | |
| CPU Resources | m | k8s_cluster_cpu_cores k8s_cluster_cpu_cores_requests k8s_cluster_cpu_cores_limits k8s_cluster_cpu_cores_allocatable |
The capacity and the Requests / Limits / Allocatable of the CPU | K8s kube-state-metrics |
| Memory Resources | Gi | k8s_cluster_memory_total k8s_cluster_memory_requests k8s_cluster_memory_limits k8s_cluster_memory_allocatable |
The capacity and the Requests / Limits / Allocatable of the memory | K8s kube-state-metrics |
| Storage Resources | Gi | k8s_cluster_storage_total k8s_cluster_storage_allocatable |
The capacity and allocatable of the storage | K8s kube-state-metrics |
| Node Status | k8s_cluster_node_status | The current status of the nodes | K8s kube-state-metrics | |
| Deployment Status | k8s_cluster_deployment_status | The current status of the deployment | K8s kube-state-metrics | |
| Deployment Spec Replicas | k8s_cluster_deployment_spec_replicas | The number of desired pods for a deployment | K8s kube-state-metrics | |
| Service Status | k8s_cluster_service_pod_status | The services current status, depending on the related pods' status | K8s kube-state-metrics | |
| Pod Status Not Running | k8s_cluster_pod_status_not_running | The pods which are not running in the current phase | K8s kube-state-metrics | |
| Pod Status Waiting | k8s_cluster_pod_status_waiting | The pods and containers which are currently in the waiting status, with reasons shown | K8s kube-state-metrics | |
| Pod Status Terminated | k8s_cluster_container_status_terminated | The pods and containers which are currently in the terminated status, with reasons shown | K8s kube-state-metrics |
Kubernetes Cluster Node Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| Pod Total | k8s_node_pod_total | The number of pods in this node | K8s kube-state-metrics | |
| Node Status | k8s_node_node_status | The current status of this node | K8s kube-state-metrics | |
| CPU Resources | m | k8s_node_cpu_cores k8s_node_cpu_cores_allocatable k8s_node_cpu_cores_requests k8s_node_cpu_cores_limits |
The capacity and the requests / Limits / Allocatable of the CPU | K8s kube-state-metrics |
| Memory Resources | Gi | k8s_node_memory_total k8s_node_memory_allocatable k8s_node_memory_requests k8s_node_memory_limits |
The capacity and the requests / Limits / Allocatable of the memory | K8s kube-state-metrics |
| Storage Resources | Gi | k8s_node_storage_total k8s_node_storage_allocatable |
The capacity and allocatable of the storage | K8s kube-state-metrics |
| CPU Usage | m | k8s_node_cpu_usage | The total usage of the CPU core, if there are 2 cores the maximum usage is 2000m | cAdvisor |
| Memory Usage | Gi | k8s_node_memory_usage | The totaly memory usage | cAdvisor |
| Network I/O | KB/s | k8s_node_network_receive k8s_node_network_transmit |
The network receive and transmit | cAdvisor |
Kubernetes Service Monitoring
K8s Service Monitoring provides observabilities into service status and resources from Kubernetes.
K8s Service as a Service in OAP and land on the Layer: K8S_SERVICE.
Kubernetes Service Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| Service Pod Total | k8s_service_pod_total | The number of pods | K8s kube-state-metrics | |
| Service Pod Status | k8s_service_pod_status | The current status of pods | K8s kube-state-metrics | |
| Service CPU Resources | m | k8s_service_cpu_cores_requests k8s_service_cpu_cores_limits |
The CPU resources requests / Limits of this service | K8s kube-state-metrics |
| Service Memory Resources | MB | k8s_service_memory_requests k8s_service_memory_limits |
The memory resources requests / Limits of this service | K8s kube-state-metrics |
| Pod CPU Usage | m | k8s_service_pod_cpu_usage | The CPU resources total usage of pods | cAdvisor |
| Pod Memory Usage | MB | k8s_service_pod_memory_usage | The memory resources total usage of pods | cAdvisor |
| Pod Waiting | k8s_service_pod_status_waiting | The pods and containers which are currently in the waiting status, with reasons shown | K8s kube-state-metrics | |
| Pod Terminated | k8s_service_pod_status_terminated | The pods and containers which are currently in the terminated status, with reasons shown | K8s kube-state-metrics | |
| Pod Restarts | k8s_service_pod_status_restarts_total | The number of per container restarts related to the pods | K8s kube-state-metrics |
Customizations
You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found in /config/otel-rules/k8s/k8s-cluster.yaml，/config/otel-rules/k8s/k8s-node.yaml, /config/otel-rules/k8s/k8s-service.yaml.
The K8s Cluster dashboard panel configurations are found in /config/ui-initialized-templates/k8s.
The K8s Service dashboard panel configurations are found in /config/ui-initialized-templates/k8s_service.