Elasticsearch monitoring

SkyWalking leverages elasticsearch-exporter for collecting metrics data from Elasticsearch. It leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System.

Data flow

  1. The elasticsearch-exporter collect metrics data from Elasticsearch.
  2. OpenTelemetry Collector fetches metrics from elasticsearch-exporter via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
  3. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.

Setup

  1. Setup elasticsearch-exporter.
  2. Set up OpenTelemetry Collector. The example for OpenTelemetry Collector configuration, refer to here.
  3. Config SkyWalking OpenTelemetry receiver.

Elasticsearch Monitoring

Elasticsearch monitoring provides multidimensional metrics monitoring of Elasticsearch clusters as Layer: ELASTICSEARCH Service in the OAP. In each cluster, the nodes are represented as Instance and indices are Endpoints.

Elasticsearch Cluster Supported Metrics

Monitoring Panel Metric Name Description Data Source
Cluster Health meter_elasticsearch_cluster_health_status Whether all primary and replica shards are allocated elasticsearch-exporter
Tripped Of Breakers meter_elasticsearch_cluster_breakers_tripped Tripped for breaker elasticsearch-exporter
Nodes meter_elasticsearch_cluster_nodes Number of nodes in the cluster. elasticsearch-exporter
Data Nodes meter_elasticsearch_cluster_data_nodes Number of data nodes in the cluster elasticsearch-exporter
Pending Tasks meter_elasticsearch_cluster_pending_tasks_total Cluster level changes which have not yet been executed elasticsearch-exporter
CPU Usage Avg. (%) meter_elasticsearch_cluster_cpu_usage_avg Cluster level percent CPU used by process elasticsearch-exporter
JVM Memory Used Avg. (%) meter_elasticsearch_cluster_jvm_memory_used_avg Cluster level percent JVM memory used elasticsearch-exporter
Open Files meter_elasticsearch_cluster_open_file_count Open file descriptors elasticsearch-exporter
Active Primary Shards meter_elasticsearch_cluster_primary_shards_total The number of primary shards in your cluster. This is an aggregate total across all indices elasticsearch-exporter
Active Shards meter_elasticsearch_cluster_shards_total Aggregate total of all shards across all indices, which includes replica shards elasticsearch-exporter
Initializing Shards meter_elasticsearch_cluster_initializing_shards_total Count of shards that are being freshly created elasticsearch-exporter
Delayed Unassigned Shards meter_elasticsearch_cluster_delayed_unassigned_shards_total Shards delayed to reduce reallocation overhead elasticsearch-exporter
Relocating Shards meter_elasticsearch_cluster_relocating_shards_total The number of shards that are currently moving from one node to another node elasticsearch-exporter
Unassigned Shards meter_elasticsearch_cluster_unassigned_shards_total The number of shards that exist in the cluster state, but cannot be found in the cluster itself elasticsearch-exporter

Elasticsearch Node Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
Node Rules meter_elasticsearch_node_rules Node roles elasticsearch-exporter
JVM Memory Used MB meter_elasticsearch_node_jvm_memory_used Node level JVM memory used size elasticsearch-exporter
CPU Percent % meter_elasticsearch_node_process_cpu_percent Node level percent CPU used by process elasticsearch-exporter
Documents meter_elasticsearch_node_indices_docs Count of index documents on this node elasticsearch-exporter
Segments meter_elasticsearch_node_segment_count Count of index segments on this node elasticsearch-exporter
Disk Free Space GB meter_elasticsearch_node_all_disk_free_space Available space on all block device elasticsearch-exporter
Open Files meter_elasticsearch_node_open_file_count Open file descriptors elasticsearch-exporter
Process CPU Usage Percent % meter_elasticsearch_node_process_cpu_percent Percent CPU used by process elasticsearch-exporter
OS CPU usage percent % meter_elasticsearch_node_os_cpu_percent Percent CPU used by the OS elasticsearch-exporter
Load Average meter_elasticsearch_node_os_load1
meter_elasticsearch_node_os_load5
meter_elasticsearch_node_os_load15		 Shortterm, Midterm, Longterm load average elasticsearch-exporter
JVM Memory Usage MB meter_elasticsearch_node_jvm_memory_nonheap_used
meter_elasticsearch_node_jvm_memory_heap_used
meter_elasticsearch_node_jvm_memory_heap_max		 JVM memory currently usage by area elasticsearch-exporter
JVM Pool Peak Used MB meter_elasticsearch_node_jvm_memory_pool_peak_used JVM memory currently used by pool elasticsearch-exporter
GC Count meter_elasticsearch_node_jvm_gc_count Count of JVM GC runs elasticsearch-exporter
GC Time ms/min meter_elasticsearch_node_jvm_gc_time GC run time elasticsearch-exporter
All Operations ReqRate meter_elasticsearch_node_indices_*_req_rate All Operations ReqRate on node elasticsearch-exporter
Indexing Rate reqps meter_elasticsearch_node_indices_indexing_index_total_req_rate
meter_elasticsearch_node_indices_indexing_index_total_proc_rate		 Indexing rate on node elasticsearch-exporter
Searching Rate reqps meter_elasticsearch_node_indices_search_fetch_total_req_rate
meter_elasticsearch_node_indices_search_query_time_seconds_proc_rate		 Searching rate on node elasticsearch-exporter
Total Translog Operations meter_elasticsearch_node_indices_translog_operations Total translog operations elasticsearch-exporter
Total Translog Size MB meter_elasticsearch_node_indices_translog_size Total translog size elasticsearch-exporter
Tripped For Breakers meter_elasticsearch_node_breakers_tripped Tripped for breaker elasticsearch-exporter
Estimated Size Of Breaker MB meter_elasticsearch_node_breakers_estimated_size Estimated size of breaker elasticsearch-exporter
Documents Count KB/s meter_elasticsearch_node_indices_docs Count of documents on this node elasticsearch-exporter
Merged Documents Count count/s meter_elasticsearch_node_indices_merges_docs_total Cumulative docs merged elasticsearch-exporter
Deleted Documents Count meter_elasticsearch_node_indices_docs_deleted_total Count of deleted documents on this node elasticsearch-exporter
Documents Index Rate calls/s meter_elasticsearch_node_indices_indexing_index_total_req_rate Total index calls per second elasticsearch-exporter
Merged Documents Rate MB / s meter_elasticsearch_node_indices_merges_total_size_bytes_total Total merge size per second elasticsearch-exporter
Documents Deleted Rate docs/s meter_elasticsearch_node_indices_docs_deleted Count of deleted documents per second on this node elasticsearch-exporter
Count Of Index Segments meter_elasticsearch_node_segment_count Count of index segments on this node elasticsearch-exporter
Current Memory Size Of Segments MB meter_elasticsearch_node_segment_memory Current memory size of segments elasticsearch-exporter
Network bytes/sec meter_elasticsearch_node_network_send_bytes
meter_elasticsearch_node_network_receive_bytes		 Total number of bytes sent and receive elasticsearch-exporter
Disk Usage Percent % meter_elasticsearch_node_disk_usage_percent Used space on block device elasticsearch-exporter
Disk Usage GB meter_elasticsearch_node_disk_usage Used space size of block device elasticsearch-exporter
Disk Read KBs meter_elasticsearch_node_disk_io_read_bytes Total kilobytes read from disk elasticsearch-exporter
Disk Write KBs meter_elasticsearch_node_disk_io_write_bytes Total kilobytes write from disk elasticsearch-exporter

Elasticsearch Index Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
Documents Primary meter_elasticsearch_index_indices_docs_primary Count of documents with only primary shards on all nodes elasticsearch-exporter
Deleted Documents Primary meter_elasticsearch_index_indices_deleted_docs_primary Count of deleted documents with only primary shards elasticsearch-exporter
Data Primary GB meter_elasticsearch_index_indices_store_size_bytes_primary Current total size of stored index data with only primary shards on all nodes elasticsearch-exporter
Data GB meter_elasticsearch_index_indices_store_size_bytes_total Current total size of stored index data with all shards on all nodes elasticsearch-exporter
Segments Primary meter_elasticsearch_index_indices_segment_count_primary Current number of segments with only primary shards on all nodes elasticsearch-exporter
Segments Memory Primary MB meter_elasticsearch_index_indices_segment_memory_bytes_primary Current size of segments with only primary shards on all nodes elasticsearch-exporter
Segments meter_elasticsearch_index_indices_segment_count_total Current number of segments with all shards on all nodes elasticsearch-exporter
Segments Memory MB meter_elasticsearch_index_indices_segment_memory_bytes_total Current size of segments with all shards on all nodes elasticsearch-exporter
Indexing Rate meter_elasticsearch_index_stats_indexing_index_total_req_rate
meter_elasticsearch_index_stats_indexing_index_total_proc_rate		 Indexing rate on index elasticsearch-exporter
Searching Rate meter_elasticsearch_index_stats_search_query_total_req_rate
meter_elasticsearch_index_stats_search_query_total_proc_rate		 Searching rate on index elasticsearch-exporter
All Operations ReqRate meter_elasticsearch_index_stats_*_req_rate All Operations ReqRate on index elasticsearch-exporter
All Operations Runtime meter_elasticsearch_index_stats_*_time_seconds_total All Operations Runtime/s on index elasticsearch-exporter
Avg. Search Time Execute / Request s meter_elasticsearch_index_search_fetch_avg_time
meter_elasticsearch_index_search_query_avg_time
meter_elasticsearch_index_search_scroll_avg_time
meter_elasticsearch_index_search_suggest_avg_time		 Search Operation Avg. time on index elasticsearch-exporter
Search Operations Rate req/s meter_elasticsearch_index_stats_search_query_total_req_rate
meter_elasticsearch_index_stats_search_fetch_total_req_rate
meter_elasticsearch_index_stats_search_scroll_total_req_rate
meter_elasticsearch_index_stats_search_suggest_total_req_rate		 Search Operations ReqRate on index elasticsearch-exporter
Shards Documents meter_elasticsearch_index_indices_shards_docs Count of documents per shards on index elasticsearch-exporter
Documents (Primary Shards) meter_elasticsearch_index_indices_docs_primary Count of documents with only primary shards on index elasticsearch-exporter
Documents Created Per Min (Primary Shards) meter_elasticsearch_index_indices_docs_primary_rate Documents rate with only primary shards on index elasticsearch-exporter
Total Size Of Index (Primary Shards) MB meter_elasticsearch_index_indices_store_size_bytes_primary Current total size of stored index data in bytes with only primary shards on all nodes elasticsearch-exporter
Documents (All Shards) meter_elasticsearch_index_indices_docs_total Count of documents with all shards on index elasticsearch-exporter
Documents Created Per Min (All Shards) meter_elasticsearch_index_indices_docs_total_rate Documents rate with only all shards on index elasticsearch-exporter
Total Size Of Index (All Shards) MB meter_elasticsearch_index_indices_store_size_bytes_total Current total size of stored index data in bytes with all shards on all nodes elasticsearch-exporter

Customizations

You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in /config/otel-rules/elasticsearch/elasticsearch-cluster.yaml, /config/otel-rules/elasticsearch/elasticsearch-node.yaml, /config/otel-rules/elasticsearch/elasticsearch-index.yaml. The Elasticsearch dashboard panel configurations are found in /config/ui-initialized-templates/elasticsearch.