SkyWalking
Elasticsearch monitoring
SkyWalking leverages elasticsearch-exporter for collecting metrics data from Elasticsearch. It leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System.
Data flow
- The elasticsearch-exporter collect metrics data from Elasticsearch.
- OpenTelemetry Collector fetches metrics from elasticsearch-exporter via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
- The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.
Setup
- Setup elasticsearch-exporter.
- Set up OpenTelemetry Collector. The example for OpenTelemetry Collector configuration, refer to here.
- Config SkyWalking OpenTelemetry receiver.
Elasticsearch Monitoring
Elasticsearch monitoring provides multidimensional metrics monitoring of Elasticsearch clusters as Layer: ELASTICSEARCH Service in the OAP. In each cluster, the nodes are represented as Instance and indices are Endpoints.
Elasticsearch Cluster Supported Metrics
| Monitoring Panel | Metric Name | Description | Data Source |
|---|---|---|---|
| Cluster Health | meter_elasticsearch_cluster_health_status | Whether all primary and replica shards are allocated | elasticsearch-exporter |
| Tripped Of Breakers | meter_elasticsearch_cluster_breakers_tripped | Tripped for breaker | elasticsearch-exporter |
| Nodes | meter_elasticsearch_cluster_nodes | Number of nodes in the cluster. | elasticsearch-exporter |
| Data Nodes | meter_elasticsearch_cluster_data_nodes | Number of data nodes in the cluster | elasticsearch-exporter |
| Pending Tasks | meter_elasticsearch_cluster_pending_tasks_total | Cluster level changes which have not yet been executed | elasticsearch-exporter |
| CPU Usage Avg. (%) | meter_elasticsearch_cluster_cpu_usage_avg | Cluster level percent CPU used by process | elasticsearch-exporter |
| JVM Memory Used Avg. (%) | meter_elasticsearch_cluster_jvm_memory_used_avg | Cluster level percent JVM memory used | elasticsearch-exporter |
| Open Files | meter_elasticsearch_cluster_open_file_count | Open file descriptors | elasticsearch-exporter |
| Active Primary Shards | meter_elasticsearch_cluster_primary_shards_total | The number of primary shards in your cluster. This is an aggregate total across all indices | elasticsearch-exporter |
| Active Shards | meter_elasticsearch_cluster_shards_total | Aggregate total of all shards across all indices, which includes replica shards | elasticsearch-exporter |
| Initializing Shards | meter_elasticsearch_cluster_initializing_shards_total | Count of shards that are being freshly created | elasticsearch-exporter |
| Delayed Unassigned Shards | meter_elasticsearch_cluster_delayed_unassigned_shards_total | Shards delayed to reduce reallocation overhead | elasticsearch-exporter |
| Relocating Shards | meter_elasticsearch_cluster_relocating_shards_total | The number of shards that are currently moving from one node to another node | elasticsearch-exporter |
| Unassigned Shards | meter_elasticsearch_cluster_unassigned_shards_total | The number of shards that exist in the cluster state, but cannot be found in the cluster itself | elasticsearch-exporter |
Elasticsearch Node Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| Node Rules | meter_elasticsearch_node_rules | Node roles | elasticsearch-exporter | |
| JVM Memory Used | MB | meter_elasticsearch_node_jvm_memory_used | Node level JVM memory used size | elasticsearch-exporter |
| CPU Percent | % | meter_elasticsearch_node_process_cpu_percent | Node level percent CPU used by process | elasticsearch-exporter |
| Documents | meter_elasticsearch_node_indices_docs | Count of index documents on this node | elasticsearch-exporter | |
| Segments | meter_elasticsearch_node_segment_count | Count of index segments on this node | elasticsearch-exporter | |
| Disk Free Space | GB | meter_elasticsearch_node_all_disk_free_space | Available space on all block device | elasticsearch-exporter |
| Open Files | meter_elasticsearch_node_open_file_count | Open file descriptors | elasticsearch-exporter | |
| Process CPU Usage Percent | % | meter_elasticsearch_node_process_cpu_percent | Percent CPU used by process | elasticsearch-exporter |
| OS CPU usage percent | % | meter_elasticsearch_node_os_cpu_percent | Percent CPU used by the OS | elasticsearch-exporter |
| Load Average | meter_elasticsearch_node_os_load1 meter_elasticsearch_node_os_load5 meter_elasticsearch_node_os_load15 |
Shortterm, Midterm, Longterm load average | elasticsearch-exporter | |
| JVM Memory Usage | MB | meter_elasticsearch_node_jvm_memory_nonheap_used meter_elasticsearch_node_jvm_memory_heap_used meter_elasticsearch_node_jvm_memory_heap_max |
JVM memory currently usage by area | elasticsearch-exporter |
| JVM Pool Peak Used | MB | meter_elasticsearch_node_jvm_memory_pool_peak_used | JVM memory currently used by pool | elasticsearch-exporter |
| GC Count | meter_elasticsearch_node_jvm_gc_count | Count of JVM GC runs | elasticsearch-exporter | |
| GC Time | ms/min | meter_elasticsearch_node_jvm_gc_time | GC run time | elasticsearch-exporter |
| All Operations ReqRate | meter_elasticsearch_node_indices_*_req_rate | All Operations ReqRate on node | elasticsearch-exporter | |
| Indexing Rate | reqps | meter_elasticsearch_node_indices_indexing_index_total_req_rate meter_elasticsearch_node_indices_indexing_index_total_proc_rate |
Indexing rate on node | elasticsearch-exporter |
| Searching Rate | reqps | meter_elasticsearch_node_indices_search_fetch_total_req_rate meter_elasticsearch_node_indices_search_query_time_seconds_proc_rate |
Searching rate on node | elasticsearch-exporter |
| Total Translog Operations | meter_elasticsearch_node_indices_translog_operations | Total translog operations | elasticsearch-exporter | |
| Total Translog Size | MB | meter_elasticsearch_node_indices_translog_size | Total translog size | elasticsearch-exporter |
| Tripped For Breakers | meter_elasticsearch_node_breakers_tripped | Tripped for breaker | elasticsearch-exporter | |
| Estimated Size Of Breaker | MB | meter_elasticsearch_node_breakers_estimated_size | Estimated size of breaker | elasticsearch-exporter |
| Documents Count | KB/s | meter_elasticsearch_node_indices_docs | Count of documents on this node | elasticsearch-exporter |
| Merged Documents Count | count/s | meter_elasticsearch_node_indices_merges_docs_total | Cumulative docs merged | elasticsearch-exporter |
| Deleted Documents Count | meter_elasticsearch_node_indices_docs_deleted_total | Count of deleted documents on this node | elasticsearch-exporter | |
| Documents Index Rate | calls/s | meter_elasticsearch_node_indices_indexing_index_total_req_rate | Total index calls per second | elasticsearch-exporter |
| Merged Documents Rate | MB / s | meter_elasticsearch_node_indices_merges_total_size_bytes_total | Total merge size per second | elasticsearch-exporter |
| Documents Deleted Rate | docs/s | meter_elasticsearch_node_indices_docs_deleted | Count of deleted documents per second on this node | elasticsearch-exporter |
| Count Of Index Segments | meter_elasticsearch_node_segment_count | Count of index segments on this node | elasticsearch-exporter | |
| Current Memory Size Of Segments | MB | meter_elasticsearch_node_segment_memory | Current memory size of segments | elasticsearch-exporter |
| Network | bytes/sec | meter_elasticsearch_node_network_send_bytes meter_elasticsearch_node_network_receive_bytes |
Total number of bytes sent and receive | elasticsearch-exporter |
| Disk Usage Percent | % | meter_elasticsearch_node_disk_usage_percent | Used space on block device | elasticsearch-exporter |
| Disk Usage | GB | meter_elasticsearch_node_disk_usage | Used space size of block device | elasticsearch-exporter |
| Disk Read | KBs | meter_elasticsearch_node_disk_io_read_bytes | Total kilobytes read from disk | elasticsearch-exporter |
| Disk Write | KBs | meter_elasticsearch_node_disk_io_write_bytes | Total kilobytes write from disk | elasticsearch-exporter |
Elasticsearch Index Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| Documents Primary | meter_elasticsearch_index_indices_docs_primary | Count of documents with only primary shards on all nodes | elasticsearch-exporter | |
| Deleted Documents Primary | meter_elasticsearch_index_indices_deleted_docs_primary | Count of deleted documents with only primary shards | elasticsearch-exporter | |
| Data Primary | GB | meter_elasticsearch_index_indices_store_size_bytes_primary | Current total size of stored index data with only primary shards on all nodes | elasticsearch-exporter |
| Data | GB | meter_elasticsearch_index_indices_store_size_bytes_total | Current total size of stored index data with all shards on all nodes | elasticsearch-exporter |
| Segments Primary | meter_elasticsearch_index_indices_segment_count_primary | Current number of segments with only primary shards on all nodes | elasticsearch-exporter | |
| Segments Memory Primary | MB | meter_elasticsearch_index_indices_segment_memory_bytes_primary | Current size of segments with only primary shards on all nodes | elasticsearch-exporter |
| Segments | meter_elasticsearch_index_indices_segment_count_total | Current number of segments with all shards on all nodes | elasticsearch-exporter | |
| Segments Memory | MB | meter_elasticsearch_index_indices_segment_memory_bytes_total | Current size of segments with all shards on all nodes | elasticsearch-exporter |
| Indexing Rate | meter_elasticsearch_index_stats_indexing_index_total_req_rate meter_elasticsearch_index_stats_indexing_index_total_proc_rate |
Indexing rate on index | elasticsearch-exporter | |
| Searching Rate | meter_elasticsearch_index_stats_search_query_total_req_rate meter_elasticsearch_index_stats_search_query_total_proc_rate |
Searching rate on index | elasticsearch-exporter | |
| All Operations ReqRate | meter_elasticsearch_index_stats_*_req_rate | All Operations ReqRate on index | elasticsearch-exporter | |
| All Operations Runtime | meter_elasticsearch_index_stats_*_time_seconds_total | All Operations Runtime/s on index | elasticsearch-exporter | |
| Avg. Search Time Execute / Request | s | meter_elasticsearch_index_search_fetch_avg_time meter_elasticsearch_index_search_query_avg_time meter_elasticsearch_index_search_scroll_avg_time meter_elasticsearch_index_search_suggest_avg_time |
Search Operation Avg. time on index | elasticsearch-exporter |
| Search Operations Rate | req/s | meter_elasticsearch_index_stats_search_query_total_req_rate meter_elasticsearch_index_stats_search_fetch_total_req_rate meter_elasticsearch_index_stats_search_scroll_total_req_rate meter_elasticsearch_index_stats_search_suggest_total_req_rate |
Search Operations ReqRate on index | elasticsearch-exporter |
| Shards Documents | meter_elasticsearch_index_indices_shards_docs | Count of documents per shards on index | elasticsearch-exporter | |
| Documents (Primary Shards) | meter_elasticsearch_index_indices_docs_primary | Count of documents with only primary shards on index | elasticsearch-exporter | |
| Documents Created Per Min (Primary Shards) | meter_elasticsearch_index_indices_docs_primary_rate | Documents rate with only primary shards on index | elasticsearch-exporter | |
| Total Size Of Index (Primary Shards) | MB | meter_elasticsearch_index_indices_store_size_bytes_primary | Current total size of stored index data in bytes with only primary shards on all nodes | elasticsearch-exporter |
| Documents (All Shards) | meter_elasticsearch_index_indices_docs_total | Count of documents with all shards on index | elasticsearch-exporter | |
| Documents Created Per Min (All Shards) | meter_elasticsearch_index_indices_docs_total_rate | Documents rate with only all shards on index | elasticsearch-exporter | |
| Total Size Of Index (All Shards) | MB | meter_elasticsearch_index_indices_store_size_bytes_total | Current total size of stored index data in bytes with all shards on all nodes | elasticsearch-exporter |
Customizations
You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found
in /config/otel-rules/elasticsearch/elasticsearch-cluster.yaml, /config/otel-rules/elasticsearch/elasticsearch-node.yaml, /config/otel-rules/elasticsearch/elasticsearch-index.yaml.
The Elasticsearch dashboard panel configurations are found in /config/ui-initialized-templates/elasticsearch.