SkyWalking
ClickHouse monitoring
ClickHouse server performance from built-in metrics data
SkyWalking leverages ClickHouse built-in metrics data since v20.1.2.4. It leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System.
Data flow
- Configure ClickHouse to expose metrics data for scraping from Prometheus.
- OpenTelemetry Collector fetches metrics from ClickeHouse server through Prometheus endpoint, and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
- The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.
Set up
- Set up built-in prometheus endpoint .
- Set up OpenTelemetry Collector . For details on Prometheus Receiver in OpenTelemetry Collector, refer to here.
- Config SkyWalking OpenTelemetry receiver.
ClickHouse Monitoring
ClickHouse monitoring provides monitoring of the metrics 、events and asynchronous_metrics of the ClickHouse server.
ClickHouse cluster is cataloged as a Layer: CLICKHOUSE Service in OAP. Each ClickHouse server is cataloged as
an Instance in OAP.
ClickHouse Instance Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| CpuUsage | count | meter_clickhouse_instance_cpu_usage | CPU time spent seen by OS per second(according to ClickHouse.system.dashboard.CPU Usage (cores)). | ClickHouse |
| MemoryUsage | percentage | meter_clickhouse_instance_memory_usage | Total amount of memory (bytes) allocated by the server/ total amount of OS memory. | ClickHouse |
| MemoryAvailable | percentage | meter_clickhouse_instance_memory_available | Total amount of memory (bytes) available for program / total amount of OS memory. | ClickHouse |
| Uptime | sec | meter_clickhouse_instance_uptime | The server uptime in seconds. It includes the time spent for server initialization before accepting connections. | ClickHouse |
| Version | string | meter_clickhouse_instance_version | Version of the server in a single integer number in base-1000. | ClickHouse |
| FileOpen | count | meter_clickhouse_instance_file_open | Number of files opened. | ClickHouse |
ClickHouse Network Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| TcpConnections | count | meter_clickhouse_instance_tcp_connections meter_clickhouse_tcp_connections |
Number of connections to TCP server. | ClickHouse |
| MysqlConnections | count | meter_clickhouse_instance_mysql_connections meter_clickhouse_mysql_connections |
Number of client connections using MySQL protocol. | ClickHouse |
| HttpConnections | count | meter_clickhouse_instance_http_connections meter_clickhouse_mysql_connections |
Number of connections to HTTP server. | ClickHouse |
| InterserverConnections | count | meter_clickhouse_instance_interserver_connections meter_clickhouse_interserver_connections |
Number of connections from other replicas to fetch parts. | ClickHouse |
| PostgresqlConnections | count | meter_clickhouse_instance_postgresql_connections meter_clickhouse_postgresql_connections |
Number of client connections using PostgreSQL protocol. | ClickHouse |
| ReceiveBytes | bytes | meter_clickhouse_instance_network_receive_bytes meter_clickhouse_network_receive_bytes |
Total number of bytes received from network. | ClickHouse |
| SendBytes | bytes | meter_clickhouse_instance_network_send_bytes meter_clickhouse_network_send_bytes |
Total number of bytes send to network. | ClickHouse |
ClickHouse Query Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| QueryCount | count | meter_clickhouse_instance_query meter_clickhouse_query |
Number of executing queries. | ClickHouse |
| SelectQueryCount | count | meter_clickhouse_instance_query_select meter_clickhouse_query_select |
Number of executing queries, but only for SELECT queries. | ClickHouse |
| InsertQueryCount | count | meter_clickhouse_instance_query_insert meter_clickhouse_query_insert |
Number of executing queries, but only for INSERT queries. | ClickHouse |
| SelectQueryRate | count/sec | meter_clickhouse_instance_query_select_rate meter_clickhouse_query_select_rate |
Number of SELECT queries per second. | ClickHouse |
| InsertQueryRate | count/sec | meter_clickhouse_instance_query_insert_rate meter_clickhouse_query_insert_rate |
Number of INSERT queries per second. | ClickHouse |
| Querytime | microsec | meter_clickhouse_instance_querytime_microseconds meter_clickhouse_querytime_microseconds |
Total time of all queries. | ClickHouse |
| SelectQuerytime | microsec | meter_clickhouse_instance_querytime_select_microseconds meter_clickhouse_querytime_select_microseconds |
Total time of SELECT queries. | ClickHouse |
| InsertQuerytime | microsec | meter_clickhouse_instance_querytime_insert_microseconds meter_clickhouse_querytime_insert_microseconds |
Total time of INSERT queries. | ClickHouse |
| OtherQuerytime | microsec | meter_clickhouse_instance_querytime_other_microseconds meter_clickhouse_querytime_other_microseconds |
Total time of queries that are not SELECT or INSERT. | ClickHouse |
| QuerySlowCount | count | meter_clickhouse_instance_query_slow meter_clickhouse_query_slow |
Number of reads from a file that were slow. | ClickHouse |
ClickHouse Insertion Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| InsertQueryCount | count | meter_clickhouse_instance_query_insert meter_clickhouse_query_insert |
Number of executing queries, but only for INSERT queries. | ClickHouse |
| InsertedRowCount | count | meter_clickhouse_instance_inserted_rows meter_clickhouse_inserted_rows |
Number of rows INSERTed to all tables. | ClickHouse |
| InsertedBytes | bytes | meter_clickhouse_instance_inserted_bytes meter_clickhouse_inserted_bytes |
Number of bytes INSERTed to all tables. | ClickHouse |
| DelayedInsertCount | count | meter_clickhouse_instance_delayed_insert meter_clickhouse_delayed_insert |
Number of times the INSERT of a block to a MergeTree table was throttled due to high number of active data parts for partition. | ClickHouse |
ClickHouse Replicas Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| ReplicatedChecks | count | meter_clickhouse_instance_replicated_checks meter_clickhouse_replicated_checks |
Number of data parts checking for consistency. | ClickHouse |
| ReplicatedFetch | count | meter_clickhouse_instance_replicated_fetch meter_clickhouse_replicated_fetch |
Number of data parts being fetched from replica. | ClickHouse |
| ReplicatedSend | count | meter_clickhouse_instance_replicated_send meter_clickhouse_replicated_send |
Number of data parts being sent to replicas. | ClickHouse |
ClickHouse MergeTree Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| BackgroundMergeCount | count | meter_clickhouse_instance_background_merge meter_clickhouse_background_merge |
Number of executing background merges. | ClickHouse |
| MergeRows | count | meter_clickhouse_instance_merge_rows meter_clickhouse_merge_rows |
Rows read for background merges. This is the number of rows before merge. | ClickHouse |
| MergeUncompressedBytes | bytes | meter_clickhouse_instance_merge_uncompressed_bytes meter_clickhouse_merge_uncompressed_bytes |
Uncompressed bytes (for columns as they stored in memory) that was read for background merges. This is the number before merge. | ClickHouse |
| MoveCount | count | meter_clickhouse_instance_move meter_clickhouse_move |
Number of currently executing moves. | ClickHouse |
| PartsActive | Count | meter_clickhouse_instance_parts_active meter_clickhouse_parts_active |
Active data part, used by current and upcoming SELECTs. | ClickHouse |
| MutationsCount | count | meter_clickhouse_instance_mutations meter_clickhouse_mutations |
Number of mutations (ALTER DELETE/UPDATE). | ClickHouse |
ClickHouse Kafka Table Engine Supported Metrics
When table engine works with Apache Kafka.
Kafka lets you:
- Publish or subscribe to data flows.
- Organize fault-tolerant storage.
- Process streams as they become available.
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| KafkaMessagesRead | count | meter_clickhouse_instance_kafka_messages_read meter_clickhouse_kafka_messages_read |
Number of Kafka messages already processed by ClickHouse. | ClickHouse |
| KafkaWrites | count | meter_clickhouse_instance_kafka_writes meter_clickhouse_kafka_writes |
Number of writes (inserts) to Kafka tables. | ClickHouse |
| KafkaConsumers | count | meter_clickhouse_instance_kafka_consumers meter_clickhouse_kafka_consumers |
Number of active Kafka consumers. | ClickHouse |
| KafkProducers | count | meter_clickhouse_instance_kafka_producers meter_clickhouse_kafka_producers |
Number of active Kafka producer created. | ClickHouse |
ClickHouse ZooKeeper Supported Metrics
ClickHouse uses ZooKeeper for storing metadata of replicas when using replicated tables. If replicated tables are not used, this section of parameters can be omitted.
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| ZookeeperSession | count | meter_clickhouse_instance_zookeeper_session meter_clickhouse_zookeeper_session |
Number of sessions (connections) to ZooKeeper. | ClickHouse |
| ZookeeperWatch | count | meter_clickhouse_instance_zookeeper_watch meter_clickhouse_zookeeper_watch |
Number of watches (event subscriptions) in ZooKeeper. | ClickHouse |
| ZookeeperBytesSent | bytes | meter_clickhouse_instance_zookeeper_bytes_sent meter_clickhouse_zookeeper_bytes_sent |
Number of bytes send over network while communicating with ZooKeeper. | ClickHouse |
| ZookeeperBytesReceive | bytes | meter_clickhouse_instance_zookeeper_bytes_received meter_clickhouse_zookeeper_bytes_received |
Number of bytes send over network while communicating with ZooKeeper. | ClickHouse |
ClickHouse Keeper Supported Metrics
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| KeeperAliveConnections | count | meter_clickhouse_instance_keeper_connections_alive meter_clickhouse_keeper_connections_alive |
Number of alive connections for embedded ClickHouse Keeper. | ClickHouse |
| KeeperOutstandingRequets | count | meter_clickhouse_instance_keeper_outstanding_requests meter_clickhouse_keeper_outstanding_requests |
Number of outstanding requests for embedded ClickHouse Keeper. | ClickHouse |
Customizations
You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found
in /config/otel-rules/clickhouse. The ClickHouse dashboard panel configurations are found
in /config/ui-initialized-templates/clickhouse.