SkyWalking
Pulsar monitoring
SkyWalking leverages OpenTelemetry Collector to collect metrics data in Prometheus format from the Pulsar and transfer the metrics to
OpenTelemetry receiver and into the Meter System.
Kafka entity as a Service in OAP and on the `Layer: PULSAR.
Data flow
- Pulsar exposes metrics through Prometheus endpoint.
- OpenTelemetry Collector fetches metrics from Pulsar cluster via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
- The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.`
Setup
- Set up Pulsar Cluster. (Pulsar cluster includes pulsar broker cluster and Bookkeeper bookie cluster.)
- Set up OpenTelemetry Collector. The example for OpenTelemetry Collector configuration, refer to here.
- Config SkyWalking OpenTelemetry receiver.
Pulsar Monitoring
Pulsar monitoring provides multidimensional metrics monitoring of Pulsar cluster as Layer: PULSAR Service in
the OAP. In each cluster, the nodes are represented as Instance.
Pulsar Cluster Supported Metrics
| Monitoring Panel | Metric Name | Description | Data Source |
|---|---|---|---|
| Total Topics | meter_pulsar_total_topics | The number of Pulsar topics in this cluster. | Pulsar Cluster |
| Total Subscriptions | meter_pulsar_total_subscriptions | The number of Pulsar subscriptions in this cluster. | Pulsar Cluster |
| Total Producers | meter_pulsar_total_producers | The number of active producers connected to this cluster. | Pulsar Cluster |
| Total Consumers | meter_pulsar_total_consumers | The number of active consumers connected to this cluster. | Pulsar Cluster |
| Message Rate In | meter_pulsar_message_rate_in | The total message rate coming into this cluster (message per second). | Pulsar Cluster |
| Message Rate Out | meter_pulsar_message_rate_out | The total message rate going out from this cluster (message per second). | Pulsar Cluster |
| Throughput In | meter_pulsar_throughput_in | The total throughput coming into this cluster (byte per second). | Pulsar Cluster |
| Throughput Out | meter_pulsar_throughput_out | The total throughput going out from this cluster (byte per second). | Pulsar Cluster |
| Storage Size | meter_pulsar_storage_size | The total storage size of all topics in this broker (in bytes). | Pulsar Cluster |
| Storage Logical Size | meter_pulsar_storage_logical_size | The storage size of all topics in this broker without replicas (in bytes). | Pulsar Cluster |
| Storage Write Rate | meter_pulsar_storage_write_rate | The total message batches (entries) written to the storage for this broker (message batch per second). | Pulsar Cluster |
| Storage Read Rate | meter_pulsar_storage_read_rate | The total message batches (entries) read from the storage for this broker (message batch per second). | Pulsar Cluster |
Pulsar Node Supported Metrics
| Monitoring Panel | Metric Name | Description | Data Source |
|---|---|---|---|
| Active Connections | meter_pulsar_broker_active_connections | The number of active connections. | Pulsar Broker |
| Total Connections | meter_pulsar_broker_total_connections | The total number of connections. | Pulsar Broker |
| Connection Create Success Count | meter_pulsar_broker_connection_create_success_count | The number of successfully created connections. | Pulsar Broker |
| Connection Create Fail Count | meter_pulsar_broker_connection_create_fail_count | The number of failed connections. | Pulsar Broker |
| Connection Closed Total Count | meter_pulsar_broker_connection_closed_total_count | The total number of closed connections. | Pulsar Broker |
| JVM Buffer Pool Used | meter_pulsar_broker_jvm_buffer_pool_used_bytes | The usage of jvm buffer pool. | Pulsar Broker |
| JVM Memory Pool Used | meter_pulsar_broker_jvm_memory_pool_used | The usage of jvm memory pool. | Pulsar Broker |
| JVM Memory | meter_pulsar_broker_jvm_memory_init meter_pulsar_broker_jvm_memory_used meter_pulsar_broker_jvm_memory_committed |
The usage of jvm memory. | Pulsar Broker |
| JVM Threads | meter_pulsar_broker_jvm_threads_current meter_pulsar_broker_jvm_threads_daemon meter_pulsar_broker_jvm_threads_peak meter_pulsar_broker_jvm_threads_deadlocked |
The usage of jvm threads. | Pulsar Broker |
| GC Time | meter_pulsar_broker_jvm_gc_collection_seconds_sum | Time spent in a given JVM garbage collector in seconds. | Pulsar Broker |
| GC Count | meter_pulsar_broker_jvm_gc_collection_seconds_count | The count of a given JVM garbage collector. | Pulsar Broker |
Customizations
You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found
in otel-rules/pulsar/pulsar-cluster.yaml, otel-rules/pulsar/pulsar-broker.yaml.
The Pulsar dashboard panel configurations are found in ui-initialized-templates/pulsar.