Pulsar monitoring

SkyWalking leverages OpenTelemetry Collector to collect metrics data in Prometheus format from the Pulsar and transfer the metrics to OpenTelemetry receiver and into the Meter System. Kafka entity as a Service in OAP and on the `Layer: PULSAR.

Data flow

  1. Pulsar exposes metrics through Prometheus endpoint.
  2. OpenTelemetry Collector fetches metrics from Pulsar cluster via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
  3. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.`

Setup

  1. Set up Pulsar Cluster. (Pulsar cluster includes pulsar broker cluster and Bookkeeper bookie cluster.)
  2. Set up OpenTelemetry Collector. The example for OpenTelemetry Collector configuration, refer to here.
  3. Config SkyWalking OpenTelemetry receiver.

Pulsar Monitoring

Pulsar monitoring provides multidimensional metrics monitoring of Pulsar cluster as Layer: PULSAR Service in the OAP. In each cluster, the nodes are represented as Instance.

Pulsar Cluster Supported Metrics

Monitoring Panel Metric Name Description Data Source
Total Topics meter_pulsar_total_topics The number of Pulsar topics in this cluster. Pulsar Cluster
Total Subscriptions meter_pulsar_total_subscriptions The number of Pulsar subscriptions in this cluster. Pulsar Cluster
Total Producers meter_pulsar_total_producers The number of active producers connected to this cluster. Pulsar Cluster
Total Consumers meter_pulsar_total_consumers The number of active consumers connected to this cluster. Pulsar Cluster
Message Rate In meter_pulsar_message_rate_in The total message rate coming into this cluster (message per second). Pulsar Cluster
Message Rate Out meter_pulsar_message_rate_out The total message rate going out from this cluster (message per second). Pulsar Cluster
Throughput In meter_pulsar_throughput_in The total throughput coming into this cluster (byte per second). Pulsar Cluster
Throughput Out meter_pulsar_throughput_out The total throughput going out from this cluster (byte per second). Pulsar Cluster
Storage Size meter_pulsar_storage_size The total storage size of all topics in this broker (in bytes). Pulsar Cluster
Storage Logical Size meter_pulsar_storage_logical_size The storage size of all topics in this broker without replicas (in bytes). Pulsar Cluster
Storage Write Rate meter_pulsar_storage_write_rate The total message batches (entries) written to the storage for this broker (message batch per second). Pulsar Cluster
Storage Read Rate meter_pulsar_storage_read_rate The total message batches (entries) read from the storage for this broker (message batch per second). Pulsar Cluster

Pulsar Node Supported Metrics

Monitoring Panel Metric Name Description Data Source
Active Connections meter_pulsar_broker_active_connections The number of active connections. Pulsar Broker
Total Connections meter_pulsar_broker_total_connections The total number of connections. Pulsar Broker
Connection Create Success Count meter_pulsar_broker_connection_create_success_count The number of successfully created connections. Pulsar Broker
Connection Create Fail Count meter_pulsar_broker_connection_create_fail_count The number of failed connections. Pulsar Broker
Connection Closed Total Count meter_pulsar_broker_connection_closed_total_count The total number of closed connections. Pulsar Broker
JVM Buffer Pool Used meter_pulsar_broker_jvm_buffer_pool_used_bytes The usage of jvm buffer pool. Pulsar Broker
JVM Memory Pool Used meter_pulsar_broker_jvm_memory_pool_used The usage of jvm memory pool. Pulsar Broker
JVM Memory meter_pulsar_broker_jvm_memory_init
meter_pulsar_broker_jvm_memory_used
meter_pulsar_broker_jvm_memory_committed
The usage of jvm memory. Pulsar Broker
JVM Threads meter_pulsar_broker_jvm_threads_current
meter_pulsar_broker_jvm_threads_daemon
meter_pulsar_broker_jvm_threads_peak
meter_pulsar_broker_jvm_threads_deadlocked
The usage of jvm threads. Pulsar Broker
GC Time meter_pulsar_broker_jvm_gc_collection_seconds_sum Time spent in a given JVM garbage collector in seconds. Pulsar Broker
GC Count meter_pulsar_broker_jvm_gc_collection_seconds_count The count of a given JVM garbage collector. Pulsar Broker

Customizations

You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in otel-rules/pulsar/pulsar-cluster.yaml, otel-rules/pulsar/pulsar-broker.yaml. The RabbitMQ dashboard panel configurations are found in ui-initialized-templates/pulsar.