RabbitMQ monitoring

SkyWalking leverages rabbitmq_prometheus plugin for collecting metrics data from RabbitMQ. It leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System.

Data flow

The rabbitmq_prometheus plugin collect metrics data from RabbitMQ. Note: The RabbitMQ version is required to be 3.8.0+. The rabbitmq_prometheus plugin is built-in since RabbitMQ v3.8.0.
OpenTelemetry Collector fetches metrics from rabbitmq_prometheus plugin via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.

Setup

Setup rabbitmq_prometheus.
Set up OpenTelemetry Collector. The example for OpenTelemetry Collector configuration, refer to here.
Config SkyWalking OpenTelemetry receiver.

RabbitMQ Monitoring

RabbitMQ monitoring provides multidimensional metrics monitoring of RabbitMQ cluster as Layer: RABBITMQ Service in the OAP. In each cluster, the nodes are represented as Instance.

RabbitMQ Cluster Supported Metrics

Monitoring Panel	Metric Name	Description	Data Source
Memory Available Before Publishers Blocked (MB)	meter_rabbitmq_memory_available_before_publisher_blocked	If the value is zero or less, the memory alarm will be triggered and all publishing connections across all cluster nodes will be blocked.	rabbitmq_prometheus plugin
Disk Space Available Before Publishers Blocked (GB)	meter_rabbitmq_disk_space_available_before_publisher_blocked	This metric is reported for the partition where the RabbitMQ data directory is stored.	rabbitmq_prometheus plugin
File Descriptors Available	meter_rabbitmq_file_descriptors_available	When this value reaches zero, new connections will not be accepted and disk write operations may fail.	rabbitmq_prometheus plugin
TCP Sockets Available	meter_rabbitmq_tcp_socket_available	When this value reaches zero, new connections will not be accepted.	rabbitmq_prometheus plugin
Messages Ready To Be Delivered To Consumers	meter_rabbitmq_message_ready_delivered_consumers	Total number of ready messages ready to be delivered to consumers.	rabbitmq_prometheus plugin
Messages Pending Consumer Acknowledgement	meter_rabbitmq_message_unacknowledged_delivered_consumers	The total number of messages that are either in-flight to consumers, currently being processed by consumers or simply waiting for the consumer acknowledgements to be processed by the queue. Until the queue processes the message acknowledgement, the message will remain unacknowledged.	rabbitmq_prometheus plugin
Messages Published	meter_rabbitmq_messages_published	The incoming message rate before any routing rules are applied.	rabbitmq_prometheus plugin
Messages Confirmed To Publishers	meter_rabbitmq_messages_confirmed	The rate of messages confirmed by the broker to publishers. Publishers must opt-in to receive message confirmations.	rabbitmq_prometheus plugin
Messages Unconfirmed To Publishers	meter_rabbitmq_messages_unconfirmed	The rate of messages received from publishers that have publisher confirms enabled and the broker has not confirmed yet.	rabbitmq_prometheus plugin
Messages Routed To Queues	meter_rabbitmq_messages_routed	The rate of messages received from publishers and successfully routed to the master queue replicas.	rabbitmq_prometheus plugin
Unroutable Messages Returned To Publishers	meter_rabbitmq_messages_unroutable_returned	The rate of messages that cannot be routed and are returned back to publishers.	rabbitmq_prometheus plugin
Unroutable Messages Dropped	meter_rabbitmq_messages_unroutable_dropped	The rate of messages that cannot be routed and are dropped.	rabbitmq_prometheus plugin
Queues Total	meter_rabbitmq_queues	Total number of queue masters per node.	rabbitmq_prometheus plugin
Queues Declared	meter_rabbitmq_queues_declared_total	The rate of queue declarations performed by clients.	rabbitmq_prometheus plugin
Queues Created	meter_rabbitmq_queues_created_total	The rate of new queues created (as opposed to redeclarations).	rabbitmq_prometheus plugin
Queues Deleted	meter_rabbitmq_queues_deleted_total	The rate of queues deleted.	rabbitmq_prometheus plugin
Channels Total	meter_rabbitmq_channels	Total number of channels on all currently opened connections.	rabbitmq_prometheus plugin
Channels Opened	meter_rabbitmq_channels_opened_total	The rate of new channels opened by applications across all connections. Channels are expected to be long-lived.	rabbitmq_prometheus plugin
Channels Closed	meter_rabbitmq_channels_closed_total	The rate of channels closed by applications across all connections. Channels are expected to be long-lived.	rabbitmq_prometheus plugin
Connections Total	meter_rabbitmq_connections	Total number of client connections.	rabbitmq_prometheus plugin
Connections Opened	meter_rabbitmq_connections_opened_total	The rate of new connections opened by clients. Connections are expected to be long-lived.	rabbitmq_prometheus plugin
Connections Closed	meter_rabbitmq_connections_closed_total	The rate of connections closed. Connections are expected to be long-lived.	rabbitmq_prometheus plugin

RabbitMQ Node Supported Metrics

Monitoring Panel	Unit	Metric Name	Description	Data Source
Ready Messages		meter_rabbitmq_node_queue_messages_ready	Total number of ready messages ready to be delivered to consumers.	rabbitmq_prometheus plugin
Unacknowledged Messages		meter_rabbitmq_node_unacknowledged_messages	Messages delivered to consumers but not yet acknowledged	rabbitmq_prometheus plugin
Incoming Messages		meter_rabbitmq_node_incoming_messages	The incoming message rate before any routing rules are applied.	rabbitmq_prometheus plugin
Outgoing Messages		meter_rabbitmq_node_outgoing_messages_total	The outgoing message rate before any routing rules are applied.	rabbitmq_prometheus plugin
Publishers		meter_rabbitmq_node_publisher_total	Publishers	rabbitmq_prometheus plugin
Consumers		meter_rabbitmq_node_consumer_total	Consumers currently connect	rabbitmq_prometheus plugin
Collections		meter_rabbitmq_node_connections_total	Connections currently open	rabbitmq_prometheus plugin
Channels		meter_rabbitmq_node_channel_total	Channels currently open	rabbitmq_prometheus plugin
Queues		meter_rabbitmq_node_queue_total	Queues available	rabbitmq_prometheus plugin
Allocated Used	%	meter_rabbitmq_node_allocated_used_percent	Erlang VM memory utilisation from erts_alloc perspective	rabbitmq_prometheus plugin
Allocated Unused	%	meter_rabbitmq_node_allocated_unused_percent	Erlang VM memory utilisation from erts_alloc perspective	rabbitmq_prometheus plugin
Allocated Used	MB	meter_rabbitmq_node_allocated_used_bytes	Erlang VM memory utilisation from erts_alloc perspective	rabbitmq_prometheus plugin
Allocated Unused	MB	meter_rabbitmq_node_allocated_unused_bytes	Erlang VM memory utilisation from erts_alloc perspective	rabbitmq_prometheus plugin
Allocated Total	MB	meter_rabbitmq_node_allocated_total_bytes	Erlang VM memory utilisation from erts_alloc perspective	rabbitmq_prometheus plugin
Resident Set Size	MB	meter_rabbitmq_node_process_resident_memory_bytes	Erlang VM Resident Set Size (RSS) As reported by the OS	rabbitmq_prometheus plugin
Allocators	MB	meter_rabbitmq_node_allocated_unused_bytes meter_rabbitmq_node_allocated_total_bytes meter_rabbitmq_node_process_resident_memory_bytes		rabbitmq_prometheus plugin
Allocated By Type	MB	meter_rabbitmq_node_allocated_by_type	Allocated by allocator type	rabbitmq_prometheus plugin
Multiblock Used	MB	meter_rabbitmq_node_allocated_multiblock_used	Multi block used	rabbitmq_prometheus plugin
Multiblock Unused	MB	meter_rabbitmq_node_allocated_multiblock_unused	Multi block used	rabbitmq_prometheus plugin
Multiblock Pool Used	MB	meter_rabbitmq_node_allocated_multiblock_pool_used	Multi block pool used	rabbitmq_prometheus plugin
Multiblock Pool Unused	MB	meter_rabbitmq_node_allocated_multiblock_pool_unused	Multi block pool unused	rabbitmq_prometheus plugin
Singleblock Used	MB	meter_rabbitmq_node_allocated_singleblock_used	Single block used	rabbitmq_prometheus plugin
Singleblock Unused	MB	meter_rabbitmq_node_allocated_singleblock_unused	Single block unused	rabbitmq_prometheus plugin

Customizations

You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in /config/otel-rules/rabbitmq/rabbitmq-cluster.yaml, /config/otel-rules/rabbitmq/rabbitmq-node.yaml. The RabbitMQ dashboard panel configurations are found in /config/ui-initialized-templates/rabbitmq.

Edit this page