OTLP Virtual MQ

The OTLP_VIRTUAL_MQ layer holds the brokers that OpenTelemetry-instrumented services call, reconstructed from the callers’ spans. A broker appears here when a CLIENT or PRODUCER span of an OpenTelemetry Service names it, through messaging.system and the peer address or peer name the span carries; the broker itself reports nothing, so every figure describes it as its clients see it. The layer is kept apart from the agent path’s Virtual MQ layer because its metrics come from a different engine under their own otlp_* names.

In Horizon’s sidebar this layer is named OTLP Virtual MQ (grouped under OpenTelemetry, with the OTLP Service layer). Each broker is listed as one of the Brokers. It is a virtual-target layer with a single scope: only the Service scope is enabled; there is no instance or endpoint scope, no topology, and no traces or logs tabs. The broker does appear as a node on the OpenTelemetry Service layer’s topology, where the edges into it carry the calls.

This page is the operator reference for the bundled OTLP Virtual MQ dashboard.

The widgets and metrics below are read from the bundled OTLP_VIRTUAL_MQ template; if an operator has published a customized OTLP_VIRTUAL_MQ template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates.

Service list

The landing page lists every broker with three sortable columns, sorted by Access RPM by default:

  • Access RPM — messages per minute, counted from the callers’ spans (otlp_mq_access_cpm).

  • Latency — average duration of those spans in ms (otlp_mq_access_resp_time).

  • Error Rate — percent of them that failed: status ERROR, or an HTTP status of 400 or above (100 - otlp_mq_access_sla/100).

Service dashboard

  • Access Traffic — messages per minute (otlp_mq_access_cpm).

  • Avg Response Time — mean latency of the callers’ spans in ms (otlp_mq_access_resp_time).

  • Success Rate — percent of the callers’ spans that did not fail, as in the list (otlp_mq_access_sla/100).

Destinations

The broker node is one row of the layer; the queues and topics the services published to are its endpoints. The Top 20 destinations list ranks them by messages per minute (otlp_mq_destination_cpm), mean latency (otlp_mq_destination_resp_time) or lowest success rate (otlp_mq_destination_sla), all taken from the producer spans. A broker whose client library reports no address, as the JMS API gives its clients none, is named after its messaging system, jms, and still carries its destinations here.

Requirements

The dashboard is a pure consumer of what OAP reports. To populate it, OAP needs the OTLP trace receiver with the trace-analyzer module’s shipped rule file, and client or producer spans that carry messaging.system. The broker is named by the span’s peer name, or by server.address and server.port; a span that names no address or peer is filed under a broker named after its messaging system, such as jms. For the upstream setup see the OpenTelemetry trace analysis documentation.