OTLP Service

The OTLP_SERVICE layer holds the services whose spans arrive over the OpenTelemetry protocol and are analyzed by SkyWalking’s Trace Analysis Language (TAL). OAP registers a service here from the service.name of every OTLP span it receives, derives the metrics below from the spans themselves, and draws the topology from each span’s peer. The OTLP trace receiver and the shipped TAL rule file produce the metrics and the topology; the Traces tab also needs OAP’s TraceQL module, see Requirements. These services are kept apart from the General Service layer because their metrics come from a different engine under their own otlp_* names; a process that runs both a SkyWalking agent and an OpenTelemetry SDK appears in both layers.

In Horizon’s sidebar the layer is named OTLP Service (grouped under OpenTelemetry, with the OTLP Virtual Database, OTLP Virtual Cache and OTLP Virtual MQ layers). It enables the Service, Instance and Endpoint scopes, the Topology and Traces tabs, and no profiling tabs, since those need SkyWalking agents. The Logs tab is off in the bundled template; see Traces and logs for turning it on. The API-dependency tab is off: an OTLP span carries no parent endpoint, so there are no endpoint-to-endpoint relations to draw.

This page is the operator reference for the bundled OTLP Service dashboard: what you see on each scope and what each widget means.

The widgets and metrics below are read from the bundled OTLP_SERVICE template; if an operator has published a customized OTLP_SERVICE template to OAP, the live dashboard reflects that copy instead. See Layer Dashboard Templates for how the bundled default, your local draft, and the OAP-published copy relate.

Service list

Before opening a service, the layer landing page lists every OpenTelemetry service with four sortable columns, sorted by RPM by default:

  • RPM — server and consumer spans per minute (otlp_service_cpm).

  • Apdex — the Apdex score computed from the span latencies against the service’s threshold in service-apdex-threshold.yml (otlp_service_apdex/10000).

  • Latency — average server and consumer span duration in ms (otlp_service_resp_time).

  • Error Rate — percent of server and consumer spans that failed: status ERROR, or an HTTP status of 400 or above (100 - otlp_service_sla/100).

Service dashboard

The primary drill-down for one selected service.

  • Top 20 Endpoints — the service’s endpoints ranked by traffic, by response time, or by worst success rate, one tab each (top_n(otlp_endpoint_cpm,20,des), top_n(otlp_endpoint_resp_time,20,des), top_n(otlp_endpoint_sla,20,asc)/100). An HTTP span’s endpoint is <method> <route>, so a route template such as GET /:endpoint stands for every request that matched it; a span with no route keeps its span name.

  • Traffic — server and consumer spans per minute, counted by the TAL rule before the receiver’s own head sampling, so it reflects what the SDK sent (otlp_service_cpm).

  • Error Rate — percent of server and consumer spans that failed, as in the service list (100 - otlp_service_sla/100).

  • Apdex — the Apdex score over the window (otlp_service_apdex/10000).

  • Response Time Percentile — p50 / p75 / p90 / p95 / p99 of the server and consumer span latency, from the buckets the rule declares (otlp_service_percentile).

  • Avg Response Time — mean server and consumer span duration in ms (otlp_service_resp_time).

  • Top 10 instances by load / slowest instances / instances by success rate — the service’s instances ranked by otlp_service_instance_cpm, otlp_service_instance_resp_time and otlp_service_instance_sla. Instances exist only when the SDK’s resource carries service.instance.id; a service whose SDK does not set it shows no data here and has no instance scope to open.

Instance dashboard

  • Service Instance Load — server and consumer spans per minute reported by the selected instance (otlp_service_instance_cpm).

  • Service Instance Latency — mean server and consumer span duration in ms (otlp_service_instance_resp_time).

  • Service Instance Success Rate — percent of the instance’s server and consumer spans that did not fail, as in the service list (otlp_service_instance_sla/100).

Endpoint dashboard

  • Traffic — server and consumer spans per minute on the selected endpoint (otlp_endpoint_cpm).

  • Response Time — mean duration in ms (otlp_endpoint_resp_time).

  • Success Rate — percent of the endpoint’s spans that did not fail, as in the service list (otlp_endpoint_sla/100).

  • Response Time Percentile — p50 / p75 / p90 / p95 / p99 of the endpoint’s latency (otlp_endpoint_percentile).

Topology

Nodes carry RPM, SLA and Latency from the service metrics above. Edges are observed on the client side, from the caller’s CLIENT and PRODUCER spans: RPM (otlp_service_relation_client_cpm), Avg response time (otlp_service_relation_client_resp_time) and SLA (otlp_service_relation_client_call_sla/100). The only server-side edge metric is RPM (otlp_service_relation_server_cpm), which exists for the calls a root server span receives, drawn from the User node, for the calls a consumer receives from its broker, and for the calls that reach a service through an uninstrumented gateway listed in gateways.yml; an OTLP server span carries no caller identity, so no other edge has server-side numbers. The map also shows the conjectured peers the calls reach, databases, caches and brokers, which live in the OpenTelemetry virtual layers.

A peer that a client named by address, cart:7070 or an IP, is folded onto the service that answers there once OAP has learned the address, from the server’s own span or by pairing the two spans of one call; the first calls to a freshly seen address can appear on an address-named node for a minute.

Traces and logs

The Traces tab reads the native OTLP store through OAP’s TraceQL endpoint, which is where the receiver keeps spans in its default otlp storage mode. OAP ships that endpoint off; Requirements lists what turns it on. A deployment that runs the receiver in its zipkin conversion mode stores no native spans, so switch the layer’s trace source to Zipkin in Dashboard setup → Layer dashboards there.

The Logs tab is off in the bundled template. The OAP accepts OTLP logs and stores them by service.name, so once a collector exports logs to the OAP, enabling the logs component on this layer shows them for the matching service.

Requirements

The dashboard is a pure consumer of what OAP reports and invents no data. It needs OAP 11.1.0 or later. To populate it, OAP needs the OTLP trace receiver with the trace-analyzer module’s shipped rule file, which produces:

  • Service, instance and endpoint metrics — the otlp_service_*, otlp_service_instance_* and otlp_endpoint_* families.

  • Relation metrics — otlp_service_relation_client_* and otlp_service_relation_server_cpm, which back the topology.

The Traces tab needs more, since OAP ships the TraceQL module and its OTLP datasource off: set SW_TRACEQL=default and SW_TRACEQL_ENABLE_DATASOURCE_OTLP=true on OAP, and point Horizon’s oap.traceql settings at it, see OAP Connection → TraceQL trace stores.

Each metric is queried at its own OAP scope. For the upstream setup — the rule file, its settings and the conventions it reads — see the OpenTelemetry trace analysis documentation.