11.1.0

Project

  • Update the e2e mock sender’s OTLP proto to v1.11.1. A current OpenTelemetry Collector adds fields the old copy lacked, such as a metric’s metadata, and the sender’s strict JSON parser refused a recording that carries them. The Windows e2e now also replays such a recording, of windows_exporter 0.31.8 through OpenTelemetry Collector 0.158.0 taken on a GitHub-hosted Windows runner, and asserts its memory metrics.
  • Replace the GenAI e2e cases’ Spring AI application with an in-repo e2e-spring-ai-service module pinned to the released Spring AI 2.0.1, and move the mock LLM endpoints out of e2e-service-provider into a dedicated e2e-mock-llm-server module that all three GenAI cases share. The Spring AI application was previously built at test time by cloning spring-projects/spring-ai-examples and running Maven inside the image build, which resolved spring-ai:2.0.0-SNAPSHOT: a line that has since been abandoned, whose surviving builds all predate the 2.0.0 GA by a day and whose older builds have been pruned, so the fixture was pinned to nothing released and could no longer be rebuilt from the same bytes. Both modules are built by the existing e2e reactor and published to ghcr.io/apache/skywalking alongside the other e2e service images; e2e-spring-ai-service needs Java 17 and Spring Boot 4, so it is gated behind a new jdk-17 profile in the e2e reactor and the jobs that build it now pin JDK 17 explicitly; make -C test docker refuses up front on an older JDK rather than packaging a missing or stale jar, since clean never reaches a module the profile excluded and compose would turn the absent jar into a directory.
  • Add the Envoy project’s NOTICE to NOTICE and to the binary distribution’s NOTICE, for the Envoy proto files SkyWalking bundles in receiver-proto.
  • The BanyanDB CA rotation e2e rotates the way an operator does: once the topology shows the consumer’s calls to the provider, the OAP trusts the old and the new CA, BanyanDB reloads a certificate the new CA signed, and the OAP then trusts the new CA only, before the storage assertions run. Each step writes its file once, and the OAP’s trust file is renamed into place, so the file monitor never reads it half-written. The rotation to an unrelated CA, which proves a rebuilt channel uses the trust file, moves to the end. It used to run first, and when the writes it fails included a network address alias’s first write, the alias stayed out of storage until its next refresh five minutes later, so the topology assertions timed out.
  • The BanyanDB credentials rotation e2e waits for the topology to show the consumer’s calls to the provider before it rotates, as the CA rotation e2e does. The incomplete credentials it applies fail the writes in that window, and when they included a network address alias’s first write, the alias stayed out of storage until its next refresh five minutes later, so the topology assertions timed out.
  • The BanyanDB credentials and CA rotation e2e cases now also prove that writes go through after the rotation. Their topology assertions use the default range, the last 30 minutes, which rows written before the rotation satisfy too, so they only showed that queries work on the rotated credentials or channel. Each case now keeps the time its last rotation took effect and asserts the provider topology drawn only from the minutes after it.
  • Clear the Dependabot CVE alerts in the e2e test fixtures, which no OAP artifact ships. The Go fixtures (cases/go/service and cases/profiling/ebpf/network) move to google.golang.org/grpc 1.79.3 → 1.83.2, golang.org/x/crypto 0.51.0 → 0.55.0 and github.com/klauspost/compress 1.15.9 → 1.18.7, and the Java test services to guava 32.0.0-jre → 33.7.2-jre and log4j 2.25.4 → 2.25.5.

OAP Server

  • Add customizable LLM-as-judge support for AI evaluation, with OpenAI-compatible endpoint / model / API key configuration, and persist the evaluation result as queryable GenAIEvaluationRecord rows for later inspection.
  • Support hot-reloading the BanyanDB credentials from a new secretsManagementFile setting in bydb.yml (SW_STORAGE_BANYANDB_SECRETS_MANAGEMENT_FILE), matching the ElasticSearch storage plugin. The properties file carrying user/password is watched, so a rotation performed by a 3rd party tool such as Vault is applied without restarting the OAP. The credentials are read per RPC by the gRPC auth interceptor, so they are swapped without re-establishing the channel and without interrupting in-flight queries or writes. Whatever the file contains is applied, including an incomplete pair, so that a mistake in it fails visibly rather than being masked by credentials that silently keep working; a username and a password are only ever sent together, so an incomplete file means requests carry no credentials and are answered with UNAUTHENTICATED rather than authenticating as the named user.
  • Support hot-reloading the BanyanDB TLS trust CA. The file at sslTrustCAPath is now watched, and a change rebuilds the gRPC channel so the new CA takes effect without restarting the OAP. Previously the CA was only re-read after the certificate in use had already caused requests to fail, which meant a rotation was paid for with an outage. The replacement channel is created before the old one is released, so a failed rebuild leaves the current channel serving; requests already in flight finish on the old channel. The replacement is not health-checked before the swap — a gRPC channel connects lazily — so rotating to a CA that does not validate the server interrupts traffic until valid material is written back; the storage doc describes the old+new overlap procedure that avoids this. Note the replacement re-picks an address from targets, so the OAP may connect to a different node after a rotation.
  • Fix MultipleFilesChangeMonitor being able to silently disable every file watch in the OAP. Its registry of monitors was a plain ArrayList that scanChanges() iterated from the scheduler thread without holding the lock that start() / stop() take, so starting a monitor while a scan was in flight could raise a ConcurrentModificationException from the iterator. That exception escapes past the per-monitor catch, and an uncaught exception cancels a scheduleAtFixedRate task permanently — after which no secrets file, keystore, or TLS certificate is ever reloaded again, with nothing in the log to say so. The registry is now copy-on-write. The failure log in the same scan loop also now names the monitor that failed instead of printing an empty gourp = .
  • Fix MultipleFilesChangeMonitor never honouring its watching period. lastCheckTimestamp was declared and compared against, but never assigned, so the guard always measured against 0 and passed — every registered monitor re-stat’d its watched files on each 200ms tick of the shared scheduler thread, and the watchingPeriodInSec constructor argument had no effect at all. This affects every file watch in the OAP: the ElasticSearch storage secrets / truststore / keystore watch, the BanyanDB credentials and trust CA watches, and the TLS certificate watches behind each OAP HTTP and gRPC server, all of which ask for 10 seconds. Change detection is now paced as configured, which also means it is no longer near-instant: a rotated file is picked up within the requested period rather than within ~200ms.
  • Add BanyanDB trace tail sampling metrics to the BanyanDB self-observability layer, in a new otel-rules/banyandb/banyandb-trace-sampling.yaml rule file. It covers the whole banyandb_trace_pipeline_* / banyandb_trace_tst_pipeline_* catalog a sampler plugin chain emits — pipeline reconciliation, per-plugin Decide execution rate and latency, chain batching, the trace-level evaluated / retained / dropped / immature outcomes, every fail-open guard and bounded-retention counter, drop-set capacity and finalization state, the plugin telemetry-host safety bounds, and the first-party sw-trace-sampler / zipkin-trace-sampler decision and row metrics. The plugin chain is optional, and the metrics follow it: on a cluster with no sampler configured the wire families are never registered, so every metric here stays absent rather than reading zero. Modeled at Service scope with group kept as a metric label rather than at Endpoint scope, so one cluster-wide page can render per-group series and cluster totals alike — OAP does no cross-scope rollup, so an Endpoint-scope metric could not have been aggregated back up to the cluster.
  • Fix a second CounterWindow key collision in the v2 MAL engine, this time ACROSS rules. rate() / increase() / irate() resolve their lower bound from a process-wide window keyed on the counter’s own name plus its post-.sum(...) label set, with nothing identifying the rule doing the evaluation. Two rules that read one wire family, tell their streams apart with tagEqual(...), and then .sum(...) away the label they filtered on therefore collapse onto one window slot and difference against each other’s values. The queue is ordered by (timestamp, value), so the smaller counter wins the lower-bound lookup and still reads correctly while its partner is inflated by the gap between them — which is why this went unnoticed. A collision needs the discriminating label to be DROPPED by the .sum(...): where it survives, the rules’ label values differ and the window keeps them apart. Auditing the shipped rules on that basis gives 10 colliding keys over ~25 rules — meter_activemq_cluster_gc_parallel_young_collection_count reported ~9000/min of young-gen collections from a completely idle broker (differencing against the old-gen counter); MySQL commands_* / tps rate against each other; so do the GenAI gateway input/output token rates, four Envoy cluster_* counters, APISIX matched/unmatched instance bandwidth, and BanyanDB’s own network_recv / network_sent, which drop the kind label that separates bytes-received from bytes-sent on one interface. Measured against two live scrapes of the demo cluster’s FODC proxy, that last pair was wrong on every interface: network_sent read a flat 0 B/s and network_recv read large negative values (down to -778 MB/s) from differencing against the sent counter, where both now match the byte delta exactly. No rule changes were needed for any of these – each rule already reduces to the labels it should; only the window key was wrong. The window is now keyed by (owning rule, counter name, labels). This is the complement of the within-rule collision fixed earlier by keying on the counter’s own name: neither name alone is sufficient, because the two collisions are independent. RunningContext.metricName — written on every rule evaluation and read by nobody since that earlier fix — is what supplies the rule identity, so no code generation or MAL syntax changes. Note the whole-rule-set comparison suite could not have caught this: it resets the shared window before every rule, the one condition under which the collision cannot appear.
  • Fix meter_rabbitmq_node_outgoing_messages_total double-counting one of its terms. The rule summed six delivery-rate terms but rabbitmq_global_messages_delivered_get_auto_ack_total appeared twice, so auto-ack basic.get deliveries were counted once more than the other four delivery paths and the reported outgoing rate ran high whenever polling consumers were in use. The duplicate term is removed, leaving the five distinct families (redelivered, consume auto/manual ack, get auto/manual ack).
  • Add AI agent conversations landed by the AI Sessionizer: the AI_AGENT layer, the bundled lal/ai-agent.yaml rule with the ConversationFile output builder that verifies and stores Session Data and Session Flow files, the ai_agent_session_data and ai_agent_session_flow models in a new BanyanDB group recordsAIAgent, the ai-agent-conversation module that folds a conversation into one asz.view document, the listConversations GraphQL query (with optional conversation and title conditions), and two streamed HTTP routes beside /graphql: GET /ai-agent/conversations/{conversation}/v1/view, which serves the document, and GET /ai-agent/conversations/{conversation}/v1/files, which serves up to 32 Session Data files of a session, each chosen by its landed seq, as application/vnd.skywalking.asz.files+ndjson, a line naming each file followed by the file’s bytes, so a page loads the files a step points at only when a reader opens it; there is no read of every file. Both routes require the service and the sender’s instance, as a list row names them, so every read is a full series lookup. The view and the files route leave BanyanDB’s stage selection unspecified by default and select only the cold stage when the caller explicitly sets coldStage to true. A round from before the list attributes existed lands and lists with zero counts, and the view shows as much as landed: the chain resumes after a missing, unreadable or refused round, and the absent rounds and files are named once as ranges. A file over maxFileBytes, 15 MiB by default, is rejected at ingest and counted under the reason size, because one file over BanyanDB’s 16 MiB gRPC message limit fails the bulk write it travels in and every record behind it; on MySQL the body column is LONGTEXT, since a body that size outgrows MEDIUMTEXT as Base64. A conversation’s rounds and files are read readWindow at a time, 16 by default, and each of those reads is capped at maxResponseBytes, 100 MiB by default, as a per-call option on the BanyanDB client in place of its 50 MB default, so the module’s reads are bounded by its own settings and nothing else’s read changes. The document also carries the session’s workspace changes: the Sessionizer’s Claude Code plugin lands a Session Data file of kind changes beside a stream’s transcript, streams/<stream>/changes-<stamp>-<seq>.sd, with one change record per observed tool call, stored like any other file, and the runtime’s own patch for its editing tools travels as the second data part of the call’s result record; workspace_changes lists every record joined to its step by tool-use id with the record’s fields as changes/1 lists them, summary.changes counts them, and a tool step names its records under changes. The document also names the session’s provider bodies: the request and response bodies a runtime exchanged with its model provider land as Session Data files of kind provider_body, <session>/provider_body/provider_body-<stamp>-<seq>.sd, stored like any other file; an llm.call step lists where its request and response landed under provider_bodies, as the round carries the join the Sessionizer made when it parsed the round, and summary.provider_bodies and summary.captured_prompts count them. A round’s header counts, changes, lines_added, lines_removed, llm_calls, subagents and bash_runs, land on ai_agent_session_flow and reach the conversation list as ConversationRow fields, absent rather than zero when the round did not carry them. The layer also takes the agent runtime’s metrics, in the otel-rules/ai-agent/ rule set enabled by default: agent.token.usage, which the Sessionizer derives from the landed transcripts, per service and per sender, by type, model and query source with session.id summed away, plus the cost, active time, sessions, lines of code, commits, pull requests and edit decisions that Claude Code’s own exporter sends when it is pointed at the OAP directly; its own token metric is not read, so a call is never counted twice.
  • Support querying Zipkin traces from the BanyanDB cold stage through the Zipkin HTTP query API. /api/v2/traces takes an optional coldStage parameter, and /api/v2/trace/{traceId} and /api/v2/traceMany take optional coldStage, endTs and lookback parameters; the two by-id lookups had no time range at all before. All of them are SkyWalking additions to the Zipkin API and default to the previous behavior, so existing Zipkin clients such as the Lens UI keep working unchanged, and storages other than BanyanDB ignore coldStage. The admin debugging endpoints /debugging/query/zipkin/api/v2/traces and /debugging/query/zipkin/api/v2/trace accept the same parameters. The BanyanDB stages e2e now seeds Zipkin traces into the generated cold data and verifies the three endpoints against it.
  • BanyanDB: a query without a time range now covers everything the group’s hot/warm stages retain instead of only the last 24 hours. The hard-coded day was a fallback added with the trace model, which requires a time range; it made queryTrace(traceId) without a duration, the Zipkin /api/v2/trace/{traceId} and /api/v2/traceMany lookups, and the TraceQL by-id lookup return “not found” for any trace older than a day even though it was still retained. The unbounded range is now bound instead, and BanyanDB’s own retention limits the scan. queryTraces now rejects a condition with neither queryDuration nor traceId instead of silently searching the last day; a traceId lookup without queryDuration searches everything the hot/warm stages retain and reports a 0-to-now retrievedTimeRange.
  • Fix the JDBC storages, MySQL, PostgreSQL and H2, reading only the first day’s table of a time range shorter than a day that crosses midnight. TableHelper walked the range in 24-hour steps from its first instant, so a range that began late on one day and ended early on the next never reached the second day; every time-ranged read, metrics, traces, logs, alarms, records and the AI agent conversation list, was missing the newer day’s rows for the length of the range after each midnight, the last thirty minutes read at 00:10 among them. The range is now walked by calendar day from the start of its first day.
  • Fix a minute of metrics on BanyanDB losing the points of an earlier persistence round. PersistenceTimer counted the delay to the next round from the moment a round’s writes were queued, not answered: the per-worker future completed when batchDAO.flush returned, the flush’s own future was used only to close a latency histogram, and endOfFlush ran after that. A metric enters the persistence session cache only when the storage acknowledges its write, and the BanyanDB bulk processor holds a queued write for up to flushInterval (15s) before sending it, so under the 25s persistentPeriod an acknowledgement that lagged ten seconds, as it can on a BanyanDB still busy with the boot’s schema installs, let the next round find the metric absent from the cache, take the minute for a new one and write it again with only its own points; a measure write replaces the series at that time, so the earlier points were gone. A round now completes when every request of it has been answered and the next round’s delay counts from there, and endOfFlush, which already forced the Elasticsearch bulk out, now forces the BanyanDB measure bulk out too, so the wait is one round trip and not the bulk’s interval. Observed as the first minute of a burst of OTLP metrics vanishing right after an OAP boot.
  • Fix BanyanDB bulk writes leaving skipped requests’ futures pending when a write fails to build. Complete every unfinished request in the aborted batch exceptionally, so persistence rounds and runtime-rule removal can finish with an error instead of waiting indefinitely.
  • Apply the configured max request header size to HTTP/2 as well. httpMaxRequestHeaderSize (and the AWS Firehose receiver’s maxRequestHeaderSize) only set Armeria’s http1MaxHeaderSize, so an HTTP/2 client kept the framework default header list size and the setting had no effect on the protocol the OTLP and gRPC-Web clients actually use. The value is now also applied as http2MaxHeaderListSize; 0, which means unlimited on the HTTP/1 side, leaves the HTTP/2 default in place because Armeria rejects a non-positive header list size.
  • Add a none provider to the ai-agent-conversation module, so the AI agent conversation feature can be turned off. The GraphQL query module requires the module, so the - selector cannot remove it; SW_AI_AGENT_CONVERSATION=none keeps the ai_agent_session_data and ai_agent_session_flow models out of the storage, answers every conversation query empty and registers no view route.
  • Fix the OpenTelemetry receiver folding a request that carries several minutes of points into one. A MAL rule folds every sample of an entity into one value at the first sample’s time, so a delta exporter that batches, or a sender replaying history, landed as one minute. A request is now analysed a minute at a time, oldest first; a request whose points share one minute, a scrape relayed by a collector, is analysed exactly as before.
  • Fix the runtime-rule sync tearing down every bundled MAL and LAL rule file on its first pass after each boot: StaticRuleLoader seeds a bundled rule with a RUNNING state, but the gone-keys pass recognised a boot-seeded entry only by a null state, so every shipped rule file without an operator row was torn down and reloaded from the bundle one after another. A MAL teardown waits a persistence round for the metric drain, and for that round the OpenTelemetry receiver held no converter for the file and dropped its samples without a log line, so every restart lost up to a round of metrics from each rule set within its first minute or two. A rule that runs its bundled file as shipped is now left alone; only an operator override whose row is gone falls over to the bundle.
  • Open every port that takes requests only after the boot is complete. A new ModuleProvider.notifyBootCompleted hook runs once every module’s notifyAfterCompleted has returned, and the core’s gRPC and HTTP servers, the sharing server, the Zipkin, AWS Firehose, eBPF, Envoy metrics and Zabbix receivers, the admin server and the PromQL, LogQL, TraceQL and Zipkin query servers bind there. Before, the core opened 11800 and 12800 in its own notifyAfterCompleted, ahead of the modules started after it, so a request could arrive while the runtime-rule boot pass or any later boot-time work was still running; the readiness probes, which check the port, now mean what they say.
  • Fix the Elasticsearch storage filtering events by a layer name against the numeric layer column, which made every event query carrying a layer condition fail with number_format_exception while the query was built, before any document was read. It is what Horizon UI’s per-service events popout sends, so that popout failed for every service; on an empty index as well. The condition value is now resolved to its layer value, as the JDBC and BanyanDB storages already do.
  • Store OTLP traces natively. receiver-otel gains otlpTraceStorage: otlp, which keeps each span as the ResourceSpans it arrived in (otlp_span, BanyanDB group otlpTrace) with service, span-name and peer catalogs, and the TraceQL module gains an /otlp datasource (enableDatasourceOTLP) that serves those spans to Grafana Tempo without any Zipkin conversion. otlp is the new default; otlpTraceStorage: zipkin keeps the previous conversion to Zipkin v2 through receiver-zipkin, and that path is unchanged.
  • Bring the TraceQL module closer to the Tempo API. Every datasource lists only the spans that matched a search in each span set, capped at spss, fills serviceStats, filters span.http.status_code, honours scope and the per-scope limit on the tag endpoints, whose intrinsic scope lists what each datasource can filter, accepts Tempo’s unscoped .key tag names on the tag-value endpoints, and answers a tag-name or tag-value lookup filtered by q from a sample of the newest 50 matching traces, so Grafana’s dependent dropdowns narrow down; the status and kind dropdowns always list their full enum. The /otlp datasource also accepts Tempo’s span:name, span:kind, span:status and span:duration spellings. /api/metrics/query_range and /api/metrics/query answer 501 Not Implemented with an error body and accept Tempo’s q parameter, instead of 200 with a string Grafana cannot parse.
  • Refuse unsupported TraceQL with 400 instead of dropping the predicate and answering with unfiltered traces: syntax errors (|| inside a spanset, regex, an attribute without a scope or leading dot), != and other unsupported operators, negation, attribute-existence checks, multiple spansets, unknown intrinsics and values, and conditions a datasource cannot filter (kind on Zipkin and SkyWalking, resource.instance on Zipkin, resource.remote.service and status = unset on SkyWalking, resource.instance or name without a service on SkyWalking). The deprecated tags search parameter is parsed as logfmt on every datasource and goes through the same mapping and refusals. kind = server and status = error accept the bare keyword as Tempo does. Every search-result span carries a status attribute (error, ok, unset) next to service.name and span.kind, so a trace list can show failures. A trace the storage matched but none of whose spans satisfies every condition is left out instead of listed with every span, and the /otlp datasource compares names the way the receiver indexed them and keeps the resource. and span. scopes apart in its tag index, so span.env no longer matches a resource attribute. duration > and < are strict at microsecond precision. (#14093)
  • Support the OpenTelemetry Collector hostmetrics receiver as an alternative source for Linux and Windows host monitoring. vm.yaml and windows.yaml map it to the same meter_vm_* / meter_win_* metrics as node-exporter and windows_exporter, whose metrics keep their meaning, and add metrics only hostmetrics provides: CPU core count and normalized CPU usage, plus CPU load, file-system usage, system handle count and pagefile usage on Windows. node-exporter metrics without a matching hostmetrics source (tcp_alloc, sockets_used, udp_inuse, filefd_allocated) stay node-exporter only. The new process-hostmetrics-linux and process-hostmetrics-windows rules, enabled by default, report each process name as an instance of its host (mp_process_linux_* / mp_process_windows_*: process count, threads, CPU, resident memory, open handles and oldest process uptime). Reference Collector configurations for both systems are under docs/en/setup/backend/; they set the job_name and normalized process_name labels the rules route and group on, and sum same-named processes before export.
  • Windows monitoring through windows_exporter now requires windows_exporter 0.29.0 or later. windows_exporter 0.31.0 removed the cs collector and the deprecated os memory metrics (windows_cs_physical_memory_bytes, windows_os_physical_memory_free_bytes, windows_os_virtual_memory_bytes, windows_os_virtual_memory_free_bytes) that windows.yaml read, so on 0.31.0+ meter_win_memory_total / _available / _used and the meter_win_memory_virtual_memory_* commit metrics were empty. The rules now read their replacements from the memory collector (windows_memory_physical_total_bytes, windows_memory_physical_free_bytes, windows_memory_commit_limit, windows_memory_committed_bytes), all of which windows_exporter provides since 0.29.0; on an older windows_exporter these memory metrics stay empty until it is upgraded. Also fix meter_win_cpu_total_percentage counting overlapping Windows CPU modes twice: windows_exporter’s privileged already includes interrupt and dpc time, and OpenTelemetry hostmetrics’ system already includes interrupt time, so busy CPU is now user + privileged or user + system.
  • Fix the OpenTelemetry receiver creating no Linux or Windows host service from an OpenTelemetry Collector 0.127.0 or later. The Collector’s Prometheus receiver sends a scraped target’s host only as server.address from that release, no longer as net.host.name, and vm.yaml and windows.yaml name a host by node_identifier_host_name, which the receiver took only from net.host.name or host.name. It now falls back to server.address when neither is present.
  • AI agent conversations adopt AI Sessionizer 130601c. Calls to MCP servers: the execution kind the Sessionizer’s Claude Code plugin writes, streams/<stream>/execution-<stamp>-<seq>.sd, is stored like any other file; the document lists every execution/1 record under tool_executions, joined to its step by tool-use id and kept once by its own id, a tool step names its records under executions, and a call to an MCP server carries mcp_server and mcp_tool in its attrs. The new otel-rules/ai-agent/mcp_endpoint.yaml gives one endpoint per MCP server and tool, <server>/<tool>, under the agent’s service, with meter_ai_agent_mcp_calls, meter_ai_agent_mcp_calls_by_outcome and meter_ai_agent_mcp_duration from the Sessionizer’s agent.mcp.calls and agent.mcp.duration. The token rules read agent.token.usage, the Sessionizer’s name for the metric. The document follows the Sessionizer’s: an llm.call takes its provider bodies from the round’s provider_bodies attribute and the session its count from provider_bodies_landed, and a round whose bodies do not read or point past its range is refused; talks are in the order they began across streams; only a child stream’s talk is a child’s, not an auxiliary one’s; a stream’s opened_by, the relations and a step’s edges are in the order they happened, by record position inside one stream or workflow run and by time across them, never by id; change and execution records of one instant are in the order they were read; a record is read by the fields its format lists, and one of another shape is skipped; a round with a frame field of another type, or a negative sequence, row or round number, does not read; a record time is RFC 3339 as Session Data defines it, to the second, and anything else is no time; record times compare as instants.
  • Fix the Elasticsearch BulkProcessor treating a bulk write as fully successful whenever the overall HTTP status was 200, even when Elasticsearch rejected individual items in the same response, for example a 429 cluster_block_exception when an index is switched to read_only_allow_delete by the flood-stage disk watermark. The response body’s errors flag and per-item status/error are now decoded and checked on every 200 response; a rejected request’s future now completes exceptionally instead of being treated as a success, while the rest of the batch still completes normally. One ERROR line per _bulk request summarizes the rejections grouped by status code and error type, never a document id or the raw ES error reason. A response whose items array is shorter than the request (a malformed or truncated response) also fails those requests instead of silently treating them as succeeded. This also fixes a related batching bug where, once a flush was split into multiple _bulk HTTP requests by batchOfBytes, the completion of one chunk’s request completed or failed every request in the whole flush instead of just its own chunk, and a bug where requests whose bulk failed to even build (e.g. an encoding error) were left with a future that never completed, which could block a persistence round forever.
  • AI agent conversations adopt AI Sessionizer 6691c36, and withhold what the runtime sent the model. The Sessionizer names the records that carry the system prompt and the tool schemas with the flags system_prompt and tool_schemas; the hide setting (SW_AI_AGENT_CONVERSATION_HIDE) names what this OAP withholds from every reader; the routes take no parameter for it. Withholding happens as a conversation is served, and only then: the storage holds every file whole, and both routes rewrite what they serve. In the document a step read from a named record keeps its node, flags and size, loses its text and has state omitted, no such injection names a talk or a stream, a person’s input names its talk whatever it carries, summary.withheld counts the named records once per id, and every count, every call’s provider_bodies and captured_prompts stay. A file is served with what it holds of the names masked: a named transcript record loses its content, and a request body is rebuilt, what it holds of the names replaced by markers at any depth, by the shapes the Sessionizer’s LangChain page lists, and written back as a body of its own with its own digest; the naming line says so with withheld and served_digest. A name nothing withholds is refused at startup. summary.withheld is {} when nothing is withheld, and a model call carries the flags of its record a reader may withhold, as the Sessionizer’s documents now do.
  • Fix endpoints registered from logs missing from findEndpoint (the endpoint list) when it is queried with a time range. The query filters on an endpoint’s last_ping, which endpoints registered from logs, from virtual MQ spans, from cross-thread segment references and from Cilium Hubble never set, so they kept last_ping at 0, and an endpoint also seen in traces was reset to 0 whenever one of those sources wrote it last. Every endpoint source now sets last_ping, and an update keeps the latest last_ping instead of the last one written. (#14121)
  • Add alarm.default.httpTimeout (SW_ALARM_HTTP_TIMEOUT, default 12 seconds) in application.yml, the deadline of each HTTP request an HTTP-based alarm hook (Webhook, Slack, WeChat, DingTalk, Feishu, WeLink, PagerDuty, Discord) sends, covering connecting, waiting for the response and reading the response body; the gRPC hook is unchanged. The previous fixed 12-second request timeout stopped once the response headers arrived, so a hook endpoint that stalled while sending the body could hold the alarm delivery thread indefinitely.
  • AI agent conversations adopt AI Sessionizer aa67971. A segment in the document is an activity window, with id, talks, from and to: the Sessionizer stopped writing a segment’s state and committable, which could not be decided without a clock, so the document no longer lists them. Its rounds also stopped repeating what they say elsewhere, a talk’s loops, runs and trigger among them, and a step’s attrs carry what the round wrote, as before. A provider body’s manifest no longer carries run or why, and the reader still accepts a manifest that does; the unused Manifest.why() and SessionFlowRound.Node.attrBool are removed. The test fixtures are that commit’s output.
  • AI agent conversations show where a model call’s history changed, as the AI Sessionizer draws it (apache/skywalking-ai-sessionizer#77). The Sessionizer compares each call’s history with the call before it and writes what it found in the call’s attrs; where the history changed, it adds a history.change node just before the call, under the same parent, at the call’s first record. The document lists that node as a step that carries no text of the record it stands on, and orders it among the nodes at one record by the call it names, and before that call, as the Sessionizer orders it.
  • AI agent conversations follow the AI Sessionizer’s id and round rules (apache/skywalking-ai-sessionizer#80). The Sessionizer writes a % in a part of an id as %25 and a / as %2F, so the document finds a session’s node by that spelling, and takes the session’s time range and provider bodies from it, where it looked for the old one. It lists the session the round headers carry, and the conversation’s files are read for that session, instead of a session read back out of a node’s id. A round holding an empty line, anywhere but after the newline that ends its last line, is refused, as Session Flow says.
  • Add the Trace Analysis Language (TAL), a compiled DSL in the new trace-analyzer module that turns the spans arriving over OTLP into metrics, topology and records through the meter system, in both OTLP trace storage modes and ahead of the receiver’s head sampling. The shipped tal/otlp-general.yaml analyzes the spans whose resource carries service.layer=OTLP_GENERAL, which an application sets through OTEL_RESOURCE_ATTRIBUTES, while every other span is stored and queryable but not analyzed; it reads the OpenTelemetry semantic conventions, current spellings first and the deprecated ones as fallbacks, and registers those services under the new OTLP_GENERAL layer with otlp_service_*, otlp_service_instance_*, otlp_endpoint_* and otlp_service_relation_* metrics, named and counted as the agent path’s core.oal metrics are, their databases, caches and brokers under OTLP_VIRTUAL_DATABASE, OTLP_VIRTUAL_CACHE and OTLP_VIRTUAL_MQ with otlp_database_access_*, otlp_cache_access_* and otlp_mq_access_*, and per destination otlp_mq_destination_*, a User node for root server spans, and slow statements as DatabaseSlowStatement records. Peers that clients name by address rather than by service are folded onto the service: the caller’s prepareNetworkAlias and the callee’s buildNetworkAlias pair the two halves of one call, the callee being the SERVER span whose parent is the caller’s span, across the OAP cluster, to learn a network address alias keyed by the caller and the address it dialed, frontend->cart:7070, so one address can name different services for different callers and a proxy keeps its own node; resolveNetworkAlias reads it, and the service topology query folds an address-named node by the caller’s own alias before the plain address; an address listed in gateways.yml is never aliased, and the shipped rule draws it as a VIRTUAL_GATEWAY node between the caller and the callee, as the agent path does, through the new isGateway function. A database or broker that a span names by no address is named by its db.system and db.namespace or by its messaging.system. A SERVER span whose OTLP span flags say its parent ran in the same process is not counted as a request. A root CONSUMER messaging receive, a poll whose span the Java agent times from the start of the wait, and any CLIENT receive are not counted, while a CONSUMER receive with a parent, a consumer’s only record of the message as the .NET Kafka instrumentation reports it, is counted as the consumer’s request. New trace-analyzer settings talFiles, dedupWindow, dedupCapacity and pairingSamplePercent, the share of traces paired, 0 for none; an empty talFiles, or removing the module with selector: -, stores the OTLP traces without analyzing them; component-libraries.yml gains Connect_RPC and the catalog answers lower-case names; MeterEntity relations may carry a layer per side; the meter system’s labeled functions take an optional per-metric limit of label sets per row, which TAL sets to 50 by default and MAL leaves unset. See the TAL concept doc, the trace analyzer setup doc and the OpenTelemetry Trace Analysis Practices doc.
  • Fix the max and min meter functions keeping the first value of a minute when two values for one entity are merged, at the L1/L2 aggregation and at the hour and day downsampling, and latestLabeled dropping every label set the merged value does not carry. MAL rules using max, min or latestLabeled reported wrong values whenever samples of one entity arrived in more than one batch.
  • Fix the OTLP/HTTP JSON trace endpoint reading the hex traceId, spanId and parentSpanId the OTLP/JSON encoding specifies as base64, the protobuf JSON mapping. The bodies parsed without an error into ids of the wrong length, so a trace exported as JSON matched neither the ids its SDK logged nor the same trace exported over gRPC or OTLP/HTTP protobuf.
  • Fix the heatmap query failing on a meter histogram whose bucket bounds exceed an int, such as the 9223372036854775807 key MAL gives an OTel histogram’s +Inf bucket: the bucket keys are read as longs.
  • Cut the slow statements a LAL SlowSQL output writes, as the default mysql-slowsql, pgsql-slowsql and redis-slowsql rules do, to the 2000 characters of the top_n_database_statement column, as the agent path already cuts them to maxSlowSQLLength. A longer statement was refused by a strict SQL store, and stored whole on Elasticsearch and BanyanDB; it is now cut there too.
  • Fix a UI template update on BanyanDB leaving a disabled template disabled. PUT /ui-management/templates wrote only the configuration there, while on ElasticSearch and JDBC it replaces the whole record and enables it. An update is the only way to bring a disabled template back on those two stores, where adding refuses an existing id; the BanyanDB update now writes the whole record too, so the same call works on every storage.
  • Fix the Elasticsearch storage reading metrics from the wrong daily indices when dayStep and superDatasetDayStep differ. The top N query, the labeled metrics read without an entity, the entity listing of a metric and the tag autocomplete query located their indices with superDatasetDayStep, though metrics are written to indices rolled by dayStep, and Elasticsearch ignores the missing indices, so these queries returned nothing or only part of the range. They now use dayStep.
  • Read the Kubernetes and Istio metadata used by Envoy ALS k8s-mesh, the eBPF access log, MAL retagByK8sMeta and the MAL process names from the API server in the background, one read at a time in each of two lanes, one for the reads MAL waits for and one for the rest, instead of on the thread analyzing the data. A slow or unreachable API server held one gRPC handler thread, with the batch it was analyzing, per access log stream until it answered, which could exhaust OAP’s memory. Envoy ALS and the eBPF access log no longer wait: an IP seen for the first time is analyzed as unknown while it is read, and is found by the later lookups; an eBPF connection resolves its unknown addresses again on its later messages. MAL queues the reads of a retagByK8sMeta’s pods first and waits for them up to the new core.default.malMetadataReadTimeout in all (2 seconds by default, under the OpenTelemetry Collector’s 5-second export timeout). The reads waiting at once are of distinct keys, and the new core.default.k8sMetadataReadQueueSize (als=<size>,mal=<size>, no limit by default) can limit them per lane, with the refused reads summarized at WARN. Lookup reads are sent without the Kubernetes client’s retries, so a failing read holds the reads queued behind it for one attempt instead of up to ten retries with backoff, and the HTTP and TCP access log analyzers share their lookups. The API calls are the same as before, and are made only for what the lookups use: data is read again when used core.default.k8sMetadataRefreshPeriod (3 minutes) after it was written and dropped core.default.k8sMetadataCacheTime after its last write (als=21,mal=21 minutes by default, per cache); finding nothing is read again when used core.default.k8sMetadataNotFoundRetryPeriod (1 minute) later, and the endpoints or services read are read again for a pod not in them once that old, as the pod may be new; a failed read, such as for a 403 or the API server not answering, is retried when used the refresh period later, and keeps the data read before. Failed reads are logged as one ERROR summary a minute after the first failure and then every five minutes instead of a line per failure, and a resource the API server does not serve, such as ServiceEntries without Istio, is not logged. ServiceEntry hostname resolution failures are summarized at WARN, and a failing resolution retries with a growing backoff instead of immediately once three attempts failed. ServiceEntries are read through the shared Kubernetes client instead of a new client per read.
  • Fix the Elasticsearch Zipkin trace search listing the daily indices of every day since 1970 when the query starts at or before the epoch, as a lookback equal to endTs makes it. With the default day step of 1 that is over 20,000 index names, which can exceed the request line Elasticsearch accepts. The search now uses the whole zipkin_span index family in that case, as its time filter already did. Queries with a usual lookback are not affected.
  • Fix Elasticsearch reads of a metric a peer OAP reinstalls without a schema change, such as a bundled MAL rule restored after the runtime rule overriding it is deleted. The peer kept the index it had registered for the runtime rule. This only matters with logicSharding enabled and when the two rules use different functions, since the function decides the index there; with the default merged indices every metric stays in metrics-all.
  • Build the dated Elasticsearch index names in one place, IndexNames, with the day step decided once per physical table from its models, replacing TimeSeriesUtils and TimeRangeIndexNameGenerator. Apart from the fixes above, index names are unchanged.
  • Fix a service instance dropping out of the instance list queried with a time range when a record carrying an older minute updates it, such as the segment of a request that started minutes earlier arriving after the instance’s heartbeat. InstanceTraffic took the last_ping of the last record written instead of keeping the latest one, as EndpointTraffic now does.
  • Fix eBPF access-log timestamps losing all sub-second precision. NodeInfo.parseTimestamp truncated the nanosecond-since-boot offset to whole seconds and also dropped bootTime.nanos before converting to millis, so short mesh HTTP latency (end - start) was quantized to 0 or 1000ms depending on whether the request crossed a whole-second boundary, and minute buckets could land in the wrong minute near minute boundaries. The conversion is now Instant.ofEpochSecond(bootSeconds, bootNanos).plusNanos(offset).toEpochMilli(), the same Instant-plus-nanos style Envoy ALS uses for wall time.
  • Fix the Envoy ALS sidecar internal request and response latencies dropping whole seconds: only the nanos part of each protobuf Duration was read. The internal response latency is now left out of the internal latency metrics when the access log lacks the upstream or downstream timing it is computed from, such as when Envoy answers an upstream timeout with its own 504, instead of counting the whole wait or going negative.
  • Sync the Envoy access log proto, envoy/data/accesslog/v3/accesslog.proto, with Envoy v1.40.0, adding the envoy/annotations/deprecation.proto it imports. With it, the Envoy ALS sidecar internal latencies leave out what the access log cannot measure instead of counting it: the internal request latency of an HTTP request Envoy retried or hedged, or of a TCP connection that took more than one connect attempt (upstream_request_attempt_count above 1), as Envoy measures the upstream timings from the start of the downstream request or connection, so the time to the first upstream byte can include the earlier attempts and the delays between them; and the internal response latency when the response details are set to anything but via_upstream, as for Envoy’s own 504 of a timeout during a retry backoff, which can carry an earlier attempt’s upstream timing, and, conservatively, for an upstream response that a later reset or timeout ended. The adapters report such a latency as -1, and the *_sidecar_internal_req_latency_nanos and *_sidecar_internal_resp_latency_nanos metrics in mesh.oal filter it out with filter(... >= 0). A customized mesh.oal needs the same filters, or the -1 values are averaged in.
  • Fix OAL >, >=, < and <= filters on a long or double source field, and == on a long one, failing to compile when they compare it with a number, such as filter(sideCar.internalRequestLatencyNanos >= 0). The generated code passed the number as an int, and the comparison matchers have no match(long, int) or match(double, int) overload, as Javassist does not widen an argument. The number is now written in the field’s type, 0L or 0.0; filters on int fields, which already worked, generate the same code.
  • Add ZIPKIN to the GenAI evaluation record’s traceRef.type, for spans received by receiver-zipkin and OTLP spans converted with otlpTraceStorage: zipkin, so the type names the store holding the judged span: SKYWALKING_NATIVE, OTLP for OTLP spans stored natively, or ZIPKIN. Zipkin and OTLP spans are both addressed by span ID, and one OAP can hold both, so an OTLP reference alone could not tell a client which store to read the trace from. The type is decided when the span is received, so records written under different otlpTraceStorage settings each name their own store.
  • Fix the Zabbix receiver dropping the sub-second part of agent sample times. Each agent data item carries clock (Unix seconds) and ns (nanoseconds within that second), but the sample time used only clock * 1000; it is now clock * 1000 + ns / 1_000_000. The sub-second precision can affect values produced by custom Zabbix rules using MAL rate, irate or increase; results from the bundled Zabbix rules and the assignment of samples to minute buckets are unchanged.
  • Fix expired records never being dropped on receipt. RecordStreamProcessor decoded a record’s second-precision time bucket as a minute bucket, which put the record’s time far in the future, so no record was ever treated as expired. It also compared the record’s age against recordDataTTL converted to microseconds instead of milliseconds.
  • Fix Kong and APISIX latency percentiles being about 1000 times too high. Kong’s kong_kong_latency_ms, kong_request_latency_ms and kong_upstream_latency_ms, and APISIX’s apisix_http_latency, are already millisecond histograms, but the rules called .histogram() with no unit, which treats each le bucket as seconds and multiplies it by 1000 before histogram_percentile stores that scaled bucket as the percentile. A 100ms request was shown as 100 seconds. The Kong service, instance and endpoint rules and the APISIX rules now call histogram("le", TimeUnit.MILLISECONDS).
  • Fix the service topology drawing a service as a different one when its name equals a network address alias. The topology query renamed any call’s destination whose name was an alias address, a real service included, so an OTLP service frontend was drawn as agent::frontend once an agent had registered the host frontend for agent::frontend. It now folds only a destination that is not a registered service, such as an address an agent or a TAL rule could not resolve yet, so those calls still fold onto the callee once its alias lands, and a self-call whose alias names another service, which a caller dialing its downstream by its own service name records until the downstream’s alias lands.
  • Add receiver-otel.otlpServiceGroupKeys (SW_OTEL_RECEIVER_OTLP_SERVICE_GROUP) to group OTLP services in the group::name convention: the first of the listed resource attributes a resource carries, such as service.namespace or k8s.namespace.name, groups cart as shop::cart in the trace analysis, the stored spans and the OTLP logs, while the resource attributes stay as sent. The trace analyzer’s alias is keyed by the caller, so cart:7070 dialed by shop::frontend reaches shop::cart and dialed by outlet::frontend reaches outlet::cart, even with one application deployed in both groups. TraceQL search lists the grouped service as each span’s service.name and a trace fetched by id carries it as its resource’s service.name, and the OTLP span listeners (SpanListener.onOTLPSpan / onNativeOTLPSpan, which gain a serviceName parameter) name it too. This changes the SpanListener SPI: an implementation that overrides the old four-argument onOTLPSpan / onNativeOTLPSpan must move to the new signature, or it is no longer called. Empty, the default, groups nothing.
  • Fix pprof task log records using seconds for a duration that is in minutes. PprofServiceHandler.recordPprofTaskLog used TimeUnit.SECONDS.toMillis(task.getDuration()) for the record timestamp and time bucket, while the GraphQL API, PprofTaskCommand and overlap checks treat duration as minutes. The log now uses TimeUnit.MINUTES, the same as profile task logs, so the record TTL stays aligned with the task.
  • Fix Kafka broker request_queue_time_ms, remote_time_ms, response_queue_time_ms and response_send_time_ms reporting observation rate instead of latency. Those rules used histogram *_count.rate, so under load the chart showed requests per second while the metric names and Kafka’s *TimeMs histograms are millisecond latency. The Prometheus JMX exporter kafka-2_0_0.yml does not expose *_sum for these histograms, so the rules now use the exported quantile="0.50" gauge.
  • Fix Kafka broker throughput and topic metrics. meter_kafka_broker_topic_bytesin_per_second read kafka_server_brokertopicmetrics_bytesout_total, so per-topic inbound traffic matched outbound; it now reads bytesin_total. The Prometheus JMX exporter exports Kafka’s BrokerTopicMetrics message, byte and request meters per topic and again as the all-topics total without a topic label, so meter_kafka_broker_messages_per_second, meter_kafka_broker_bytes_in_per_second and meter_kafka_broker_bytes_out_per_second summed both and reported twice the real rate, and every meter_kafka_broker_topic_* metric carried an extra topic="" series holding the broker total, which the Topic Rates chart drew as an extra line. Broker rules now read only the all-topics total and topic rules only the per-topic series. meter_kafka_broker_network_processor_avg_idle_percent applied rate to Kafka’s idle-ratio gauge, so it reported how fast the idle percentage changed; it now reports the idle percentage. Values stored before the upgrade keep the old numbers, so these charts step at the upgrade.
  • Fix RabbitMQ node singleblock allocator meters reading multiblock carriers. meter_rabbitmq_node_allocated_singleblock_used / _unused copied kind=mbcs, so the Singleblock charts matched Multiblock; they now use kind=sbcs, and _unused subtracts the used blocks size from the carriers size, as RabbitMQ’s Erlang memory allocators dashboard does, instead of reporting the whole carriers size. Values stored before the upgrade keep the old numbers, so these charts step at the upgrade.
  • Fix PostgreSQL temporary-files rate reading temp bytes. meter_pg_temporary_files_rate / meter_pg_instance_temporary_files_rate used pg_stat_database_temp_bytes, so the Temporary Files / s charts showed byte throughput; they now use pg_stat_database_temp_files, and no chart shows the temporary-file byte throughput. Values stored before the upgrade keep the old numbers, so these charts step at the upgrade.
  • Fix Zabbix agent filesystem usage percentage reading inode usage. meter_vm_filesystem_percentage used vfs.fs.inode pused, so the File System Mountpoint Usage chart showed inode occupancy; it now uses the agent’s vfs.fs.size[/,pused], the space usage df and the Telegraf VM rule report, which leaves the space reserved for root out (the OTEL VM rule counts that space as used, so it reads higher on the same disk). Add vfs.fs.size[/,pused] to the required item keys, and drop unused vfs.fs.inode[/,pused] and the duplicate vm.memory.size[pavailable]. Values stored before the upgrade keep the old numbers, so this chart steps at the upgrade.
  • AI agent conversations read a tool call as its arguments, as the AI Sessionizer does (apache/skywalking-ai-sessionizer#94). A step on one part of a record reads that part alone. A tool call that shares a record with the agent’s text read as that text, and now reads as its arguments. A queued command reads as the prompt inside its own part, never as text another part of the record holds.
  • Add the OTLP_MESH layer for the spans of service mesh sidecars that report over OTLP, such as Istio’s Envoy. No shipped TAL rule analyzes it, so the sidecars’ spans are stored and searchable ({resource.service.layer="OTLP_MESH"} in TraceQL) but not analyzed.
  • The OTLP service catalog keeps each service’s service.layer, UNDEFINED when absent or unregistered, in the otlp_service_layer column, and the stored span carries the same value. A TraceQL q on resource.service.layer alone lists the services from the catalog.
  • Write a network address alias only when its mapping changes, or once per five minutes to keep it in the alias cache’s ten-minute reload window; a changed mapping is written at the next flush. The alias’s last_update_time_bucket is the minute OAP received it rather than the span’s time, so an agent’s clock skew or a late segment cannot leave a changed alias outside the cache reload. NetworkAddressAlias#combine used to rewrite every alias at every persistence period, about 140 writes an hour for a stable alias, now about twelve.
  • Fix RabbitMQ node multiblock and multiblock-pool unused meters reporting whole carriers size. meter_rabbitmq_node_allocated_multiblock_unused / _multiblock_pool_unused now subtract the used blocks size from the carriers size, matching the singleblock unused formula and RabbitMQ’s Erlang memory allocators dashboard. Values stored before the upgrade keep the old numbers, so these charts step at the upgrade.
  • Fix RabbitMQ node publisher count using channels minus channel consumers. meter_rabbitmq_node_publisher_total computed rabbitmq_channels - rabbitmq_channel_consumers, which counts channels without a consumer rather than publishers: a basic.get polling channel counted as a publisher, a channel that both publishes and consumes did not, and a channel with several consumers drove the value below zero, stored as 0. It now reads rabbitmq_global_publishers, which RabbitMQ counts correctly from 3.10.11 and 3.11.3. Values stored before the upgrade keep the old numbers, so this chart steps at the upgrade.
  • Fix MySQL connection-errors-for-max-connections reading a mistyped exporter label. meter_mysql_connection_errors_max_connections / meter_mysql_instance_connection_errors_max_connections filtered error=max_connection, while mysqld_exporter exposes Connection_errors_max_connections as error=max_connections, so live scrapes never matched and the chart stayed empty; the filter now uses max_connections.
  • Add server clusters, server-clusters.yml and its dynamic configuration item core.default.serverClusters, to show a message broker, a database or a cache that clients reach at more than one address as one service. An address listed under a cluster’s members or aliases, or a list of addresses, such as a Kafka client’s bootstrap servers, whose every element is listed under one cluster in any order, is renamed to the cluster by the native trace analysis where no network alias maps the address, and by the OpenTelemetry trace analysis through the new TAL function findServerClusterName, which the shipped tal/otlp-general.yaml calls for brokers, databases and caches. A cluster may declare the scheme its clients write before an address, as written, such as pulsar://, and kinds, names of the user’s choosing that a custom TAL rule may pass as findServerClusterName(address, kind) to match only the clusters of that kind; the native trace analysis and the shipped TAL rules do not use kinds. The file is empty by default, so nothing is renamed until a cluster is listed.
  • Bump dependencies to clear CVE alerts on shipped OAP jars: jackson 2.18.8 → 2.18.12, guava 32.0.1-jre → 33.7.2-jre, micrometer 1.14.4 → 1.16.8, log4j 2.25.4 → 2.25.5, freemarker 2.3.31 → 2.3.35 and postgresql 42.7.11 → 42.7.14. Micrometer 1.14.x has no open-source fix, and Armeria 1.34.2 is built against micrometer 1.16.0, which the old pin downgraded.
  • Bump dependencies to clear CVEs on shipped OAP jars that Dependabot does not see, because their versions come from a BOM or a transitive dependency: netty 4.2.15.Final → 4.2.19.Final (with netty-tcnative-boringssl-static 2.0.77.Final → 2.0.84.Final, the version netty 4.2.19.Final is built with), and, pulled in by kafka-clients, lz4-java 1.10.1 → 1.11.4 and snappy-java 1.1.10.5 → 1.1.10.12.
  • Add HTTP alarm hook delivery self-observability metrics, alarm_http_delivery_total, alarm_http_delivery_status_total and alarm_http_delivery_latency_seconds, on telemetry.prometheus, labeled by hook_type and by result or status_class. result names why a delivery failed (auth_failed, rate_limited, rejected, server_error, token_error, timeout, io_error, interrupted) and reads the error code DingTalk, WeCom, Feishu and WeLink return in a 2xx body, so such a refused message is now logged as a failure instead of passing as delivered. A WeLink access-token response without a token now skips the messages of that webhook URL instead of sending them with an empty token. The otel-rules/oap.yaml rules add meter_oap_instance_alarm_http_delivery_count, meter_oap_instance_alarm_http_delivery_status_count and meter_oap_instance_alarm_http_delivery_latency_percentile to the OAP self-observability layer, and the OAP Grafana dashboards chart the raw series.

UI

  • Add a Virtual GenAI evaluation-record page and evaluation-score chart in Horizon UI, so operators can inspect evaluation result, level, reason, judge model, timestamp, trace linkage, and the gen_ai_model_evaluation_score_ppm trend for evaluated records.

Documentation

  • The Windows monitoring doc’s example OpenTelemetry Collector configuration uses the debug exporter. The logging exporter it used was removed from the Collector, so the example no longer loaded.
  • Document the BanyanDB trace tail sampling metrics in the BanyanDB self-observability dashboard catalog, and point the “Operating it” section of the trace tail sampling guide at them — the OAP-collected metrics show what the sampler plugins proposed next to what storage committed, which the data node’s raw metrics endpoint alone does not.
  • Rename Envoy AI Gateway to Agent Router in the monitoring docs, following the project’s move to the Agentic AI Foundation. Only the prose changes: the ENVOY_AI_GATEWAY layer, the job_name=envoy-ai-gateway routing tag, the envoy-ai-gateway rule files and the meter_envoy_ai_gw_ metric names are kept, as Agent Router itself kept its deployed names and the telemetry it emits.
  • Give the AI agent metrics a catalog that names the source of each: the AI Sessionizer sends the token and MCP metrics, and Claude Code’s own exporter, pointed at the OAP directly and not at the Sessionizer, sends cost, active time, sessions, lines of code, commits, pull requests and edit decisions. Neither source supplies the other’s, so a metric whose source does not send stays empty. The exporter’s three resource attributes are named, with what goes wrong without each, and the OpenTelemetry receiver page lists mcp_endpoint.yaml.
  • The OpenTelemetry receiver page names the metrics handler otlp-metrics, the name enabledHandlers takes. It named it otlp, which matches no handler. The page also no longer says the receiver reads only the group, defaultMetricLevel and metricsRules nodes of a rule file: it reads the whole scheme the MAL page describes.

All issues and pull requests are here