11.1.0

Project

OAP Server

• Add customizable LLM-as-judge support for AI evaluation, with OpenAI-compatible endpoint / model / API key configuration, and persist the evaluation result as queryable GenAIEvaluationRecord rows for later inspection.
• Support hot-reloading the BanyanDB credentials from a new secretsManagementFile setting in bydb.yml (SW_STORAGE_BANYANDB_SECRETS_MANAGEMENT_FILE), matching the ElasticSearch storage plugin. The properties file carrying user/password is watched, so a rotation performed by a 3rd party tool such as Vault is applied without restarting the OAP. The credentials are read per RPC by the gRPC auth interceptor, so they are swapped without re-establishing the channel and without interrupting in-flight queries or writes. Whatever the file contains is applied, including an incomplete pair, so that a mistake in it fails visibly rather than being masked by credentials that silently keep working; a username and a password are only ever sent together, so an incomplete file means requests carry no credentials and are answered with UNAUTHENTICATED rather than authenticating as the named user.
• Support hot-reloading the BanyanDB TLS trust CA. The file at sslTrustCAPath is now watched, and a change rebuilds the gRPC channel so the new CA takes effect without restarting the OAP. Previously the CA was only re-read after the certificate in use had already caused requests to fail, which meant a rotation was paid for with an outage. The replacement channel is created before the old one is released, so a failed rebuild leaves the current channel serving; requests already in flight finish on the old channel. The replacement is not health-checked before the swap — a gRPC channel connects lazily — so rotating to a CA that does not validate the server interrupts traffic until valid material is written back; the storage doc describes the old+new overlap procedure that avoids this. Note the replacement re-picks an address from targets, so the OAP may connect to a different node after a rotation.
• Fix MultipleFilesChangeMonitor being able to silently disable every file watch in the OAP. Its registry of monitors was a plain ArrayList that scanChanges() iterated from the scheduler thread without holding the lock that start() / stop() take, so starting a monitor while a scan was in flight could raise a ConcurrentModificationException from the iterator. That exception escapes past the per-monitor catch, and an uncaught exception cancels a scheduleAtFixedRate task permanently — after which no secrets file, keystore, or TLS certificate is ever reloaded again, with nothing in the log to say so. The registry is now copy-on-write. The failure log in the same scan loop also now names the monitor that failed instead of printing an empty gourp = .
• Fix MultipleFilesChangeMonitor never honouring its watching period. lastCheckTimestamp was declared and compared against, but never assigned, so the guard always measured against 0 and passed — every registered monitor re-stat’d its watched files on each 200ms tick of the shared scheduler thread, and the watchingPeriodInSec constructor argument had no effect at all. This affects every file watch in the OAP: the ElasticSearch storage secrets / truststore / keystore watch, the BanyanDB credentials and trust CA watches, and the TLS certificate watches behind each OAP HTTP and gRPC server, all of which ask for 10 seconds. Change detection is now paced as configured, which also means it is no longer near-instant: a rotated file is picked up within the requested period rather than within ~200ms.
• Add BanyanDB trace tail sampling metrics to the BanyanDB self-observability layer, in a new otel-rules/banyandb/banyandb-trace-sampling.yaml rule file. It covers the whole banyandb_trace_pipeline_* / banyandb_trace_tst_pipeline_* catalog a sampler plugin chain emits — pipeline reconciliation, per-plugin Decide execution rate and latency, chain batching, the trace-level evaluated / retained / dropped / immature outcomes, every fail-open guard and bounded-retention counter, drop-set capacity and finalization state, the plugin telemetry-host safety bounds, and the first-party sw-trace-sampler / zipkin-trace-sampler decision and row metrics. The plugin chain is optional, and the metrics follow it: on a cluster with no sampler configured the wire families are never registered, so every metric here stays absent rather than reading zero. Modeled at Service scope with group kept as a metric label rather than at Endpoint scope, so one cluster-wide page can render per-group series and cluster totals alike — OAP does no cross-scope rollup, so an Endpoint-scope metric could not have been aggregated back up to the cluster.
• Fix a second CounterWindow key collision in the v2 MAL engine, this time ACROSS rules. rate() / increase() / irate() resolve their lower bound from a process-wide window keyed on the counter’s own name plus its post-.sum(...) label set, with nothing identifying the rule doing the evaluation. Two rules that read one wire family, tell their streams apart with tagEqual(...), and then .sum(...) away the label they filtered on therefore collapse onto one window slot and difference against each other’s values. The queue is ordered by (timestamp, value), so the smaller counter wins the lower-bound lookup and still reads correctly while its partner is inflated by the gap between them — which is why this went unnoticed. A collision needs the discriminating label to be DROPPED by the .sum(...): where it survives, the rules’ label values differ and the window keeps them apart. Auditing the shipped rules on that basis gives 10 colliding keys over ~25 rules — meter_activemq_cluster_gc_parallel_young_collection_count reported ~9000/min of young-gen collections from a completely idle broker (differencing against the old-gen counter); MySQL commands_* / tps rate against each other; so do the GenAI gateway input/output token rates, four Envoy cluster_* counters, APISIX matched/unmatched instance bandwidth, and BanyanDB’s own network_recv / network_sent, which drop the kind label that separates bytes-received from bytes-sent on one interface. Measured against two live scrapes of the demo cluster’s FODC proxy, that last pair was wrong on every interface: network_sent read a flat 0 B/s and network_recv read large negative values (down to -778 MB/s) from differencing against the sent counter, where both now match the byte delta exactly. No rule changes were needed for any of these – each rule already reduces to the labels it should; only the window key was wrong. The window is now keyed by (owning rule, counter name, labels). This is the complement of the within-rule collision fixed earlier by keying on the counter’s own name: neither name alone is sufficient, because the two collisions are independent. RunningContext.metricName — written on every rule evaluation and read by nobody since that earlier fix — is what supplies the rule identity, so no code generation or MAL syntax changes. Note the whole-rule-set comparison suite could not have caught this: it resets the shared window before every rule, the one condition under which the collision cannot appear.
• Fix meter_rabbitmq_node_outgoing_messages_total double-counting one of its terms. The rule summed six delivery-rate terms but rabbitmq_global_messages_delivered_get_auto_ack_total appeared twice, so auto-ack basic.get deliveries were counted once more than the other four delivery paths and the reported outgoing rate ran high whenever polling consumers were in use. The duplicate term is removed, leaving the five distinct families (redelivered, consume auto/manual ack, get auto/manual ack).
• Add AI agent conversations landed by the AI Sessionizer: the AI_AGENT layer, the bundled lal/ai-agent.yaml rule with the ConversationFile output builder that verifies and stores Session Data and Session Flow files, the ai_agent_session_data and ai_agent_session_flow models in a new BanyanDB group recordsAIAgent, the ai-agent-conversation module that folds a conversation into one asz.view document, and the listConversations (with optional conversation and title conditions) / getConversationRawFiles GraphQL queries and the streamed GET /ai-agent/conversations/{conversation}/v1/view route that serves the document. A round from before the list attributes existed lands and lists with zero counts, and the view shows as much as landed: the chain resumes after a missing, unreadable or refused round, and the absent rounds and files are named once as ranges. A file over maxFileBytes, 15 MiB by default, is rejected at ingest and counted under the reason size, because one file over BanyanDB’s 16 MiB gRPC message limit fails the bulk write it travels in and every record behind it; on MySQL the body column is LONGTEXT, since a body that size outgrows MEDIUMTEXT as Base64. Each window read is capped at maxResponseBytes, 100 MiB by default, as a per-call option on the BanyanDB client in place of its 50 MB default, so the module’s reads are bounded by its own settings and nothing else’s read changes.

UI

• Add a Virtual GenAI evaluation-record page and evaluation-score chart in Horizon UI, so operators can inspect evaluation result, level, reason, judge model, timestamp, trace linkage, and the gen_ai_model_evaluation_score_ppm trend for evaluated records.

Documentation

• Document the BanyanDB trace tail sampling metrics in the BanyanDB self-observability dashboard catalog, and point the “Operating it” section of the trace tail sampling guide at them — the OAP-collected metrics show what the sampler plugins proposed next to what storage committed, which the data node’s raw metrics endpoint alone does not.

All issues and pull requests are here