Release Apache SkyWalking BanyanDB 0.11.0

SkyWalking BanyanDB 0.11.0 is released. Go to downloads page to find release tars.

Features

• Breaking Change: Upgrade the API version to 0.11 for the schema revision fields and SchemaBarrierService operations added in this release. External clients and internal node communication must use API 0.11; mixed BanyanDB 0.10 and 0.11 clusters are not supported.
• Add batch-write phase metrics for client-held-open time, post-seal admission time, and admission deadlines, plus per-part liaison-wqueue-to-data-node file-sync latency for measure, stream, and trace.
• Add bounded trace plugin overhead metrics: chain decision batches by result, per-plugin execution count and duration, batch trace-count distribution, and per-plugin fail-open bypasses by reason. The configured plugin_name distinguishes links in a multi-plugin chain; segment, shard, and schema labels remain absent. The catalog works with pipeline_traces_evaluated for per-plugin time/calls per trace, latency percentiles, timeout/circuit-open rates, and link-health monitoring. The Phase 10 merge benchmark mirrors chain batches and named-link timing into JSON and HTML and rejects missing, failed, bypassed, timed-out, or circuit-open execution records.
• Add trace finalization sampling (PIPELINE_EVENT_FINALIZE), a best-effort background backstop for the in-merge trace retention filter. A single node-wide, concurrency-1 scanner periodically sweeps cooled segments (segEnd < now − finalize_grace, reopening idle-closed segments via SelectSegments(reopenClosed=true)) and force-merges each shard’s un-finalized parts through the group’s registered sampler chain, reusing the hot merge path (mergeParts + per-sidx Merge + the shard’s introducer loop) so it never touches the hot merge semaphore. Selection uses a per-part finalizeGen stamp (min-propagated on ordinary merges, written to disk before the part metadata so a crash cannot double-sample on replay); per-shard state (finalize.json) holds the generation, cooldown clock, round count, terminal flag, and an arrival-based unsampled-bytes counter incremented O(1) on the flush path. Re-rounds are bounded by an absolute FLOOR (8 MiB), a RATIO (0.10 of segment size), a cooldown (= finalize_grace), and a hard max_finalize_rounds (8). Coverage is best-effort — misses (e.g. a segment TTL-deleted before finalization) are accepted, not guaranteed. Enabled per group via enabled_events containing PIPELINE_EVENT_FINALIZE; the settling window is finalize_grace with a fixed 5m engine fallback. Finalize rounds emit the merge metrics under type="finalize"/lane="finalize".
• Bound the trace merge drop-set so a large sampled merge cannot exhaust the heap. The merge-wide dropped-trace-ID collector was unbounded: because a shard’s first finalize round selects every cooled part into one merge, an 18M-drop set reaches ~1.3 GiB live and ~2.6 GiB reserved heap (append doubling plus a single contiguous lookup index), inside a process also serving queries. The bound is applied to the sampling decision, not to the pruning predicate: once a merge’s drop set is full every further proposed drop is retained instead of recorded, so the set stays complete with respect to the drops actually performed and each sidx still contains exactly its part’s retained traces — no orphan entries, none missing. The fragment guard is skipped for a trace that will be retained, since it exists only to confirm drops. Every merge is charged the same ceiling, finalize included, resolved from the memory protector as limit/(16*CPUs) so the aggregate across concurrent merges stays near limit/16; bytes-per-entry is derived from the observed trace-ID length rather than a constant, so a deployment with long service-prefixed IDs cannot overshoot the budget. A node whose protector is disabled (no --allowed-bytes and no valid cgroup limit) gets a flat fallback ceiling regardless of RAM — there the lever is the protector limit, not the machine’s memory. The accepted cost is bounded deletion: a capped finalize round leaves traces undeleted and they are only revisited by the existing re-round path, which is capped at max_finalize_rounds (8) per shard for its lifetime, so a very large shard can retain traces the sampler wanted dropped. Finalize I/O is unchanged, because the round count is unchanged. New group-level metrics make the loss visible without segment or shard cardinality: pipeline_drop_set_budget_bytes, a lane-split pipeline_drop_set_entries histogram emitted whether or not a merge caps, pipeline_traces_retained_by_ceiling, pipeline_merges_ceiling_reached{lane}, and scan-level finalize_rounds/finalize_terminal worst-case gauges. Design: docs/design/trace-drop-set-bounding.md.
• Redesign queue (queue_pub/queue_sub) metrics around a uniform model with per-message and per-batch-stream catalogs. Use operation and group labels, add error_type and remote-endpoint labels, and propagate sender identity and group on the wire. [Breaking Change] Previous queue metric families, chunk-ordering metrics, and the topic label are removed; update dashboards and alerts.
• Stamp the lifecycle’s tier-migration publisher’s identity onto the wire so the receiving data node records a non-empty remote_node/remote_role/remote_tier on its banyandb_queue_sub_total_finished series. The lifecycle’s parseGroup resolves the lifecycle’s self identity by matching the lifecycle pod’s hostname (POD_NAME via the K8s downward API, falling back to os.Hostname() — same precedence as nativeNodeContext at banyand/backup/lifecycle/service.go) against the data-node registry’s GrpcAddress (host-portion match with loopback-alias and IP-literal normalization, via hostMatches at banyand/backup/lifecycle/steps.go) — Metadata.Name becomes remote_node, Labels["type"] becomes remote_tier — and calls SetSelfNode(senderNode, "lifecycle", senderTier) on the migration publisher. The previous --grpc-addr address-match (Pass 1) and --node-labels superset-match (Pass 2-3) fallbacks are replaced by this single host-based match because they failed on the production cluster where the data pod’s GrpcAddress is a headless-service FQDN but the lifecycle’s --grpc-addr is the loopback. Mirrors the liaison’s existing SetSelfNode(node.NodeID, "liaison", liaisonTier) call in pkg/cmdsetup/liaison.go:170-171; no new CLI flags are introduced.
• Add labeled lifecycle health metrics (banyandb_lifecycle_cycles_total, banyandb_lifecycle_last_run_timestamp_seconds, banyandb_lifecycle_last_run_success) keyed by remote_node, remote_role, remote_tier, and group; stamp the last-run gauges atomically at cycle end and remove their previous label tuple. [Breaking Change] Update dashboards and alerts for the labeled series.
• Remove banyandb_lifecycle_self_identity_resolution_total. The regression-detection role moves to the now-labeled banyandb_lifecycle_cycles_total{remote_node!=""} (an empty remote_node series means the registry match failed for every group, the bug the old counter caught), plus the existing receiver-side count of empty remote_node on lifecycle banyandb_queue_sub_total_finished series. The wire-level cluster.v1.SendRequest sender-identity fields are unchanged.
• Vectorized measure query path is now enabled by default. The columnar pipeline replaces per-row protobuf serialization in NewMIterator, cutting allocations and ns/op for scan-heavy measure queries; gRPC wire format (*measurev1.InternalDataPoint) is byte-identical. Single-node coverage is complete: scan, GroupBy+Agg via BatchAggregation, scalar reduce (Agg without GroupBy), raw GroupBy (without Agg), implicit projection coverage for GroupBy/Agg fields, TopN/BottomN, order_by (via logical.ParseOrderBy, mirroring the row path’s PushDownOrder rule), queries with hidden criteria tags, and boundary-error parity (nil time range, unknown projection, empty result) all resolve through the vec dispatch with row-path-equivalent semantics (SUM/COUNT/MIN/MAX/MEAN type semantics, first-seen carry-forward of non-key projected tags, canonical validation errors). Validated by a 6h production soak (byte-identical parity, zero divergences) and the per-workload bench gates. Distributed Map-mode partial aggregation (multi-node GroupBy+Agg / TopN), multi-measure (multi-group) requests, and non-vectorized backends continue to flow through the row path pending the distributed vectorized query work. Rollback: pass --measure-vectorized-enabled=false on the standalone or data-node command line and restart; the row path resumes immediately.
• Enable vectorized stream and trace queries by default. Stream uses the columnar pipeline and native liaison-to-data frame for supported timestamp-, index-, criteria-, and multi-group queries; unsupported shapes and traced queries fall back to row or protobuf execution. Roll back with --stream-vectorized-enabled=false or --trace-vectorized-enabled=false.
• Add banyandb_vec_frame_encoded{engine} and banyandb_vec_frame_decoded{engine} gauges (engine: measure/stream/trace), reporting how many query responses this process has put on the wire as a native columnar frame and how many it has decoded from one. Which wire format a node is actually speaking was otherwise invisible from outside the process, and the 0.11 upgrade-ordering rule depends on exactly that: a data node emits frames only when it is distributed AND its engine flag is on, so a standalone server reports zero encodes however the flag is set. Both series are cumulative and, being labelled, appear on /metrics only after the first collection tick. A run whose encode count stays flat on the data nodes is falling back to protobuf.
• [Breaking Change] Upgrade liaison nodes before data nodes when rolling out vectorized query paths. A 0.11 liaison accepts both columnar frames and protobuf, but an older liaison cannot decode frames emitted by an upgraded data node. Standalone deployments are unaffected.
• Add validation to ensure Measure’s ShardingKey contains all Entity tags to guarantee entity locality.
• Organize access logs under a dedicated “accesslog” subdirectory to improve log organization and separation from other application data.
• Collect BanyanDB data on e2e test failure for CI debugging.
• Add log query e2e test.
• Sync lifecycle e2e test from SkyWalking stages test.
• Add noDuplicates verification to all e2e expected files to detect duplicate data in query results.
• Add a program-generated trace query integration-test framework under test/cases/trace/cmd/{generate,capture}: layered case generation (criteria leaves, AND/OR trees, and feature-pairwise across the traceID-lookup and order-based query modes), env-gated golden capture, and a shared SeedAll seeder — mirroring the measure test-case framework.
• Persist segment end time in per-segment metadata so boundaries don’t shift across restarts or config changes.
• Introduce fair fast/slow lane scheduling for trace part merges to prevent short merges from being blocked by long-running merges; expose queue wait time as total_merge_queue_latency.
• [Breaking Change] Remove etcd components. The property-based schema registry is now the only supported mode.
  • All --etcd-* CLI flags have been removed.
  • The --namespace CLI flag has been removed (it previously configured the etcd key prefix).
  • The --node-discovery-mode flag no longer accepts etcd (supported values: none, dns, file).
  • The --schema-registry-mode flag only accepts property.
• Implement panic diagnostics and the FODC crash reporting pipeline.
• Schema consistency (Phase 1): introduce client-observable revision and propagation primitives. All gates are opt-in and zero-valued requests preserve prior behavior.
  • Add mod_revision to Group / IndexRule / IndexRuleBinding / TopNAggregation Create and Update responses.
  • Add delete_time to all *ServiceDeleteResponse messages so clients can observe tombstones.
  • Add created_at to Stream / Measure / Trace / Property / IndexRule / IndexRuleBinding / TopNAggregation / Group; preserved across updates.
  • Add STATUS_SCHEMA_NOT_APPLIED (10) for writes/queries whose mod_revision is ahead of the server cache.
  • Add three-way write-path ModRevision gate on Stream and Measure write RPCs (< expired, == succeed, > not-applied, 0 skipped).
  • Add per-group query-path gate via QueryRequest.group_mod_revisions and QueryResponse.group_statuses for Stream / Measure / Trace queries.
  • Add automatic time-range clamp max(time_range.start, schema.created_at); multi-group uses the maximum across queried groups; nil for pre-upgrade schemas is a no-op.
  • Add SchemaBarrierService with AwaitRevisionApplied, AwaitSchemaApplied, AwaitSchemaDeleted (10 000-key cap, timeout-bounded, returns laggards on expiry).
  • Add tombstone retention/GC (default 7 days, configurable via --schema-server-tombstone-retention) with a per-cache count cap to bound memory under bulk deletes.
  • Reject Create with updated_at <= tombstone.delete_time to prevent replayed creates from overwriting newer deletes.
  • Guard pkg/schema/cache against out-of-order EventDelete events; expose monotonic LatestModRevision watermark.
• Schema consistency (Phase 2): add cluster-wide barriers that fan out AwaitRevisionApplied, AwaitSchemaApplied, and AwaitSchemaDeleted across liaison and data nodes through NodeSchemaStatusService, while safely handling mixed-version, membership, and timeout cases.
• Align schema barriers and write/query revision gates with each node’s executor cache, including correct watermark advancement and no-op update revisions.
• Add schema-barrier metrics, structured access logs, integration coverage, and a load-test harness.
• Introduce a migration tool with copy, verify, and analyze subcommands for measure and stream data.
• Support displaying a measure’s indexed tags in the dump tool, resolved per part so peak memory is bounded by the part rather than a segment-wide series map.
• Make segment lifecycle safe for snapshots, backups, and inspection: segments retain a dormant state while idle, and these operations no longer reopen closed segments or race idle cleanup.
• Add opt-in vectorized measure query tracing over raw-frame distributed queries, including a trace envelope and fixed trace-label vocabulary.
• Speed up GCS backup uploads: write each object and its checksum metadata in one request, dropping the per-object Update round-trip.
• Lifecycle migration now archives rows whose measure/stream schema was deleted from the registry, instead of aborting the group.
• Canopy M3 metadata CRUD: add IndexRule + IndexRuleBinding create/edit/delete forms with admin-only edit/delete, pure validate*() module covering 8 input groups (group, stream, measure, trace, index rule, index rule binding, interval, lifecycle stages), useFocusTrap + useDirtyGuard modal hooks, live m3-indexrule e2e (CRUD round-trip + MF2 server-authority negative), and opt-in m3-handoff e2e + zero-dep static server for the design-bundle comparison.
• Prevent lifecycle migration receiver OOMs with memory-efficient external series-index deduplication, receiver load shedding and bounded memory waits, and sender retries for transient failures; incomplete parts resume in the next cycle.
• Add a storage-node in-merge trace-retention filter (PIPELINE_EVENT_MERGE) that evaluates per-group sampler chains and safely drops non-retained traces from core and secondary-index parts. Configure samplers dynamically through each group’s pipeline, with bounded staging, fail-open error handling, and runtime register, update, and removal support.
• Add FODC memory-pressure pprof capture: agent grabs heap/goroutine profiles when RSS nears the cgroup limit, served via the proxy.
• Add fuzzy resource search to the From row: type a name and jump to any measure / stream / trace / Top-N across every group, with contiguous-run + word-boundary scoring, highlighted matches, arrow/Enter/Escape navigation, and click-outside dismiss.
• Add bounded plugin telemetry (meter + logger) to the trace-pipeline sampler SDK. A sampler plugin opts in by implementing sdk.HostAware; the engine calls UseHost(h) once per group before the first Decide, delivering a scoped sdk.Host the plugin may cache and use in Decide. The host enforces all resource limits: metric names are forced under the banyandb_trace_pipeline_plugin_ prefix; {group, plugin_name} labels are always present; per-plugin cardinality is capped at 100 series (overflow counted under _overflow); log throughput is capped at 50 lines/s burst 100, messages at 1 KiB, and 16 structured fields per call. Per-group delivery means a HostAware plugin receives a distinct Host per group so attribution is correct by construction. Series are reclaimed on pipeline teardown via Delete. No ABI bump — delivery is via type assertion (sdk.ABIVersion remains 1). Reference plugins _telemetrysampler (well-behaved) and _faultysampler (adversarial bounds verification) are in test/plugins/; build with make build-trace-pipeline-telemetry-plugins.
• Add the first-party trace-retention sampler plugins sw-trace-sampler.so and zipkin-trace-sampler.so (plugins/skywalking/), implementing the Scenario 6.1/6.2 keep logic on SkyWalking’s two trace schemas. Both compile the shared plugins/skywalking/internal/tracesampler engine and differ only in a Schema value naming where each schema keeps the rule inputs: the flattened searchable-tag array (tags for segments, query for Zipkin), the error signal (a real is_error column vs. Zipkin’s conventional error key inside query), and the duration columns (start_time/latency in ms vs. timestamp_millis/duration in µs, normalized via Schema.DurationTagNanosPerUnit). Config keys are durationThresholdMs, keepErrors (with an errorTag override), keepTagRules, and healthySampleRate: a trace is kept when any sure-keep rule matches, otherwise a deterministic FNV-1a hash of the trace ID admits healthySampleRate of the remainder — stable across the merge and finalize passes, so a trace is never half-kept. durationThresholdMs tests the end-to-end envelope max(start + duration) − min(start) over the trace’s rows, catching traces slow only through sequential spans, rather than any single span’s duration or the intrinsic MaxTS − MinTS (which is the spread of per-row start timestamps, and 0 for a single-row trace). Because OAP flattens every searchable tag into one string-array column, every keepTagRules entry resolves there; a rule naming a first-class column could never match and is rejected at admission, with the array-column workaround named in the error. keepTagRules also accepts a compact key=value,key=~regex,key string so a rule set fits in an environment variable. Unknown and empty configs are rejected rather than ignored — every option is a keep rule, so a key that silently missed would drop the whole group — and any predicate that cannot be evaluated keeps the trace. Built by make build-plugins and shipped in the -plugins-carrier image; see plugins/README.md for the config reference.
• Remove the latencystatussampler first-party plugin. It projected duration and status: status exists in neither shipped trace schema and duration only in Zipkin (in µs, so a thresholdMs there was off by 1000x), so it could never match live data and — failing open — kept every trace it was given. It shipped in the -plugins-carrier image beside the two real samplers, leaving three first-party plugins of which one silently retained everything. The near-identical copy under test/plugins/_latencystatussampler is unaffected and still backs the trace-pipeline integration suites, the plugin-sidecar test and sdktest’s LoadSO coverage; the worked-example references in docs/operation/plugins*.md now point at sw-trace-sampler.
• Add a Claude/Codex plugin packaging the BanyanDB MCP server with a parse-only validate_bydbql tool (backed by the mcp/tools/bydbql-parse Go validator) and a BydbQL skill for read-only natural-language-to-BydbQL generation over STREAM/MEASURE/TRACE/PROPERTY resources.
• Introduce positional parameter binding (? placeholders) into BydbQL to eliminate QL injection.
• Add reusable BydbQL binding and prepared-statement caching on the gRPC query path, with bounded cache and top-K retention, cache and slow-query observability, and redacted bound parameters in the latest slow-query log.
• Add an FODC schema consistency check comparing registry, cache, and runtime fingerprints across all nodes via lifecycle inspection.

Bug Fixes

• Prevent long-lived client write streams from failing whole liaison batches when the write timeout expires; start the liaison admission timeout only after the batch is sealed and raise its default to two minutes. (apache/skywalking#13982)
• Stabilize measure, stream, and trace snapshot tests by waiting for persisted manifests and all flushed memory parts rather than directory creation or the latest snapshot creator, eliminating asynchronous-flush and fallback races.
• Prevent merge-time trace sampling from dropping fragments when the same trace ID may remain in unselected parts. Provisional drops are now checked against time bounds and trace-ID Bloom filters in candidate outside parts whose time bounds intersect the trace’s max-fragment-gap-expanded range, revalidated before publication, and retained on uncertainty; the default merge grace is two hours.
• FODC: namespace dynamic node labels as node_*, continuously resolve node roles and labels, and prevent oversized metric uploads and ghost ROLE_UNSPECIFIED series.
• Close BanyanDB merge write-path durability gap that allowed torn parts to be created by a crash between data write and metadata commit. Metadata files (metadata.json for trace/measure/stream, manifest.json for sidx, plus traceID.filter and tag.type) now go through a new WriteAtomic (write-tmp + fsync + rename + fsync-dir) sequence; data writers (seqWriter.Close, localFileSystem.Write) now propagate fdatasync errors instead of silently dropping them. mustOpenFilePart / mustOpenPart in each engine cleans up safe post-rename .tmp leftovers on open. (#13862, root cause for #13861)
• Fix bydbctl command tests using global stdout capture, which caused race-enabled runs to corrupt captured command output.
• Fix flaky trace query filtering caused by non-deterministic sidx tag ordering and add consistency checks for integration query cases.
• Fix index-mode measure queries returning documents outside the requested time range when a widened segment overlaps the query window.
• MCP: Add validation for properties and harden the mcp server.
• Make property-schema client connections recover after data-node restarts and add periodic health checks.
• Fix flaky on-disk integration tests caused by Ginkgo v2 random container shuffling closing gRPC connections prematurely.
• Fix snapshot error when there is no data in a segment.
• ui: fix query editor refresh/reset behavior and BydbQL keyword highlighting.
• Disable the rotation task on warm and cold nodes to prevent incorrect segment boundaries during lifecycle migration.
• Prevent epoch-dated segment directories (seg-19700101) from being created by zero timestamps in distributed sync paths.
• Fix SIDX streaming sync sending SegmentID as MinTimestamp instead of the actual timestamp, causing sync failures on the receiving node.
• Fix handoff controller TOCTOU race allowing disk size limit bypass, and populate sidx MinTimestamp/MaxTimestamp during replay to prevent corrupt segment creation on recovered nodes.
• Delete orphaned parts when no snapshot references them during tsTable initialization.
• Extract shared LocateAll on NodeRegistry to ensure resolveAssignments and syncer GetNodes always produce identical node lists, preventing liaison from enqueuing parts to online/healthy data nodes.
• Add validation for MATCH and IN conditions in inverted index query builder, and handle nil OR branch when all entities are specific.
• Fix wrong backup path of schema property.
• Fix lifecycle migration failure when the target stage has close: true.
• Fix stale sync request blocking watch session channel, causing repeated “channel full, skipping session” errors when a watch stream is in backoff.
• Fix nil pointer panic in disk monitor when group schema is not yet initialized during early startup, and ensure monitor loop survives recovered panics.
• Fix FileSystemError not satisfying errors.Is(err, io/fs.ErrNotExist), which prevented the segment controller from cleaning up half-born segment directories and left groups in a permanent zombie state after a crash or partial sync.
• Fix lifecycle migration panic when a stream shard’s snapshot has no element index (idx/) directory.
• Avoid FODC lifecycle inspection failing on busy data nodes by raising the per-broadcast CollectDataInfo / CollectLiaisonInfo deadline from 5s to 30s and parallelizing per-group inspection in the cluster-internal InspectAll.
• Fix deadlock when fodc-agent reconnects to fodc-proxy after a pod rotation.
• Fix flaky TestCollectWithPartialClosedSegments by raising SegmentIdleTimeout so wall-clock variance on slow CI does not mark still-open segments as idle.
• Fix FODC lifecycle cache poisoning where transient InspectAll failures were cached for 10 minutes and masked liaison recovery; raise FODC agent and proxy timeouts from 10s to 40s.
• Fix FODC /cluster/lifecycle dropping zero-valued group fields (e.g. replicas=0, close=false) under encoding/json + omitempty; switch to protojson so all fields are emitted (nil nested messages serialize as null).
• Fix trace block_writer panic on out-of-order timestamps within the same traceID, which dropped one trace-write batch per panic in multi-agent SkyWalking deployments. Spans of a single trace originate from independently-clocked services, and trace storage is organized by traceID rather than timestamp, so per-traceID timestamp monotonicity is not a writer invariant.
• Add GroupLifecycleInfo.errors to surface per-group collection failures from FODC InspectAll instead of silently dropping the affected node entry.
• Fix CollectDataInfo and CollectLiaisonInfo not handling CATALOG_PROPERTY groups.
• Fix lifecycle migration where the receiving node could create segments shorter than the configured SegmentInterval.
• Fail fast on incompatible storage version at boot. Previously the server would start in a degraded SERVING state with affected groups un-loaded because the property schema-registry retry loop swallowed the version-incompatibility panic. Compatible versions are listed in banyand/internal/storage/versions.yml.
• Release bluge index writers on segment rotation so analysisWorker pools sized from GOMAXPROCS don’t accumulate across rotations. Two layered defects kept the existing idle-segment reclaim path from running: segmentIdleTimeout defaulted to 0 (which disabled the 10-minute reclaim ticker), and incRef refreshed lastAccessed on every rotation tick so closeIdleSegments never observed an idle segment. Defaults to time.Hour, moves the lastAccessed bump to real read/write call sites, and rewrites closeIdleSegments to take its own CAS-bumped snapshot so a concurrent reopen cannot have its only ref dropped under the reclaimer (apache/skywalking#13874).
• Fix incorrect counts and missing trace fields in the lifecycle migration report.
• Fix lifecycle migration placing data in the wrong target segment when the source segment interval is not a multiple of the target stage’s interval, by row-level replaying parts that straddle a target-segment boundary instead of chunk-copying them into a single segment.
• Fix trace query identity-tag projection: when trace_id/span_id are explicitly projected, reconstruct them from span identity at response build time instead of requesting them as stored tags, and preserve tag order with null-filled per-span value alignment in the distributed trace result iterator.
• Fix measure, stream, and trace queries returning data from segments already expired by the TTL. Retention removes a segment only on its next scheduled run, so a fully expired segment can linger on disk and keep serving TTL-expired data; queries now skip segments whose whole time range is past the retention deadline, matching retention’s own removal condition.
• Trace storage metrics now expose the storage sub-scope, matching the stream_storage_* naming. The StorageMetricsFactory for trace switched from the root trace scope to trace.storage, so per-segment inverted-index metrics (inverted_index_total_updates, inverted_index_total_doc_count, inverted_index_total_term_searchers_started) are now emitted as banyandb_trace_storage_* instead of banyandb_trace_*, aligning the dashboard query names. Other trace metrics (trace_tst_*, trace_scheduler_*) are unchanged.
• Fix FODC proxy corrupting Prometheus metric types. The agent dropped the # TYPE line while parsing banyandb /metrics, the StreamMetrics proto carried no type field, and the proxy guessed the type from a name-suffix heuristic — downgrading counters to gauge, mislabeling _count-suffixed counters as histograms, and splitting summaries into two conflicting # TYPE lines. Capture the type with the Prometheus expfmt parser, store it in the flight recorder, thread it through a new Metric.type enum over gRPC, and emit the real type from the proxy; pre-upgrade (untyped) samples fold into the matching typed family so a mixed-version rollout never emits two conflicting # TYPE lines for one metric.
• Fix lifecycle row-replay OOM on large measure parts by streaming the dump reader, pooling size-classed marshal buffers, and bounding in-flight batch bytes (default 32 MiB); peak heap drops ~1.5 GB→~296 MB.
• Consolidate lifecycle migration report errors into a single flat list of structured, stage-aware entries.
• Fix backup container OOM from overlapping scheduled runs; serialize runs and upload small snapshot files concurrently.
• Fix block metadata reset before unmarshal for stream and trace.
• Lifecycle only handles stream/measure/trace snapshots, skipping other catalogs.
• Align code-mode editor with the handoff toolbar + footer (move “← Builder” back button into the toolbar, switch the footer to qb-foot/qb-foot-row, restore the ⌘/Ctrl+↵ kbd hint on Run, switch markup to the handoff class names, and drop the syntax-highlight <pre> that double-rendered side-by-side with the textarea).
• Fix liaison goroutine leak from uncanceled gRPC SyncPart stream context in write-queue part sync.
• Fix group update overwriting a warm/cold node’s stage-resolved segment interval, TTL and shard count with the group default.
• Re-index bound resources when an index rule or binding is deleted, clearing stale index config.
• Deleting one TopN aggregation no longer tears down sibling aggregations on the same source measure.
• Purge a deleted group’s resource, index-rule and binding cache entries to avoid dangling references.
• Clear a trace subject’s index when its last index rule or binding is removed.
• Allow * as a non-initial character in BydbQL identifiers, so resources named with * become queryable, not just writable.
• Enable periodic health checks on the queue client (--<prefix>-client-health-check-interval, default 10s), evicting dead data nodes proactively.
• Build the plugin .so with -tags slim to match the host binary’s ABI. Without this, pkg/pool has build-tag-dependent code (tracker.go vs tracker_stub.go) and the plugin-sidecar E2E failed with “plugin was built with a different version of package pkg/pool”.
• Make the trace merge loop survive a persistently failing merge instead of destroying the node. A production cluster wedged when a few unreadable parts made every merge selection fail: each failed attempt leaked its partially-written output directory (the block writer creates it before the merge runs), which exhausted the volume’s inodes at 8% byte usage; the resulting disk-full panic then killed the merge lane workers, which are recovered but never respawned, so the dispatcher blocked on the lane channel forever and the parts stayed pinned in flight. Four layered fixes: (1) mergeParts removes the output directory on any error return or panic, in trace, stream, measure, and sidx; (2) merge-read failures are wrapped in a typed error carrying the failing part’s identity, and a part accumulating three consecutive attributable failures is excluded from both the hot merge selector and the finalize-round selector — it stays in the snapshot (queryable, TTL-deleted as usual) and the registry is swept when parts leave the snapshot; (3) merge dispatch backs off exponentially from 1s to a 60s cap after consecutive failures, reset on any success, with wave mode (operator-driven controlled merges) bypassing it; (4) merge-execution panics are converted to ordinary merge errors at the lane worker and the dispatch cycle, so the semaphore release, in-flight unpin, and failure accounting always run and the goroutine survives. New metrics: total_merge_part_quarantined, merge_quarantined_parts, total_merge_backoff_seconds, and total_merge_panic_recovered.
• Report panics recovered by the gRPC servers through panicdiag, so they reach banyandb_panic_total{component}, the crash reporters and the artifact writer. Each server (queue/sub, liaison/grpc, property/gossip, metadata/schema/schemaserver) hand-rolled a recovery handler that only logged and returned codes.Internal, bypassing every panicdiag signal: during a production incident one data node logged 737 recovered request-path panics while banyandb_panic_total stayed at 4 (the only goroutine panics, which route through run.Go). The panics most worth alerting on were the ones nothing counted. The four copies are replaced by panicdiag.GRPCRecoveryHandler(logger, component), which reports via the new exported panicdiag.RecoverExternal – the entry point for a panic the caller recovered itself – and is labelled per server (grpc.queue-sub, grpc.liaison, grpc.property-gossip, grpc.schema-server). The stack is captured inside the interceptor’s recovering defer, so artifacts still show the panic site rather than the recovery machinery.
• Fix the measure schema-change merge test, which was time-of-day dependent rather than flaky. It wrote its two batches at now-2h and now-1h and waited for the part count to drop, but sw_metric uses a 1-day segment aligned to local midnight and a merge only ever combines parts within one segment, so any run starting within two hours of midnight split the batches across yesterday’s and today’s segments and the count could never drop. The budget had been raised twice (10x, then 20x) blaming merger starvation, which no budget could have cured; both offsets are now compressed into the elapsed part of the current segment.
• Fix the flaky distributed schema clamp test: it queried once immediately after AwaitRevision/AwaitApplied, which confirm the schema cache but not that the data node has loaded the group’s query topology, so the query could still fail with “group not found”. It now retries until the query resolves, the same treatment the multi-group spec in the same file already had.
• Stop the TopN test overriding the configured eventually timeout. It passed flags.EventuallyTimeout and then chained WithTimeout(10s), which takes precedence, so the spec always ran on a 10s budget even though CI builds the integration suites with -X ...flags.eventuallyTimeout=30s to give slow runners room. It was the only case in test/cases opting out.
• Await SchemaBarrier applied state after standalone schema preload so measure/stream/trace writes no longer race local caches.
• Fix wrong log level in trace lifecycle migration.

Document

• Add a code-accurate, API-first “Storage & File Format” doc and correct stale storage/format descriptions: fix the on-disk hierarchy to group → segment → shard → part (in tsdb.md, data-model.md, clustering.md, disk-management.md, including the dump CLI path examples), correct the measure field-values file name (fv.bin, not fields.bin), clarify that the GORILLA/ZSTD enums are schema hints (the engine uses delta/dictionary + size-thresholded zstd), document the measure index_mode two-engine split and the trace span-store/sidx layout, and fix the property repair Merkle-tree SHA/snapshot-state descriptions. Replace the file-structure diagrams with inline mermaid.
• Refresh observability for FODC-proxy scraping: add two dashboards built for the deployment where Prometheus scrapes the FODC proxy /metrics as the single target, split by aggregation dimension — grafana-fodc-nodes.json for node/pod-level health and resources aggregated by pod_name (fleet overview, per-node health table, a “Topology: Pod-to-Pod Flows” table joining the publisher’s and subscriber’s views of each directed source → target flow into one row, resources, disk-by-path, Go runtime) and grafana-fodc-workload.json for business/data-level throughput and latency aggregated by group (cluster workload summary, liaison ingestion/query/publish + write-queue backlog, data storage/inverted-index, and the internal queue with per-operation throughput & p99 by group) — and remove the stale direct-scrape grafana-cluster.json. Audit and validate the observability prose/PromQL against a live cluster, replace the out-of-date self-observability-write image with an inline mermaid write-flow diagram, add a “Key Signals to Watch” section, and split the oversized observability.md into an observability/ folder (overview, logging, metrics, providers, profiling, tracing). Reconcile the FODC overview’s HTTP API surface (/metrics, /metrics-windows, /cluster/topology, /cluster/lifecycle, /diagnostics) with the proxy code and apis.md.
• Add a “Cluster Topology Rendering” doc (docs/operation/fodc/topology.md) that joins the FODC /cluster/topology node inventory with cluster signals to render a directed topology with two edge layers — the request pipeline (liaison→data, weighted from banyandb_queue_pub_* metrics) and the lifecycle tier migration (hot→warm→cold; path from the lifecycle service’s entries in /cluster/topology calls, weight from the banyandb_lifecycle_migration_* family). A data node and its lifecycle sidecar render inside one pod boundary with the containers labeled data / lifecycle. Includes the per-edge PromQL, three captured sample inputs (topology, queue metrics, and the migration metrics taken while the daily run was in flight), and a stdlib-only join script (render_topology.py) that emits Graphviz DOT and Mermaid; cross-linked from the FODC overview and the metrics queue reference.

Chores

• Upgrade Go, npm, OpenTelemetry, AWS SDK, and Google Cloud dependencies.
• Regenerate expired TLS test certificates with 100-year validity.
• Set Ginkgo --repeat to 0 in the flaky-test workflow so the hourly run completes within the 50-minute timeout.
• Refactor the dump tool into a reusable banyand/dump parser library.
• Strip macOS AppleDouble (._*) and __MACOSX/ metadata from every release tarball (src, banyand, bydbctl, fodc-agent, fodc-proxy) so downstream users running make generate from a downloaded source tarball no longer hit “invalid control character” errors when buf generate walks the resource-fork files; export COPYFILE_DISABLE=1 and filter ._* files at the source.
• Bump Go and ui/mcp dependencies to clear Dependabot advisories: golang.org/x/net v0.52.0→v0.56.0 (CVE-2026-25680), opencontainers/runc v1.3.3→v1.3.6 (CVE-2026-41579). Refresh mcp/ui lockfiles and license attribution.
• Optimize the SkyWalking and Zipkin trace samplers and shared var-array decoder through deferred decoding, zero-copy string and tag handling, early tag rejection, direct scalar reads, cached rule prefixes, and escape-free decoding paths.