Lifecycle Management in BanyanDB

BanyanDB uses a hot-warm-cold data tiering strategy to optimize storage costs and query performance. The lifecycle management feature automatically migrates data between storage tiers based on configured policies.

Overview

A dedicated lifecycle agent (an independent Go process) is responsible for:

• Identifying data eligible for migration based on lifecycle stages.
• Extracting data from the source node.
• Applying configured source-stage trace sampling before sending retained spans.
• Migrating data to designated target nodes.
• Removing the original data after a successful migration.

This process ensures data flows correctly through hot, warm, and cold stages as defined in your group configurations.

Prerequisite: node discovery must be configured. The lifecycle agent resolves the target nodes of every stage by listing all data nodes through the cluster’s node registry and filtering them by the stage’s node_selector. In the default --node-discovery-mode=none setting the registry is empty, so the agent cannot find any destination to migrate data to and the run will be a no-op. Configure dns or file mode on the lifecycle agent using the same source of truth as the rest of the cluster; see the node discovery documentation for the full list of --node-discovery-* flags.

Data Migration Process

The lifecycle agent performs the following steps:

1. Identify Groups for Migration:

   • Connect to the metadata service and list all groups.
   • Filter groups configured with multiple lifecycle stages.
   • Skip groups already processed (using progress tracking).

2. Take Snapshots:

   • Request a snapshot from the source data node.
   • Retrieve the stream, measure and trace data directory paths.

3. Setup Query Services:

   • Create read-only services for streams, measures and traces using the snapshot paths.

4. Process Each Group:

   • Determine the current stage based on node labels.
   • Identify target nodes for the next lifecycle stage.
   • For each stream, measure or trace:
     • Query data from the current stage.
     • For traces with a matching source-stage sampling chain, persist whole-trace decisions across snapshot parts and replay only retained spans.
     • Transform data into write requests.
     • Send write requests to target nodes.
   • Delete the source data after a successful migration.

5. Progress Tracking:

   • Track migration progress in a persistent progress file.
   • Enable crash recovery by resuming from the last saved state.
   • Clean up the progress file once the migration is complete.

Configuration

Lifecycle stages are defined within the group configuration using the LifecycleStage structure.

Example Configuration

metadata:
  name: example-group
catalog: CATALOG_MEASURE
resource_opts:
  shard_num: 4
  segment_interval:
    unit: UNIT_DAY
    num: 1
  ttl:
    unit: UNIT_DAY
    num: 7
  stages:
    - name: warm
      shard_num: 2
      segment_interval:
        unit: UNIT_DAY
        num: 7
      ttl:
        unit: UNIT_DAY
        num: 30
      node_selector: "type=warm"
      close: true
    - name: cold
      shard_num: 1
      segment_interval:
        unit: UNIT_DAY
        num: 30
      ttl:
        unit: UNIT_DAY
        num: 365
      node_selector: "type=cold"
      close: true

In this example:

• Data starts in the default stage with 4 shards and daily segments.
• After 7 days, data moves to the “warm” stage with 2 shards and weekly segments.
• After 30 days in the warm stage, data transitions to the “cold” stage with 1 shard and monthly segments.
• Data is purged after 365 days in the cold stage.

Command-Line Usage

The lifecycle command offers options to customize data migration:

lifecycle \
  --grpc-addr 127.0.0.1:17912 \
  --stream-root-path /path/to/stream \
  --measure-root-path /path/to/measure \
  --trace-root-path /path/to/trace \
  --node-labels type=hot \
  --progress-file /path/to/progress.json \
  --schedule @daily \
  --node-discovery-mode file \
  --node-discovery-file-path /etc/banyandb/nodes.yaml

Command-Line Parameters

Handling Orphan (Deleted-Schema) Data

A source segment can hold data for a measure or stream whose schema was later deleted from the registry (for example, when the upstream metric definition is renamed or removed). Such rows can no longer be written to a target — their schema no longer exists — so the migration treats them as orphan rows.

Orphan handling applies to the measure and stream catalogs only. Trace migration is not wired for per-series orphan handling: a trace group has a single schema, so a deleted trace schema is a whole-group concern rather than a per-series orphan within a surviving group.

On the row-replay path (used when a source segment spans multiple target segments), the migration detects orphan rows, skips them so the rest of the group still migrates, and then — unlike rows skipped because of a series-index gap — lets the source segment be deleted normally. What happens to the orphan rows themselves is controlled by --migration-orphan-policy:

• archive (default): each orphan row is written as one JSON line so an operator can recover it. The archive lives in the --migration-orphan-archive-subdir subdirectory under each catalog’s own root path (so measure orphans land under <measure-root-path>/<subdir>/… and stream orphans under <stream-root-path>/<subdir>/…); the catalog is therefore not a path level (it is recorded inside every record and manifest instead). The per-part JSONL is gzip-compressed on disk (the rows are highly repetitive, so this is ~37× smaller — e.g. 269 MB of raw JSONL becomes ~7 MB). Files are laid out as:

  <catalog-root-path>/<subdir>/<group>/seg-<segment-suffix>/shard-<id>/part-<part-id>.jsonl.gz
  <catalog-root-path>/<subdir>/<group>/seg-<segment-suffix>/manifest.json
  

  Each line is self-describing (decoded from the part’s own column types, with no registry schema needed): it carries the group, catalog, the measure/stream name (always under the JSON key measure, regardless of catalog — the catalog field disambiguates), source location, series id, entity, timestamp (both RFC3339 timestamp and epoch-nanos timestamp_unix_nano), tags, and — for measures — version, indexed_tags, and fields; streams carry element_id instead of those three and omit version entirely (rather than emitting a misleading zero). The per-segment manifest.json (plain JSON) indexes which deleted measures/streams were archived and their row counts. A part with no orphan rows writes no file; re-running a part rewrites its file idempotently.

• discard: orphan rows are dropped; nothing is written to disk.

Orphan handling is not a migration error — the source segment is migrated and deleted normally. Instead of appearing in the report’s errors buckets, it is summarized under an orphans section that records, per catalog → group → deleted measure/stream, how many rows were archived or discarded:

"orphans": {
  "policy": "archive",
  "measure": { "sw_metricsHour": { "meter_..._hour": 1234 } },
  "stream":  { "<group>": { "<stream>": 56 } }
}

(The counts accumulate across resume cycles. If the process is hard-killed mid-group before the group finishes, that run’s not-yet-flushed counts are lost from the report — the archive files and their manifest.json are unaffected.)

Reading the archive

The manifest.json files are plain text — read them directly. The per-part data is gzip-compressed JSON Lines:

# inspect one part's rows (Linux: zcat; macOS: gzcat)
gunzip -c <measure-root-path>/<subdir>/<group>/seg-20260601/shard-0/part-000000000000003b.jsonl.gz | jq .

# project a few fields
gunzip -c part-*.jsonl.gz | jq -c '{measure, ts:.timestamp, v:.fields.value.value}'

# search without fully decompressing (Linux)
zgrep "meter_banyandb_instance_disk_usage" part-*.jsonl.gz

# what was archived for a segment (counts per deleted measure)
jq '{total_rows, total_series, measures:[.measures[]|{measure,rows}]}' .../seg-20260601/manifest.json

Limitations and operations notes:

• The archive lives under each catalog’s own root path (--measure-root-path / --stream-root-path), so it shares the durability of the volume that already holds the catalog data — no separate path to provision. --migration-orphan-archive-subdir only changes the subdirectory name within that root.
• Detection only happens on the row-replay path. On the chunk-sync path (a source segment that maps to a single target segment), whole part files are copied without per-row decoding, so orphan rows pass through to the target. They then occupy target-stage disk until that stage’s TTL expires (e.g. up to the warm-stage TTL), and whether the copy succeeds depends on the receiver’s part-acceptance behavior for a now-unregistered schema.
• The archive directory is not cleaned up automatically. Prune it periodically once the data is no longer needed.

Automatic Behavior on Warm and Cold Nodes

When a data node matches a lifecycle stage (i.e., its labels match a stage’s node_selector), two behaviors are automatically configured:

• Rotation is disabled: The rotation task (which pre-creates segments on hot nodes) is not started on warm or cold nodes. Segments are created on demand by the lifecycle migration process, ensuring correct segment boundaries for multi-day intervals.
• Retention is disabled on non-last stages: TTL-based segment deletion is disabled on all stages except the last one. Data removal on intermediate stages is managed by the lifecycle migration process, not by the built-in retention mechanism.

If a node’s labels do not match any stage, both rotation and retention are disabled as a safety measure.

The lifecycle agent also accepts the full suite of --node-discovery-* flags so it can enumerate every candidate target node in the cluster. At a minimum you need --node-discovery-mode and either --node-discovery-dns-srv-addresses (DNS mode) or --node-discovery-file-path (file mode). The set of nodes visible to the lifecycle agent must include every warm/cold data node that any stage’s node_selector could match — use the same discovery configuration that the liaison nodes use. See the node discovery documentation for the complete reference.

Best Practices

1. Node Labeling:
   • Use clear, consistent labels that reflect node capabilities and roles.
2. Sizing Considerations:
   • Allocate more shards to hot nodes for higher write performance.
   • Optimize storage by reducing shard count in warm and cold stages.
3. TTL Planning:
   • Set appropriate TTLs based on data access patterns.
   • Consider access frequency when defining stage transitions.
4. Migration Scheduling:
   • Run migrations during off-peak periods.
   • Monitor system resource usage during migrations.
5. Progress Tracking:
   • Use a persistent location for the progress file to aid in recovery.

Example Workflow

1. Setup nodes with appropriate labels:

   Every node (liaison, hot, warm, cold, and the lifecycle agent itself) must point at the same node-discovery source so that the lifecycle agent and liaison nodes see the identical topology. The example below uses file mode with a shared /etc/banyandb/nodes.yaml; see the node discovery documentation for DNS mode.

   # Liaison node
   banyand liaison \
     --node-discovery-mode=file \
     --node-discovery-file-path=/etc/banyandb/nodes.yaml
   
   # Hot data node
   banyand data \
     --node-labels type=hot \
     --grpc-port 17912 \
     --node-discovery-mode=file \
     --node-discovery-file-path=/etc/banyandb/nodes.yaml
   
   # Warm data node
   banyand data \
     --node-labels type=warm \
     --grpc-port 18912 \
     --node-discovery-mode=file \
     --node-discovery-file-path=/etc/banyandb/nodes.yaml
   

2. Create a group with lifecycle stages.

bydbctl connects the liaison node to create a group with lifecycle stages:

   bydbctl group create -f - <<EOF
metadata:
   name: example-group
catalog: CATALOG_MEASURE
resource_opts:
   shard_num: 4
   segment_interval:
      unit: UNIT_DAY
      num: 1
   ttl:
      unit: UNIT_DAY
      num: 7
   stages:
      - name: warm
      shard_num: 2
      segment_interval:
         unit: UNIT_DAY
         num: 7
      ttl:
         unit: UNIT_DAY
         num: 30
      node_selector: "type=warm"
      close: true
EOF

3. Run the lifecycle agent:

lifecycle connects to the hot node (as its snapshot source via --grpc-addr) and uses node discovery to locate the warm/cold target nodes:

lifecycle \
  --node-labels type=hot \
  --grpc-addr 127.0.0.1:17912 \
  --stream-root-path /data/stream \
  --measure-root-path /data/measure \
  --trace-root-path /data/trace \
  --progress-file /data/progress.json \
  --node-discovery-mode=file \
  --node-discovery-file-path=/etc/banyandb/nodes.yaml

4. Verify migration:
   • Check logs for progress details.
   • Query both hot and warm nodes to ensure data availability.

This process automatically migrates data according to the defined lifecycle stages, optimizing your storage usage while maintaining data availability.

Trace retention sampling

Selecting a policy through the group API

The lifecycle agent applies the source stage’s pipeline.stages plugin chain before migrating trace rows. Migration sampling is independent of enabled_events; the destination stage and top-level plugin chain are not run. An applicable stage policy requires an explicitly enabled native plugin host. A missing plugin or incompatible ABI fails migration before sending data.

For hot -> warm, configure pipeline.stages[].stage as hot; a warm rule applies when leaving warm for the next stage. For an implicit hot stage, the source identity comes from the agent’s type node label. Stage names must match that resolved identity exactly. Sampling currently requires exactly one trace schema in the group. schemaNames and schemaNameRegex, when present, select that schema; either selector may match.

The following example updates an existing sw_trace group through the liaison HTTP API. It assumes the group already has hot-to-warm lifecycle stages and a SkyWalking segment trace schema. It preserves the group’s other settings but replaces its pipeline, so include any existing top-level plugins or additional stage rules you want to retain.

curl --fail --silent --show-error \
  http://localhost:17913/api/v1/group/schema/sw_trace > group-current.json

jq '{group: .group} | .group.pipeline = {
  enabled: true,
  schemaNames: ["segment"],
  stages: [{
    stage: "hot",
    plugins: [{
      name: "sw-trace-sampler",
      sampler: {
        path: "sw-trace-sampler.so",
        abiVersion: 1,
        config: {
          keepErrors: true,
          durationThresholdMs: 500,
          healthySampleRate: 0.01
        }
      }
    }]
  }]
}' group-current.json > group-sampling.json

curl --fail --silent --show-error -X PUT \
  -H 'Content-Type: application/json' \
  --data-binary @group-sampling.json \
  http://localhost:17913/api/v1/group/schema/sw_trace

The same pipeline can be submitted as part of a complete group document with bydbctl group update -f group.yaml, or through GroupRegistryService.Update with GroupRegistryServiceUpdateRequest.group. The example keeps errors and traces meeting the duration threshold, plus a 1% sample of healthy traces; completeness safeguards can retain additional traces. See the sampler configuration reference for supported schemas and rule semantics.

A disabled or absent pipeline, an unmatched schema selector, or no nonempty source-stage chain preserves ordinary unsampled migration. enabledEvents controls data-node merge/finalization only. This example has no top-level plugins, so it configures lifecycle sampling without a data-node sampling chain. Deploy a capable lifecycle agent before enabling stage rules: older agents may ignore them. Existing unsampled migration attempts finish without retroactive sampling.

Enabling the lifecycle plugin host

Use the -plugins host image for the lifecycle agent and mount the matching plugin carrier binaries read-only at the trusted directory. That image builds /lifecycle, /banyand, and the plugin carrier from the same Go toolchain, CGO setting, SDK module graph, libc and build flags. The ordinary static image cannot load native Go plugins. See plugin deployment for carrier image volumes and init-container delivery.

For example, a Compose lifecycle service can use the existing cluster networking and discovery options with these additional settings:

image: apache/skywalking-banyandb:<TAG>-plugins
entrypoint: ["/lifecycle"]
command:
  - --trace-sampling-enabled
  - --trace-sampling-trusted-dir=/plugins
  - --trace-sampling-timeout=1s
  - --trace-sampling-circuit-break=3
  - --progress-file=/state/lifecycle-progress.json
volumes:
  - ./plugins:/plugins:ro
  - lifecycle-state:/state

Persist <progress-file>.sampling together with the progress file and immutable snapshots. For /state/lifecycle-progress.json, the artifact directory is /state/lifecycle-progress.json.sampling. Prepared decisions and delivery checkpoints remain required until source removal succeeds. Restart reuses those decisions even if schema or plugin configuration changes; do not downgrade an agent while sampled attempts remain pending. Sampler timeouts retain affected traces. Native calls that outlive their deadline retain their input buffers and worker slots until returning; plugins are closed only afterward.

Delivery, small parts, and recovery

The agent assembles each trace across all core parts in a snapshot’s source segment/shard before deciding whether to keep it. It retains traces whose completeness is uncertain, including source-segment boundary fragments, and traces that exceed decision budgets. This is a local snapshot decision, not a guarantee that a trace is complete across nodes, segments, or future arrivals.

Sampling writes durable decision tables, indexes, manifests, and delivery checkpoints to disk; bounded scans can also use temporary metadata merge files. It does not write a reduced native data part for every input part. Retained spans flow through continuous bounded batches across source-part boundaries. The destination rebuilds indexes through its normal write path and merges small parts through normal compaction. A small final part is valid; migration does not wait for compaction or retain extra traces to fill it. An all-drop shard sends no data and counts as completed only after its decision state is durable.

Restart with the same progress-file path and snapshot volumes. Do not delete .sampling state to retry a failed migration or change policy for an already prepared shard. The agent discovers pending sampling artifacts even if the legacy progress file is missing. Only successful delivery of all required work allows source removal; artifacts are cleaned up after source removal succeeds.

A missing plugin, incompatible ABI, or disabled host with an applicable chain fails preflight before sending. Runtime plugin errors, panics, or invalid verdicts bypass that link; other successful links still apply. A batch timeout or open circuit retains the affected batch. Source read errors and artifact durability failures stop migration rather than treating unreadable data as a sampling drop.

Verifying sampling

Use --report-dir to retain migration reports. The top-level trace_sampling map records each artifact’s manifest, manifest_bytes, replayed_bytes, and resume_count. Manifest fields include source_stage, target_stage, traces, kept_traces, spans, kept_spans, fail_open, and preparation_nanos. Calculate dropped counts by subtracting kept counts from input counts. Logical sampling counts describe the prepared snapshot; replay counters can include retried deliveries.

Query the destination both by trace ID and by indexed tags/time range to verify that retained spans are available and sampled-out traces are absent. After successful migration, confirm source removal and sampling artifact cleanup. The regular lifecycle integration suite includes native sampling and a process-kill recovery case that restarts with the sampler binary unavailable, verifies reuse of committed decisions, and checks public destination queries. See the design for the durable protocol.