Data Export and Import
Status: the export dry run, the session RPCs (Plan create/read, Sessions) and the CSV export (Export RPC, bydbctl data export) are implemented (2026-10-10), and so is the CSV import (ImportService, bydbctl data import, 2026-10-19) — see 5.4 CSV import as implemented. The native export and the native import (ImportService.ImportParts, 2026-11-03) are implemented too — see 5.5 Native import as implemented; what is still design is listed in 4.9 Remaining work.
NIDX-03/04/05 note. This document was written while the per-segment series index (
seg-*/sidx/) was still backed by a third-party index library. NIDX-03 cutseg-*/sidx/over to the native ICE v3 engine (pkg/index/native) and removed the legacy series store entirely: there is no index-local lock file, no legacy writer, and no engine flag for it any more, so every claim below that describesseg-*/sidx/as “a legacy inverted directory” or routes its snapshot/export path through the retiredpkg/index/invertedpackage’sReadOnlyWalkDocuments/ReadOnlyDocCount(nowpkg/index/native’s own read-only API) needs re-validation against the native engine before this design is implemented. The Stream element index (shard-N/idx/) is a separate cutover from NIDX-03: PR #1390 already moved the live element-index write/query path onto the same native ICE v3 engine, so it is not “unaffected”, and every claim below that describesshard-N/idx/itself as a legacy inverted directory needs the same re-validation asseg-*/sidx/above. NIDX-04 then cut the offline migration tool (banyand/stream/migration_element_index.go) that bridges pre-native element-index data onto the native writer too, and NIDX-05 removed the retired third-party index library (and its dependency packages) from the module entirely, so no claim below should assume that library’s presence at runtime, on disk, or ingo.mod.
BanyanDB already supports snapshots, backup/restore, and lifecycle management, but it still lacks an operations-oriented, cluster-level data import/export workflow. Backup moves a whole node’s files to remote storage; lifecycle moves data between stages inside one cluster. Neither lets an operator take a named set of groups out of one cluster and put it into another.
This document specifies that workflow as two bydbctl subcommands, a pair of gRPC services, and
an on-disk artifact format. Two things shape most of the design. First, the client never touches
a data node directly — everything crosses a liaison, which relays frames without storing them.
Second, the same logical shard routinely exists on several nodes, so the artifact carries every
copy and the decision about which copies to push happens on the import side, where a wrong guess
costs a re-run rather than data.
1. Goals
- Support all data types: import/export for stream, measure, trace, and property, including groups managed by lifecycle.
- Connect through liaison nodes: the client talks to the liaison endpoint, which routes to the corresponding data nodes.
- Support multiple formats: native and CSV. Native is efficient; CSV is readable.
- Support schemas: metadata and schemas travel with the data.
2. Architecture
Two subcommands are added to bydbctl:
# Export
bydbctl data export --plan plan.yaml # Use --plan to specify the export configuration file.
bydbctl data export --plan plan.yaml [--output <dir>] # Override part of the export configuration.
# Import
bydbctl data import --plan plan.yaml # Use --plan to specify the import configuration file.
# Import and export can share the same configuration file.
bydbctl data import --plan plan.yaml [--input <dir>] # Override part of the import configuration.
2.1 Plan file
All import and export operations are driven by a plan file holding the key configuration.
connection: # Shared by both commands
nodes: [liaison-1:17912, liaison-2:17912] # Liaison node list; every usable one shares the export streams
nodesTLS: {enable: false, insecure: false, cert: ""} # TLS for liaison node port 17912
rateLimit: 0 # Per-liaison byte rate limit (bytes/s), enforced by each liaison,
# 0 = unlimited, e.g. "800MiB/s"
export: # Export only: scope and options
output: ./backup-20260708 # Export directory; CLI --output overrides this. Error if both are missing
format: csv # native | csv
compress: gzip # none | gzip; default gzip for CSV, none for native (native accepts only none)
maxFileSize: 1GiB # Shared by both formats: limit per output file in on-disk bytes (native: hard, >= 64KiB; CSV: soft)
# on disk after compression; 0 = unlimited
includeSchema: true # false = no schema/ files and no schema revision check
schemaOnly: false # true = write schema/ and manifest.json only: no session, no data
csvAnnotations: false # CSV only: write a "#datatype" line before every header
parallelism: max # Data nodes exported concurrently; max = the selected node count,
# clamped to min(N, nodes)
nodeRateLimit: 0 # per-node byte rate limit, 0 = unlimited, e.g. "50MiB/s"
nodeRowRateLimit: 0 # CSV only: per-node row rate limit (rows/s), 0 = unlimited
nodeIdleTimeout: 10m # Fail a node stream that delivers nothing for this long; 0 = never
sendRetryTimeout: 15m # Window for reopening a refused export stream on the next liaison
selectors: # Empty or omitted = all catalogs and groups in the cluster, but the
# groups the server creates for itself (_deletion_task,
# _property_gossip), which no export includes
- catalog: stream
groups: [sw_record, sw_log] # One selector can include multiple groups under the same catalog
- catalog: measure
groups: [sw_metric]
- catalog: property
groups: [sw_prop]
import: # Import only: import policy; scope is read from the artifact manifest
input: ./backup-20260708 # Import directory; CLI --input overrides this. Error if both are missing
includeSchema: true
schemaConflictPolicy: # compare schemas before import and apply the policy to each resource
default: error # error (default) | ignore | overwrite
rules: # override the default policy for specific resources
# match = <kind>/<group>[/<name>]; kind is one of group, stream,
# measure, trace, property, indexRule, indexRuleBinding,
# topNAggregation (case-sensitive; group takes two segments only)
- match: group/sw_metric # Example: allow topology differences only for this group
policy: ignore
parallelism: max # Work items imported concurrently; max = the union of target data
# nodes reported by Targets, capped at the number of work items
writeRateLimit: 0 # CSV only: row rate limit (rows/s) shared by CSV row replay and property import;
# 0 = unlimited
nodeRateLimit: 0 # CSV only: byte rate each target data node admits from the imports,
# however many liaisons write to it, e.g. "50MiB/s"; 0 = unlimited
nodeRowRateLimit: 0 # CSV only: row rate each target data node admits from the imports
# (rows/s); 0 = unlimited
partRateLimit: 0 # Native only: total outbound byte rate limit, e.g. "50MiB/s"; 0 = unlimited
partChunkSize: 1MiB # Native only: bytes per ImportParts frame; must be within [64KiB, 4MiB],
# out of range is a usage error (exit 1)
writeBatchSize: 1000 # CSV only: rows per ImportRows batch; 1-10000, out of range is a
# usage error (exit 1); see §5.4.7 for writeBatchBytes and the rest
verify: true # Reconcile counts after the import; CLI --verify overrides
workDir: /data/import-work # Local scratch directory, used only to unpack .tar/.tar.gz artifacts
In the tree today (CSV export step, 2026-10-10): the plan file is strict and accepts
connection.nodes,connection.nodesTLS,connection.rateLimitand, underexport,output,format,compress,maxFileSize,includeSchema,schemaOnly,csvAnnotations,parallelism,nodeRateLimit,nodeRowRateLimit,nodeIdleTimeout,sendRetryTimeoutandselectors. Every option is declared once inpkg/transfer/configwith struct tags naming its flag, plan path,BYDBCTL_*variable, default and grammar; one-shot switches (--dry-run,--preempt,--strict-coverage,--output-format) have a flag and a variable but no plan field. Underimport(CSV import, 2026-10-19) it acceptsinput,includeSchema,schemaOnly,schemaConflictPolicy.{default, rules}(rulesis plan-only),parallelism,writeRateLimit,nodeRateLimit,nodeRowRateLimit,writeBatchSize,writeBatchBytes,writeInflightBatches,sendRetryTimeout,verifyandverifyTimeout; the native import (2026-11-03) addspartRateLimitandpartChunkSize;import.workDirbelongs to archives and is rejected until those land, as is any other key not listed here.format: nativeexports.bnccontainers (Native export). Explicit flags override the file, which overrides theBYDBCTL_*environment.
An import reproduces what the export captured, so there is no duplicate policy: everything is pushed as-is. Running an import again stores native stream and trace parts, element index segments and CSV stream / trace rows a second time (What a duplicate costs, per catalog). Overlapping sources on a first run are reported as a WARN; --strict-coverage turns that into an error.
workDir is not a rebuild workspace. Neither import channel reconstructs parts locally: the native channel relays opaque bytes and the CSV channel parses rows into write requests. Unpacking an archived artifact is the only local scratch requirement.
3. Dry run
Dry run walks the real workflow without importing or exporting anything. It is a strict prefix of the real run with two deliberate exceptions: Plan goes out without a session, so no snapshot is created; and no schema beyond the group list is listed, so nothing is written to schema/.
3.1 Export dry run
- Connection check — connect to one liaison and call
GetCurrentNodeto confirm theLIAISONrole, thenGetClusterStatefor the other nodes’ IDs, labels, versions, and timezones. - Fetch group list — list every group through the liaison’s group registry, so a selector that names an unknown group, or a group of another catalog, fails before anything is planned. The dry run loads no other schema.
- Plan request — send one
Plan{}request (nosession) to that liaison, which filters allDATA-role nodes, forwards the request, and relays each response frame as it arrives. A dry run creates no session and no snapshot, so it inventories the online catalog and its numbers have a shelf life; the real export inventories a frozen snapshot. - Client aggregation — aggregate and print what every node returned.
NODES the cluster reports 4 data node(s) [data-1 data-2 data-3 data-4]; no inventory came back from [data-3] ⚠ (their data would not be exported)
MULTI-SRC 1 unit(s) have more than one source; all of them are exported, the importer decides which to use
stream/sw_record/hot/20260706/shard-1 <- data-1(48.9M rows, 07-06 ~ 07-06) + data-4(48.9M rows, 07-06 ~ 07-06)
NODE CATALOG GROUP STAGE SEGMENTS SHARDS PARTS EST-ROWS EST-SIZE TIME-RANGE
data-1 measure sw_metricsMinute hot 7 2 34 118.2M 18.3 GiB 06-30 ~ 07-07
data-1 stream sw_record hot 7 2 61 342.7M 96.1 GiB 06-30 ~ 07-07
data-2 measure sw_metricsMinute warm 28 4 52 1.4G 201.5 GiB 05-30 ~ 06-30
data-2 property sw_prop warm - 1 - 12.3K 4.1 MiB -
data-4 stream sw_record hot 7 2 58 339.1M 95.2 GiB 06-30 ~ 07-07
SNAPSHOT each data node pins its own snapshot; the largest node (data-2) needs about 201.5 GiB; check df there first
The NODES line has two other forms: NODES N data node(s) [...], all answered, and NODES standalone process [self] for a standalone server. MULTI-SRC appears only when some unit is held by more than one node (replicated shards): it lists each key — catalog/group/stage/segment/shard-N — with every source’s rows and time range. Property groups are not listed, nor are index-mode measure segment-level units: they carry no shard id, so the dry run cannot tell replicas of a segment from different shards of it. The SNAPSHOT line budgets the largest per-node compressed subtotal, because each node pins its own snapshot on its own disk.
The table lists only data that exists; it never derives “which shards ought to be here” from the schema’s shardNum, because shards and segments are created lazily and comparing against an expected set produces large-scale false reports. The NODES line is therefore the only completeness check — it lists the nodes the liaison reported unreachable or not holding the session (unreachable_nodes), which cannot produce a false positive. It is a WARN by default and an error under --strict-coverage. A dry run inventories the live directories, and a session snapshot contains what was flushed to disk when it was taken: rows still in memory (up to one flush interval) are in neither.
3.2 Import dry run
CSV import as implemented (2026-10-19). The CSV dry run runs steps 0–5 of the implemented workflow — preflight, artifact check, read-only schema comparison,
ImportService.Targets, the puredecide()and the rendered verdict table — and exits0(would proceed) or2(would be refused) without writing anything. Its table is byte-identical to the one the real import prints before it writes. Below the main table the CSV table prints SCHEMA, one line per group with its resource count and how many resources get each disposition; PROBLEMS, only when there are any, one sentence each: the groups of the missing resources together, every conflict with its differences and thematch:rule that would accept it, the work items a route, TTL, coverage or row check refuses, and the warnings; and RESULT, the conclusion in one line. The main table’sSTATUScolumn isok,refusedorexpired. Step 4 also runs the read-only row check (gateROWS) over every row of the artifact, so the CSV dry run reads all the data, not only the manifest and schema. Of the gates below only [1] schema, [4] TTL andROWSapply to CSV: the version and timezone gates are no-ops, and there is no topology gate because rows are re-routed by the target’s own stage, shard number and replicas. The TTL gate follows item 6 below: it asks whether the target segment holding the unit’s newest row ends beforenow − cumulative TTL of the mapped target stage(see TTL gate). The CSV table has aTARGET-NODEScolumn, read fromTargets, which the liaison computes with the same route the import uses. A group the import creates or overwrites has no route before its schema is written, so itsTARGET-NODESisafter-schema, explained by aNOTEline below the table; the re-check after the schema write routes it. A standalone target shows-. The native dry run as implemented (5.5) prints the same table, withTARGET-NODESandSTATUS; itsCHANNELispart-push(property/partsfor a property group), and its refusals (TOPOLOGY,TZ,VERSION,INDEX_RULE) appear inPROBLEMS. The rest of this section, including the table below, itsCHANNEL/GATEvalues and the paragraph aboutLocateAll, is the original native-aware design and is superseded where it differs.
Steps 1–7 of the import workflow plus the channel decision, then exit — the same code path, stopping before step 8, the schema write, so not one byte of the target changes. Exit code 0 means the run would proceed and 2 that a gate would reject it, which makes it directly usable as a CI gate. --input is required here, unlike the export dry run, because the artifact has to be read.
- Connection check — poll
connection.nodesuntil one liaison answers, confirm theLIAISONrole, thenGetClusterStateonce for every target data node. - Artifact check — validate
manifest.json, the artifact version, and every file’s size and CRC32, then derive the multiple-source decisions frommanifest.units. - Schema comparison — fetch the target schema and compare it with the artifact’s. Read-only: because no schema is created, the later gates run against a hypothetical topology — existing groups use the target’s stages, groups this run would create use the artifact’s.
- Version gate — the unit’s segment version must be in the target’s
compatible_file_format_version. - Timezone gate (per unit) — compare IANA names, not current UTC offsets. A no-op for CSV.
- TTL gate (per unit) — the effective TTL is cumulative:
group.ttlplus everystage.ttlup to that stage, aspub.ResolveStageResourceOptscomputes it. The test is whether the whole target segment has expired (segmentEnd < now − effectiveTTL), projected onto the target’s segment grid and timezone — not the unit’sminTimestamp. Stages with retention disabled are exempt, as is property. - Topology gate (per unit) — the target
segmentIntervaland bothshardNumvalues — the mapped stage’s and the group-levelResourceOpts.ShardNum— must equal the artifact’s. Comparing only the stage-level one lets a unit through and then fails in the liaison’s routing withunknown shard; comparing only the group-level one gets it silently dropped byloadShardson the next restart. - Channel decision and guards — pick native / CSV / property / rejected per unit, then apply the schema guards.
TARGET data-1:17912 data-2:17912 (from --plan ./restore.yaml) | 0.10.1 | supported segment formats [1.4.0 1.5.0] | api 0.10 | tz Asia/Shanghai
liaison liaison-1:17912 (ROLE_LIAISON, active)
SOURCE ./backup-20260707 native artifactVersion=1 exported at 2026-07-07T12:00:00Z | 0.9.0 | tz Asia/Shanghai
BYDBCTL 0.10.1 | segment format 1.5.0 | api 0.10
CATALOG GROUP STAGE SEGMENTS SHARDS(src→dst) ROWS SIZE SCHEMA CHANNEL GATE TIME-RANGE
stream sw_record hot 7 2→2 342.7M 96.1GiB CONFLICT ① part-push BLOCK 06-30 ~ 07-07
measure sw_metric hot 7 2→4 118.2M 18.3GiB IGNORE REJECT(topology) BLOCK 06-30 ~ 07-07
measure sw_metric warm 28 4→4 1.4G 201.5GiB OK ① part-push TTL! 05-30 ~ 06-30
measure sw_traffic hot 7 - 0.4M 12.3MiB CREATE ① sidx only ok 06-30 ~ 07-07
trace sw_trace hot 7 4→2 12.1M 4.1GiB CONFLICT REJECT(topology) BLOCK 06-30 ~ 07-07
property sw_prop - - 1→1 12.3K 4.1MiB CREATE property/repair ok -
TOTAL 56 1.87G 320.0GiB 05-30 ~ 07-07
of which BLOCKED: 3 rows / 42 segments / 473.0M rows
There is no TARGET-NODES column: the liaison picks target nodes at execution time from the live routing table, and LocateAll is not on the dry run’s read-only call list. Never compute them on the client as nodes[(shardID + replicaID) % len(nodes)] either — the real roundRobinSelector.Pick indexes a globally sorted lookup table rather than the shard ID, so that formula breaks as soon as there is more than one group.
All three verdict columns take closed value sets:
SCHEMA—CREATE/OK/MISSING/CONFLICT/IGNORE/OVERWRITE/INVALID.CHANNEL—① part-push/① sidx only(index-mode measure) /② row replay/property/repair/REJECT(<gate>).② row replaycan only appear for a CSV artifact: a native artifact has no fallback across a topology change, so it rendersREJECT(topology).GATE—ok/TTL!/BLOCK. Any row whoseCHANNELisREJECT(...)isBLOCK, as is any row whoseSCHEMAisCONFLICTorINVALID, whatever its channel;TTL!marks a row that clears every other gate and is stopped by the TTL gate alone, which--force-ttloverrides.
TOTAL counts every row, blocked ones included, and SEGMENTS / ROWS / SIZE all use that same basis; the blocked subtotal is reported on its own line.
measure/<group>/_top_n_result is never written and not compared as an ordinary resource: the target creates it automatically from its own hardcoded schema, so comparing it would only produce conflicts the user can neither fix nor should fix. Its rows are still imported (see Top-N results), so its columns get a separate compatibility check: matching columns make it OK, counted with the identical resources and not listed, and an artifact _top_n_result whose tag or field names or types differ from the target’s auto-created one is a CONFLICT.
4. Export
4.1 Client
bydbctl data export without --dry-run runs these steps (pkg/transfer/exporter.RunExport):
- Check the output directory — it must not exist or must be empty; anything else is exit
1. An export therefore never mixes with what an earlier run left behind. - Connection check — probe every address in
--nodes(GetCurrentNode,GetClusterState). An unreachable candidate, or one that has not discovered any data node yet, is skipped with aWARN; refused credentials or an endpoint that is not a liaison fail at once (exit2). The first usable liaison is the primary: the schema,PlanandSessionscalls go through it, and the dry run andrelease-sessionuse only it. - Fetch and write the schema — list the groups, validate the selectors, then list every resource of the selected groups through the liaison’s registry services and write
schema/<kind>.jsonl(Schema files). The mod revision of every stream, measure, trace and property is kept in memory for the CSV check in step 6. With--include-schema=falsethis step lists nothing beyond the groups and writes noschema/files, and step 6 skips the revision check. With--schema-onlythe client writesmanifest.json(schemaOnly: true, no units) here and stops: no session, no snapshot, no data. - Create the session — one
Plan{selectors, create{preempt}}. The session id from the first (created) frame is printed at once, before the inventory arrives. A node holding another session refuses withALREADY_EXISTS; the client lists the sessions once, prints every occupant with its start time, last heartbeat, catalogs and nodes, and exits2(Taking over an existing session). Transient failures (INTERNAL,UNAVAILABLE,ABORTED) are attempted up to three times with jittered back-off. A create whose summary still names unreachable nodes is released again and fails, because their data would be missing. - Heartbeat —
Sessions(ACTION_HEARTBEAT)every 5 minutes in the background. A failed heartbeat is aWARNline and is retried on the next tick;FAILED_PRECONDITION(the session expired or its lease is unreadable) cancels every worker and exits3. - Export — fold the Plan frames into one job per
node_idand run up to--parallelismworkers, oneExportstream per node (Full workflow). The nodes, sorted by id, are dealt round-robin over the usable liaisons, so nodeistreams through liaisoni mod L; a stream that cannot be opened (UNAVAILABLE,RESOURCE_EXHAUSTED, orNOT_FOUNDfrom a liaison that does not see the node) moves on to the next liaison within--send-retry-timeout. Each file lands as<path>.tmp, is checked against itsfile_finished(size and CRC32, and for CSV also the schema mod revisions the node encoded with; a native file binds no column to the schema, so its revisions are not checked), fsynced, renamed, and its directory fsynced. The first failing worker cancels the others. - Write the manifest — once every node finished,
manifest.jsonis built from what was received and written atomically (manifest.json); every warning is also printed as aWARNline. - Release —
Sessions(ACTION_RELEASE). A failed release is a warning only: the artifact is complete and the lease reclaims the snapshot. The client then prints what was actually written — oneNODE CATALOG GROUP STAGE UNITS FILES ROWS SIZE TIMErow per node, catalog, group and stage (a property row shows its unit’s stage, that of the unit’s first source;TIMEadds up the time its units took) — and one line with the units, files, bytes and warnings written and the time of the whole export. The snapshot is released at this point, so the dry run’sSNAPSHOTbudget line is not repeated.
Once the session exists, a failure keeps it and prints bydbctl data release-session --id <id>; the run exits 3. Files already renamed stay in the output directory, a file in flight stays as .tmp, and no manifest.json is written, so a partial directory never passes for an artifact. Ctrl-C or SIGTERM is handled the same way: the export stops, the session is kept and the same bydbctl data release-session --id <id> line is printed. After a successful export, a failed release (as a whole or on some node) is a WARN with exit 0, followed by the same bydbctl data release-session --id <id> line.
The export side performs no source selection. Every source node’s copy of a unit is carried away into its own nodes/<node-dir>/ subtree, and all multi-source judgement happens on the import side (Multiple shard sources). This is deliberate: a wrong judgement on the export side loses bytes permanently, while a wrong judgement on the import side costs a re-run. Every source node is included, but a CSV copy is a reconstruction, not a byte copy: rows or values the format cannot represent under the current schema are left out with a warning (CSV export), and the export still succeeds. An export that exits 0 is therefore complete for migration only when manifest.warnings is empty, or when every warning in it is acceptable.
4.1.1 Node IDs are not path-safe
A node ID is not a friendly name. GenerateNode builds it as net.JoinHostPort(host, port) (api/common/id.go), so a normal ID is 10.0.0.5:17912, and an IPv6 one is [::1]:17912. Those contain :, [ and ], which are illegal in a Windows path component and awkward everywhere. The artifact must therefore never use the raw ID as a directory name.
Two values are kept, and they have different jobs:
| Value | Where it lives | |
|---|---|---|
| Raw ID | 10.0.0.5:17912 |
manifest.source.nodes[].id, and every node_id field on the wire. This is the identity |
| Directory component | 10-0-0-5-17912-8a05db98 |
manifest.source.nodes[].dir, and the nodes/<node-dir>/ path. This is only a filename |
The directory component is derived from the raw ID by:
- lowercase the raw ID;
- replace every run of characters outside
[a-z0-9]with a single-; - truncate to 32 characters;
- trim leading and trailing
-; if nothing is left, usenode; - append
-followed by the first 8 hex characters ofSHA-256over the raw UTF-8 bytes.
Reversal is by lookup in the manifest, not by decoding. Steps 1 to 3 are lossy on purpose — the bounded readable prefix exists so an operator can tell subtrees apart, and the manifest carries the exact ID for anything that needs it. Import resolves a subtree by matching dir, and uses the paired id wherever identity matters.
The hash suffix keeps IDs apart that the readable prefix alone would merge:
| Raw IDs | Directory component |
|---|---|
10.0.0.5:17912 |
10-0-0-5-17912-8a05db98 |
[::1]:17912 |
1-17912-…, usable where the raw ID is not |
Data-1:17912, data-1:17912 |
data-1-17912-03d2b6d6, data-1-17912-98a53f33 — macOS and Windows fold case, the suffix does not |
a.b:17912, a-b:17912 |
a-b-17912-613e0e89, a-b-17912-b5233c34 |
| two ports on one host name longer than 32 characters | one prefix, distinct suffixes |
The client still refuses the export if two nodes in one Plan produce the same directory component, naming both IDs, so a 32-bit hash collision fails the export instead of mixing two nodes’ files in one subtree.
4.2 Native export
As implemented (2026-11-03).
bydbctl data export --format nativeexports.bnccontainers through the same session,PlanandExportRPCs as CSV; the data node’s native producer (banyand/transfer/export/native) packs each unit straight from the session snapshot. A segment unit gives<group>/seg-<suffix>/metadata(the raw segment metadata file),sidx-NNNNN.bnc(the segment series index, when the snapshot holds one) and oneshard-<N>-NNNNN.bncper shard; a shard container holds the parts the newest snapshot file names (and their trace secondary index parts), that snapshot file, the files of the newest committed generation of the stream element indexidx/and any other file in the shard directory, so the leftovers a closed segment can carry (unnamed parts, older snapshot files, olderidx/generations) are not exported. A property unit gives oneshard-<N>-NNNNN.bncper shard. Compression is alwaysnone(--compress gzipwith native is exit1);--max-file-sizesplits containers into volumes and must be 0 or at least 64 KiB.FileFinished.entry_countis filled;FileFinished.row_countsstays empty, so the manifest takes a shard’stotalCountfrom thePlanstatistics. With--format native,--compressother thannone,--node-row-rate-limitand--csv-annotationsare exit1when given as a flag or inplan.yaml, and are ignored when they come fromBYDBCTL_*environment variables.FileStarted.segment_levelmarks the segment metadata file and thesidx-NNNNN.bncvolumes, andFileStarted.seqis the volume number within its container. The progress lines of a native export print no row counts. The table and the BNC format below are the design it follows.
Exporting in native format:
- Create snapshot for the required data types.
- Traverse segments of every group.
- Transfer data — read files per the gRPC protocol and write them into BNC (BanyanDB Native Container) files in the stream, one frame per 1 MiB. Rate limiting applies here.
- Complete export — finish the gRPC stream.
The storage layout differs by data type and mode.
| Type | Source directory | Output |
|---|---|---|
| Normal group | <group>/seg-20260907/metadata |
metadata (raw file, not packed into a container) |
<group>/seg-20260907/sidx |
sidx-00001.bnc (segment-level container, one per segment) |
|
<group>/seg-20260907/shard-0 |
shard-0-00001.bnc (shard-level container, one per shard) |
|
| Index-mode measure | <group>/seg-20260907/metadata |
metadata |
<group>/seg-20260907/sidx |
sidx-00001.bnc |
|
| Property | property/data/<group>/shard-0 |
shard-0-00001.bnc |
Two different directories are called sidx, and they are not the same thing:
| Source path | Actual content | Container |
|---|---|---|
seg-XXX/sidx/ |
Series index — a native index directory (%012x.seg segments and a %012x.snp manifest), no parts |
segment-level sidx-NNNNN.bnc |
shard-N/sidx/<rule>/<016x>/ |
Trace secondary index parts — real parts, one directory per index rule, each with manifest.json |
shard-level shard-N-NNNNN.bnc |
4.2.1 BNC file format
The container library is pkg/transfer/format/bnc; its package documentation is the authoritative definition. All integers are little endian:
[member 0][member 1]…[member n-1][footer][footer CRC-32 u32][footer length u32]["BNCA"]
footer:
uvarint volume_seq number of this volume in its container, from 1
byte last 1 when this is the last volume of the container, else 0
uvarint count number of members, at least 1
repeat count times:
uvarint path_len
bytes path UTF-8, '/'-separated, relative to the container root
u64 size bytes of this member
u32 crc32 IEEE CRC-32 of those bytes
uvarint file_offset where these bytes start in their file, 0 for a whole file
uvarint file_size size of the whole file, equal to size for a whole file
- No offsets are stored. A member starts where the one before it ends, so overlaps, gaps and out-of-range offsets cannot be expressed; the members must add up to the file size minus the footer and the 12-byte trailer. Every uvarint is minimally encoded, so a footer has exactly one encoding.
- The magic carries the version. It is
BNCplus a single format-version letter, so a future incompatible layout becomesBNCBwithout needing a separate version field. - Opening reads the tail only. Two reads (trailer, then footer) check the magic, the footer CRC-32, every bound before allocating, every path, and where each member sits in its file. Member bytes are checked against their CRC-32s when they are read or verified.
- Limits: footer ≤ 16 MiB, at most 2^20 members per volume, paths ≤ 4096 bytes with components ≤ 255 bytes. Zero-length files are allowed (CRC-32 0); no member is compressed, since column blocks are already zstd.
Volumes. max_file_size caps every volume file on disk, footer and trailer included; 0 means no limit, and any other value below 64 KiB is refused. Each volume is a complete container named <prefix>-00001.bnc, -00002.bnc, … (five digits, at most 99999). The writer fills each volume up to the limit regardless of part boundaries; a file that crosses the end of a volume is split: its first piece is the last member of that volume and the rest continues as the first member of the next one, so one file — and one part — can span several volumes. file_offset/file_size place every piece in its file, and each piece has its own CRC-32.
Reading. A reader opens all volumes of one container, in sequence order, as one volume set. The set requires volume numbers 1, 2, … with last set on the final volume only, and every unfinished file at the end of a volume to continue at exactly the next byte as the first member of the next one. A missing, reordered or repeated volume, a missing tail or a volume of another container is therefore rejected before any data is read. Files are then read whole across volumes; the read that returns the last byte of a corrupt piece fails, and closing the reader reports that failure again, because helpers that copy an exact length (io.CopyN, io.ReadFull) drop it.
What the container cannot prove. A container carries no identity — a valid container from another artifact, unit or node passes every check — and a prefix that ends at an inner boundary (a first member that is itself a container, or volumes whose last one says it is last) opens and verifies. A file the producer never packed leaves nothing to detect. The manifest is the final authority: the importer compares the size and CRC-32 of every volume with the manifest entry of that path before it trusts a container.
There are only three gRPC frame types:
file_started{ relative_path, shard_id, segment_level, shard_index, seq }
→ Client creates <path>.tmp
chunk{ content, rows } × N → Client appends data and updates the progress
file_finished{ bytes, crc32, entry_count, row_counts, schema_mod_revisions }
→ Client validates → fsync → rename
4.3 CSV export
Each data node produces CSV straight from the session snapshot, read-only (banyand/transfer/export): parts are opened through the dump readers banyand/internal/dump/{stream,measure,trace} over a read-only file system, and the segment series index and the property store through pkg/index/native’s OpenReadOnlyGeneration, which takes no lock and writes nothing into the snapshot. Directories are listed with os.ReadDir. Per unit:
- Locate the unit in the snapshot:
<root>/<group>/seg-<suffix>/shard-<N>(property:<root>/<group>/shard-<N>). A missing segment or requested shard directory isINTERNAL: the snapshot is frozen, so that is a node-side fault. A file that is missing when it is read is taken to mean the session was preempted or released, and the stream fails withFAILED_PRECONDITION; the session is checked once more beforeunit_finished, so a release or takeover during the unit is caught even when no read failed. This covers detected read failures and session loss only, not snapshot damage in general: a snapshot-listed part directory that vanished before the unit was listed is skipped like a merge source, and a shard whose snapshot file vanished lists no parts. As a lightweight consistency check,unit_started.total_partsis counted the same way as the sessionPlan, which inventoried the same snapshot, and the client fails the export (exit3) when the two differ. A shard’s live parts are the ones its newest readable<epoch>.snpnames, as when the engine opens the table: a closed segment is linked into the snapshot as a whole, so it can still hold the source parts of a merge the engine had not yet removed, and those are skipped. The Plan inventory counts parts the same way. - Build the column layout of the group from the node’s current schema (Columns). A group the layout cannot describe — a
TIMESTAMPtag in a stream or measure, or two traces of one group with the same tag set — is refused withFAILED_PRECONDITION. - Encode rows — one CSV stream per shard (index-mode measure: one per segment), gzipped on the fly unless
compressisNONE, sent as chunks of at most 1 MiB of on-disk bytes. A stream rolls to the next file after a complete row once the file’s on-disk bytes reachmax_file_size(0never rolls), and every file repeats the header. A row whose encoded size reaches the limit gets a file of its own, and the unit reports the number of such rows in one warning. - Finish the unit —
unit_finishedcarries the unit’s warnings. A unit without any row produces no file, and an empty file is never written. The warningunit has no rowsis given only when the unit listed shards, whose parts then yielded no row; a unit without shards (a segment the engine created ahead of time) is silent.
How each model is read:
| Model | Rows come from | _name |
|---|---|---|
| stream | the parts of every requested shard | the series subject decoded from the row’s EntityValues; the entity tags are filled from the same series. The segment series index stands in for parts without smeta.bin |
| measure | as stream; indexed tags, which the write path keeps only in the series index, are decoded from it | as stream |
| index-mode measure | the documents of the segment-level series index (seg-<suffix>/sidx), one row per document of an index-mode measure; the series entries of the group’s regular measures in the same index are skipped |
the document’s subject |
| trace | the parts of every requested shard | matched by the row’s tag-name set, without the trace-id and span-id tags and without tags no trace of the group declares, against the group’s traces. A group with one trace takes it directly; a row whose set equals one trace’s takes it; otherwise a row (from a part written before a tag was removed) goes to the only trace whose tags include all of its tags, and a row that no trace or several traces include is dropped and counted; two traces with the same tag set make the group fail (use native, once it exists). Matching by tag set cannot tell an old row from a row of another trace with the same tags |
| property | the live documents of every shard’s store, decoded from _source and streamed one at a time; tombstones included |
Property.metadata.name |
_top_n_result is exported like any other measure: its binary value field carries the engine-internal Top-N encoding, base64-encoded. A point stored in several versions is exported as one row per point, the versions merged the way a query or a part merge combines them (MergeTopNResults in banyand/measure/export_topn.go), with the highest version. The import writes those rows verbatim and never recomputes them. The warnings report dropped rows and why, stored tag and field values the current schema no longer declares or declares with another type, and property timestamps outside the int64 nanosecond range (counted per resource and tag), tags only a multi-tag index rule stores, a single row larger than the file size limit, and a unit whose shards yielded no row. A column that is null on every row is not reported: the source schemas commonly declare tags they never fill. None of them fails the export; all of them land in manifest.warnings, and each of them is data the artifact does not carry.
The schema the rows were encoded with is checked. Columns are bound by position and a group’s layout is built from all of its resources, so every file_finished carries the ModRevision of every resource the node built the layout from, whether or not it wrote rows. The client compares that set with the revisions it saved in schema/ (including _top_n_result); a missing or extra resource or any different revision fails the export (exit 3, “schema … changed during export; export again”). With --include-schema=false there is nothing to compare against and the check is skipped.
Rate limiting has three knobs; two are enforced by the data node, one by the liaison. All three are set by the client and travel in init; neither server has a configuration of its own for them. They throttle different resources, so none subsumes another and whichever binds first wins:
| Knob | Flag / plan field | Protobuf field | Measured |
|---|---|---|---|
| Node bytes | --node-rate-limit / export.nodeRateLimit |
RateLimit.bytes_per_second |
Each chunk before it is sent, so after gzip |
| Node rows | --node-row-rate-limit / export.nodeRowRateLimit |
RateLimit.rows_per_second |
Each row the producer encodes. CSV only (a usage error with native): native moves opaque bytes and has no row concept |
| Liaison bytes | --liaison-rate-limit / connection.rateLimit |
Init.liaison_bytes_per_second |
Each chunk the liaison relays, waited for before it is forwarded to the client; below 1 MiB/s a chunk is forwarded in pieces of one second’s worth, each waited for on its own. Nothing is prefetched, so the wait back-pressures the node |
The node knobs travel in init.rate_limit and apply to that one stream, that is, per data node. The liaison knob is process-wide on each liaison: every Export stream a liaison relays shares one bucket, and the value of the stream opened last wins — 0 included, which lifts the limit — until no Export stream is left, when the bucket goes back to unlimited. One export session at a time per cluster means every stream on a liaison normally carries the same value. With several liaisons in --nodes each enforces the value on its own share, so the total can reach L × the limit. A node bucket’s burst is one second’s worth (at least one row), and a node byte rate below 1 MiB/s sends chunks of one second’s worth, so a chunk always fits the burst; a node about to wait for its row bucket first sends the rows it holds, flushing gzip — right away when the wait is 100 ms or longer, so a low row rate delivers every row as it is written, otherwise once they are a second old — so a throttled stream still delivers a chunk at least about every second; the liaison bucket’s burst is one second’s worth too, at most one chunk (1 MiB), and a larger chunk is relayed in pieces of the burst’s share of one relay (the burst divided by the relays in flight) carrying the chunk’s rows, with its bytes in order, so every node stream still gets a frame about every second; 0 disables a knob entirely. The byte knobs protect the liaison and the network; the row knob protects the data node, whose cost for CSV is row iteration rather than transfer. A CSV export that sets only a byte limit can still saturate a data node’s CPU on a highly compressible group, which is why the row knob exists as well as, not instead of, the byte knob.
Rates take <quantity>[/<period>]: the period is s (the default), min or h (m is refused as ambiguous); a byte quantity is bare bytes or a size unit (50MiB/s, 3GiB/min), a row quantity a plain integer (50000/s). A negative rate, a byte unit on a row rate, and a non-zero rate below one unit per second are usage errors (exit 1).
4.3.1 Value encoding
| Type | Encoding |
|---|---|
STRING (also trace_id, span_id, property _id) |
Written via encoding/csv; a value containing \r is downgraded to \B + base64, and a value starting with a backslash gets one extra leading backslash (the escape algorithm) |
INT (also _version, _mod_revision, _create_revision) |
Decimal |
_element_id |
The 16 lowercase hex digits of the stored uint64 element id’s big-endian bytes, the form a stream query returns as element_id (csvfmt.EncodeElementID). An import parses it back with csvfmt.DecodeElementID and replays the row with that value as raw_element_id, so the target stores the same id |
| FLOAT (measure field) | strconv.FormatFloat(v, 'f', -1, 64). Precision must be -1, or the round trip loses precision. NaN and infinities are written NaN, +Inf, -Inf |
DATA_BINARY / _span payload |
base64.StdEncoding, standard alphabet, with padding; an empty value is an empty cell, distinct from null |
| STRING_ARRAY | JSON array string (HTML characters not escaped; invalid UTF-8 becomes U+FFFD) |
| INT_ARRAY | Semicolon-separated decimal int64 values; an empty array is an empty string |
All time columns (_timestamp, _updated_at, _delete_time) and TIMESTAMP tags |
Decimal int64 UnixNano. TIMESTAMP tags exist only for trace and property; a stream or measure group declaring one is refused |
| null | The \N sentinel; a cell whose value the row does not carry is null. Literals are kept distinct by the leading-backslash rule in the escape algorithm |
NaN and ±Inf are exported.
encodeFloat64Columncannot encode them losslessly and falls back toencodeDefault, which writes a dictionary or plain block beneath an outer plain marker. The measure dump reader now decodes measure columns the way the engine does (readColumnValues, mirroringcolumn.decodeColumnValues), so such blocks read back and the values reach the CSV asNaN/+Inf/-Inf, whichParseFloataccepts.
'f' is chosen over 'g' because 'g' renders 1000000 as 1e+06, and six of the nine non-test call sites in the repository already use 'f', -1, 64. Both round-trip bit-exactly. The cost of 'f' is length: 1e300 becomes a 301-character field, so the export side must not assume an upper bound on field width.
4.3.2 Escape algorithm
Two sentinels share one cell: \N for null and \B<base64> for a value downgraded because it contains \r. They must therefore share one escaping rule, or the mapping stops being injective.
An earlier draft used two rules of different granularity — “a literal \N is written \\N” (whole-value equality) plus “a value starting with \B is written \\B” (prefix match) — and neither handled a value that already starts with a backslash. That collapses real values: the literal \\N and the literal \N both come out as \\N, and a CRLF-bearing string and the literal \BDQo= both come out as \BDQo=. The reader cannot tell them apart, so one of them is silently corrupted.
The rule: a single leading-backslash escape, decided on the first character only. No whole-value equality, no pairing, no recursion.
Writing, in this order:
- value is null → emit
\N; - value is a STRING containing
\r→ emit\B+base64.StdEncoding(value); - value starts with a backslash → emit one extra leading backslash, then the value verbatim;
- otherwise → emit the value verbatim.
Reading, in this order — the four branches are mutually exclusive:
- the cell equals
\N→ null; - the cell starts with
\\→ drop exactly one leading backslash; the rest is the literal; - the cell starts with
\B→ base64-decode the remainder; - otherwise → the cell is the literal.
The three written prefixes \N / \B… / \\… are pairwise disjoint and the reader accepts every cell the writer produces, so the mapping is injective.
Round-trip cases that must be in the test suite, each asserted byte-for-byte:
| Original value | On disk | Read back |
|---|---|---|
| null | \N |
null |
\N |
\\N |
\N |
\\N |
\\\N |
\\N |
\Bx |
\\Bx |
\Bx |
\\Bx |
\\\Bx |
\\Bx |
a\r\nb |
\BYQ0KYg== |
a\r\nb |
\BDQo= (literal) |
\\BDQo= |
\BDQo= |
Why
\rneeds the downgrade at all:encoding/csv’s reader converts every\r\nin its input to a plain\n, regardless of whether it sits inside quotes. CRLF is routine in log bodies and stack traces, and the CRC32 is computed over the compressed bytes on disk, so it cannot catch this kind of logical rewrite. The base64 alphabet contains neither CR nor LF, so the downgrade sidesteps it by construction.STRING_ARRAY does not participate in the downgrade — it is already a JSON array string, and JSON escapes CR itself.
4.3.3 Columns
| Model | Fixed columns | Additional columns |
|---|---|---|
| stream | _name, _element_id, _timestamp |
all tagFamily.tag columns |
| measure (normal) | _name, _timestamp, _version |
all tagFamily.tag and all _field.* columns |
| measure (index-mode) | _name, _timestamp, _version |
all tagFamily.tag columns, no _field.* |
| trace | _name, trace_id, span_id, _span |
all tags, without the family level |
| property | _name, _id, _mod_revision, _create_revision, _updated_at, _delete_time |
all tags |
A group has one layout, shared by every file of every unit and by all of its resources: the fixed columns, then the union of the resources’ tag columns (family.tag for stream and measure, the bare tag name for trace and property), then, for measures, the union of the _field.<name> columns. Resources are taken in name order (_top_n_result included) and tags in declaration order; the first occurrence fixes a column’s position, and a row leaves the columns of other resources null (\N). When two resources declare the same tag or field with different types, or its header would repeat another column’s (a fixed column, a _field.<name> column, or family a tag b.c against family a.b tag c), each resource declaring it gets its own column headed <resource>::<family.tag> (or <resource>::_field.<name>), so every header is unique. Every measure group holds the auto-created _top_n_result, whose binary field is named value, so a group whose measures declare a numeric value field gets split headers such as cpm::_field.value and _top_n_result::_field.value. The layout is a pure function of the schema (pkg/transfer/format/csvfmt), so an importer rebuilds it from schema/.
With --csv-annotations (init.csv_annotations) every file starts with one extra line before the header: #datatype, then one token per column — string, long, double, base64Binary, stringArray, longArray or timestamp. That line has one field more than the header, so a reader must accept a variable number of fields per record.
stream
_name,_element_id,_timestamp,searchable.trace_id,searchable.service_id,searchable.state,data.data_binary
sw_service_log,000000110ff1a035,1783419300123000000,af1c...beef,c2VydmljZS1h,1,ZGF0YQ==
measure — _version is preserved to maintain deduplication semantics.
_name,_timestamp,_version,default.entity_id,default.scope,_field.total,_field.value,_field.percentile
service_cpm_minute,1783419300000000000,1783382401000000,c2VydmljZS1h,SERVICE,\N,87,\N
service_resp_time,1783419300000000000,1783382401000000,c2VydmljZS1h,SERVICE,1024,\N,W1BMQUNFSE9MREVSXQ==
_field.percentile has type DATA_BINARY. FieldValue has no array branch, so it must use base64 rather than a JSON array such as "[50,75,90,95,99]".
trace
_name,trace_id,span_id,_span,service_id,start_time,duration
sw_segments,af1c...beef,1,CgZzZWdtZW50EgQ...,c2VydmljZS1h,1783382400123000000,150
property
_name,_id,_mod_revision,_create_revision,_updated_at,_delete_time,name,value
ui_template,ui-dashboard-1,1783382400000000000,1783382400000000000,1783382400000000000,0,dashboard,"{""layout"":""grid""}"
_delete_time is a required column — a non-zero value is a tombstone. _create_revision and _updated_at must also be carried, or version merging and gossip repair on the target cannot decide correctly.
4.4 Artifact file format
The top-level artifact structure is identical for both formats; only the leaf files differ.
| Path | Description |
|---|---|
manifest.json |
Written last |
schema/ |
One JSON Lines file per kind, one protojson object per line |
nodes/<node-dir>/ |
Organized by source node, keyed by the derived directory component (Node IDs are not path-safe), never the raw node ID |
nodes/<node-dir>/<catalog>/<group>/ |
Further organized by catalog and group |
Below that level:
| Native | CSV | |
|---|---|---|
| Normal group | seg-<suffix>/metadata (raw file)seg-<suffix>/sidx-00001.bncseg-<suffix>/shard-<N>-00001.bnc |
seg-<suffix>/shard-<N>-00001.csv.gz |
| Index-mode measure | seg-<suffix>/metadataseg-<suffix>/sidx-00001.bnc |
seg-<suffix>/segment-00001.csv.gz |
| Property | shard-<N>-00001.bnc (no seg- layer) |
shard-<N>-00001.csv.gz (no seg- layer) |
| File splitting | -00001, -00002, …; each volume is a complete container of at most max_file_size bytes, and a file crossing the end of a volume continues in the next |
-00001, -00002, …; each volume repeats the header |
In CSV, --compress none drops the .gz suffix (shard-0-00001.csv). The <catalog> component is lowercase (stream, measure, trace, property), and the group name is escaped into a safe path component: every byte outside [A-Za-z0-9._-] becomes %XX (upper-case hex), and the names . and .. are escaped in full. The client also rejects any relative_path from a node that is absolute, contains .. or a backslash, or lies outside the unit’s <catalog>/<group>/ prefix, and refuses to overwrite an existing file. Directories are created 0700 and files 0600.
4.4.1 Schema files
schema/ always holds eight JSON Lines files, one per kind, in this order: group.jsonl, stream.jsonl, measure.jsonl, trace.jsonl, property.jsonl, indexRule.jsonl, indexRuleBinding.jsonl, topNAggregation.jsonl. Each line is one resource as protojson with unpopulated fields emitted. Only the selected groups and their resources are listed, and only the kinds a group’s catalog can hold (index rules and bindings for stream, measure and trace; Top-N aggregations for measure). A kind with nothing to list still gets an empty file, so the file set does not depend on the data. The files are listed with their size and CRC32 in manifest.schemaFiles.
--include-schema=false (or export.includeSchema: false) leaves the schema out entirely: no registry listing beyond the groups, no schema/ files (schemaFiles: []), no mod revision check, and groups[].indexMode is recorded as false with a manifest warning that it is unknown. An import then reads the CSV columns with the target’s schema, which must be the schema the rows were exported with (see CSV import as implemented). --schema-only together with --include-schema=false is a usage error. --schema-only writes schema/ and a manifest.json with "schemaOnly": true and no units, without creating a session.
4.5 manifest.json
manifest.json records the format, export time, source cluster, group information, and per-unit results. It is written last, atomically (temporary file, fsync, rename, fsync of the directory), mode 0600; until it exists the directory is not an artifact. A CSV export writes:
{
"artifactVersion": 1,
"format": "csv",
"compress": "gzip",
"exportedAt": "2026-09-07T12:00:00Z",
"source": {
"nodes": [{"id": "10.0.0.5:17912", "dir": "10-0-0-5-17912-8a05db98",
"timezone": "Asia/Shanghai"}]
},
"groups": [{
"name": "sw_metric", "catalog": "MEASURE", "indexMode": false,
"stages": [{"name": "hot", "shardNum": 2, "segmentInterval": "1d", "ttl": "7d", "replicas": 1}]
}],
"units": [{
"catalog": "MEASURE", "group": "sw_metric", "stage": "hot", "segment": "20260907",
"segmentVersion": "1.5.0",
"minTimestamp": 1788739200000000000, "maxTimestamp": 1788825599999999999,
"sources": [{
"node": "10.0.0.5:17912",
"shards": [{
"shard": 0,
"minTimestamp": 1788739200000000000, "maxTimestamp": 1788825599999999999,
"totalCount": 123456,
"parts": [
{"id": 1, "minTimestamp": 1788739200000000000,
"maxTimestamp": 1788760799999999999, "totalCount": 54321},
{"id": 2, "minTimestamp": 1788760800000000000,
"maxTimestamp": 1788825599999999999, "totalCount": 69135}
],
"rowCounts": {"service_cpm_minute": 100000, "service_resp_time": 23456},
"estimatedCompressedBytes": 10737418,
"estimatedUncompressedBytes": 41943040,
"files": [
{"path": "nodes/10-0-0-5-17912-8a05db98/measure/sw_metric/seg-20260907/shard-0-00001.csv.gz",
"bytes": 536871012, "crc32": "0x9f2a1b34"},
{"path": "nodes/10-0-0-5-17912-8a05db98/measure/sw_metric/seg-20260907/shard-0-00002.csv.gz",
"bytes": 11834400, "crc32": "0x77e0aa15"}
]
}]
}]
}],
"schemaFiles": [{"path": "schema/group.jsonl", "bytes": 812, "crc32": "0x05f5e0fe"}],
"warnings": []
}
stageslists the group-level options first, namedhot(the default tier), then every declared lifecycle stage;stages[0]is always the group’s own opts, and one lifecycle stage may also be namedhot. A stage records its ownshardNum,replicas(0means no replicas) andsegmentInterval; itsttlis the stage’s own, not the cumulative effective TTL.indexModeis true when the group holds an index-mode measure.- A segment unit is keyed by
(catalog, group, stage, segment)and holds one source per node that exported it. A property unit’s key degrades to(catalog, group): every node’s shards of the group are sources of one unit, whosestageis the stage of its first source (in node-id order). Property units have nosegmentand no time range; their shards carrydocCount(the Plan’s document count) and an emptyparts. An index-mode measure unit has its rows insegmentFilesonly and no shards; its time range is the one the series index records (SidxStat.min_timestamp/max_timestamp), and is zero only when the index is missing or unflushed. compressisgzipornone; an unspecified compression is the CSV defaultgzip.- In a CSV artifact, a shard’s
totalCountandrowCountscount the rows actually written to its files, per resource, as the data node reported each file infile_finished.row_counts; the Plan’s statistics only feedparts[]and theestimated*sizes. Files carry no row counts of their own:--verifyand the multiple-source judgement only use the totals, and a file’s integrity is itsbytesandcrc32. - An index-mode measure’s rows are segment-level: its files are listed in the source’s
segmentFilesrather than under a shard, and the source carries theirrowCountsandtotalCount. schemaOnly: truemarks a--schema-onlyartifact, which has no units.warningscollects every unit warning, prefixed with the node and unit, plus the stage warnings of the dry run.
A native export writes the same layout with "format": "native" and "compress": "none". A shard’s totalCount is the Plan’s aggregate ShardStat.total_count (split pieces already merged), its parts come from the Plan’s part statistics and its rowCounts is empty. A segment source’s totalCount is the Plan’s SidxStat.doc_count, computed only when the group declares an index-mode measure: an upper bound of the index-mode rows, since it also counts the series entries of the group’s regular measures. The unit’s minTimestamp/maxTimestamp also cover SidxStat.min_timestamp/max_timestamp, the range the series index records for those rows (both formats), so an index-mode unit without shards still has a time range. A property shard’s totalCount is its doc_count, which includes superseded versions and tombstones. Every container volume carries entryCount (required, above 0; the import refuses a volume without one), and the segment-level files sit in the source’s segmentFiles (abridged):
{
"artifactVersion": 1, "format": "native", "compress": "none",
"units": [{
"catalog": "MEASURE", "group": "sw_metric", "stage": "hot", "segment": "20260907",
"sources": [{
"node": "10.0.0.5:17912",
"segmentFiles": [
{"path": "nodes/10-0-0-5-17912-8a05db98/measure/sw_metric/seg-20260907/metadata",
"bytes": 24, "crc32": "0x1c2d3e4f"},
{"path": "nodes/10-0-0-5-17912-8a05db98/measure/sw_metric/seg-20260907/sidx-00001.bnc",
"bytes": 1048576, "crc32": "0x5a6b7c8d", "entryCount": 12}
],
"shards": [{
"shard": 0, "totalCount": 123456, "parts": [{"id": 1, "totalCount": 123456}],
"rowCounts": {},
"files": [
{"path": "nodes/10-0-0-5-17912-8a05db98/measure/sw_metric/seg-20260907/shard-0-00001.bnc",
"bytes": 536870912, "crc32": "0x9f2a1b34", "entryCount": 48}
]
}]
}]
}]
}
A native export records a segment’s raw segment-level files (metadata, sidx-NNNNN.bnc) in the source’s segmentFiles, and every container volume carries entryCount, the footer entry count of that volume (a piece of a split file counts as one). sidxCoversShards is not recorded: the import derives series-index targets from the unit’s shards, see Native import as implemented.
shards[].parts[] is written unconditionally, for every shard of every source. The export side does not decide which shards need it: that array is the input to the import-side multi-source judgement (Detection basis), and a shard that looks single-sourced today becomes multi-sourced as soon as a second artifact is merged in.
As implemented:
sidxCoversShardsbelow is the original design and is not written to the manifest; the import sends a segment’s series index to the union of the nodes of the unit’s shards, or to every node of the target stage when the unit has none (Native import as implemented).
sidxCoversShards (native only) is the union of the route-derived shard set and the shard directories actually present on disk. It is used only to work out which nodes must receive this segment’s series index (Data transfer); it never selects which sources to push — that follows the core-part selection (Multiple-source push behavior). Both halves of the union are required:
- Using only the disk fails for index-mode measure, which has no shard directories at all even though the segment-level sidx is its only payload.
- Using only the routing table fails after topology drift: a node can route-own shard
[1]while physically holding shard[3], with an empty intersection. - The union costs nothing. Over-reporting a shard opens one extra stream whose bytes are absorbed by receiver-side deduplication; under-reporting makes
Lookup(series)miss and renders the whole segment unqueryable.
4.6 Protocol
A new gRPC export service carries the workflow. The data node generates the stream directly and the client writes it to local files. The same service backs dry run.
The listings below are illustrative. Field numbers, validation rules and comments are authoritative in
api/proto/banyandb/transfer/v1/export.proto.
service ExportService {
// Plan enumerates the per-segment units (per-group for property) a selector hits.
// Its behavior is determined by the request fields.
rpc Plan(PlanRequest) returns (stream PlanResponse);
// Sessions runs one lifecycle action (list, heartbeat, release) on every data node
// and streams one frame per node, stamped with node_id by the liaison.
rpc Sessions(SessionsRequest) returns (stream SessionsResponse);
// Export streams the files of one data node's units out of an export session
// snapshot. The first request frame is init; every later frame carries exactly one
// unit. The node produces units one at a time in request order, and the client may
// send the next unit before unit_finished for the previous one. The liaison forwards the
// stream to init.target_node unchanged. Export neither creates nor removes the snapshot.
rpc Export(stream ExportRequest) returns (stream ExportResponse);
}
message Selector {
common.v1.Catalog catalog = 1;
repeated string groups = 2;
}
// SegmentUnit is the key of a physical unit of a stream, measure or trace group.
message SegmentUnit {
common.v1.Catalog catalog = 1; // never PROPERTY
string group = 2;
// Example: "20260907" (day) or "2026090712" (hour).
string segment_suffix = 3;
repeated uint32 shard_ids = 4;
}
// UnitInventory is one unit returned by Plan. Property groups have no segments, so they
// are inventoried per group instead of per segment.
message UnitInventory {
oneof kind {
SegmentInventory segment = 1;
PropertyInventory property = 2;
}
}
// SegmentInventory is a SegmentUnit with the statistics calculated by the server.
message SegmentInventory {
SegmentUnit unit = 1;
// Version from the segment root metadata file.
// Used as the version gate during import.
string segment_version = 2;
// Per-shard statistics.
repeated ShardStat shards = 3;
// Segment-level series index (the sidx/ directory).
SidxStat segment_level = 4;
}
// PropertyInventory is one property group on one node: its shards, each a document store
// without parts or timestamps.
message PropertyInventory {
string group = 1;
repeated PropertyShardStat shards = 2;
}
message PropertyShardStat {
uint32 shard_id = 1;
// On-disk size of the shard's document store: property has no compressed form, so this
// is both the compressed and the uncompressed estimate.
uint64 estimated_bytes = 2;
// Documents in the store, including superseded versions and tombstones.
uint64 doc_count = 3;
}
// SidxStat describes the segment-level series index.
message SidxStat {
uint64 estimated_bytes = 1;
// Documents of the series index, computed only for groups that hold an index-mode measure
// (0 otherwise). Those documents are the index-mode rows plus the series entries of the
// group's regular measures, so the value is an upper bound of the rows.
uint64 doc_count = 2;
// "_timestamp" range the index's segment footers record, read alongside doc_count; also a
// superset (deleted documents and series entries count too). 0 when not computed or absent.
int64 min_timestamp = 3;
int64 max_timestamp = 4;
}
// ShardStat aggregates the parts of one shard. For trace, estimated_uncompressed_bytes
// counts the span payload only (tags are excluded), so it can be below
// estimated_compressed_bytes; estimated_compressed_bytes is the bytes the part writer
// recorded and is a lower bound of the on-disk size.
//
// A shard whose part list is too large for one frame arrives as several ShardStat pieces
// with the same shard_id and disjoint parts. Only the first piece carries the aggregates
// (fields 2-7); the others set only shard_id and parts.
message ShardStat {
uint32 shard_id = 1;
// Full time range of this shard in UnixNano.
int64 min_timestamp = 2;
int64 max_timestamp = 3;
uint64 estimated_compressed_bytes = 4;
uint64 estimated_uncompressed_bytes = 5;
uint32 parts_count = 6;
uint64 total_count = 7;
repeated PartStat parts = 8;
}
message PartStat {
uint64 id = 1;
int64 min_timestamp = 2;
int64 max_timestamp = 3;
uint64 total_count = 4;
}
// SessionsRequest runs one session lifecycle action on every data node; the liaison
// streams back one SessionsResponse per node.
message SessionsRequest {
enum Action {
ACTION_UNSPECIFIED = 0;
// Read-only: every node reports the session it holds (the newest) or none. An unreachable node
// fails the whole call, so "clean" and "silent" cannot look alike.
ACTION_LIST = 1;
// The heartbeat: renew the lease of session_id on every node.
ACTION_HEARTBEAT = 2;
// Delete the snapshot of session_id on every node; a node that does not hold it still
// succeeds (answering `none`), so the action is idempotent.
ACTION_RELEASE = 3;
}
Action action = 1;
// Required by ACTION_HEARTBEAT and ACTION_RELEASE, ignored by ACTION_LIST.
string session_id = 2;
}
// One frame per data node; node_id is stamped by the liaison. Exactly one outcome is set.
message SessionsResponse {
string node_id = 1;
oneof outcome {
// ACTION_LIST: the session this node holds. One session id covers the whole cluster, so
// a node reports at most one: the newest when leftovers of older sessions exist.
SessionLease session = 2;
// ACTION_LIST: the node holds no session. ACTION_RELEASE: the node did not hold
// session_id, so there was nothing to delete; this is still a success.
Ack none = 3;
// ACTION_HEARTBEAT: the node renewed the lease. ACTION_RELEASE: the node held the session
// and deleted it.
Ack done = 4;
// ACTION_HEARTBEAT and ACTION_RELEASE: the node did not complete the action; the frame is
// still sent so the client can tell which nodes failed.
// ACTION_HEARTBEAT: the node was unreachable, does not hold the session, or is a data
// node without ExportService. An expired or unreadable lease, or any other node error,
// fails the whole call instead (FAILED_PRECONDITION for the lease).
// ACTION_RELEASE: the node was unreachable, removing the session failed, or it is a data
// node without ExportService. A node that does not hold the session answers `none`.
string error = 5;
}
}
// Ack is an outcome that carries no data.
message Ack {}
// Mirrors the .lease file plus the catalogs, which the node derives from the
// <export-snapshot-path>/<id> directories it finds. expires_at drives reclamation by the
// node's sweeper; last_heartbeat_at tells a refused operator when the holder was last seen.
// Times are Unix nanoseconds of the data node's clock; an unreadable .lease reports zero times.
message SessionLease {
string session_id = 1;
int64 started_at = 2;
int64 expires_at = 3;
int64 last_heartbeat_at = 4;
repeated common.v1.Catalog catalogs = 5;
}
message PlanRequest {
// Empty = all catalogs and groups in the cluster.
repeated Selector selectors = 1;
// Unset = inventory the online directories (a dry run).
oneof session {
CreateSession create = 2; // snapshot first, then inventory that snapshot
ReadSession read = 3; // read the session's snapshot and renew its lease
}
}
message CreateSession {
// Empty at the liaison, which generates the id and forwards it; always set on a data node.
string id = 1;
// Remove every session the node holds, whatever its lease says, before creating this
// one; the removed ids come back in preempted_session_ids. Without it any other session
// on the node answers ALREADY_EXISTS naming the occupants. The client sets it for
// --preempt.
bool preempt = 2;
}
message ReadSession {
string id = 1; // required
}
// One frame of the Plan stream. Through the liaison the stream is: with CreateSession a
// `created` frame first; then `units` frames, each carrying units of exactly one
// (node, group); then exactly one `summary` frame. A data node's own stream follows the same order without `created`: it
// never sets node_id or answered_nodes, and its closing `summary`, carrying only
// preempted_session_ids, is sent only with CreateSession when anything was removed.
message PlanResponse {
oneof frame {
SessionCreated created = 1;
UnitFrame units = 2;
PlanSummary summary = 3;
}
}
message SessionCreated {
string session_id = 1;
}
message UnitFrame {
string node_id = 1; // stamped by the liaison
// The node's resolved stage for this group; empty for the default tier (no stages, no
// node labels, or no matching stage). Only groups the registry knows are listed.
string stage = 2;
repeated UnitInventory units = 3;
}
message PlanSummary {
// Sessions removed while creating because preempt was set.
repeated string preempted_session_ids = 1;
// Live directories and ReadSession only: transport failure, liaison Plan max wait,
// NotFound (no longer holds the session), or a data node without ExportService.
// CreateSession fails on any of these instead.
repeated string unreachable_nodes = 2;
// Nodes whose Plan stream completed; with unreachable_nodes it covers every planned node.
repeated string answered_nodes = 3;
}
enum Format {
FORMAT_UNSPECIFIED = 0;
FORMAT_NATIVE = 1; // .bnc containers, byte for byte; never compressed (ImportService.ImportParts imports it)
FORMAT_CSV = 2;
}
enum Compress {
COMPRESS_UNSPECIFIED = 0; // refused: the client always names one
COMPRESS_NONE = 1;
COMPRESS_GZIP = 2;
}
// In transfer/v1/common.proto, shared by Export and ImportRows; 0 = unlimited. Enforced by
// the data node: per Export stream, and for ImportRows on every row the node is sent.
message RateLimit {
uint64 bytes_per_second = 1;
uint64 rows_per_second = 2; // CSV only
}
// Property groups have no segments: the unit is the group's shards on one node.
message PropertyUnit {
string group = 1;
repeated uint32 shard_ids = 2;
}
message ExportRequest {
message Init {
string session_id = 1; // 32 lowercase hex
string target_node = 2; // a node_id from the Plan frames; the liaison routes by it
Format format = 3; // required
Compress compress = 4; // required
uint64 max_file_size = 5; // on-disk bytes, 0 = no split; CSV: soft (rolls after a row), NATIVE: hard cap per .bnc volume, 0 or >= 64 KiB
RateLimit rate_limit = 6;
bool csv_annotations = 7; // CSV only: a #datatype line before every header
uint64 liaison_bytes_per_second = 8; // the relaying liaison's process-wide rate; last stream wins
}
oneof frame { // required
Init init = 1; // first frame only
SegmentUnit segment = 2; // stream, measure, trace: SegmentInventory.unit, pieces of a split segment merged
PropertyUnit property = 3;
}
}
// Order: ready, then per unit: unit_started, (file_started, chunk*, file_finished)*, unit_finished.
message ExportResponse {
oneof frame {
Ready ready = 1;
UnitStarted unit_started = 2;
FileStarted file_started = 3;
DataChunk chunk = 4;
FileFinished file_finished = 5;
UnitFinished unit_finished = 6;
}
}
// The node located the session and renewed its lease; units may follow.
message Ready {}
message UnitStarted {
uint32 total_shards = 1; // 0 for segment-level-only units
uint32 total_parts = 4; // live parts of the unit's shards; 0 without parts
}
message FileStarted {
string relative_path = 1; // relative to nodes/<node-dir>/
uint32 shard_id = 2;
bool segment_level = 3; // CSV index-mode measure rows; native the segment metadata file and sidx volumes
uint32 shard_index = 4; // index of shard_id in the unit's shard list
uint32 seq = 5; // from 1: CSV split number within the shard; native volume number within its container
}
message DataChunk {
bytes content = 1; // at most 1 MiB
uint64 rows = 2; // CSV only: rows written into the open file so far (progress only); native sends 0
}
message FileFinished {
uint64 bytes = 1; // on-disk size
uint32 crc32 = 2; // IEEE, over the on-disk bytes
uint32 entry_count = 3; // native only; CSV sends 0
map<string, uint64> row_counts = 4; // CSV only: rows per resource in this file; native sends none
map<string, int64> schema_mod_revisions = 5; // CSV: schema revision of every resource in the layout
}
message UnitFinished {
repeated string warnings = 1;
}
4.7 Full workflow
4.7.1 Phase 1 — Plan: create session and inventory
| # | Direction | Frame | Liaison behavior |
|---|---|---|---|
| 1 | ctl → liaison | Plan{selectors=[…], create{}} |
Generate one session id for the whole fan-out. A node already holding a different session rejects the create and returns the incumbent’s ID (why session uniqueness is per node); any rejection or failure makes the liaison roll the new id back on the nodes that accepted it; nodes whose rollback failed are named in the error as leaked, and only those ids need a manual bydbctl data release-session |
| 2 | liaison → data-1 / data-2 | One Plan{…, create{id="xx…"}} per node |
— |
| 3 | Each data node | Refuse if any other session directory exists (or remove them all when preempt is set), snapshot the selected groups of every selected catalog under <export-snapshot-path>/<id>, then commit the .lease in all of them; inventory from the snapshot |
— |
| 4 | data-* → liaison → ctl | Stream PlanResponse{units{node_id, stage, units[…]}} frames. A data node bounds each frame to 4 MiB, so a large group spans several consecutive frames, and a segment too large for one frame is split into pieces with the same key and disjoint shard_ids (only the first piece carries segment_level). A single shard too large for one frame is split further into ShardStat pieces with the same shard_id and disjoint parts, in different segment pieces; only the first carries the aggregates. ctl merges the pieces back into one unit per (node, segment) as it collects the frames, concatenating each shard’s parts and adding up its aggregates |
Forward each frame as it arrives, stamping node_id, through one bounded queue drained by a single sender, so a client that stops reading cannot hold back a failed create’s rollback; the sender also stops when the call ends, so a create rejected before any unit leaves nothing running; fold a node’s summary (preempted ids) into the final one |
| 5 | ctl | Print the session_id (from the created frame) as soon as it arrives; keep the unit inventory in memory |
— |
| 6 | ctl | Group entries by node_id into one job per node, each with its own nodes/<node-dir>/ subtree |
— |
Only this phase creates snapshots; they are created once and reused for the whole export. The liaison stamps node_id on every frame from the upstream connection and does not trust a node’s self-report.
4.7.2 Phase 2 — Export: transfer unit by unit
Each data node uses one stream. The example shows data-1; the other nodes run in parallel with the same structure, up to --parallelism at a time.
| # | Direction | Frame | Liaison behavior |
|---|---|---|---|
| 1 | ctl → liaison | Init{session_id="9f2a…", target_node="data-1", format=CSV, compress=GZIP, max_file_size=512MiB, rate_limit{…}, csv_annotations, liaison_bytes_per_second} |
Read target_node; a node that is not registered answers NOT_FOUND. Apply liaison_bytes_per_second to the process-wide bucket (last stream wins). Take the pooled connection and open an upstream bidirectional Export stream |
| 2 | liaison → data-1 | Init |
Forward unchanged, without modifying any field. Both directions are pumped without prefetching, so back-pressure reaches the node |
| 3 | data-1 | Validate Init (INVALID_ARGUMENT if the first frame is not one); answer UNIMPLEMENTED for an unknown format, and INVALID_ARGUMENT for a native Init that asks for compression (anything but COMPRESS_NONE) or a non-zero max_file_size below the BNC minimum; locate <export-snapshot-path>/9f2a… by session_id and renew the lease (FAILED_PRECONDITION when it is gone or expired; NOT_FOUND is kept for a liaison that cannot see the node, which the client retries through another liaison) |
Forward errors unchanged |
| 4 | data-1 → liaison → ctl | ready{} |
Forward |
| 5 | ctl → … → data-1 | segment{catalog=STREAM, group="sw_record", segment_suffix="20260907", shard_ids=[0,1]} |
Forward. The node produces one unit at a time; ctl keeps the next request queued |
| 6 | data-1 | Renew the lease again, then check that the segment and both shard directories exist in the snapshot | — |
| 7 | data-1 → liaison → ctl | unit_started{total_shards=2, total_parts=5} |
Forward |
| 8 | data-1 → liaison → ctl | file_started{relative_path="stream/sw_record/seg-20260907/shard-0-00001.csv.gz", shard_id=0, seq=1} → chunk × N → file_finished{bytes, crc32, row_counts, schema_mod_revisions} |
Forward; each chunk waits on the liaison bucket first, and one larger than its share of the bucket’s burst goes out in pieces of that share. First of 2 shards; a shard larger than max_file_size continues in shard-0-00002.csv.gz |
| 9 | data-1 → liaison → ctl | file_started{…/shard-1-00001.csv.gz} → chunk × N → file_finished{…} |
Forward. Second of 2 shards |
| 10 | data-1 → liaison → ctl | unit_finished{warnings=[…]} |
Forward |
| 11 | ctl | Keep the unit’s files and warnings in memory for the manifest. Throughout steps 7–10 ctl prints the progress from the frames: the unit and its part count from unit_started, the file and its shard from file_started, the bytes and the rows of every chunk, and a line with the time of every finished file and unit; on a terminal one live line per node is redrawn in place |
— |
| 12 | ctl → … → data-1 | segment{…, segment_suffix="20260908", shard_ids=[0,1]} or property{group="sw_prop", shard_ids=[0]} |
Forward and repeat steps 6–10. ctl sends the next unit before the current one’s unit_finished arrives, so the node starts it without a round trip; the node still produces units one at a time in request order |
| 13 | ctl → liaison | CloseSend |
Half-close upstream |
| 14 | data-1 → … → ctl | Response stream ends normally | — |
An index-mode measure unit sends one file_started{relative_path=".../seg-20260907/segment-00001.csv.gz", segment_level=true} instead of per-shard files; a property unit sends property/<group>/shard-<N>-00001.csv.gz files.
On the client every frame is checked against that order; anything else fails the node. Two timers guard a stream:
--node-idle-timeout(default10m) fails a node that delivers nothing for that long; it bounds the gap between two frames, not the length of the export. Time between open retries does not count. Throttling does not trip it: a throttled node sends a chunk about every second, and a throttled liaison relays a chunk in pieces of one second’s worth.0disables it.--send-retry-timeout(default15m) bounds how long the client keeps reopening a stream that the liaison or node refused withRESOURCE_EXHAUSTEDorUNAVAILABLE, or that a liaison answeredNOT_FOUNDbecause it does not see the target node; every retry moves to the next liaison in--nodes, backing off from 1 s, doubling up to 30 s. Any other refusal fails at once, and so does a stream that breaks after it started transferring.
4.7.3 Phase 3 — Finalization
| # | Direction | Frame | Liaison behavior |
|---|---|---|---|
| 1 | ctl | Write manifest.json atomically, once every node has finished |
— |
| 2 | ctl → liaison | Sessions{action=ACTION_RELEASE, session_id="9f2a…"} |
Fan out to all data nodes |
| 3 | Each data node | Delete <export-snapshot-path>/xx…; if already absent, still return success for idempotency |
One frame per node; a node that failed carries error |
A frame with a non-empty error, or a failed release call, is a WARN, not a failure: the artifact is already sealed and valid, so the run still exits 0. The CLI prints one WARN release of export session <id> failed on <node>: <reason> line per failed node; run bydbctl data release-session --id <id> to retry, because until those snapshots go the nodes carry them for the rest of the lease — and on a hot node a snapshot costs roughly a second copy of the live data, not the free hard links the cold case suggests.
4.8 Session management
Each export uses one session, corresponding to one snapshot per data node. One export should own the whole cluster at a time — but that is an outcome the protocol converges to, not an invariant any single component enforces, and the difference matters for implementers.
On disk a session is one directory per catalog, <export-snapshot-path>/<session-id>/ (<catalog>/export-snapshots/<id> under the catalog root by default), where <session-id> is exactly 32 lowercase hex characters (the sweeper treats no other directory name as a session), holding the snapshot of every selected group as <group>/… plus the .lease file (hard links for stream, measure and trace; a full copy of the property index for property, see below); property sessions use the same <group>/shard-N layout, without the extra data/ level its backup snapshots keep. The directory is set per catalog with --stream-export-snapshot-path, --measure-export-snapshot-path, --trace-export-snapshot-path and --property-export-snapshot-path, and for stream, measure and trace must be on the same filesystem as the data path because their snapshots are hard links; the property directory instead needs as much free space as the property data. The export service refuses to start when one of these paths equals or lies inside any catalog’s backup snapshot directory, equals, contains or lies inside any catalog’s data path, or when two catalogs’ paths are equal or nested. These paths are compared with their symlinks resolved (through the longest existing ancestor of a path not created yet), so an alias of a live data directory is refused too, and a path that cannot be resolved refuses to start. It is deliberately not the snapshots/ directory the backup snapshots live in: the snapshot reclaimers (DeleteStaleSnapshots, DeleteOldSnapshots) only ever see that directory, so a multi-day export needs no special case in them, and the session sweeper (step 5 below) only ever sees sessions.
4.8.1 There is no coordinator, so uniqueness is per node
BanyanDB has no etcd, no lock service, and no CAS primitive anywhere in the tree; node discovery is file-, DNS- or flag-based. A session therefore cannot be claimed atomically cluster-wide before fan-out. The only real enforcement point is each data node’s own export snapshot directory.
That leaves a genuine race. Two clients calling Plan{create{}} through two different liaisons can each win on a different subset of data nodes, producing two partial sessions that block each other. Left unhandled they would deadlock each other. The protocol resolves it without a coordinator:
- A node accepts a create only if it holds no other session. The rule is purely structural: if any other session directory exists on the node, whatever its
.leasesays, the node rejects the create and returns the incumbent’s ID(s) rather than a bare error. The node takes no liveness decision — the sweeper reclaims expired sessions, the operator reclaims abandoned ones with--preempt. - Any rejection aborts the whole attempt. When even one node rejects, the liaison releases the new ID on every node that accepted it (
Sessions(ACTION_RELEASE)semantics, over the nodes the create may have reached) and fails the call with a concurrent-export error naming the incumbent IDs. The client releases by hand only the ids the error reports as leaked. - Retry with jittered backoff. Because the loser releases everything it took, the next attempt finds a clean subset. Two clients retrying in lockstep is the only livelock risk, and jitter bounds it.
Step 2 is the load-bearing one: a partial session kept after a partial win is what creates the deadlock. Releasing is cheap — deleting a snapshot only unlinks files (hard links for stream, measure and trace; the property copy’s own files).
Sessions(ACTION_LIST) streams one frame per node precisely so this state is observable. A list that comes back with two distinct session_id values, or with a session on only some nodes, is reporting a partial session, not a healthy one. The client surfaces that verbatim in the refusal message; --preempt in that state removes every session the node holds before creating its own, rather than taking over one of them.
A data node that joins the cluster after the session was created holds no snapshot for it. It answers NOT_FOUND to a Plan{read{id}}, which the liaison reports in unreachable_nodes rather than failing the call, and it answers its ACTION_HEARTBEAT frame with the error outcome, so the client sees it as a coverage gap either way. An expired session is different: until the sweeper removes it, the node answers FAILED_PRECONDITION and the call fails; once it is swept the node answers NOT_FOUND like any node without the session.
Beyond the race, a partial session left behind by a client that died mid-abort is reclaimed by the node-side expiry sweep (step 5 below) within one sweep period of the lease expiring, with no operator action. Until then it blocks the next export, which is told whose session it is and when that session was last seen; the operator decides whether to --preempt it.
4.8.2 Taking over an existing session
Starting a new export never silently removes another session, and no component classifies a session as alive or dead. A session is held until it is released, preempted, or expires (5 days after its last heartbeat) and gets swept:
- Any holder: the node answers
ALREADY_EXISTS; the client then lists the sessions once and prints who holds it, where, and when its last heartbeat was, with the hint that--preempttakes it over. The last-heartbeat time is what tells the operator whether the holder is still running (heartbeats arrive every 5 minutes) or died hours ago.--preemptis a confirmation action, not a tuning knob. An operator at a terminal (stdin and stdout both terminals) is asked instead,[y/N], and a yes is the same confirmation; a script or a pipe is never asked and gets the refusal. --preempt: the node removes every session it holds before creating the new one and reports the removed ids inpreempted_session_ids, which the client prints as a WARN. Nothing else ever sets that field.
Earlier revisions (2026-10-07 to 2026-10-09) had the data node reclaim a holder whose last heartbeat was older than 15 minutes without --preempt. That classification was dropped on 2026-10-09: a heartbeat lost to a network partition must not cost a running export its snapshot, and the heartbeat time shown to the refused operator carries the same information without the node guessing.
Lifecycle:
- Create — the liaison sends
Plan{create{id}}to all data nodes. - Export — export requests carry the
session_idto read the matching snapshot. - Heartbeat —
bydbctlsendsSessions{action=ACTION_HEARTBEAT, session_id}on a fixed interval. - Release — after the export completes and
manifest.jsonis written,bydbctlcallsSessions{action=ACTION_RELEASE, session_id}. On a fail-fast abort it deliberately does not: the session is kept, and the client prints the session ID plus thebydbctl data release-session --id <id>command; Ctrl-C andSIGTERMstop the export the same way and print the same command. An interrupted export is re-run into a new, empty directory. There is no flag for “release on abort”:release-sessioncovers it, and the lease sweep reclaims a forgotten session. - Cleanup — each data node sweeps export snapshots on a one-minute timer and removes any whose lease has expired. A session directory left without a readable lease (a crash mid-create or a failed rollback) is reclaimed once its newest directory is older than a 10-minute orphan grace; until then it still counts as an occupant, so a create without
preemptanswersALREADY_EXISTS. The first sweep runs as soon as the export service starts serving, so after a data-node restart an expired session is reclaimed at once and a live one survives.
| Constant | Value | Why it is not configurable |
|---|---|---|
| Lease | 5 days | A data-node constant: every heartbeat resets the expiry to now + 5 days, so it only decides how long after the client disappears the snapshot is reclaimed. The request used to carry a lease with a 30-day cap; both were removed because the task length is irrelevant |
| Heartbeat interval | 5 min | A client-side timer, deliberately not derived from the lease. It keeps the lease fresh and stamps last_heartbeat_at, which a refused operator reads to judge whether the holder is still running |
The body of a snapshot is hard links, so it copies zero bytes, and deletion is unlink only — a background merge that removes a source part during export leaves the snapshot’s link fully readable, which is what makes lock-free reading safe. “Hard links, therefore free” holds only for cold segments, though: the open segment’s series index and the whole property database are full byte copies (the property snapshot is the index’s own backup, written shard by shard, and also holds documents not yet flushed). The snapshot holds what was flushed to disk when it was taken; rows still in a memtable are not in it, so wait one flush interval after stopping writers for a complete export. Creating a session can be canceled only between groups and shards: the snapshot of a single group or shard is not interruptible. A large property copy can therefore outlast the liaison’s per-frame wait for a node (5 minutes); the create then fails and is rolled back on every node.
The data-node gRPC port has no authentication of its own; the export RPCs are gated by RBAC only on the liaison. Deployments must keep the data-node port on an isolated network or behind mTLS, as they already must for the snapshot and segment-deletion RPCs on that port.
4.9 Remaining work
The CSV export step (2026-10-10) completes the export path for CSV: the Export RPC on the liaison (a directed relay, also in-process for a standalone server) and on the data node, the four CSV producers, the client session library (create with retries, refusal rendering, heartbeat, release), the receive loop, the schema files, the manifest and the bydbctl data export flags. What is still to come, each with where its design lives in this document:
--archive tar|tar.gz— packing the artifact into one file.- Native format — landed with the native step (2026-11-03):
.bnccontainers, consumed only bydata import(there is no offline extract command) (Native export, Native import as implemented). - Import — the CSV import (including property) and the cross-cluster e2e landed with the CSV import step (CSV import as implemented), the native import with the native step.
- Deferred design items — a dedicated
bydbctl data sessionslisting command (declined for now: the refusal text already renders the occupant), probe-before-create, pinning the session’s node membership, skipping zombie nodes under file discovery, and a stage manifest inside the snapshot.
The end-to-end cases test/e2e-v2/cases/transfer/csv/ and test/e2e-v2/cases/transfer/native/ run the same cases, once per format, against one hot and one warm data node behind a liaison: the dry run, release-session, the export of each catalog, and the import into a second, empty cluster with its refusals and round trip. Their environment and scripts are shared in test/e2e-v2/cases/transfer/common/.
4.10 Node gRPC extension
The node response must expose supported file versions and timezone information.
message Node { // api/proto/banyandb/database/v1/database.proto
// ... // existing fields 1-9
banyandb.cluster.v1.VersionInfo version = 10;
string tz_name = 11;
}
5. Import
5.1 Workflow
The workflow below is the original, native-aware design. The CSV import as implemented runs the steps in 5.4.1: its checks are the schema and TTL gates only, and rows are re-routed by the target’s own topology, so the topology gate (item 7) does not apply to CSV.
- Connection check — call
GetCurrentNodeon each node. - Artifact check — validate
manifest.json, the artifact version, and every file’s CRC32. - Schema check — compare the artifact’s schemas with the target cluster’s. Two resources compare equal unless they differ outside the ignore set
created_at,updated_atandMetadata.{id, create_revision, mod_revision};created_atalways differs across clusters, so counting it would make every import aCONFLICT.indexRule’sMetadata.idis the exception — it is materialised into the inverted files, so the native channel compares it out of band and refuses a mismatch. - Version compatibility — for CSV, check the locally generated version against the data nodes; for native, check each shard’s part version.
- Timezone check — for native, the target node timezone must match the artifact’s, or the import fails.
- TTL check — skip data that has already expired.
- Topology check — the target stage’s
shardNumandsegmentIntervalmust equal the artifact’s; a mismatch rejects the unit for native. The default stage is always written ashot: the server reports it as an empty string, and the client normalises that so the default tier has a stable name in the manifest and the gates. A unit is routed to the target stage of the same name — both sides’ stage definitions are already known, from the artifact’sgroups[].stages[]and the target schema, so no mapping is configured — and a unit whose stage the target does not have falls back tohotand is re-checked against that stage — routing with the wrong stage’s shard number writes shard directories thatloadShardsskips on restart, so the data is written, is queryable, and vanishes on the next restart. Equality also guarantees that no source shard ID reaches or exceeds the target’sshardNum.LocateAllreturns<group>-<N> is a unknown shardfor such a shard, so the data is simply unroutable. This must be listed and blocked during dry run, not discovered halfway through a push. Topology drift frequently arrives together with ashardNumchange, so the two conditions co-occur. - Schema import — create or overwrite per policy, then rerun checks 2–7.
- Data import — per data type, as below.
Two routing failures must be distinguished: zero nodes returns nil plus no nodes available, while an out-of-range shard returns ... is a unknown shard. Separately, LocateAll deduplicates its result set, so it can silently return fewer targets than the requested copies with no error, log, or warning. The import side must compare len(targets) against the requested copies itself; --strict-replicas cannot use len(targets) as its denominator. A shortfall is a WARN by default; --strict-replicas promotes it to an error that aborts the run.
Stage routing. ImportService places every (group, stage) with the cluster’s own node selection: the hot stage with the data node selector the liaison’s write queue syncs online writes with, any other stage with the round-robin selector a lifecycle migration uses for it; see Stage-aware routing. Multi-stage CSV import is therefore supported. The paragraph above about LocateAll and --strict-replicas describes the native design; the CSV import writes to every available distinct node when a stage has fewer than copies nodes and reports the actual count.
5.2 Native fast push
Fast push is available only when all of the following hold:
- The artifact format is native.
- The target shard number equals the source shard number after stage mapping, and both the stage-level
shardNumand the group-levelResourceOpts.ShardNummatch. Comparing only the former lets a unit through the gate and then fails in the liaison’sPickwithunknown shard; comparing only the latter gets it silently dropped byloadShards. - The segment interval and timezone are exactly the same.
- The segment version is in the supported compatibility list.
- The schema is either created successfully or already consistent.
5.2.1 Protocol
The messages are defined in api/proto/banyandb/transfer/v1/import.proto (API reference), which is authoritative. One ImportParts stream carries one payload, ImportPartsRequest.Kind: a part with its trace secondary index parts (KIND_PART), a segment’s series index (KIND_SERIES_INDEX), a stream shard’s element index (KIND_ELEMENT_INDEX) or a property shard (KIND_PROPERTY). The client names the core part with an empty part_type, which the liaison relays as "core" on the cluster wire; routing uses the stage route of 5.4.3. requested_copies is per shard (replicas + 1); for a series index, whose targets are the union of several shards’ nodes, it is not comparable with len(targets). The semantics as implemented are in 5.5; the rest of this section records the design reasoning.
Duplicate-delivery validation must be split by ImportPartsRequest.Kind. (id, part_type) uniqueness can only be enforced on PART streams: index streams carry fixed zero values with id = 0 and an empty part_type, so applying the same rule there — or trying to tell them apart by part_type — makes the receiver reject the stream with a does not support part type error.
The upstream id is a framing signal, not an identifier. The receiver always allocates its own part ID with atomic.AddUint64(&tsTable.curPartID, 1), and fillFromSyncContext copies six scalars but deliberately not the ID. Its only real use is the receiver’s createNewContext := session.partCtx == nil || session.partCtx.ID != partInfo.Id check, which decides where one part ends and the next begins. Mixing two part IDs in one stream therefore causes a split plus an orphan deletion (MustRMAll), not a silent merge.
5.2.2 Transfer flow
| # | bydbctl → liaison (import/v1) | liaison → data-* (cluster.v1) | Liaison behavior |
|---|---|---|---|
| ⓪ | Read the manifest and target schema/topology | — | — |
| ① | Open every volume of each container as one volume set, group its files by part and split into streams by part. A part — even a single file — may span volumes, so streams are cut on part boundaries, never on volume boundaries | — | — |
| ② | Init{catalog, group, shard_id, stage, kind, segment_version, parts{…}} |
— | Validate, route with the stage route of 5.4.3, open SyncPart streams |
| ③ | The first Chunk |
SyncPartRequest{chunk_index=0, metadata=SyncMetadata{…}, …} |
Build SyncMetadata, fill session_id, and assemble version_info: file_format_version is copied from the client’s Init.segment_version — it is the guard on the source data — while api_version and compatible_file_format_version are the liaison’s own. A liaison that fills file_format_version with its own value turns that guard into a tautology |
| ④ | Chunk frames |
SyncPartRequest frames × N |
Forward to all N target nodes |
| ⑤ | Completion |
SyncPartRequest{completion=SyncCompletion{…}} |
Forward |
| ⑥ | ImportPartsResponse{requested_copies, targets[]} |
Final responses from all streams | Wait for all N results before returning |
The fan-out must be concurrent with independent per-target reporting: one failed target must not cut off the others, and every target reports its own result. Do not copy the fan-out shape of banyand/stream/syncer.go: syncPartsToNodesHelper — it is a sequential loop with a fail-fast return, the opposite of what is needed. banyand/liaison/grpc/property.go: replaceProperty is closer in spirit (concurrent publish, wait for all futures, partial-success verdict), but it goes through bus.Publish rather than a gRPC stream.
Liaison memory stays flat only under a per-frame synchronous barrier — send one frame to all N targets, wait for all N acks, then read the next. An asynchronous or buffered fan-out makes memory grow with the speed gap between the fastest and slowest target.
5.2.3 Build the import plan
This step reads only manifest.json and the target cluster’s topology; it opens no BNC files. It decides which segments qualify for fast push, and which source node to select for each segment when several hold copies (Multiple shard sources).
5.2.4 Read and split BNC files
nodes/data-1/measure/sw_metric/seg-20260907/
metadata ← Raw file, not a container
sidx-00001.bnc ← Contains seg-20260907/sidx/**
shard-0-00001.bnc ← Contains shard-0/**
shard-1-00001.bnc ← Contains shard-1/**
| Artifact file | Container | Source data | Generated streams |
|---|---|---|---|
metadata |
No | seg-XXX/metadata, holding {Version, EndTime} |
None — not sent. The receiver generates it locally when creating the segment |
sidx-NNNNN.bnc |
Yes | seg-XXX/sidx/**, a native index directory |
SERIES_INDEX stream, one per target node (Data transfer) — not one per shard |
shard-N-NNNNN.bnc |
Yes | shard-N/**, <016x>.snp; also idx/, and for trace sidx/<index-name>/<016x>/ |
PART and ELEMENT_INDEX streams |
Pushing runs in three rounds with a global barrier between them: ① segment-level series index, ② core parts, ③ element index. The barrier cannot be per-subtree, because the ordering dependency is per-shard and the same shard routinely appears under several source subtrees.
5.2.5 Data transfer
Segment-level sidx files go first.
Init{
catalog=MEASURE,
group="sw_metric",
shard_id=0,
stage="hot",
kind=SERIES_INDEX,
segment_version="1.5.0",
parts=[{
id=0,
part_type="", ← fixed zero values for index streams
compressed_size_bytes=0,
uncompressed_size_bytes=0,
total_count=0,
blocks_count=0,
min_timestamp=1788739200000000000,
max_timestamp=1788825599999999999,
min_key=0,
max_key=0
}]
}
Footer of sidx-00001.bnc, relative to the segment sidx/ directory:
000000000012.seg offset=0 size=4_194_304
000000000013.seg offset=4_194_304 size=1_048_576
000000000013.snp offset=5_242_880 size=1_024
(a series index is a native index directory — only .seg and .snp, and no metadata.json)
Chunk{
data=<1MiB>,
files=[
{name="000000000012.seg", size=1048576}
]
}
Chunk{ … }
... one frame per 1 MiB ...
Completion{
total_bytes=…,
total_chunks=…
}
The series index is segment-scoped, not shard-scoped, so it is delivered once per target node. <segment>/sidx is a single native index directory shared by every shard in that segment (storage/segment.go builds it from the segment location), and the receive side confirms it: syncSeriesCallback.CreatePartHandler — identical in stream, measure and trace — resolves the destination from ctx.Group and ctx.MinTimestamp only, calls CreateSegmentIfNotExist, and never reads ctx.ShardID. Compare the element index, whose handler does exactly the opposite: syncElementIndexCallback.CreatePartHandler calls segment.CreateTSTableIfNotExist(common.ShardID(ctx.ShardID)).
So sidxCoversShards has one job only: resolving which nodes need this index. Run LocateAll(group, shard, copies) for each shard in the list, take the union of the node sets, deduplicated, and open one SERIES_INDEX stream per node. shard_id on that stream is only a routing hint for the liaison; the receiver ignores it for placement.
An earlier draft said “one stream per shard in
sidxCoversShards, sending the same bytes several times to different shards”. That was wrong on its premise. Because the receiver ignoresshard_id, those extra streams re-introduce the identical external segment into the identical index. Deduplication absorbs them, so nothing breaks — it is pure waste. Measured on the real store: a source covering 4 shards sent 6,084 bytes under the per-shard rule versus 1,521 once per node, 4.0×, with byte-identical query results.
Shard-level data follows this pattern:
Init{
catalog=MEASURE,
group="sw_metric",
shard_id=0,
stage="hot",
kind=PART,
segment_version="1.5.0",
parts=[
{
id=1,
part_type="", // the core part; the liaison sends "core" on the SyncPart wire
compressed_size_bytes=47972566,
uncompressed_size_bytes=198000000,
total_count=123456,
blocks_count=892,
min_timestamp=1788739200000000000,
max_timestamp=1788825599000000000,
min_key=0,
max_key=0
}
]
}
Entries under prefix 0000000000000001/ in shard-0-00001.bnc:
metadata.json size=214 ← not sent; scalar fields above are parsed from it
meta.bin size=1_048_576
primary.bin size=8_912_896
timestamps.bin size=4_194_304
searchable.tf size=33_554_432
searchable.tfm size=262_144
fv.bin size=65_536
6 files are transferred, totaling 48,037,888 bytes.
Chunk{ data=<1MiB>, files=[{name=meta.bin, size=1048576}] }
Chunk{ data=<1MiB>, files=[{name=primary.bin, size=1048576}] }
... primary.bin spans multiple frames ...
Chunk{
data=<1MiB>,
files=[
{primary.bin, 524288},
{timestamps.bin, 524288}
]
}
... cross-file chunk ...
Chunk{
data=<832KiB>,
files=[
{searchable.tf, 524288},
{searchable.tfm, 262144},
{fv.bin, 65536}
]
}
Completion{
total_bytes=48037888,
total_chunks=46
}
→ ImportPartsResponse{
requested_copies=2,
targets=[
{node:"data-a", success:true},
{node:"data-b", success:true}
]
}
Parts are sent in ascending part-ID order, which is also the byte order inside the container. That keeps reading sequential and keeps the relative ordering of source and target part IDs aligned.
Trace cannot be split across sessions. Its core part and its secondary index parts share one locally allocated part ID: NewPartType reuses the ID within the same syncPartContext (if s.partID == 0 before allocating), and the secondary part is written under that same ID. Across two sessions they receive different IDs and lose each other — and the two halves then fail differently, neither of them as an error:
- A secondary part without its core is deleted.
FinishSyncsees an empty core path and returns early.Close()then skips the coreMustRMAll— that path is empty — but still callsClose()on everysidx.SyncPartContext, and that is what removes the secondary part directory. The bytes are gone before the stream answerssuccess=true. - A core without its secondaries is kept, which is worse. It is introduced normally with an empty secondary-part map. The spans stay queryable by time, but the ordered-by-key path over that index never returns them, silently and permanently.
- A late secondary cannot repair it. Reopening the tsTable passes the core parts’ IDs as
AvailablePartIDs, and any on-disk secondary part whose ID is not in that list is removed.
So a PART stream carrying any trace secondary part must also carry its core part, in the same session.
5.2.6 Liaison handling
On Init the liaison resolves the data topic (for example MEASURE + PART = "measure-part-sync"), resolves the stage and schema to find the target data nodes, and opens one SyncPart stream per target. That needs a new queue.Client method: the existing NewChunkedSyncClient hands back a client whose only send path is SyncStreamingParts, which re-reads parts from disk. Add NewChunkedSyncRelayClient(node) alongside it, borrowing the pooled connection exactly as NewNodeSchemaStatusClient already does, then drive the raw SyncPart(ctx) stream.
The relay context must derive from the downstream client stream’s context. Deriving it from context.Background() leaves zombie upstream streams when the client cancels.
| Time | bydbctl → liaison | liaison → data-a / data-b | Response |
|---|---|---|---|
| t0 | Chunk{data=<1MiB>, files=[{name=meta.bin, size=1048576}]} |
SyncPartRequest{session_id, chunk_index=0, chunk_data=<same 1MiB>, chunk_checksum="3a91c7e2", parts_info=[{id=1, part_type="core", files=[{name:"meta", offset:0, size:1048576}]}], metadata=SyncMetadata{…}, version_info} |
Each node returns CHUNK_RECEIVED |
| t1–t8 | Chunk{…, files=[{name=primary.bin, size=1048576}]} (8 chunks) |
Same structure with increasing chunk_index; files=[{name:"primary", offset:0, size:1048576}]; no repeated metadata |
Each node returns CHUNK_RECEIVED |
| t9 | Chunk{…, files=[{primary.bin, 524288}, {timestamps.bin, 524288}]} |
Convert to absolute offsets: files=[{name:"primary", offset:0, size:524288}, {name:"timestamps", offset:524288, size:524288}] |
Each node returns CHUNK_RECEIVED |
| t10–t44 | Full frames containing one file slice | Same structure with one files entry |
Each node returns CHUNK_RECEIVED |
| t45 | Chunk{data=<832KiB>, files=[…3 entries…]} |
files=[{name:"tf:searchable", offset:0, size:524288}, {name:"tfm:searchable", offset:524288, size:262144}, {name:"fv", offset:786432, size:65536}] |
Each node returns CHUNK_RECEIVED |
| t46 | Completion{total_bytes=48037888, total_chunks=46} |
SyncPartRequest{session_id, chunk_index=46, completion=SyncCompletion{total_bytes_sent=48037888, total_parts_sent=1, total_chunks=46}, version_info}, no chunk_data or parts_info |
Final response |
The liaison stores nothing: it converts one frame at a time and forwards it, which is why it stays memory-flat at line rate. The memory protector is not a backstop here — it only observes memory, and a relay saturating the NIC has a flat memory curve.
5.2.7 When only some replicas succeed
ImportPartsResponse.targets reports {node, success, error} per node, and the liaison waits for all N before returning. It does not retry the part itself — it holds one chunk at a time, and re-pushing a part means re-reading the container, which only the client can do.
The client counts the part as partially complete, not done, and retries only the failed targets.
Re-pushing must target only the failed replicas, through Init.retry_nodes. Re-sending the whole part is not equivalent: data-a would receive a second copy. Measure still converges by (seriesID, timestamp, version), but stream and trace are append-only, so the second copy is a permanent, visible duplicate, and the element index re-introduces a whole segment. So the set actually pushed again is:
retry_nodes = targets of the payload ∖ targets that answered success
Pushing to every target again re-delivers to replicas that already succeeded. A failed target is not by itself proof that the node lacks the data: the receiver commits before it answers, so a stream or trace part, or an element index, whose target failed after its completion was relayed (completion_sent) is not pushed again; see the Retries item of 5.5.
5.3 CSV data rewrite
For CSV, bydbctl reads the files directly, rebuilds the corresponding requests, and redistributes them to the target data nodes.
Superseded by the implementation. The protocol below is the original draft. The implemented
ImportService(api/proto/banyandb/transfer/v1/import.proto) differs: one stream per(catalog, group, stage)with the resource name on every row, nosuccess/errorin the acknowledgement (a failed batch ends the stream), a read-onlyTargetsRPC, and the liaison fans every row out to all replicas itself instead of writing replica 0. See CSV import as implemented.
service ImportService {
// Bidirectional: the client keeps a bounded window of unacknowledged batches,
// so it needs an acknowledgement per batch, not one response at end of stream.
rpc ImportRows(stream ImportRowsRequest) returns (stream ImportRowsResponse);
}
message ImportRowsRequest {
oneof frame {
Init init = 1;
RowBatch batch = 2;
}
message Init {
common.v1.Catalog catalog = 1;
string group = 2;
string name = 3; // Resource name
string stage = 4; // Target stage
}
message RowBatch {
// Monotonic within one stream, starting at 0. Echoed back in the
// acknowledgement so the client can correlate without assuming
// that responses arrive in request order.
uint64 batch_seq = 5;
repeated StreamRow stream_rows = 1;
repeated MeasureRow measure_rows = 2;
repeated TraceRow trace_rows = 3;
repeated PropertyRow property_rows = 4;
}
message StreamRow {
stream.v1.ElementValue element = 1;
uint64 raw_element_id = 2; // the _element_id cell, decoded by csvfmt.DecodeElementID
}
message MeasureRow {
measure.v1.DataPointValue point = 1;
}
message TraceRow {
// Tags and span of the row. The engine does not store a span's write version, which
// must be positive, so the importer sends 1.
trace.v1.WriteRequest span = 1;
}
message PropertyRow {
property.v1.Property property = 1;
int64 delete_time = 2;
}
}
// One acknowledgement per RowBatch, so the client can advance its record
// cursor as batches land instead of only at end of stream.
message ImportRowsResponse {
uint64 batch_seq = 1; // Echoes RowBatch.batch_seq
bool success = 3;
string error = 4;
}
The RPC is bidirectional, not client-streaming. A client-streaming RPC delivers a single response after the client closes the stream, which cannot carry the per-batch acknowledgements the client needs: a batch counts as done only once the server has confirmed it, the client bounds the batches sent and not yet acknowledged (5.4.7), and a failed batch is named by its batch_seq. Acknowledgements carry batch_seq so the client correlates them explicitly rather than assuming they arrive in request order.
A dedicated import RPC is required rather than reusing the public write API, because the public path regenerates fields the artifact already carries. In particular a measure version of 0 is silently replaced with the message ID on the liaison, destroying the deduplication semantics the CSV _version column exists to preserve.
| # | bydbctl → liaison | liaison → data-* | Liaison behavior |
|---|---|---|---|
| ⓪ | Read the manifest and target schema/topology | — | — |
| ① | Open .csv.gz files and build the column mapping from the schema |
— | — |
| ② | Init{catalog, group, name, stage} |
— | Locate the schema of the target resource |
| ③ | Send RowBatch{…_rows} in batches |
InternalWriteRequest per row |
Resolve (EntityValues, ShardID), call Locate(group, name, shard, 0) and send one internal write request to replica 0 only — the write path replicates internally, so the liaison does not fan out. property_rows are the exception: they go to TopicPropertyRepair and fan out over all copies replicas |
| ④ | Receive the batch response | — | Aggregate success and failure for the batch |
This channel is not idempotent on replay: re-sending a batch duplicates stream and trace rows.
Property rows must be applied with db.Repair(ctx, id, shardID, property, deleteTime). Update has no delete_time parameter at all — its underlying call hardcodes 0 — so a tombstone imported through Update comes back to life. Delete searches for an existing document first and is meaningless against a fresh database.
5.4 CSV import as implemented
The CSV import landed on 2026-10-19 (bydbctl data import, pkg/transfer/importer, the liaison’s ImportService). Where it differs from the design above, this section is authoritative. The native import is described in 5.5.
5.4.1 Workflow
bydbctl data import --input DIR [--plan plan.yaml] [--dry-run] [-o table|yaml|json] ...
0 preflight reuse the export preflight (several liaisons, standalone fallback, ROLE_LIAISON)
1 artifact check manifest.json, format=csv|native, artifactVersion <= 1, non-empty schemaFiles,
every schema and unit file present with its size and CRC32 (under
--schema-only, data files by size only); the server's
internal groups an older export kept are left out with a warning
2 schema compare read-only, artifact schema/*.jsonl against the target registry
3 Targets ImportService.Targets for every (group, stage) involved, read-only
4 decide() [1] schema gate, [4] TTL gate, multiple-source selection, and the ROWS
row check that reads every row (skipped with --schema-only) -> ImportPlan
5 render verdict table, byte-identical for --dry-run and the real run
-- a dry run exits here: 0 = would proceed, 2 = would be refused --
6 schema write dependency order group -> stream/measure/trace/property -> indexRule
-> indexRuleBinding -> topNAggregation, through the registry Create / Update
7 wait SchemaBarrierService.AwaitSchemaApplied, including every group's auto-created
_top_n_result
8 re-check collect + Targets + decide again; a changed data plan is printed and the
run continues, or exits 5 with --abort-on-plan-change (schema written, no data);
a route failure of a created or overwritten group exits 3 here
9 verify baseline with --verify
10 push worker pool, one ImportRows stream per (catalog, group, stage) per worker
11 verify poll and reconcile; a mismatch exits 4
The column layout is rebuilt from all resources of the group in the artifact’s schema/, so it matches the export bit for bit whatever --include-schema says for the import, and the import logs that the schema is read from those files. An artifact exported with --include-schema=false carries no schema: the import logs an info line and rebuilds the layout from the target’s resources of the group instead, compares and writes no schema (as --include-schema=false; a group the target lacks is MISSING, and --schema-only is a usage error), and refuses a file whose header or #datatype line differs from that layout. The target’s schema must therefore be the one the rows were exported with; a column whose type alone changed is caught only by the #datatype line or by a row that fails to parse.
5.4.2 Protocol
api/proto/banyandb/transfer/v1/import.proto, served on the liaison gRPC port in cluster and standalone mode. Both methods require PermissionClusterAdmin at global scope.
service ImportService {
rpc ImportRows(stream ImportRowsRequest) returns (stream ImportRowsResponse);
rpc Targets(TargetsRequest) returns (TargetsResponse); // read-only
}
message ImportRowsRequest {
message Init {
common.v1.Catalog catalog = 1; // not CATALOG_UNSPECIFIED
string group = 2;
string stage = 3; // empty = default stage; an undefined stage resolves as "hot"
// (the explicit hot stage if defined, else the default); ignored for property
uint64 liaison_bytes_per_second = 4; // process-wide import bucket, last stream wins; 0 = unlimited
RateLimit rate_limit = 5; // per target data node, enforced by the node; last import wins
}
message StreamRow { string name = 1; stream.v1.ElementValue element = 2; uint64 raw_element_id = 3; } // id > 0
message MeasureRow { string name = 1; measure.v1.DataPointValue point = 2; } // point.version != 0
message TraceRow { string name = 1; repeated model.v1.TagValue tags = 2; bytes span = 3; } // 1..512 tags
message PropertyRow { property.v1.Property property = 1; int64 delete_time = 2; } // group == Init.group
message StreamRows { repeated StreamRow rows = 1; } // each list at most 10000
message MeasureRows { repeated MeasureRow rows = 1; }
message TraceRows { repeated TraceRow rows = 1; }
message PropertyRows { repeated PropertyRow rows = 1; }
message RowBatch {
uint64 batch_seq = 1; // monotonic from 0
oneof rows { // the catalog of Init
StreamRows stream = 2;
MeasureRows measure = 3;
TraceRows trace = 4;
PropertyRows property = 5;
}
}
oneof frame { Init init = 1; RowBatch batch = 2; } // required; init first
}
message ImportRowsResponse { uint64 batch_seq = 1; }
message TargetsRequest { string group = 1; string stage = 2; } // one (group, stage) per call keeps every response small
message TargetsResponse {
message Shard { uint32 shard_id = 1; repeated string node_ids = 2; }
string group = 1;
string stage = 2; // the stage actually used: "hot" after a fallback
uint32 shard_num = 3;
uint32 copies = 4; // replicas + 1
repeated Shard shards = 5;
repeated string inactive_node_ids = 6; // route nodes this liaison is not connected to
bool group_deleting = 7;
}
- Rows carry tags and tag families in the declaration order of the target schema and no
TagFamilySpec/DataPointSpec, like the lifecycle row replay. Every internal message carries itsMetadata. - An acknowledgement means every target data node accepted the batch at batch level. A data node only logs per-row failures, so
--verifyis the only check that rows landed. - A failed batch ends the whole stream with a gRPC status whose message starts with
batch_seq=<n> node=<id>:. Codes: disk full →ResourceExhausted, unknown schema →NotFound, unreachable node →Unavailable, anything else →Internal. A row the liaison refuses ends the stream withInvalidArgumentand an ErrorInfo (reasonROW_INVALID) whose metadata holdsbatch_seqandrow; its message starts withbatch_seq=<n>: row <i>:. A batch naming a missing resource getsNotFoundwith reasonSCHEMA_NOT_FOUNDandbatch_seqin the metadata. Only a frame refused by message validation, or anInitsent after the first frame, carries nobatch_seq, and the client attributes it to the oldest unacknowledged batch. - A trace data node that answers
STATUS_SERVER_BUSYis retried by the liaison with backoff for that node, bounded by the liaison’s write pressure timeout; if it is still busy the batch fails withResourceExhausted. - The liaison writes rows as
InternalWriteRequests wrapped intransfer.v1.ImportWriteframes straight to the data nodes’ import topics, whose listener hands them to the catalog’s write path, so the public write path’s rewrites (a measureversionof0replaced by a message ID) never apply;version == 0is refused withInvalidArgumentinstead. A row the write path cannot store is only logged for a regular write. The import listener puts acommon.WriteRecorderin the context it calls the catalog’s write listener with, and the write listener reports to it where it logs, its reply unchanged (without a recorder the calls do nothing): a row that failed before it was stored (a decode failure, its schema or TSDB not loaded on the node, a segment or table it could not create) by its index in the message (common.RecordRowFailure); an index write after the rows were stored is wrapped incommon.RetryWrite, which under the recorder retries it with back-off, since every such write may be repeated (series and metadata documents are inserted if absent, index-mode and trace series documents upserted, a refused element-index batch admits nothing, a sidx instance is created once); what still fails is recorded withcommon.RecordWriteFailure. The node writes the rows not stored again, alone, with back-off, for at most30sshared with the index retries, so a retry stores no row twice. Rows still not written are answered as an error naming them (k of n imported row(s) were not written after retries (rows …)), and a failure after storing ends the batch at once, its message saying that importing the rows again would store stream and trace rows twice; either ends the import withNODE_FAILED. - The import listener hands measure rows to the write path wrapped in
data.ImportedMeasureWrite, so it skips Top-N for them (see Top-N results).
5.4.3 Stage-aware routing
The liaison computes a route snapshot on every Init and every Targets call with the node selection API the cluster already places shards with, so imported rows land where the cluster’s own writes and migrations put them:
route(catalog, group, stage):
g := the target group (absent -> NotFound)
property: shardNum/replicas from g.ResourceOpts, nodes from the property node registry; done
want := lower(stage) or "hot"; a stage the group does not define -> "hot"
"hot" (the default stage, or a stage explicitly named hot):
shardNum/replicas from g.ResourceOpts (group level)
nodes := the catalog's data node selector LocateAll -- the selector the liaison's write queue
syncs online writes with, over the --data-node-selector nodes
any other stage:
shardNum/replicas from the stage
nodes := a round-robin selector over the registered nodes matching the stage's node selector,
laid out by every group -- the selector a lifecycle migration places the stage with
targets(shard) := nodes.LocateAll(group, shard, replicas + 1) // sorted, distinct
a shard the selector cannot place -> FailedPrecondition
- The hot stage follows the online write placement exactly, the
--data-node-selectorincluded: a liaison without one places hot rows on every data node, as its online writes do. - A stage other than hot follows the lifecycle migration’s placement. The round-robin selector is laid out by the group-level shard number, so, as for a migration, a stage with more shards than the group is refused (
unknown shard). - The route borrows the selectors: it registers no handler on the shared tier-2 client, which would overwrite the online handlers. The data node selectors are the ones the write queues keep up to date; a stage’s selector is built per call from the registered nodes and the listed groups.
- A node that is down fails the batch instead of silently rerouting it.
- Replica fan-out: stream, measure and trace rows go to every node of
targets(shard)through oneBatchPublisherper batch (TopicStreamImport/TopicMeasureImport/TopicTraceImport), each node’s stream opening with an init frame. Property rows go toTopicPropertyImportthe same way, one batch per node holding the revisions of an entity in batch order; the node’s listener repairs them one at a time in that order and stops at the first failure. When a stage has fewer nodes thancopies, every available distinct node is written,Targetsreports the actual count, and the client warns. - Shard ids: stream and measure via the entity / sharding-key locator with the route’s shard number (the group’s for hot, the stage’s otherwise); trace via
TraceShardID(traceID, routeShardNum)after checking the tag count, a non-empty string trace and span id and a non-null, valid timestamp tag; property via the entity and the group shard number. - Standalone:
Targetsreports the local node, a non-hot stage falls back tohot, and rows go through the local queue. - The liaison rate limit (
Init.liaison_bytes_per_second) is a bucket of its own, independent of the export’s, charged once per received batch before the fan-out. - The node rate limit (
Init.rate_limit) is one byte bucket and one row bucket per target data node, kept by the liaison and shared by every import stream it serves; the stream opened last sets the rates and they are lifted once no import stream is left. Before a batch is written to a node the liaison pays, in full, the internal write requests that node receives (proto.Sizefor bytes, one row per request), so every replica is paid on its own node; a property repair is paid per node the same way. A busy node’s retry is not paid again: the node refused the batch before writing any of it. A bucket’s burst is one second’s worth (at most 1 MiB, at least one row) and a larger payment is paid in steps of the burst.
5.4.4 Rows
| Catalog | Handling |
|---|---|
| stream | _element_id → csvfmt.DecodeElementID → raw_element_id, kept as is by the receiver (0 means a corrupt artifact) |
| measure | _version must be present and non-zero; fields in target schema order |
| index-mode measure | as measure, no fields; when several sources of a segment are pushed, the client keeps the highest _version per (name, entity) within the unit, the later timestamp winning a tie |
| trace | tags in target schema order with trace_id / span_id as string tags in their positions; _span → bytes; a null timestamp tag means a corrupt artifact |
| property | _updated_at \N → unset; _delete_time 0 (or \N) = live, > 0 = tombstone, negative = corrupt. Only each entity’s latest state is kept: a property work item is read once, source by source and file by file, and every row is sent as it is read, batched like the other catalogs, so memory does not depend on the item’s size; nothing is indexed, sorted or folded on the client. The target’s repair makes the result independent of the order: a revision older than the newest one the entity holds is refused, and a same-revision conflict resolves deterministically by the repair rule below, so every entity converges to its latest revision. Whether older revisions leave documents on the target is not guaranteed. The liaison keeps property batches, and the rows of one entity within a batch, sequential |
The repair rule (banyand/property/db/shard.go), now applied to every caller — gossip anti-entropy, liaison read repair, RepairSchema and the import — for an incoming row with the same mod_revision as the local latest: a local tombstone beats an incoming live row (nothing is written); an incoming tombstone replaces a local live row with the incoming deleteTime; between two tombstones the smaller deleteTime wins, so replicas converge whatever the order; identical rows are a no-op.
A resource name the target schema lacks, a header that does not match the layout, or a cell that does not parse or fails the liaison’s row checks is a corrupt artifact. The row check (gate ROWS) reads every row before anything is written, in the dry run too, and refuses the import with exit 2, naming the file and record number in the [d] rows details.
5.4.5 Gates
[1] Schema. Resources compare equal unless they differ outside created_at, updated_at and Metadata.{id, create_revision, mod_revision}. Each difference is classified as identity, topology, shape or behavior, and resolved by import.schemaConflictPolicy{default, rules[{match, policy}]} (match = <kind>/<group>[/<name>], first match wins; --on-schema-conflict sets only the default) to error, ignore or overwrite. Every CONFLICT prints a ready-to-paste match: line. Guards that no policy overrides:
- G1 — an identity difference is always
CONFLICT. - G4 —
overwriteof a non-empty target group needs--force; “non-empty” is any count above zero in the target’sPlan, failing closed when the inventory is incomplete. Rows not yet flushed are invisible to it. - G5 — a measure
index_modeorintervaldifference is alwaysCONFLICT. - G6 — an
overwritethe registry’sUpdatewould refuse isCONFLICTup front, so no run writes half a schema: a changed stream or measure entity, a removed entity tag or its family, a changed existing measure field spec, a removed reserved trace tag, or enabling a sampler pipeline on a group with several traces. ignorewith a type difference in a tag or field present on both sides isCONFLICT; extra or missing columns are dropped or sent as null and listed.
Artifact resources are re-validated (INVALID), and a missing dependency is MISSING. Any CONFLICT, INVALID or MISSING exits 2 with nothing written. CREATE uses the registry Create and OVERWRITE its Update, with created_at, updated_at and the metadata revisions cleared so the server stamps them. A resource someone else created between steps 2 and 6 makes Create fail with AlreadyExists (exit 3). G0 (tombstones) is not implemented because the registry does not expose them; G2/G3 are native only.
[4] TTL. The effective TTL is the group TTL plus every stage TTL up to the mapped target stage (CumulativeTTL, checked against pub.ResolveStageResourceOpts); the default stage, including a fallback, has the group TTL only; property is exempt. A unit’s newest time is the largest maxTimestamp of its selected sources, or, for index-mode segment files, the segment end parsed in the source node’s timezone. Retention removes a whole segment, and queries hide it, only once the segment’s end is before now − effectiveTTL; a row older than that in a segment ending later stays visible. So the gate projects the newest time onto the target stage’s segment grid (IntervalRule.Standard: buckets of the stage’s segment interval anchored at 1970-01-01 in the target’s time zone: for native, the source node’s, which the TZ gate makes equal to the target’s; for CSV, that of the first target data node of the route that reports one, else the source node’s) and refuses a unit (STATUS expired) unless --force-ttl only when that segment ends before now − effectiveTTL. A unit just inside the window may still land in a segment that expires soon after. Some units refused and the rest imported exits 0; all refused exits 2.
5.4.6 Top-N results
_top_n_result is exported and imported verbatim, never recomputed. A standalone server would recompute it from the imported source measures, and the recomputed values, carrying newer versions, would overwrite the imported ones; the import listener therefore hands each measure row to the write path wrapped in data.ImportedMeasureWrite, an in-process marker that skips the Top-N flow (a cluster data node never computes Top-N on writes anyway). The imported _top_n_result therefore equals the artifact’s exactly in both deployments. The schema itself is not created (the target auto-creates it with each measure group, and step 7 waits for it), but its columns must match the target’s: a mismatch is a CONFLICT that refuses the whole import. The cost: a target whose TopNAggregation differs keeps the source’s historical Top-N; new traffic is computed by the target as usual.
5.4.7 Batches, parallelism and sources
-
A batch is sent when it reaches
writeBatchSizerows (1–10000, default 1000) orwriteBatchBytes(64 KiB–64 MiB, default 4 MiB), whichever comes first; a single row above the byte limit travels alone with a warning. Out-of-range values are usage errors. AResourceExhaustedreply names the file and record and both possible causes (the liaison’s--max-recv-msg-sizeor a full disk). -
Work items: one per shard of a segment unit; one per segment for index-mode segment files (decided by where the files are, not by
groups[].indexMode); property grouped by source shard id.--parallelism maxis the size of the union of target nodes returned byTargets, clamped to the number of work items; an explicit N is clamped the same way. -
Each worker opens one
ImportRowsstream per(catalog, group, stage)lazily and reuses it; workeriuses liaisoni % L. Only the first send or receive of a stream is retried (within--send-retry-timeout:Unavailableand load-protectionResourceExhaustedmove to the next liaison;SCHEMA_NOT_FOUND, or aNotFoundwithout a detail, waits on the same liaison;NOT_WRITTEN(the liaison refused before writing anything, such as on a failed schema read) moves to the next liaison even after a stream or trace batch was sent;NODE_FAILED(including a full disk),MESSAGE_TOO_LARGEandROW_INVALIDare never retried); a later error is terminal, because re-sending a whole batch duplicates stream and trace rows. A failed worker cancels the rest (exit3). -
After that first exchange (init plus
batch_seq0, the only batch that may be retried), a worker keeps up towriteInflightBatches(1–8, default 2) batches sent and not yet acknowledged on each stream: it builds and sends the next batch while the liaison works on the previous one, and takes the acknowledgements inbatch_seqorder. A batch is done only once acknowledged and a work item once all its batches are; a failed batch is named by thebatch_seqthe liaison reports. A worker holds at mostwriteInflightBatches × writeBatchBytesof rows. -
Rate limiting has four knobs, set by the client; whichever binds first wins:
Knob Flag / plan field Protobuf field Enforced by, measured on Client rows --write-rate-limit/import.writeRateLimit— The client: one row bucket for the whole import, paid before each batch is sent Liaison bytes --liaison-rate-limit/connection.rateLimitInit.liaison_bytes_per_secondEach liaison, process-wide: the serialized size of every received batch, before the fan-out Node bytes --node-rate-limit/import.nodeRateLimitInit.rate_limit.bytes_per_secondEach target data node: the encoded rows it is sent, replicas included Node rows --node-row-rate-limit/import.nodeRowRateLimitInit.rate_limit.rows_per_secondEach target data node: the rows it is sent, replicas included The node knobs reuse the export’s
RateLimitand flag names, and like the export the data node enforces them. The liaison writes import rows on their own topics, one per catalog (stream-import,measure-import,trace-import,property-import), each frame atransfer.v1.ImportWrite: every batch stream to a node opens with an init frame carryingrate_limit. The node’s import listener charges the batch to one byte pace and one row pace that every import on the node shares, then hands the rows to the catalog’s write path; only the import topics pay, so other writes are never limited. A pace is the time by which the node has paid out what it admitted, moved on by each batch’s cost at the rate that batch names, so the import that wrote last sets the rate and a change carries no debt; one second’s worth (at most 1 MiB) passes on an idle node without waiting. The rates hold on the node whichever and however many liaisons write to it; a process that runs several data nodes, as a test harness does, keeps one pace per node. A batch waits for its own share and for the batches admitted before it, but no more than its own share plus10s: a batch that would wait longer, or past its request’s deadline, is answeredSTATUS_RATE_LIMITEDbefore anything is charged or written. The liaison therefore waits2 minplus the batch’s share (the slower of its bytes over the byte rate and its rows over the row rate) plus10sfor an acknowledgement, and resends a batch refused by the rate limit to that node with backoff for as long as the stream lasts; unlike memory pressure, the rate limit is not bounded by the write pressure timeout, since it is expected to hold batches back. A standalone server admits the rows the same way in process. A data node older than the import topics refuses them, so a cluster must run one version for an import. The node knobs are ignored by--dry-run, as are the other two. -
The schema write prints one summary line,
schema: <n> resource(s) written and applied on every node; the plan table above it already lists every resource. During the push the client prints, like the export, a line for every finished work item (<catalog> <group>/<stage> [done/total] <position> <rows> rows <time>) and one for every finished(catalog, group, target stage)(✓ … done: <n> work item(s), <rows> rows in <time>); on a terminal it also redraws one live line per group in flight with the rows written (acknowledged by the data nodes), the rows left of the planned ones, the row rate and the elapsed time (<catalog> <group>/<stage> <n> rows written, <m> left <r> rows/s <time>). The rows written move one acknowledged batch at a time, so a smallwriteBatchSizemakes them move more often. -
Multiple sources follow §6.3, keyed by
(group, stage, segment, shard): coverage fromparts[], counts fromshards[].totalCount. Property and index-mode units push every source.--all-sourcesskips the selection;--strict-coveragerefuses (exit2) a shard work item, or a CSV stream/trace segment-level item, that would push more than one source; property units, native series indexes (whose receiver keeps the first copy of each document) and measure segment items, which deduplicate or merge, are exempt.
5.4.8 Verify
--verify (default on, --verify-timeout 2m, any positive value) counts with ExportService.Plan without a session, narrowed to the imported groups, polling every 5 s (the default flush interval) because only flushed parts are counted; on a terminal a live line (verify <elapsed> / <timeout> waiting for the target to flush: <what has not reconciled yet>) is redrawn after every poll and erased when the check ends; without a terminal every poll that does not end the check logs one verify: waiting for the target to flush (...) line instead. For each (catalog, group, stage, node, shard) it takes Δ = post − pre over the segments present after the import (segments removed by retention are excluded with a warning). Any target node in unreachable_nodes makes verify incomplete.
- stream / trace: for every shard with Δ > 0, exactly its target nodes have Δ > 0 and the same Δ, and the shards’ Δ sum to the rows pushed.
- measure: once the sum is stable across two samples a flush interval apart, rows pushed > 0 ⇒ ΣΔ > 0, Σ_nodes Δ ≤ rows pushed × copies, and a shard with Δ > 0 has Δ > 0 on all its target nodes; a negative Δ (merge folding) is only a warning.
- index-mode measure: a target node that gained no segment document although rows were sent, and held none before, is a warning; nothing else is checked (one document per series and segment, and rows replayed onto held documents gain nothing).
- property: not counted, since the target keeps each entity’s latest revision; a failed repair already fails the import.
Verify assumes no other writes reach the target during the import, does not cover index structures, and in standalone uses copies = 1, stage hot. Only a reconciliation difference or an incomplete coverage exits 4. The push ends with IMPORTED <n> rows of <m> work item(s) in <time>, and the report is one line, VERIFY ok in <time> or VERIFY FAILED in <time> (exit 4), followed by one line per problem: a check that could not complete (not checked: …), a count mismatch, or a warning (warning: …). Work items refused by a gate are already listed under the plan’s PROBLEMS, so verify does not repeat them.
5.4.9 Exit codes
| Code | Meaning |
|---|---|
0 |
success, warnings included (also: some units refused by the TTL gate) |
1 |
usage error |
2 |
refused before anything was written (preflight, schema gate, row check ROWS including a corrupt artifact, all units refused, --strict-coverage) |
3 |
failure after writing started (schema write, wait, re-check route of a created or overwritten group, push) |
4 |
--verify failed |
5 |
the re-check found a different data plan and --abort-on-plan-change was given |
Errors before step 6 map through exitcode.For; every error from step 6 on is 3.
5.4.10 Known limitations
_top_n_resultis transferred verbatim and never recomputed, so it keeps the source’s Top-N caliber; an incompatible target_top_n_resultrefuses the import.- An acknowledgement is a batch-level acceptance; per-row failures on a data node are only logged and surface in
--verify. - Stream and trace row replay is not idempotent: re-running after a failure duplicates what had landed.
- G4’s emptiness check cannot see unflushed rows; G0 is not implemented.
- No
.tar/.tar.gzinput; it arrives with the export’s--archive. --verifydoes not cover index structures and needs a quiet target; content equality is proven by the integration and e2e tests, which re-export the target and compare logical rows with the source artifact (pkg/transfer/rowdiff). A property is compared by its entity(group, _name, _id)at its latest revision only (the largest_mod_revision, a tombstone winning over a live row of the same revision;_delete_timecompared as live or tombstone); older revisions are ignored on both sides.- The same data held by source nodes in different timezones, or caught mid lifecycle migration, forms different units the multiple-source selection cannot match, so it is pushed more than once (duplicate trace spans), never lost.
- Pre-existing and not fixed here: the lifecycle row replay writes replica 0 only, and builds its route table from the group-level shard number while computing shards with the stage’s.
5.5 Native import as implemented
The native import (2026-11-03) is the CSV import’s workflow (5.4.1) with another push: the preflight, the artifact check (every file’s size and CRC-32; each .bnc volume is verified once, member checksums included, and under --schema-only data files are checked by size, though each .bnc container’s volume footers are still read: the volume set must be complete and every volume must hold the entryCount the manifest registers), the schema comparison and write, Targets, the source selection, the TTL gate, the re-check after the schema write and --verify are shared. What differs:
- No row check. The part files are relayed as they are, so the
ROWSgate does not run. - Four more gates, each a refusal of the whole import (exit
2), with the gate and its reason inPROBLEMS:TOPOLOGY— the shard number of the target stage (the routed one, or the predicted group’s) and of the group, and the segment interval of the target stage, must equal the source’s frommanifest.groups[].stages. For a property group only the shard number counts.TZ— the time zone of every target data node must equal the source node’s (manifest.source.nodes[].timezone). A source or target node that reports no time zone is not refused.VERSION— every target data node must read the unit’ssegmentVersion(itscompatible_file_format_version, or its own version); a target node that reports no file format version is not refused, and property units carry no segment version and are not checked. TZ and VERSION apply to stream, measure and trace groups only, and, like theTARGET-NODEScolumn, check only the nodes the liaison relays each payload to: a shard or property item the nodes of its shard, a segment item (series index) the union of the nodes of the shards it covers, or every node of the stage when it covers none; when that set is empty, every node of the stage; and when the stage has no route yet (a group the import creates) or no nodes, every data node of the target cluster. A CSV import checks every node of the stage.INDEX_RULE— every index rule of the group that the target already has must carry the artifact’s id: the native index files key the stored tag values by the rule id. An index rule the import creates keeps the artifact’s id (the schema write passes it on the create instead of letting the registry assign one); an existing rule with another id refuses the import whateveronSchemaConflictsays, because an overwrite keeps the target’s id andignoreleaves it. Letting the import create the schema avoids this; a schema created beforehand by hand does not. An artifact exported without its schema skips this gate with a warning. The table’sCHANNELcolumn sayspart-push(property/partsfor a property group).
- Payloads. One
ImportPartsstream carries one payload:KIND_PART(a core part and, for trace, the secondary index parts sharing its id, so the two halves never land in different sessions),KIND_SERIES_INDEX(the.segsegments of a segment’s series index),KIND_ELEMENT_INDEX(the.segsegments of a stream shard’sidx/) orKIND_PROPERTY(a property shard’s index directory as it is). The importer reads every part’smetadata.json(or a secondary part’smanifest.json) out of the container intoInit.parts; neither is sent, norsmeta.bin, nor an index’s.snpmanifest — the receivers introduce each.segas an external segment, as a lifecycle migration sends them. Each chunk lists its file slices by their on-disk names; the liaison maps them to the names the receivers expect (primary.bin→primary,<tag>.tf→tf:<tag>, …).pkg/transfer/format/nativefmtholds the disk-to-wire table, shared by the importer and the liaison, and the one table of the (catalog, kind) pairs a native import carries (which kinds each catalog takes, which are duplicable, which catalog has secondary parts), which the liaison’s topic table is tested against. Each data-node receiver keeps its own dispatch; engine contract tests (banyand/{stream,measure}TestNativeSyncFileNameMatchesReceiver,banyand/traceTestNativeSyncFileNameMatchesStreamingMapping) pin the two together. An element index of a catalog that has none fails the import before it is sent. - Rounds. All series index payloads go first, then the parts, the element indexes and the property shards, so no part is visible before the index naming its series. A segment’s series index is pushed from every source whose core parts some shard item of the unit pushes; in a group that holds an index-mode measure it is pushed from every source, even when the unit has shard items, because the index also holds the index-mode rows, which no shard item carries. Its targets are the union of the nodes of the unit’s shards, or every node of the target stage when the unit has none (index-mode measure: the manifest records no series membership).
- Relay. The liaison validates the payload, routes it with the same stage-aware route as
ImportRows(5.4.3), opens one clusterSyncPartstream per target throughqueue.Client.NewChunkedSyncRelayClient(a standalone process feeds its own chunked sync handlers in-process), and forwards each chunk to every live target, waiting for all acknowledgements before it reads the next. A chunk carries only its bytes and file slices: the liaison computes each relayed chunk’s CRC-32 itself, and refuses a chunk over 4 MiB (nativefmt.MaxChunkSize; a data node’s--max-recv-msg-sizemust stay above that plus framing overhead, as the default 10 MiB does) or aKIND_PARTpayload in which a listed part received no file. A frame over the liaison’s own receive limit is answered with theMESSAGE_TOO_LARGEdetail. A target that answers nothing to one frame within twice the receivers’ default memory wait (10 minutes;--{stream,measure,trace}-lifecycle-receive-mem-wait-timeoutdefaults to 5) is dropped, so a busy receiver’s own answer reaches the client rather than a relay timeout.file_format_versionis the client’sInit.segment_version(the liaison’s own version for a property payload, which carries none). Each payload holds an in-flight request on its group from open until close, so a group deletion waits for the native payloads in flight on it. A target that fails is dropped and reported; when all failed, the liaison answers at once. The response carriesrequested_copiesand one result per target. A target that failed after its completion was relayed hascompletion_sentset: it may have committed the payload. - Retries. A payload the liaison refuses before answering is sent again within
--send-retry-timeout: through the next liaison when it is unavailable or busy, through the same one when a schema is not live yet. A frame the liaison refuses as too large (MESSAGE_TOO_LARGE) is final, not retried. A payload some targets failed is sent again to those targets only (Init.retry_nodes: the liaison narrows the route to those nodes and refuses, with INVALID_ARGUMENT, a retry node that is not in the route), up to three attempts, with the back-off doubling from 1 s to 30 s that stream reopening uses; still failing, it fails the import (exit3) with the work items done and not done. A stream or trace part, or an element index, that may already be committed (a failed target withcompletion_sent, or a stream that broke after the completion was sent, reported as UNAVAILABLE/ABORTED without a detail) is not pushed again, through the same or another liaison, since a second copy would store it twice; that fails the import at once.requested_copiesis per shard, so the copies warning is not given for a series index, whose targets span several shards. - Index deletion masks do not travel. An index payload carries the
.segsegments of the source’s live generation without its.snpmanifest, which also holds the masks that hide superseded or deleted documents. The importer sends the segments newest first, so the series index receiver, which keeps the first copy of a document, keeps the newest one; a document the source masked without a newer copy is live again on the target. Carrying the masks (re-encoding live-only segments on export, or a mask with each external segment) is a follow-up inpkg/index/native. - Same version on every node. Empty part files are relayed as zero-length slices, which only a data node of this version writes; as the design requires, do not import during a rolling upgrade.
- Reruns. A rerun of a native import stores stream and trace parts, and element index segments, again, as a CSV rerun stores stream and trace rows again.
- Options.
--part-rate-limit(import.partRateLimit) caps the bytes per second the whole import sends;--part-chunk-size(import.partChunkSize, 64 KiB to 4 MiB, default 1 MiB) sizes the frames; a size out of that range is a usage error (exit1).--write-rate-limit,--liaison-rate-limit(connection.rateLimit),--node-rate-limitand--node-row-rate-limitdo not apply to the part push: the liaison relays part bytes without a bucket of its own, so--part-rate-limitis the only native throttle (the import warns when another limit is set). Why a user sets them: a native import copies whole segments, often many GiB, into a cluster that is usually serving live traffic, and every byte the client sends is written by the liaison to each replica’s disk, so an unthrottled import can saturate the target’s network and disks and slow its queries and writes;--part-rate-limitbounds that cost, as--write-rate-limitdoes for CSV.--part-chunk-sizetrades memory for round trips: the liaison holds one frame per target until every target acknowledges it, so a larger chunk means fewer acknowledgements on a high-latency link and a smaller one less memory on the liaison and the receivers, and a finer-grained rate limit; the 4 MiB cap keeps a frame under a data node’s default receive limit. - Progress. The live progress of a shard item moves after each acknowledged part.
- Verify counts the rows of the parts that landed per (catalog, group, target stage); the target’s
Plancounts the same parts, so stream and trace grow by exactly the rows sent. The push ends withIMPORTED <m> work item(s) of native files, <n> rows in the imported parts, in <time>; series index, element index and property payloads count no rows.
6. Multiple shard sources
6.1 When it happens
| Cause | Description |
|---|---|
| Replicas | When replicas > 0, the same shard naturally exists on multiple nodes |
| Topology drift | After the node count in a stage changes, the same shard number may remain on old nodes |
| Orphan shard | A node is no longer the shard’s owner, but the data still remains on disk |
| Lifecycle migration window | Lifecycle migrates an expired segment to the next stage and only then deletes the source; on failure it returns and skips the delete. The same data sits in two stages at once, and both go into the artifact. This is deliberately not detected: the source cluster genuinely holds two copies at that moment, so reproducing it is correct. Pause lifecycle before exporting |
Topology drift has a wider trigger surface than it appears. The mapping is nodes[(index + replicaID) % len(nodes)], where index is the position of the (group, shardID) pair in the cluster-wide sorted lookup table — not the shard ID. Creating a new group whose name sorts early shifts every subsequent entry, and when the shift is not a multiple of the node count, 100% of the following (group, shard) pairs change owners.
All three causes look identical in the manifest — they are simply a second entry under sources[]. There is no replicaId, no creation time, and no cross-node comparable version. So the design does not classify the cause; it classifies the coverage.
6.2 Detection basis
For each part the manifest records {minTimestamp, maxTimestamp, totalCount}. The coverage of one source is the union of the time ranges of all its parts.
data-1 [00:00,06:00] + [06:00,12:00] + [12:00,23:59] → [00:00,23:59] 120,000 rows
data-2 [00:00,23:59] → [00:00,23:59] 118,000 rows
These two are replicas of the same data despite holding different numbers of parts, because one has already been merged. Therefore neither the part ID nor the part count can be used for comparison:
- Part IDs come from each node’s local counter. Three replicas receiving the same part were measured to assign it
1,4, and8. - Part counts drift with merge progress. During a merge sweep the Jaccard similarity of two nodes’ part sets fell to 0.45 at 50% merged and below 0.3 at 70%.
The decision rests on time coverage and row count. The union is computed over closed intervals, and adjacent intervals must be merged as well as overlapping ones (next.Min <= cur.Max + 1); otherwise every pair of parts leaves a 1 ns phantom hole and real holes vanish into the noise.
Sources with no parts[] or totalCount == 0 are dropped before the union is computed. Leaving them in pollutes the coverage with [0,0] and adds a phantom source that makes the unit look disjoint.
6.3 Rules
Sort sources by coverage duration, descending, then compare each against the already-selected ones.
| Condition | Meaning | Action |
|---|---|---|
| Time ranges do not overlap, or only partially overlap | The source holds time ranges nobody else covers | Push |
| Time range fully contained, row-count difference large (relative > 5% and absolute > 64) | Volume does not match; may be an orphan left by topology drift | Push + warning |
| Time range fully contained, row-count difference small | Treat as a replica | Skip |
Both thresholds must be exceeded to call the difference large. Using only 5% would make a 100-row shard differ significantly on 6 rows; using only 64 rows would make a 120,000-row shard differ significantly on 65 rows, about 0.05%.
“Fully contained” means contained by one single already-selected source. A source covered only by the union of several selected sources has no counterpart to compare row counts against, so it falls into the first row: push with a warning.
Example, with data-1 [00:00,23:59] / 120,000 rows already selected:
| Source | Range | Rows | Decision |
|---|---|---|---|
data-2 [00:00,23:59] |
Fully contained | 118,000, difference 1.67% | Skip; replica |
data-3 [08:00,10:00] |
Fully contained | 5,000, difference 95.8% | Push + warning; possible orphan |
data-4 [00:00,06:00] + [18:00,next day 04:00] |
Extends beyond 23:59 | — | Push; new time coverage |
measure does not apply the row-count condition — a fully contained source is skipped outright. Its background merge collapses multiple versions of the same (seriesID, timestamp), so the legitimate row-count skew is (K−1)/K for a source pushed K times: 66.7% at K=3. Any threshold below 100% is broken by the entirely normal “pushed a few times” case — and that skew is permanent rather than awaiting compaction, because equally sized parts are never merged under the production policy. The cost is that a measure orphan whose time range happens to be contained is skipped silently.
Every decision and its reason is printed in the import table’s DUPLICATES section (<key>: import from <node>; skip <node> (same data as <node>), with the reason when more than one copy is imported), because this is the only place in the import that can decide to send less data. --strict-coverage promotes a multi-source unit to an error and writes nothing; --all-sources skips the judgement and pushes everything. --all-sources must disable both the shard-level and the segment-level judgement — the escape hatch is one switch, not two. It then adds, for every source whose core parts were not selected, one SERIES_INDEX stream per target node in that source’s LocateAll union.
6.4 Multiple-source push behavior
| Catalog | Import behavior | Query | Disk |
|---|---|---|---|
| measure | Each part from every selected source becomes its own stream; no merging | Correct; the latest version is selected | Grows with the number of copies, and is not merged back |
| stream | Same as measure, plus the complete element index from every source | Correct; deduplicated by element ID | Permanently grows with the number of copies |
| trace | Same as above, but each core part and its secondary indexes must be sent together | ⚠️ Correct when the selected sources have disjoint coverage; duplicate spans are visible only in the overlapping cases below | Grows with the number of copies |
| property | The whole shard directory is sent and every record from every source is written | Correct. On an equal mod_revision shard.repair now lets the tombstone win (a live row never resurrects it), keeps the smaller deleteTime between two tombstones and never re-stamps one with now (fixed with the CSV import). The “two physical documents” split no longer reproduces since the native store folds same-docID writes |
No amplification |
| index-mode measure | Only segment-level indexes exist. The import side must deduplicate in memory by _version per docID across every source of a segment, keep the maximum, then issue a single UpdateSeriesBatch — that primitive is last-write-wins and InsertSeriesBatch is keep-existing, so neither alone gives a reproducible winner across sources. The docID is the series’ EntityValues, not the uint64 series.ID |
Correct; the highest version is selected | No amplification |
-
trace is the only catalog whose query result can be wrong, and it does not self-heal. But the exposure is narrower than “any multi-source unit”, and every case is closed on the import side — the read path is not changed.
Selecting two sources duplicates only where they overlap. A second source is pushed because its coverage is not contained in the first, and a span carries its own timestamp, so only the time both sources cover can repeat. A source chosen to fill a coverage gap may still partly overlap the first one; that overlap is pushed twice, and the CSV dry run warns about it for stream and trace. Overlap also arises when a source is rescued by
count-spread/no-single-covering(its coverage is contained) or when--all-sourcesskips the judgement. Both are pushed as-is and reported;--strict-coveragestops instead.Row-level deduplication was designed and rejected. The key is cheap —
_element_idfor stream,(trace_id, span_id)for trace, both fixed columns needing no schema, measured at 12.1M keys / 92 MB / 2.2 s. It loses because a content key cannot tell where a duplicate came from: if the source already held the same span twice (a lifecycle migration window, say), deduplicating removes the source’s copy too, and the target stops reproducing the source. So no channel deduplicates rows — native cannot decode them, and CSV will not. -
The element index cannot be reduced to a covering set. Each source’s
idx/may be complementary rather than duplicated, and a dropped index entry never reappears. It was measured at exactly ×3.000 for three sources (4,000 → 12,000 documents, 765 KB → 2,295 KB), and the legacy engine’s merge does not reclaim duplicates because the element index does not enable deduplication. Extrapolated to a 167-segment group with a 200 MB per-shard baseline, that is 65 GB atreplicas=1and 98 GB atreplicas=2. Provisionidx/at(replicas + 1)times. -
The segment-level series index follows the core-part selection, source by source. Push the series index of every source whose core parts were selected, and in a group that holds an index-mode measure — whose rows only the index carries — push every source’s. Do not compute a separate minimum shard-covering set over
sidxCoversShards.Why shard coverage is the wrong containment test. Two sources can both advertise shard 0 and still hold different series: that is exactly what topology drift produces. Their core parts cover disjoint time ranges, so the selection rules rule 1 selects both — but a minimum shard-covering set sees
{0}⊇{0}and keeps only one. The pruned source’s series then have core data with no series-index entry, andLookup(series)never finds them: the rows are on disk and undiscoverable.sidxCoversShardsrecords shard membership, not series membership. The manifest carries no series-document inventory, so series containment cannot be proven from it, and an unproven containment must not be used to drop an index.This costs nothing in the common case. With pure replicas the redundant source is already skipped at the core-part step, so its index is skipped too — the same saving the covering set would have produced. Only the drift case pushes more, which is precisely the case where the extra push is load-bearing. On disk the redundant bytes are free anyway: the receiving store enables deduplication.
-
Only the series index is segment-scoped; every other index is per shard. This is what decides whether “merging shards” is even a question for a given index:
Index On-disk scope Receiver keys placement on Merge axis Series index seg-XXX/sidx/segment — one directory for all shards (group, segment);ShardIDunusedacross sources Element index shard-N/idx/(stream)shard CreateTSTableIfNotExist(ShardID)across sources, within one shard Trace secondary index shard-N/sidx/<rule>/<016x>/shard, and it is a real part the core part’s tsTable follows its core part Property property/data/<group>/shard-Nshard shard (entity, mod_revision)viaRepairSo for the series index there is nothing to merge across shards — one segment already has exactly one index covering all of them. The axis that does need merging is across sources, which is the rule above. For the other three the shard is part of the identity, and merging across shards would matter — but it never arises, because native fast push requires the source and target shard counts to be equal (Native fast push), and the CSV channel rebuilds every index from rows on the target.
-
What deduplication on that index does and does not do: the winner is the first arrival, not the newest. The discarded copy’s index tag values become unqueryable —
Lookup(entity)still matches, because the legacy engine’s document_idis the entity value itself, while a filter on a non-entity indexed tag returns nothing for the losing value. -
measure’s storage does not converge. The production default merge policy requires a write-amplification score of
≥ max(maxParts/2, minMergeMultiplier) = 4, and equally sized parts score exactly their own count — three equal parts score 3 and are never merged. Queries are correct immediately regardless, because the read path deduplicates by version before any merge runs.
6.4.1 What a duplicate costs, per catalog
Duplication is not a trace problem; it is a problem whose cost differs per catalog. A duplicate comes from overlapping sources on a first run or from running an import again: nothing on the target recognises a re-imported part, because the same payload imported twice produces two directories with different part IDs and partMetadata carries no hash, checksum or fingerprint. The costs hold for both channels; CSV stream and trace rows behave as their native parts.
| Catalog / payload | What a duplicate costs |
|---|---|
| trace | Duplicate spans, visible to users; there is no deduplication anywhere and nothing self-heals |
| stream | Queries stay correct (the vectorized path deduplicates by element ID in pkg/query/vectorized/stream’s Distinct). Disk grows permanently |
| measure | Queries stay correct (the read path keeps the larger version by (seriesID, timestamp) before any merge). Disk grows, and under the production merge policy equally sized parts are never merged back |
| property | Union by (entity, mod_revision); idempotent, no amplification |
| index-mode measure | docID overwrite; idempotent, no amplification |
| series index | First write wins; later writes are discarded silently |
| element index | No deduplication: documents and disk grow linearly and are never reclaimed — measured at exactly ×3.000 for three sources, extrapolating to 65 GB at replicas=1 and 98 GB at replicas=2 for a 167-segment group |
There is deliberately no overwrite anywhere. A part has no overwrite semantics and there is no API to delete a single part, so “replace the existing copy” cannot be built. Row-level deduplication on the CSV channel was designed and rejected: a content key cannot tell a duplicate the source cluster already held from one this import created, so deduplicating it would stop the target reproducing the source.