Self-Stored Logs

Status: phase 1 (standalone and data nodes) implemented, with its buffering in the queue module (section 3.5); phase 2 (liaison and remote routing) is design.

BanyanDB self-stores its metrics into the _monitoring group but has no equivalent for its own logs: pkg/logger writes to a single io.Writer fixed at logger.Init time, so logs leave the process on stderr and are gone when a pod recycles. This document specifies a second, optional destination for logs — native self-storage into a _monitoring_log stream group — alongside normal logging, which is never disabled.

It follows the native metrics path where that path fits, and departs from it where logs are events rather than sampled state: a missed metric flush costs nothing because the next one carries the current value, while a dropped log line is data loss. That single difference is the source of most departures here.

1. Feature introduction and scope

BanyanDB already self-stores its own telemetry, but only half of it. Metrics have a native path into the database; logs do not.

During an incident the metrics are in the database and the logs are not. When a pod recycles, they are gone.

This feature adds a second, optional destination for logs: native self-storage into a new _monitoring_log stream group, behind --logging-native-enabled (bool, default false). Normal logging keeps its current behaviour — one zerolog writer, stderr today, though zerolog also supports file writers — and is never disabled.

The sections that follow go roles, approach, configuration, failure modes.

2. Role → function → log destination

One binary, several roles. One property decides the destination: whether the process owns a storage engine or must reach one.

A liaison has local disk: it runs a two-tier write queue whose normal StreamService/Write route ends in part files. What it lacks is the storage engine, so it can hold a batch in flight but can never be where logs are read back from.

Three shapes: in-process; the liaison selecting a data node by shard; and every other producer publishing into the generic receiver, subscribed on data nodes’ cluster server and on the liaison.

Restore and migration use normal logging only by construction, not as a gap to close later: a tool that runs while the database is down cannot log into it. The native flags are registered only by the server, backup, lifecycle and FODC binaries.

3. Proposed approach

3.1 The writer seam

One event, two level-gated writers under zerolog’s MultiLevelWriter. The zerolog level becomes an admission floor — the more verbose of the enabled sinks — and each sink then applies its own threshold, letting native run at info while normal logging stays at error.

flowchart LR
  E["log event"] --> F{"admission floor: min over enabled sinks"}
  F -->|below| X["dropped"]
  F -->|at or above| M["MultiLevelWriter"]
  M --> C["gate: --logging-level"] --> S["stderr or file"]
  M --> N["gate: --logging-native-level"] --> R["buffered publisher (queue)"]
  R --> K["delivery"] --> P["shard and node selection"] --> D["stream log"]

Worked example, normal logging at error and native at info, so the floor is info.

Native policy is never found by parsing JSON per line. Every logger writes through one writer that knows its module, gates the console on the logger’s own threshold, and resolves native policy for its module against the configuration in force, once per configuration. A package-level logger built with GetLogger before initialization therefore follows the configuration that initialization installs; callers keep GetLogger, DefaultLevel and ToZapConfig as they were, and the latter two act on the console threshold.

Native policy is resolved on the logger’s complete module, every component it was named with, not on the module the console prints. A --logging-modules override shortens what is printed — --logging-modules=measure prints a MEASURE.BLOCK logger as MEASURE — and native policy must not follow it. The most specific native override wins, and an exclusion covers a module and its children: measure covers MEASURE.BLOCK, not MEASUREMENT.

3.2 Buffer and transport

The buffered publisher of the queue module owns the buffer (section 3.5). Its delivery goroutine selects over the stop signal, the flush ticker and the queue, so both triggers fire.

When native is enabled the ring is live from logger initialisation; service start only begins draining it, so no startup line is stranded in a buffer nobody drains.

3.3 Routing and shard selection

Phase 1 follows native metrics. A standalone or data node writes its own events through its local write pipeline, always to shard 0 of _monitoring_log, which is created with one shard. No entity hash selects a shard and no event goes to another node, so each node’s logs stay on that node. A group created by an earlier build with more shards still works, because every group has shard 0. This aligns the first implementation with native metrics; it does not define node-local, non-replicated shards as a general contract.

The liaison route below is phase 2 and predates the single-shard group: a selection by shard over one shard would send every liaison’s events to one data node, so it is revisited with phase 2.

Decided: keep the selection, replace the transport. Reusing the liaison’s existing shard-based routing and never touching disk cannot both hold, because that route terminates in the disk-backed write queue and writes part files. The liaison therefore keeps the shard and node selection and publishes directly to the selected data node over its tier-2 client, bypassing the write queue entirely. The new native log topic serves the receiver’s foreign producers, not the liaison’s own route. Two facts make this modest rather than novel: nothing enforces shard ownership on ingest, and lifecycle already publishes straight into a colocated data node in production today.

The receiver is deliberately thin.

--logging-native-enabled governs only a process’s own collection: a node with it false still accepts, routes and persists forwarded writes.

3.4 Schema

level is an entity tag; entity tags accept only equality and set membership, so selecting several levels uses IN rather than a negation. module stays an ordinary tag — module strings splice in group, measure and task names, so entity membership would give unbounded series cardinality. Nothing stores the original line verbatim: known keys become tags, and only the remainder lands in fields.

Having no index rules costs more than scan speed. An analyzer is configured on an index rule, so with none defined MATCH has no analyzer to tokenize with and falls back to comparing the value whole — MATCH "timeout" does not find write timeout exceeded, and it reports no match rather than an error. Full-text search over message is therefore an index rule away, not available now.

What narrows a query in v1 is the time range and equality or IN on the entity tags, node_id and level: they choose the series before the scan. A filter on any other tag, such as module or message, does not narrow it. It runs on the rows the scan returns, and a timestamp-ordered scan stops at the query’s limit before the filter sees them. So a filter on module can return fewer rows than the limit while more matching rows exist. A reader looking for a module narrows by time and node first.

3.5 Buffered publishing in the queue module

Status: implemented, in answer to the review of #1419. Sections 3.2 and 5 describe the behaviour; this section describes who owns it.

Phase 1 first kept the whole buffering lifecycle in the logging module: a bounded channel, byte accounting, a consumer goroutine, batching, retries and the shutdown drain, which then published through queue.Client. That lifecycle now lives in the queue module, so that the distributed native logging of phase 2 reuses the same admission and lifecycle contract instead of a second copy.

Logger -> filter/encode -> BufferedPublisher.Offer      (logging)
                               |
                        bounded admission               (queue)
                               |
                         batch consumer                 (queue)
                               |
                    Delivery: local now, remote in phase 2

Contract

BufferedPublisher is a new interface in banyand/queue. BatchPublisher and its callers do not change: the local batch publisher still delivers synchronously when Close runs, and the distributed one still sends during Publish.

// Package queue.
type BufferedPublisher interface {
	// Offer accepts or refuses req without waiting for storage, the network
	// or free capacity, and reports the decision it made. On Accepted the
	// publisher owns req until it calls req.Release. On any refusal the caller
	// keeps req.
	Offer(req *BufferedRequest) Admission
	// Start begins delivery. Requests accepted before it are held.
	Start(d Delivery)
	// Close stops admission, drains within the drain timeout and returns even
	// when a delivery is stuck.
	Close()
	// Bytes reports the bytes charged against the allowance, and the part of
	// them in a delivery.
	Bytes() (charged, inFlight int64)
}

// Admission is the decision Offer made, taken at the moment it made it.
type Admission uint8

const (
	Accepted        Admission = iota // the publisher owns the request
	RefusedPressure                  // the memory pressure check reported high pressure
	RefusedFull                      // the allowance had no room for the request
	RefusedClosed                    // Close had already started
)

type BufferedRequest struct {
	Data    any           // the message body
	Done    func(Outcome) // terminal accounting: exactly once, by the drain deadline at the latest
	Release func()        // resource release: exactly once, after Done has returned, when neither Deliver nor Close can still read the request
	Bytes   int64         // what it holds, charged against the allowance until Release
}

type Outcome struct {
	Err       error  // the last delivery error, when there was one
	Reason    string // why it was not delivered: delivery_failed, retries_exhausted, shutdown_deadline
	Delivered bool   // only a completed delivery; acceptance never counts as written
}

type Delivery interface {
	// Deliver writes one batch. A failure wrapped with Retryable is retried;
	// any other ends the batch.
	Deliver(ctx context.Context, batch []any) error
}

func Retryable(err error) error // errors.Is finds both ErrRetryable and err
func NewBufferedPublisher(ctx context.Context, cfg BufferedConfig) BufferedPublisher
func NewLocalDelivery(client Client, topic bus.Topic, timeout time.Duration) Delivery

Phase 1 implements one Delivery, the local adapter: it publishes a batch to its topic through the local pipeline’s batch publisher, turns the error map of Close into the batch’s error, and marks a topic with no subscriber yet as retryable. The logging module wraps it to check the schema and stamp the node identity on each batch. A remote adapter is phase 2.

Guarantees

Flush interval, batch limits, retry count and deadlines remain internal, as in section 4.

What stays in the logging module

Filtering, the reserved exclusions, encoding into pooled requests, identity stamping, the schema retry, and the native_log_* counters. The counters take their reasons from the Admission that Offer returns and from Outcome values, so the closed reason set of section 5 does not change:

Done can run while a stuck Deliver still holds the request, so what the delivery writes on the request and Done reads is atomic.

Tests

The queue package tests the guarantees through the BufferedPublisher interface: concurrent Offer and Close; a writer paused after it obtained the publisher and resumed after Close returned, which gets RefusedClosed; a writer paused inside Offer, which Close waits for, and one that outlasts the deadline, whose request is still settled; each refusal returning its own Admission; queued and in-flight accounting until Release; holding before Start; the size of every retried batch and retries paced once per interval; and a stuck Deliver: Close returns at the deadline, the stuck requests receive Done with shutdown_deadline by then, and their Release runs only after Deliver returns; and a stuck Deliver that returns while Close is still inside the Done it runs at the deadline, whose Release waits for that Done to return. The logging tests keep to filtering, encoding, identity, recursion protection, and real storage and query behavior.

4. Parameters and configuration

Two independent namespaces; neither inherits from the other.

Every dashed flag binds to BYDB_<UPPER_SNAKE>, and the env value applies only when the flag was not set on the command line, so an explicit flag wins.

Level semantics are in section 3.1; the four existing --logging-* flags keep their names and defaults. Buffering, batching, write handling, shutdown and memory are internal: flush every 1s or at 100 events, 5s per publish, 5s for the final drain, the bounds in section 3.2 and the memory protector’s policy in section 5.1. None of them had an operator requirement that a flag would serve. The schema pass against metadata has its own internal timeout, separate from the write timeout: a slow destination is not a reason to wait longer on metadata.

4.1 Feedback protection

Logging caused by storing native logs must never be stored, or one stored line produces the next; it still reaches the console. Whole modules are not excluded for it, because the stream and storage modules serve every group and their logs about ordinary writes are worth keeping:

• The delivery publishes under a context marked by logger.WithNativeWrite, which carries a NATIVE-LOG logger. The stream write callback takes its logger from the context with FetchOrDefault, the existing way a logger is passed downstream, so what it logs about a native write is a child of NATIVE-LOG. Other writes log as before. Nothing is switched off process-wide.
• The storage of the log group logs under _MONITORING_LOG, including its asynchronous flushes and merges: a storage logger is named after its group.
• NATIVE-LOG and _MONITORING_LOG are reserved. An operator’s exclusion list adds to them.

Measured on an in-process standalone node at native level debug, idle after startup: about 0.5 stored events per second with the reserved set, and the same without it. The reserved set removes the node’s logging about its own log storage — about 700 lines of a 2200-line startup — and guards the paths that log on failure.

With --logging-native-enabled=false — the default — no schema and no counters are created, nothing is admitted, and the cost is one atomic pointer load per event. The ring itself is allocated either way: the sink is built while the command tree is, before the flags are parsed, so the buffer exists for the lines emitted during startup. It is one channel of 8192 request pointers and holds nothing.

5. Failure modes

event -> producer -> buffer (queue) -> delivery -> schema -> selection -> publish -> destination
                                                                                       |
                      GracefulStop: stop admission, then bounded drain ----------------+

Every stage that can fail has a row, carrying one behaviour and one reason from the closed reason set on native_log_dropped_total. Nine of the reasons exist today; locate_failed, marked phase 2, is a stage a single node does not have and arrives with remote routing.

The metric names carry the native_log_ prefix. The native meter provider names each measure with the bare metric name inside the one _monitoring group, so a name must be unique across every component that reports there.

No row blocks, retries forever, or falls back to disk. Loss is bounded, and counted everywhere except the known gap above.

5.1 Memory

The sink follows the memory protector’s own policy rather than a second one. On each admission it calls State(), the protector’s nonblocking pressure check: it reads counters and never waits, so it is safe on every goroutine that logs. While State() is high, the newest event is dropped and counted as memory_pressure. The blocking AcquireResource is never called from the logging path. Where no protector is registered, State() reports low pressure and the buffer’s own bounds hold.

The buffer still needs its own bound: protecting storage writes does not bound what waits in the buffer. It holds at most 8MiB, queued and in flight, and an event is charged its line, the memory its pooled write request holds, about 2KiB, and the capacity of that request’s fields buffer, which a request reused from a larger event keeps up to 4KiB. A typical event is charged about 2.1KiB, so the bound admits about 3800 events. At the bound the newest event is dropped as buffer_full.

Normal logging is never degraded by the sink. Native is the more verbose sink, so at normal error and native info a dropped native info event has no normal copy anywhere — the drop counters are the only record it existed.

With --observability-modes=native alone those counters travel the same transport as the logs, so the same outage loses both. Keep prometheus enabled alongside, which is what makes a loss visible during an outage. A periodic drop summary on stderr was considered and is not implemented: it would print on the path the outage already degrades, and the counters cover the same ground.