Logging

BanyanDB uses the zerolog library for logging. The log level can be set using the logging-level flag. The supported log levels are debug, info, warn, error, and fatal. The default log level is info.

Every shipped BanyanDB binary accepts the same four logging flags: the server (standalone, data, liaison), backup, restore, lifecycle, migration, the FODC proxy and the FODC agent. Each flag is also bound to an environment variable named after it, prefixed with BYDB_ and upper-cased with dashes replaced by underscores, which is the usual way to configure them in a container:

A process logs a little before the configured logger exists: the command tree registers its flags, and GOMAXPROCS is resolved. Those early lines honor --logging-level / --logging-env and their environment variables all the same – the level is read straight from the command line and the environment at that point, with the flag winning over the environment, as everywhere else. Only --logging-modules and --logging-levels are not applied that early; they take effect once the logger is initialized. Without any of them the early lines stay at debug, which is the historical behavior. The ASCII banner is printed directly to the console and is not a log line, so no level silences it.

logging-env is used to set the logging environment. The default value is prod. The logging environment can be set to dev for development or prod for production. The logging environment affects the log format and output. In the dev environment, logs are output in a human-readable format, while in the prod environment, logs are output in JSON format.

logging-modules and logging-levels are used to set the log level for specific modules. The logging-modules flag is a comma-separated list of module names, and the logging-levels flag is a comma-separated list of log levels corresponding to the module names. The log level for a specific module can be set using these flags. Modules on the server include storage, distributed-query, liaison-grpc, liaison-http, measure, stream, trace, metadata, property-schema-registry, metrics, pprof-service, query, server-queue-sub, server-queue-pub. The other binaries name theirs differently: migration in banyand-migration, fodc-proxy in the FODC proxy, and fodc, server and watchdog in the FODC agent.

Those lists are not exhaustive. A module name is whatever scope the code passes to logger.GetLogger, and every log line carries it in its module field, so reading one off a log line is the reliable way to find the name you need. Matching is case-insensitive. For example, to set the log level for the storage module to debug, you can use the following flags:

--logging-modules=storage --logging-levels=debug

Native Self-Storage

BanyanDB can store its own log events in BanyanDB, alongside the measures native observability already writes into _monitoring. It is off by default and adds a second destination rather than replacing the normal one, which stays enabled and is never degraded by it.

banyand standalone --logging-native-enabled --logging-native-level=info

Events land in the _monitoring_log group as elements of a stream named log, queryable through the ordinary stream API:

bydbctl stream query -f - <<EOF
name: "log"
groups: ["_monitoring_log"]
projection:
  tagFamilies:
    - name: "searchable"
      tags: ["node_id", "module", "level", "message"]
EOF

What is stored where

A log line is split across two tag families, and nothing is stored twice.

The split matters when you go looking for a value. A call site that writes

l.Warn().Str("group", "sw_metric").Dur("took", d).Msg("flush took longer than expected")

stores message in searchable and {"group":"sw_metric","took":1200} in fields. The query above would not show group at all — project the data family to get it:

bydbctl stream query -f - <<EOF
name: "log"
groups: ["_monitoring_log"]
projection:
  tagFamilies:
    - name: "searchable"
      tags: ["node_id", "module", "level", "message"]
    - name: "data"
      tags: ["fields"]
EOF

fields is a binary tag holding JSON, so it is returned as an opaque blob and is not filterable. A call site that adds a new key needs no schema change for it to be kept.

Filtering

node_id and level are the entity, so filtering on them selects series directly. Entity tags accept only = and IN, which is why several levels are selected with IN rather than by negating one.

Every other tag is filterable but unindexed in this version: there are no index rules on the stream, so a condition on module or message is evaluated after the scan rather than through a posting list. The scan stops at the query’s limit before that condition runs, so such a query can return fewer rows than the limit while more matching rows exist. Narrow the time range and the entity first.

MATCH on message does not do what it does elsewhere. An analyzer comes from an index rule, and this stream has none, so the value is compared whole instead of being tokenized: searching for timeout will not find write timeout exceeded. The condition does not error — it just fails to match — so an empty result may be this rather than an absence of logs. Use = on the full message, or project message and filter outside the database.

The flags live in their own namespace, --logging-native-* with BYDB_LOGGING_NATIVE_*, which inherits nothing from --logging-*. An explicit flag beats its environment variable, as everywhere else.

Native policy is resolved on a logger’s complete module, every component it was named with. A --logging-modules override can shorten the module the console prints — --logging-modules=measure makes a MEASURE.BLOCK logger print MEASURE — but it does not change what native storage collects: --logging-native-modules=measure.block and --logging-native-exclude-modules=measure.block still apply. Exclusions match a module and its children, so measure covers MEASURE.BLOCK and not MEASUREMENT.

Buffering, batching and memory are not configured. Each node writes its own events through its local write path to shard 0 of _monitoring_log, a single-shard group, as native metrics do: no event is sent to another node. The buffer holds at most 8MiB, an event over 64KiB is dropped whole, a batch is written every second or at 100 events, and a stuck write cannot hold shutdown for more than about 5 seconds. While the memory protector reports high pressure, new events are dropped and counted rather than buffered.

Because the two destinations have independent thresholds, native can be the more verbose of the two. Running normal logging at error and native at info keeps stderr quiet while the database retains the info and warn events that describe what a node was doing beforehand:

Note the consequence: an info event dropped by the buffer has no copy on stderr. Losses are counted rather than silent, under banyandb_logging_native_log_dropped_total{reason}, with banyandb_logging_native_log_written_total and the buffer gauges alongside. banyandb_logging_native_log_buffer_bytes has a state label: queued is waiting to be written and in_flight is being written, so a stalled write shows as in_flight that does not fall. Those counters travel the same transport as the events they count, so keeping --observability-modes=prometheus enabled is what makes a loss visible during an outage.

The reason label takes one of nine values, so a loss is always attributable to a stage:

Logging caused by storing native logs is never stored, or one stored line could produce the next. That logging still reaches the console. Two modules are reserved for it, and --logging-native-exclude-modules cannot remove them:

• NATIVE-LOG is the native log service. While the write path handles a native write, it logs as a child of NATIVE-LOG, for example NATIVE-LOG.STREAM. The same write path logs as before for every other write, so ordinary STREAM and STORAGE logs are collected.
• _MONITORING_LOG is the log group’s own storage. A storage logger is named after its group, so its segments, shards, flushes and merges log under it.

A blank entry in --logging-native-exclude-modules is ignored.

An event logged without a level is stored with the level none, because the level is half the series key and an empty one cannot be queried. Such an event arrives with no severity at all — a library writing through a standard log.Logger produces one — so it is admitted as an informational event and obeys --logging-native-level like any other. The buffer is bounded and in-memory only – no queue files, no write-ahead log, no disk fallback – and it never blocks the caller: at its bound the newest event is dropped and counted, so a burst keeps the head that explains it.

restore and migration do not offer these flags at all. Both run when the data tier is unavailable, and a tool that runs while the database is down cannot log into it.

Slow Query Logging

BanyanDB supports slow query logging. The slow-query flag is used to set the slow query threshold. If a query takes longer than the threshold, it will be logged as a slow query. The default value is 0, which means no slow query logging. This flag is only used for the data and standalone servers.

The dst-slow-query flag is used to set the distributed slow query threshold. This flag is only used for the liaison server. The default value is 5s; set it to 0 to disable distributed slow query logging.

When query tracing is enabled, the slow query log won’t be generated.

What these two logs contain. Both slow-query and dst-slow-query write the entire query request, including the tag filter values it was executed with. Those values come from the client and may be user data. There is no redaction option on these two flags — if that matters for your deployment, control it by raising the threshold or by routing the log somewhere with appropriate access.

BydbQL Slow Queries

BydbQL queries are tracked separately, because they are parameterized: the query text is a reusable template and the values arrive alongside it as ? parameters. Instead of logging every slow occurrence, the liaison keeps a bounded top-K of the slowest templates and dumps it periodically, so a hot bad query is reported once per interval rather than once per request. --bydbql-slow-query-threshold sets what counts as slow and --bydbql-topk-log-interval sets how often the list is dumped.

Because the template and its values are separate here, how much of the values reaches the log is a choice — --bydbql-topk-param-mode:

Numeric, timestamp and null parameters render verbatim under both fingerprint and raw: time-window width, LIMIT and thresholds are usually what explains why a query is slow, and they carry no user-identifying content. Only str and str_array are subject to the mode; binary is always digested and never rendered verbatim, not even under raw.

The reported sample is the most recent slow occurrence of that template, which is why the field is named last_params and not params — it is not necessarily the occurrence that produced max_latency.

For the complete flag list and the caveats on the fingerprint (it is unsalted, so it is not a cryptographic guarantee), see Configuration. If you need the full, unredacted parameters of a query, enable the query access log (--enable-query-access-log) instead: it records every request in full, in its own file, with its own retention and permissions.