Trace Pipeline Merge Performance Test

Status

Proposed iterative benchmark design. The benchmark is local and opt-in because it requires the downloaded real-trace shard and creates a 24-hour merge workload on local disk. Results from each phase are used to find and fix performance problems before the next phase begins; this is not a single final benchmark run.

The first generated fixtures and their reports are invalid for performance conclusions because their timestamp mapper used publication - mergeGrace + 1ns as a fallback. Although their absolute bounds covered nearly 24 hours, 1,146 of 3,219 writes had a maximum timestamp exactly one nanosecond inside the two-hour maturity frontier, and 52 writes had a maximum timestamp later than publication. Those reports remain discarded.

The corrected two-times fixture has now been regenerated from the frozen 29 July shard backup. It contains 74,576 traces, 325,570 rows, and 3,219 writes, and the default SkyWalking sampler deletes 35.3733% of its complete traces. All 3,219 write maxima are no later than publication and strictly inside the hot side of the two-hour frontier when first published. None uses the old frontier + 1ns fallback. Write maxima span 23.985 logical hours and publications span 23.993 logical hours, so the workload now represents the intended day rather than a frontier-concentrated timestamp artifact.

No-Pre-Roll Experiment

The historical pipeline-disabled iteration deliberately omitted pre-roll and used two independent production-shaped write streams in one 24-hour window. It preserves observed liaison write shapes while assigning unique IDs to every trace instance. The resulting fixture contains 74,576 traces, 325,570 rows, and 3,219 writes. Its untimed default SkyWalking validation deletes 35.3733% of complete traces.

The original requirement for ten mature selections per repetition is replaced before the next run because selection count had 23.6% variation while mature row and byte work varied by only 8.4% and 8.3%. The frozen compound maturity gate requires, in every throughput repetition:

mature input rows of at least 30% of fixture rows;
mature core input bytes of at least 30% of raw fixture core bytes;
at least three mature selections;
at least 95% of mature row work in selections containing an input with merge depth one or greater; and
no more than 15% coefficient of variation for mature rows and mature core bytes across five fresh processes.

A mature selection whose every input has depth one or greater remains visible as a non-blocking diagnostic. It is not a performance-readiness gate because mixed selections still execute the plugin over earlier merge outputs and represent the production picker trajectory. Complete logical ledgers for core, latency, and start_time, exact row counts, zero sampler calls, the Docker resource envelope, and five repetitions remain blocking gates.

The first clean run after freezing these gates remains HOLD. Mature selections processed 25.3%-49.4% of fixture rows and 24.9%-49.0% of raw core bytes. One repetition fell below both 30% volume floors, while cross-run row and byte coefficients of variation were 22.0% and 22.2%, above the 15% limit. Selection diversity and merged-output exposure passed: every run had 6-13 mature selections and 97.1%-99.9% exposure. Only one run contained a fully precompacted mature selection. Do not lower the frozen gates based on this result; it demonstrates that the two-times, no-pre-roll topology does not provide repeatable mature plugin work and is not yet a valid baseline for plugin comparisons.

Corrected Serialized Pilot Outcome

The corrected empty-shard pilot serialized each publication with its following merge drain while keeping the controller and data node in separate processes. It reconciled the core and both secondary-index ledgers exactly, published all 3,219 parts, completed 286 production-picker merges, and returned every publication barrier with zero queued, running, or in-flight merge work. It therefore validates the fixture boundary, logical clock, cross-process barrier, and lossless baseline path.

The original report incorrectly treated a merge as mature only when every selected part was mature. A mature merge is a selection containing at least one part whose maximum row timestamp is at or before logical_now - 2h. A selection may therefore be both mature and hot. Reconstructing every selected input from the immutable schedule and preceding merge events gives six mature merges, all of which were mixed with at least one hot part; all 286 merges contained a hot part. The six mature selections processed 224,788 mature input rows across their merge rounds.

This definition is deliberately independent of whether sampling executes. The event’s mature/hot input-part and row counters describe the selected data, while its sampling classification, bypass reason, and plugin-call count describe what the pipeline did. Under the current whole-selection grace guard, a mixed merge is mature for workload coverage but still bypasses sampling because it also contains a hot part.

The corrected report with per-input maturity counters confirms six mature-containing selections, so the pilot is below the unchanged ten-mature-round gate but does not establish that the empty-shard topology produces no mature work. Do not infer pre-roll necessity from the obsolete zero count. Decide the next fixture iteration from the corrected six-round result. The mature-round gate remains ten and is not replaced by merge-depth, row-volume, or input-depth gates.

Why 280 Selections Are Pure Hot

Every one of the 286 pilot selections has a newest input maximum equal to its dispatch publication clock. Usually this is the raw part just published; in recursive picker passes it is an output created earlier under the same fixed logical clock. Consequently, every selection contains a hot input. A selection becomes mature-containing only when the same size-compatible picker window also includes an older part whose maximum is at or before the two-hour frontier.

The first merge occurs at 00:08:03 after 19 publications and combines 14 raw parts. The first 24 merge rounds have no mature input available in the active inventory. A mature part first exists at the 02:20:54 dispatch, but that merge selects 11 newer raw parts and leaves the mature part behind. Across 256 later pure-hot rounds, mature parts are available but not selected; the median such inventory has six mature parts and 118,329 mature rows.

This behavior follows the unmodified size-based policy rather than timestamp priority. The picker sorts by compressed size, breaking equal-size ties by newer minimum timestamp, and searches size-balanced windows of 8-15 parts. In pure-hot rounds with mature inventory, the median mature-part size is about 3.10 MB while the median selected-part size is about 1.4 KB, a median ratio of roughly 1,765:1. The older parts are predominantly large outputs from earlier compactions, so including them with the new small writes would violate the policy’s write-amplification balance. They remain parked until enough comparable outputs accumulate.

Only six size windows bridge into mature data, at logical times 05:44:00, 08:42:29, 12:21:14, 14:51:33, 17:51:50, and 21:55:11. All six are mixed selections. The largest mature rounds are the depth-five merges at 12:21:14 (10 mature plus 5 hot inputs) and 21:55:11 (11 mature plus 4 hot inputs); the latter contains only previous merge outputs and no raw part. It is still hot because four of those outputs carry data newer than the frontier.

Revised Two-Track Test Strategy

The primary performance comparison uses one frozen, production-shaped mature merge rather than requiring the continuously writing replay to produce a stable number of mature selections. Every framework and plugin variant clones the same pre-dispatch shard snapshot, advances the test clock until every selected input is mature, and executes exactly one production-picker merge. This controlled round is the pipeline-disabled baseline and the common comparison unit for the minimal framework plugin, trace assembly, and native plugin variants.

The seed is not a synthetic part fabricated for the benchmark. Discover it by running the real two-times fixture through the unmodified picker with the pipeline disabled and pausing immediately before a representative high-generation selection. Freeze the complete core and secondary-index snapshot, selected input IDs, selection checksum, logical clock, part metadata, and logical ledgers. Preserve the whole snapshot, not only the selected directories, because conjunction parts outside the selected set can affect trace-fragment boundary protection.

The strongest current candidate is pilot sequence 263 at 21:55:11. Its production selection contains 15 merge outputs, no raw parts, input depths two through four, 147,126 rows, and 36,948,453 compressed core input bytes. Eleven inputs and 113,898 rows were already mature; four inputs and 33,228 rows were hot. For the controlled benchmark, keep the snapshot and selection unchanged but set logical time to exactly selected_max_timestamp + 2h, making all 15 selected inputs mature without changing picker ordering. A discovery rerun must capture this pre-dispatch state because the completed pilot has already consumed those input parts.

The controlled discovery rerun has now captured that exact state after 2,941 publications and 262 completed merges. The selection checksum is 45aa16460d7ca36fc0ddaa1bc1d2e73f45109dbaab86328012798d5011e60b7b. The active snapshot contains 1,621 files and 81,894,144 bytes with tree checksum 24dd0abcadafec97d80fbf99e069777365dbfdfaef07c79a05117f8a8a4e3ddf. Its manifest also freezes every active part’s in-memory merge depth because merge generation is not persisted in the production part format and would otherwise reset when a seed clone reopens.

The first five pipeline-disabled controlled repetitions all selected the same 15 inputs, matured all 15 parts, merged 147,126 input rows to 147,126 output rows, completed exactly two secondary-index children, and reconciled all three logical ledgers. Median core-round wall time is 7.061 seconds (2.59% CV), median attributed CPU is 7.176 seconds (2.66% CV), median allocation is 215.5 MiB (2.19% CV), and median peak RSS is 135.2 MiB (1.99% CV). Compressed logical output over core plus both indexes is 1.0009 times compressed input. These values establish the controlled pipeline-disabled baseline; filesystem write-byte counters remain a noisier diagnostic at 6.66% CV.

The first minimal-framework comparison now loads the real metadata-only alwayskeepsampler.so through the production SDK loader before timing. Its plugin checksum is 0966c63513576d0f7d35176f1dec172bef760ed40dfaf475faa4b9f2d6baac8c. Five fresh pipeline-disabled processes and five fresh retain-all processes used the same frozen selection in alternating order. Every retain-all repetition made two plugin calls, evaluated and retained exactly 33,353 trace IDs, dropped no trace, preserved all 147,126 span rows, merged both secondary indexes, and reconciled all three logical ledgers. Compressed logical write amplification remained identical at 1.0009 times.

This opening comparison exposes a framework memory issue. Relative to the disabled medians, retain-all allocated 56.2% more bytes and made 38.0% more allocations; median peak Go heap increased 83.4% and median peak RSS increased 72.8%. The exact-interval allocation profiles identify raw-trace staging as the first optimization target: stageRawTrace, blockMetadata.copyFrom, traceEvaluationStager.stage, and assembleStagedEvaluationBatch account for the dominant plugin-only allocation samples. The profiler is written immediately after the controlled merge, before output-ledger verification, although its cumulative allocation view also contains the identical pre-merge input-ledger scan.

Do not interpret the retain-all median wall time (6.262 seconds) or CPU time (6.462 seconds) as a speedup over the disabled medians (6.590 and 6.647 seconds). The disabled wall series has 9.87% CV and spans 5.937-7.608 seconds, while the CPU profiles are dominated by variable filesystem syscalls. Physical process write-byte counters are likewise not a semantic comparison: the two modes produce byte-identical logical core and index outputs even though those counters differ. The timing conclusion remains HOLD; the stable allocation and memory deltas are actionable. Optimize staging, then rerun both modes from the unchanged seed before adding the deterministic-drop matrix.

The controlled-round gates are:

the cloned opening snapshot and all three logical ledgers match the frozen seed;
the unmodified production picker chooses the exact frozen input-ID set and selection checksum;
every selected input maximum is at or before the two-hour frontier;
exactly one core merge and its two secondary-index child merges complete, with no recursive merge included in timing;
the pipeline-disabled baseline is lossless, while enabled variants invoke the expected plugin and reconcile against their independently generated output oracle; and
repeated processes use identical inputs and report wall time, CPU, allocation, peak memory, read/write I/O, and write amplification for that one round.

Plugin logic still has independent unit benchmarks over complete trace batches at the frozen 35% SkyWalking deletion ratio. The controlled merge measures framework, reader/writer, trace assembly, fragment guard, and plugin integration cost; the unit benchmark isolates the sampler algorithm itself.

The 24-hour serialized write-and-merge replay remains the final SkyWalking-settings integration test, not the primary performance comparison. It validates real publication, repeated policy selection, hot bypass, secondary-index updates, trace correctness, and end-to-end resource behavior. Its acceptance gate must require actual SkyWalking plugin calls; otherwise the current whole-selection grace guard can bypass all six mixed mature selections and the test can pass without exercising SkyWalking sampling. If the final replay starts from an empty shard, it therefore needs either an explicit mature controlled opening round or a sampling policy that evaluates mature data inside mixed selections. This choice must be fixed before treating the final replay as plugin integration coverage.

This two-track strategy replaces the unstable ten-mature-round requirement as the performance-comparison gate. Mature selection counts in the full replay remain an integration diagnostic rather than a prerequisite for comparing plugin costs.

Full-Replay Integration Gates

Every full-replay integration iteration starts from the same frozen opening snapshot and serializes each publication with the following merge drain while keeping the controller and data node in separate processes. The measured 24-hour two-times workload remains unchanged.

Only four conditions block a report:

Same test boundary: fixture, empty opening snapshot, binary, resource envelope, one-shard setting, and repetition count match the frozen suite identity.
Correct output: core and both secondary-index logical ledgers reconcile with the variant’s expected result, with exact row counts for lossless baseline and retain-all variants.
SkyWalking exercised: the configured SkyWalking plugin receives traces, returns decisions, and produces the expected independently verified deletions; a replay containing only grace bypasses is invalid.
Sustainable execution: every publication-to-idle barrier and the final drain complete within their timeout without OOM, unexpected sampling, leaked in-flight parts, or unfinished merge work.

Mature merge counts, mature rows and bytes, merge-depth composition, run-to-run CV, write amplification, CPU, memory, and I/O remain full-replay report metrics rather than performance-comparison gates. Merge generation is not part of maturity validity: it only describes a part’s history.

The previous mature-round instability analysis was polluted by the defective timestamp mapper. A part positioned one nanosecond inside the frontier became mature after any positive logical-clock advance, so asynchronous picker timing changed its classification. The corrected mapper preserves each trace’s internal deltas but selects the latest constant offset for which every fragment timestamp is no later than its publication. Consequently, at least one fragment of each trace is publication-aligned, all fragments are initially hot, and the two-hour grace is a real aging interval. The corrected rerun yields six mature-containing selections, below the ten-round gate; it does not by itself choose between a warmed opening state and another production-shaped fixture adjustment.

The first discovery replay at 1,400x completed with exact ledgers but used the defective fixture. Its five mature rounds and merge-depth results are discarded rather than treated as a failed readiness gate.

Objective

Measure and remove the cost added by the plugin framework, complete-trace assembly, and native trace plugins during real file-part compaction. First establish a stable pipeline-disabled baseline from the revision under test. Then measure the framework with the simplest plugin, benchmark plugin logic independently, and finally validate the complete SkyWalking configuration. The legacy plugin implementation and historical pre-plugin revisions are outside the comparison boundary.

The test must exercise the production merge picker, dispatcher, worker lanes, core and secondary-index readers and writers, trace assembly, plugin call, fragment guard, and merged-part introduction. It must not include trace ingestion, seed decoding, fixture generation, plugin compilation, or plugin loading in the measured interval.

Questions Answered

What is the correctness and performance baseline of the current ordinary merge path with the pipeline disabled?
How much wall time and CPU does the minimal plugin framework add to the merge loop?
How much additional allocation and peak memory comes from assembling complete traces?
Does the plugin change merge throughput, write amplification, or the merge-policy trajectory?
Does the two-hour merge grace prevent plugin evaluation for the hot tail while allowing evaluation for mature data?
Does the plugin see one assembled input per trace ID, including traces split between seed and boundary parts?
Is the logical output identical to the input when the benchmark plugin always retains traces?
After framework optimization, how expensive is each proposed plugin independently?
Can the final SkyWalking configuration sustain the downloaded shard’s one-shard workload inside the resource limit?

Boundary

Included

File-part selection by the default production merge policy.
Fast- and slow-lane dispatch, queueing, and merge workers.
Core trace-part reads and writes.
Shard-local secondary-index merges associated with the selected core part IDs.
Trace-ID grouping and complete-trace assembly.
Native plugin invocation through the production registry and chain.
The merge-grace maturity check and fragment-guard session.
Snapshot introduction and deletion of merged input parts.
All merge rounds caused by one serialized 24-hour logical replay.
Exactly one trace shard; the fixture is never cloned into additional shard IDs to manufacture concurrency.

Excluded

Network ingestion and WAL work.
Liaison raw-write conversion, part transfer, and data-node receipt.
Query load, retention deletion, finalization, handoff, and segment-level series-index work.
Plugin compilation and initial plugin.Open cost.
Fixture cloning and logical-output verification.
Performance of a future query-time trace-latency field.

The shard-local secondary indexes are part of the primary end-to-end merge workload. A separate core-only diagnostic may disable them to attribute CPU more precisely, but it must not replace the production-shaped primary result.

The untimed fixture builder uses the production data path only to create valid inputs. A liaison-side encoder converts each frozen raw-write batch directly into a core part and its secondary-index parts. The production part-transfer handler delivers them to an isolated data-node table, which persists each received part immediately. No accumulation or timed flush policy is modeled. The resulting files become immutable merge inputs, and none of this setup is measured.

Resource-Limited Container

Every measured run must execute in a fresh resource-limited Docker container. Running the benchmark directly on the host is permitted only for development and cannot produce a reportable result.

Only the data-node container belongs to the measured cgroup. A separate fixed-version controller process or container drives logical time and part publication, and its CPU, heap, and allocations are never mixed with data-node results. The controller is pinned away from the data-node CPUs and reports its own scheduling lag and failures separately.

The canonical container profile is:

four pinned logical CPUs with a four-CPU quota;
GOMAXPROCS=4;
8 GiB memory limit;
memory swap limit equal to the memory limit, preventing additional swap use;
a fixed PID limit of 512;
the same read-only seed mount and a new writable fixture volume for every run;
the same image digest, benchmark binary, plugin binary, filesystem, and storage device for all variants.

The profile may be changed for an explicit resource-scaling matrix, but baseline and plugin variants in one comparison must have identical limits. Every report records both the requested Docker limits and the CPU and memory limits detected inside the container. A mismatch invalidates the run because merge concurrency and staging budgets are derived from the detected cgroup resources.

The four-CPU limit is an upper resource envelope, not a utilization target. Low utilization from the single shard is a valid result and must not be corrected by cloning the workload, increasing shard count, or forcing extra merges. This design measures plugin impact on the downloaded shard; it does not establish whole-node saturation capacity.

Fixture generation uses the same image, binary, schema, encoders, and cgroup profile as the measured runs. Generate and validate the immutable fixture once, then clone it for every measured variant. Part boundaries come from the frozen liaison input batches, not from resource-derived memory limits or a flush timer.

The monitor captures cgroup v2 CPU usage and throttling, memory current/peak/events, I/O bytes and operations, PID events, and CPU, memory, and I/O pressure. CPU throttling, OOM events, or nonzero swap use do not automatically invalidate a stress run, but they must be highlighted and cannot be compared with an unthrottled run as if the environments were equivalent.

Docker containers share the host page cache. Fresh containers alone do not create cold-cache runs. Cache policy must therefore remain unchanged across variants, and reports must label the series as warm, cold under externally controlled host cache, or uncontrolled. Results from different cache policies must not be combined.

Real-Data Seed

The fixture separates write-batch, tail-fragment, and semantic seed roles. Each smallest source part is the persisted result of one real liaison write batch. These parts provide empirical batch templates plus 254 real hot-tail traces and observed cross-part fragment cases; they are not the sole semantic row population. Their initial manifest is:

0000000000002266
000000000000226b
0000000000002265
0000000000002259
0000000000002233
000000000000222c
0000000000002256
0000000000002264
0000000000002223
0000000000002248
000000000000223a
00000000000021f1

The twelve observed batches total 301,026 compressed bytes and 254 distinct trace IDs. Individually they contain 3-49 blocks, 14-262 rows, 6,171-121,692 uncompressed span bytes, and 4,542-54,450 compressed core bytes. Eighteen selected trace IDs also occur outside this selection in the original shard. At the shard’s logical end time, all twelve parts fall inside the two-hour grace window and must merge without invoking a plugin.

The batch template uses only the observed post-encoding block count and row count. In data-node receive order, represented by ascending part ID, the frozen template cycle is:

Part ID	Blocks	Rows
`00000000000021f1`	45	262
`0000000000002223`	46	117
`000000000000222c`	13	31
`0000000000002233`	9	36
`000000000000223a`	49	188
`0000000000002248`	32	180
`0000000000002256`	26	164
`0000000000002259`	7	33
`0000000000002264`	32	137
`0000000000002265`	6	28
`0000000000002266`	4	14
`000000000000226b`	3	24

Fixture generation repeats this ordered cycle. The counts are batch-composition targets, not BanyanDB configuration. Compressed and uncompressed byte fields remain reported outcomes and are not used as batching limits.

Batch boundaries use a deterministic closest-boundary rule over the chronologically ordered, indivisible physical fragments. For template boundary k, let (B_k, R_k) be the cumulative target blocks and rows through that boundary and let (b, r) be cumulative realized counts. Compare the boundary immediately before the next fragment with the boundary immediately after it using:

distance = abs(b - B_k) / templateBlocks_k + abs(r - R_k) / templateRows_k

Choose the lower distance. A tie chooses the earlier boundary, except that a batch must consume at least one fragment to make progress. Using cumulative targets prevents one batch’s underfill or overshoot from drifting through later batches. The generator never reorders, splits, or drops a fragment to improve the match.

When the measured input ends before the next template boundary, emit all remaining fragments as one partial_tail batch. Do not pad it, carry it into another logical day, or redistribute it into earlier batches. It remains part of the merge workload and all correctness and resource totals, but it is excluded from batch-template fit statistics. A fixture has at most one measured partial-tail batch.

The twelve batches contain 1,214 rows, so one observed write contains an average of 1,214 / 12 = 101.1667 rows. The generated day’s actual row count determines the write rate:

writesPerDay = ceil(generatedRows * 12 / 1,214)

writesPerMinute = writesPerDay / 1,440

The downloaded shard contains 162,238 rows. At that reference volume, the mock writer introduces 1,604 parts per day, or 1.1139 parts per logical minute, with one part due every 53.8653 logical seconds. The final hybrid fixture recalculates the rate from its immutable row ledger rather than changing its trace population to force the reference number.

Create exactly writesPerDay evenly spaced logical publication slots across the half-open measured day. Repeat the twelve (blocksCount, totalCount) templates across those slots; use the closest-boundary rule for the first writesPerDay - 1 boundaries and put every remaining fragment in the final batch. The final batch is marked partial_tail only when it does not satisfy the next complete template. The writer introduces each already-generated part at its slot while the production merge loop remains continuously active.

The semantic seed is the two-hour-mature production population from these six parts:

00000000000005ee;
0000000000000b5b;
0000000000001002;
0000000000001191;
000000000000165a; and
0000000000001c36.

They contain 31,844 physical blocks, 31,832 distinct trace IDs, 138,686 rows, and 34,856,465 compressed bytes. Their trace IDs do not overlap the 254 structural-seed IDs. Thirty-one mature trace IDs also have 245 rows in 00000000000021d4; those rows are part of the semantic closure even though the unrelated rows in that hot part are not.

The measured payload contains all 31,832 mature traces, all 254 closed structural-seed traces, and one additional copy of 5,202 mature traces selected by an independent stable hash. Every copied trace receives a new ID and includes all of its physical fragments. This produces exactly 37,288 distinct traces while retaining mature production variety and a small amount of the real hot-tail fragmentation pattern.

Before fixture generation, re-read metadata and reject the source when any part ID, count, byte total, closure allowlist, or source-manifest hash changes. The source remains read-only throughout the test.

The fixture closes those 18 traces by extracting only their missing rows from five outside source parts:

0000000000002218: eight trace IDs and 63 rows;
000000000000222b: six trace IDs and 28 rows;
0000000000002267: one trace ID and six rows;
000000000000226a: one trace ID and 15 rows; and
000000000000226c: two trace IDs and 18 rows.

Every affected trace occurs in exactly one of these outside parts, for 130 extracted rows in total. Unrelated rows from the five carrier parts are excluded. The mature closure similarly extracts only the 245 rows for its 31 allowlisted trace IDs from 00000000000021d4. Generated closure fragments retain separate boundary roles so the affected trace IDs remain split across normal parts. These parts participate in the production snapshot, picker, merge, and fragment guard; they are not hidden test-only guard inputs. Their generated bytes are reported separately because extraction changes their size relative to the original carrier parts.

Daily Volume Calibration

The primary 24-hour workload matches the complete downloaded shard rather than merely extrapolating one copy of the small-part seed. The reference targets are 37,288 distinct traces, 162,238 rows, and 40,879,799 compressed core bytes. The downloaded secondary-index trees occupy 2,798,752 bytes for latency and 2,404,977 bytes for start_time. Their part-metadata compressed payload totals are 2,793,388 and 2,398,716 bytes respectively. Generated raw tree bytes are reported independently, while density gates compare production-consolidated payload bytes to these like-for-like source payload totals.

The mature population, structural seed, and 5,202-copy extension have an estimated source-equivalent core size of 40,853,750 bytes, within 0.1% of the downloaded daily target before production re-encoding. The 5,202 templates are the lowest stable hashes over the complete mature trace IDs. This choice is independent of sampler class, error, latency, tags, trace size, and performance results; freeze its allowlist and hash in the manifest.

Distribute the mature semantic population across the 24-hour write schedule while preserving each trace’s intra-trace timestamp deltas, duration, fragment gaps, relative row order, errors, and tags. Absolute timestamps may shift by one constant per trace to place it in the measured day; this does not change the sampler’s latency or error decisions. Every raw part must still be hot when introduced under the two-hour grace. Divide chronological fragments into the calculated number of liaison raw-write batches, convert every batch through the production part encoder, transfer it through the production part-sync contract, and persist it in an isolated merge-disabled data-node table. There is no fixture flush clock or flush threshold. The mature rows supply the dominant semantics; the smallest parts contribute real batch templates, tail traces, and fragment cases. Repeat the twelve ordered (blocksCount, totalCount) targets while assigning whole physical fragments; never split a fragment to hit a target. Batch boundaries come from those observed templates, never from a synthetic byte budget that BanyanDB does not use.

Before freezing the fixture, run the actual default SkyWalking sampler over every complete generated trace after trace-ID mapping. Its deletion ratio must be between 34.5% and 35.5%. This is a validity gate, not an optimization target: do not change trace IDs, the 5,202-template allowlist, content, or plugin configuration after observing verdicts. Failure means the seed does not represent the mature calibration and must be redesigned. The controlled DeterministicDrop plugin remains configured at 35% and reports its realized per-selection ratios without a result-selected hash salt.

Encode the complete candidate through the production liaison encoder and data-node receiver. Require generated rows and production-consolidated core bytes to be within 2% of the 162,238-row and 40,879,799-byte references. Require each combined production-consolidated secondary-index bytes to be within 5% of their reference total and each individual index to be within 10% of its reference. The per-index allowance covers key-distribution compression skew from the independently hash-selected complete trace copies, while the tighter combined gate preserves total secondary-index I/O. The consolidation is an untimed, lossless production merge into the source shard’s 26-part cardinality; it validates logical data density without conflating it with the unavoidable framing cost of the approximately 1,604 raw input parts. Preserve and report the raw fixture byte totals separately because those bytes, not the consolidated calibration outputs, are the merge workload. Remove the temporary consolidation outputs and prove that the raw fixture remains queryable. Do not retune input batches, timestamps, trace mapping, or content to satisfy these gates after observing output. The fixture may not add padding, copy unrelated rows, or drop partial traces. If the gates cannot be met, the volume model must be redesigned rather than accepting an unreported scale mismatch.

Daily trace, row, and byte volume remain the primary scale gates. The twelve write batches guide input grouping, but they are a small tail sample rather than proof of the shard-wide production batch distribution. Full-loop results compare plugin variants under one identical frozen topology; they cannot estimate production merge frequency or fan-in without broader batch evidence. Reports must expose generated batch count, part-size histogram, publication gaps, and picker trajectory.

The downloaded shard also contains shard-local secondary-index trees under sidx/<index-name>/<part-id>. These are different from the segment-level sidx/*.seg and sidx/*.snp files, which are the series index. The fixture manifest must pair every generated core part with every secondary-index part having the same part ID and record the source role, index schema, and bytes separately.

The manifest should also record seed span count, block count, minimum and maximum timestamp, per-part compressed size, and a checksum of the logical trace rows. These values make later benchmark runs comparable and make accidental fixture changes visible.

Default SkyWalking Ratio Calibration

The realistic DeterministicDrop point is derived from the actual SkyWalking sampler rather than an arbitrary 50%. SkyWalking PR 13965 leaves the pipeline disabled by default because it deletes stored data. For calibration only, the pipeline was enabled while retaining its shipped sw_trace sampler defaults: a 500 ms duration threshold, error retention enabled, no tag rules, and a 10% deterministic sample of the remaining healthy traces. This distinction is important: the literal disabled configuration deletes nothing.

The calibration used the downloaded 24-hour shard with 26 core parts, covering 2026-07-29 00:00:00.587 through 23:59:56.171 UTC. Its source-manifest SHA-256 is 7291ff8abedb1c6d31bb98356a89ed0bee4020d1ff0f602376b9028d0cf8c510. The input contained 37,404 physical trace blocks and 37,288 distinct trace IDs; 113 trace IDs occurred in more than one part. Because the files predate persisted block timestamp bounds, the probe first performed a lossless production merge, asserted that no trace was dropped, and asserted that the normalized output contained exactly one block for each of the 37,288 trace IDs. It then invoked the actual native sw-trace-sampler through the merge path.

The source-manifest hash covers the sorted relative path and content hash of every file in the shard. The clean mature-only replay opened only the six eligible part IDs listed below. Its eligible-input manifest SHA-256 was 445e46f7d2a0d17976b8efb45b9479121e333a907ea6bfbd5a36b5e9f9bef2fa, sampler binary SHA-256 was 5768529e6087c00acc246bea4aceb4876dc9d3b954d6bb1a1a95d1c2fe550618, and exact OAP-shaped configuration SHA-256 was 6466931a246d608cd5994feec5daec6091f36e91e659437b01e62ce413ecb2ce. The configuration passed healthySampleRate in the quoted-string form produced by OAP’s environment-placeholder loader.

The complete-shard verdict breakdown was:

29 traces retained by the duration rule;
22,789 additional traces retained by the error rule;
14,470 healthy, fast traces reaching the hash rule, of which 1,475 were retained;
24,293 traces retained in total; and
12,995 traces deleted, a 34.8504% deletion ratio.

With logical time fixed at 2026-07-29 23:59:56.171 UTC, the two-hour maturity frontier is 21:59:56.171 UTC. Exactly these six downloaded source parts are mature by their maximum timestamps:

00000000000005ee;
0000000000000b5b;
0000000000001002;
0000000000001191;
000000000000165a; and
0000000000001c36.

The next large part, 00000000000021d4, reaches 23:41:48.898 UTC, so any selection containing it is hot even though the part begins at 18:07:40.302 UTC. A single hot part bypasses sampling for the whole merge.

After lossless normalization, the six mature parts contained 31,832 distinct trace IDs. The actual default sampler retained 26 traces by duration, 19,377 additional traces by error, and 1,287 of the remaining 12,429 by the healthy hash. It retained 20,690 and deleted 11,142, a 35.0025% deletion ratio. The complete-shard ratio of 34.8504% is consistent with that result. The 35.0025% value is a production-data-calibrated sampler verdict and is suitable for shaping the Phase 2 DeterministicDrop workload. It is not an observed production merge deletion ratio.

The twelve smallest parts produced a 40.9449% deletion ratio when forced through the sampler, but that result is not used for performance load shaping because a real two-hour-grace merge would bypass the plugin. They contribute only their 254 closed traces and real tail-fragment cases. The mature population supplies the dominant semantic content of the generated fixture, so the default SkyWalking plugin naturally sees the production-calibrated mix instead of a synthetically reweighted sample. Their original part sizes do not control the generated topology.

The current production picker selects no parts from this frozen snapshot. For the six mature parts, the best admissible size window has a merge multiplier of 3.92791341, below the default policy’s required 7.5. Including all 26 parts still produces only 5.52848485, also below 7.5. Therefore the mature-only 35.0025% result is a clean offline plugin verdict but does not describe a merge that production dispatched at that instant. This picker result does not invalidate 35% as a representative deletion-load input for the framework benchmark; it limits the claim that can be attached to it.

The benchmark distinguishes three ratios:

R_calibrated is the 35.0025% actual-plugin verdict over the clean mature production-data population and controls the Phase 2 DeterministicDrop setting;
R_merge_verdict is the trace-weighted plugin verdict over exact mature selections dispatched during replay; and
R_effective is the ratio actually removed after grace, fragment-guard, bypass, retry, and publication handling.

R_calibrated is an external day-long production calibration. Never infer it from the small-part seed and never reweight latency, error, tag, or span content to manufacture it. The hybrid seed combines mature semantic data with the real small-part tail traces and must naturally pass the 35% actual-plugin validity gate. The fixed 35% DeterministicDrop policy controls Phase 2 deletion load.

Measure the latter two from the serialized production loop. Run the pipeline-disabled loop first, capture every selection for which all parts are beyond the two-hour frontier, and replay those selections offline through the actual sampler without changing the baseline picker trajectory. Freeze the selection-manifest hash and report any difference from R_calibrated; do not silently retune the Phase 2 matrix after seeing performance results.

The ratio calibration must be rerun when the source manifest, trace-ID mapping, merge schedule, merge policy, or SkyWalking sampler configuration changes. Reports must show the eligible plugin verdict ratio separately from the effective deletion ratio after merge grace and fragment-guard decisions.

Building a 24-Hour Fixture

Fixture construction happens once before profiling and is never timed.

Decode the six mature semantic parts through the production trace block reader.
Decode 00000000000021d4, extract exactly the 245 allowlisted rows needed to complete the 31 mature boundary trace IDs, and reject any missing or unrelated row.
Decode the twelve structural parts and the five structural boundary carriers. Extract exactly the 130 allowlisted rows needed to complete the 18 structural boundary trace IDs, excluding every unrelated carrier row.
Build two immutable complete-trace catalogs. Verify 31,832 mature IDs, 254 structural IDs, zero overlap between the catalogs, and a logical checksum for every trace and physical fragment.
Select the 5,202 mature templates with the lowest values under the frozen independent hash. Add one complete copy of each selected template. Map every measured trace instance to a unique fixed-width ID; every physical fragment of an instance receives the same mapped ID. Derive the fixed-width ID from an independent source-ID-only family prefix and an instance digest. This preserves the repeated producer-prefix compression present in real SkyWalking IDs without consulting trace content, sampler verdicts, index size, or benchmark output; copied instances share only the family prefix with their source and retain a unique instance suffix.
Calculate writesPerDay from the immutable generated-row ledger and the observed 101.1667 rows per write. Create that many evenly spaced slots in one measured logical day. For each complete trace, calculate the latest constant timestamp offset for which every fragment maximum is no later than its assigned publication. Preserve intra-trace deltas, fragment gaps, and relative row order. Reject the schedule unless every fragment is non-future and inside the two-hour grace when introduced. This makes trace maxima follow the 24-hour publication schedule instead of concentrating at the grace frontier. Generate a deterministic preceding pre-roll at the same part rate only after validating this mapping.
Divide physical fragments in deterministic event order into immutable liaison raw-write batches. Repeat the twelve observed (blocksCount, totalCount) templates in ascending source part-ID order. Assign only whole physical fragments. At every cumulative template boundary, compare stopping immediately before or after the next fragment with the frozen normalized-distance formula and choose the closer boundary. Do not reorder or split a fragment, reset cumulative error between batches, or introduce a byte-budget flush rule. Schedule each allowlisted boundary fragment in a later batch than its companion so the resulting parts remain separate and the production fragment guard can observe them. Use exactly the calculated number of publication slots and emit every final remainder in the last batch, marking it partial_tail when it is not a complete template match.
Convert each batch through the production liaison part encoder and send the resulting core, latency, and start_time parts through the production part-sync contract to an isolated data-node receiver. The receiver persists each part immediately; merge workers, plugins, and WAL are disabled. Capture the received files without rewriting or manually splitting them, then freeze their boundaries and logical publication times before sampling or profiling. The downloaded shard supplies the already-resolved secondary-index series IDs and post-rule SIDX tags for each row; fixture generation preserves those records and sends them through the same SIDX mem-part encoder used after liaison rule processing, rather than reconstructing a potentially different historical schema or dropping projected tags.
Persist a schedule manifest containing source catalog and role, source trace ID, copy ordinal, mapped trace ID, part ID, template ordinal, full or partial_tail status, target and realized block and row counts, cumulative targets and counts, before/after boundary distances and choice, logical publication slot, timestamp bounds, core and per-index bytes, trace counts, content checksums, source-manifest hash, closure-allowlist hashes, copy-allowlist hash, liaison-encoder and schema hashes, raw-write-batch template and schedule hashes, and received-topology hash.

The fixed-width trace-ID mapping prevents template copies from becoming one false logical trace. It also keeps key lengths stable and prevents a changing string prefix from distorting compression more than necessary. The mapping and copy allowlist are chosen before any sampler verdict and cannot be tuned to reach the ratio gate.

The complete fixture must satisfy these gates before a measured run:

the configured publication window is exactly one half-open 24-hour logical day, every publication and mapped trace maximum falls inside it, mapped maxima span the publication day, and the timestamp mapping preserves every intra-trace delta and fragment gap; an earlier trace row may precede the publication-day boundary only by its preserved trace duration;
the measured window contains exactly 37,288 distinct mapped trace IDs;
the generated row ledger is within 2% of the 162,238-row source reference, and writesPerDay exactly equals ceil(generatedRows * 12 / 1,214);
the population ledger contains exactly 31,832 mature base instances, 254 structural instances, and 5,202 additional mature instances, with no duplicate mapped ID or overlap between the source catalogs;
the mature source closure contains exactly 245 allowlisted rows for 31 trace IDs from 00000000000021d4, and the structural source closure contains exactly 130 allowlisted rows for 18 trace IDs from the five declared carriers; neither catalog contains an unrelated carrier row;
every base or copied instance is complete, includes all applicable closure rows, maps all physical fragments to the same ID, and preserves its source intra-trace timestamp deltas, fragment gaps, and relative row order;
the actual default SkyWalking plugin deletes between 34.5% and 35.5% of all complete generated trace instances and its per-rule counts, configuration hash, binary hash, mapped-ID manifest hash, and verdict checksum are recorded;
production-consolidated core bytes are within 2% of 40,879,799 bytes, combined consolidated secondary-index bytes are within 5% of their downloaded-shard total, and each index is within 10% of its own reference; raw fixture bytes and 26-part calibration bytes are both reported;
the pre-roll ends at a dispatch barrier immediately before the first mature production-picker selection;
every merge completed before that barrier is hot under the two-hour grace, while the pending selection is entirely mature and satisfies the unmodified production merge policy;
all part bounds enclose the timestamps decoded from their blocks;
no mapped trace ID is reused in pre-roll or crosses a logical-day boundary;
every generated instance of an affected structural or mature trace remains split across separate normal part roles;
every raw-write batch was converted by the production liaison encoder and immediately persisted through the production data-node part receiver; no configurable accumulation or flush deadline affected the topology;
every generated batch follows the frozen contract extracted from the twelve observed source batches, and the fixture contains no synthetic byte-budget or flush-threshold parameter;
the template sequence repeats in ascending source part-ID order, no source physical fragment is split to improve a batch match, every boundary is reproducibly the closest chronological boundary under the frozen formula, and target-versus-realized block and row deviations are recorded;
the measured schedule has at most one partial_tail batch, it is last, it contains every remaining fragment exactly once, and only this batch is excluded from template-fit statistics;
the fixture contains exactly writesPerDay evenly spaced publication slots, each part is introduced once at its slot, every newly introduced raw part is inside the two-hour merge grace, and no fragment timestamp is later than its publication;
no compaction, plugin evaluation, or WAL write occurred while creating source parts, and transfer time is outside the benchmark interval;
the logical checksum before liaison conversion equals the checksum reopened from the received core and secondary-index parts;
every benchmark variant uses the exact frozen part boundaries, publication schedule, and received-topology hash;
all parts can be reopened by the production file-part reader;
every generated core part has the expected secondary-index parts with the same part ID;
trace lookups for every generated ID return the expected row count; secondary-index full scans return the exact query-semantic projection of the rewritten row multiset (including native per-block data deduplication), and keyed lookups cover every distinct encoded trace ID with only valid ledger rows and never exceed the corresponding physical row multiplicity (the keyed implementation deduplicates data within each scanner batch, not globally across batches);
the default merge policy can select at least one set of parts;
total input rows and the logical checksum match the schedule manifest.

Logical Clock and Serialized Publication

The test must not wait 24 wall-clock hours. Add a test-only clock seam used by the merge maturity check; production still uses the real clock. The logical clock starts at the pre-roll’s first timestamp and advances directly to the next publication only after the preceding merge-idle barrier completes. Before introducing a part scheduled at P, the controller sets the data node’s logical current time to exactly P; logical time remains fixed at P until every merge triggered by that publication has drained.

Only if the corrected empty-shard serialized pilot fails the mature-round gate for a demonstrated startup-state reason, evaluate pre-roll as a separate redesign; it is not part of the current run. Determine any proposed pre-roll length in an untimed discovery pass. Publish the preceding hybrid schedule through the real production merge loop with the pipeline disabled, advance the logical clock, and inspect each proposed dispatch. Stop when the picker first proposes a selection for which every part maximum timestamp is at or before logical_now - 2h. A benchmark-only dispatch barrier must pause before that merge starts. Fail fixture generation if no such selection appears within 24 logical hours.

Rebuild the pre-roll from the immutable schedule to the discovered barrier and verify its opening inventory, logical checksum, picker-selection fingerprint, and logical duration against the discovery manifest. Every merge executed during pre-roll must be hot, making pipeline-disabled behavior equivalent to an enabled pipeline’s grace bypass. Freeze and clone this warmed state for every variant. Timing begins when the barrier is released, so the first controlled merge is both production-dispatchable and mature. Record opening core and secondary-index inventories separately from the 24-hour measured input.

All file parts are generated before measurement. During a run, the driver publishes scheduled parts by atomic rename and snapshot introduction, sets the logical clock to that part’s publication, notifies the production merge loop, and waits for verified merge-idle before continuing. Merge-output notifications may cause recursive production picker passes, but the controller cannot introduce the next write or advance logical time until all of them finish. There is no concurrent publisher mode in a reportable run.

The driver is an external controller, not a goroutine in the measured data-node process. Before timing, every immutable part is staged on the same filesystem as its destination so publication is an atomic rename rather than a timed byte copy. The controller performs the rename and calls a test-only introduction/clock endpoint. The data node still performs the real part open, snapshot introduction, picker notification, and merge work; those operations remain measured. The controller cannot rewrite part contents or invoke the merger directly.

The merge-idle barrier is epoch-aware: it may return only after the dispatcher has observed the snapshot epoch created by the publication, found no selectable parts for that epoch, and both worker lanes have no queued, running, or in-flight work. Checking only empty queue counters is insufficient because the dispatcher might not yet have consumed the publication notification. Fixed sleeps are not acceptable.

The controller and data node remain different processes. The external controller owns file publication and logical-time advancement; the resource-limited data-node process owns snapshot introduction, production picking, core and secondary- index merging, and plugin execution. Controller CPU and memory remain outside measured data-node resources.

The merge grace is exactly two logical hours. At a 12:10 publication, the maturity frontier is exactly 10:10; a part whose maximum timestamp is at or before 10:10 is mature. A merge containing that part is a mature merge even when it also contains the newly written 12:10 part. Such a selection is both mature and hot. Under the current whole-selection grace guard, the hot part causes sampling to be bypassed; the maturity classification and sampling decision are separate reported dimensions. The driver records mature and hot input-part counts, their row counts, the frontier, and the sampling decision before dispatch.

After the last scheduled publication and its merge-idle barrier, advance only to the measured day boundary and drain one final picker epoch. This completes the primary 24-hour phase without prematurely maturing the tail. Record a phase boundary and resource-counter snapshot.

The cooldown phase then advances logical time by two hours, triggers the merge loop without creating a synthetic data part, and drains to merge-idle again. This trigger requires a test-only notifier seam. Cooldown work is reported separately because it represents deferred tail processing rather than continuous 24-hour writing. The sum of primary and cooldown work is a secondary end-to-end result used for complete-output accounting.

This schedule deliberately includes two states:

rolling hot-tail compaction, in which newly published parts bypass sampling;
mature compaction of parts produced by earlier hot merges, in which the plugin processes complete traces.

The approximately 1,604 raw parts at the reference row volume must create numerous policy-driven merges while writing continues. Require at least ten grace-bypassed core merges and ten mature plugin-executed core merges during the 24-hour publication interval, before the artificial cooldown. Every plugin-executed input part must have merge depth of at least one, proving that raw write parts compacted while hot and sampling ran only on their later merged outputs. A raw part sampled only after the final cooldown does not satisfy this gate. If natural policy selection cannot meet it, redesign the fixture rather than forcing a selection.

Per-Merge Sampling Classification

The benchmark monitor must classify every core file merge independently. Inferring sampling from a difference between global counters is insufficient because sampled and unsampled merges can occur in the same logical replay.

Each merge receives a benchmark-local sequence number used only in the JSON event stream, not as a metrics label. At dispatch, the monitor records selected part count and bytes, part time envelope, logical time, maturity frontier, lane, input merge-depth range, and the initial pipeline decision. Raw writer parts have depth zero; every merge output has one plus the maximum input depth. At completion, the monitor records plugin calls, evaluated traces, output bytes, secondary index child merges, duration, CPU, allocation, memory, and I/O deltas.

Every event has a low-cardinality sampling classification:

executed: at least one plugin Decide call evaluated at least one complete trace;
enabled_no_evaluation: a filter was installed, but no trace reached Decide, for example because all traces were oversized-bypassed;
not_executed: no filter was installed or the merge took a lossless retry path.

not_executed and enabled_no_evaluation also carry exactly one reason:

pipeline_disabled;
no_sampler;
event_disabled;
merge_grace;
fragment_gap_contract;
guard_unavailable;
all_traces_oversized;
lossless_retry;
empty_input;
other.

The binary report groups executed as sampling and the other two classifications as no sampling, while preserving the reason breakdown. A merge is never marked executed merely because a sampler was registered; actual Decide calls and evaluated-trace counts are required.

Secondary-index merges inherit the sequence number and sampling classification of their parent core merge. They are reported as child work rather than independent sampling decisions because the plugin runs only against the core trace merge.

The monitor emits both a JSONL event file for per-merge analysis and aggregate counters/histograms labeled only by sampling classification, reason, and lane. Part IDs and merge sequence numbers must not become Prometheus labels.

Attribution Run

Per-merge CPU, memory, and I/O cannot be attributed reliably when sampled and unsampled merges overlap. Therefore the mandatory attribution run limits active merge concurrency to one while retaining the four-CPU container. The monitor captures process resource counters immediately before and after each merge and samples memory during the interval. With query, flush, finalization, and ingestion work disabled, the deltas belong to that merge and its secondary-index child work. The driver also waits for merge-idle before publishing the next scheduled batch.

The pipeline-disabled and AlwaysKeep attribution runs must produce the same ordered selection fingerprint, including input part lineage, fan-in, lane, and logical output contents. Wall time may differ, but closed-loop publication prevents that latency from changing which parts are available to the picker. Any trajectory divergence is a framework issue and invalidates attribution until explained and fixed. DeterministicDrop is exempt because deleting output intentionally changes later policy inputs.

Serialized Full-Loop Resource Run

A second full-loop run uses production-derived merge concurrency for the four-CPU container. It reports aggregate daily resource use. Per-merge wall time, selection, sampling classification, and bytes remain valid, but CPU and peak-memory attribution by sampling class is advisory when merge workers overlap within one drain epoch. The report must not present overlapping cgroup deltas as additive per-class CPU totals. The controller publishes the next part only after the current merge loop reaches epoch-aware idle. Report logical input volume per data-node CPU second and active merge wall time; do not interpret serialized end-to-end wall time as production ingestion capacity.

Only the downloaded shard is active. Report effective CPU utilization and available headroom, but do not interpret unused CPUs as evidence that the data node could sustain a proportional number of additional shards.

All variants start from the same opening state and serialized publication schedule. Plugin latency cannot change which raw parts coexist at a picker epoch. Pipeline-disabled and AlwaysKeep must therefore have identical selection fingerprints. A dropping plugin may diverge because its output sizes and logical contents intentionally differ; report the first divergence rather than forcing the baseline plan.

Benchmark Variants

Each variant starts from an independent clone of the same immutable generated fixture and uses the same logical clock and publication schedule.

1. Baseline: Ordinary Merge

Run only the revision under test.
Disable the native trace pipeline and register no sampler.
Do not load a plugin or perform filtering, projection, or complete-trace staging.
Use the same frozen opening state, schedule-manifest checksum, binary, Docker image digest, Go toolchain, resource limits, filesystem, storage device, cache policy, merge policy, and merge concurrency as the plugin variants.

This pipeline-disabled run is the sole baseline for the iteration. Establish its correctness, merge trajectory, repeatability, and performance envelope before plugin-enabled measurements begin. Record its commit and artifact hashes; after a production change, rerun and freeze a new baseline rather than comparing against the legacy plugin or an incompatible historical file format.

2. Minimal AlwaysKeep Plugin

Native trace pipeline enabled.
Two-hour merge grace.
Fragment-gap declaration set to a valid value no greater than the merge grace, so the guarded path can run.
A real native plugin is loaded before timing.
The plugin projects no optional columns and retains every assembled trace.
Apart from constructing the required verdict mask and maintaining test counters, it performs no application logic.

This variant measures the minimum framework cost: trace grouping, staging, SDK batch construction, native ABI dispatch, verdict handling, guard setup, core rewriting, and ordinary secondary-index merging. Its output must be logically identical to the pipeline-disabled baseline.

3. Minimal DeterministicDrop Plugin

Use the same engine configuration and empty projection as AlwaysKeep.
Decide from a stable hash of TraceID only, using a fixed configured drop percentage.
Record the configured and observed drop ratios.
Perform no tag decoding, span decoding, logging, telemetry, or other application logic in the timed path.

This variant exercises dropped-ID collection, core trace removal, secondary-index pruning, and atomic publication without mixing projection or business-logic costs into the framework measurement. The hash function and threshold must remain identical across all iterations.

Use the following drop matrix:

1% to expose fixed framework cost when output size is nearly unchanged;
35% as R_calibrated, the clean mature production-data sampler ratio; and
99% to stress dropped-ID tracking, deletion, and secondary-index pruning while minimizing output writes.

Run all three points in the controlled mature-selection comparison and short validation fixture. Run 35% for the mandatory full 24-hour Phase 2 DeterministicDrop workload. The 1% and 99% points need a full 24-hour run only when the shorter results reveal a nonlinear cost or correctness anomaly. Phase 4 separately reports R_merge_verdict and R_effective from the actual SkyWalking plugin; those observed ratios must not be relabeled as the controlled 35% case.

4. Final SkyWalking Plugin Configuration

After framework and plugin-unit optimization, run the actual SkyWalking plugins and settings, including their real projections and sampling rules. Dropped output reduces write I/O, so compare this result using retained bytes and work normalized by input traces, spans, and bytes rather than interpreting a lower elapsed time as lower framework overhead.

Iterative Execution Phases

Phase 1: Protect the Ordinary Merge Path

Run the current revision with the pipeline disabled. Validate output correctness and profile selection trajectory, wall time, CPU, allocations, memory, I/O, and amplification. Fix flaws observable on this ordinary path and repeat until at least five runs establish a stable baseline envelope. The legacy plugin implementation is scheduled for removal and is not a benchmark comparator.

Phase 2: Measure the Plugin Framework

Run both minimal native plugins with empty projections. AlwaysKeep establishes the minimum complete-trace assembly and ABI overhead while preserving all data. DeterministicDrop adds only stable trace-ID hash selection and exercises core deletion and both shard-local secondary-index pruning paths. Profile both full merge paths, fix framework bottlenecks and correctness flaws, and repeat the baseline and both framework cases after every change. Phase 2 finishes only when all known framework issues have been resolved or explicitly accepted with quantified impact. Use 35% for the full-workload DeterministicDrop run and retain the 1% and 99% controlled-round sensitivity points.

Phase 3: Benchmark Plugins Independently

Run unit benchmarks for each plugin outside storage merging. Use representative assembled batches captured from the real fixture, including trace-count and spans-per-trace distributions. Measure projection, decision logic, allocations, latency calculation, and any output construction independently so plugin-local costs are not confused with storage I/O. Fix plugin-local performance problems and repeat until stable.

Phase 4: SkyWalking Integration

Configure the actual SkyWalking pipeline, projections, thresholds, merge grace, fragment-gap contract, core indexes, and secondary indexes. Run the serialized 24-hour publication schedule with production-derived merge concurrency inside the same Docker resource envelope. This is the final one-shard integration result, not a whole-data-node capacity claim.

Every optimization iteration preserves the fixture and environment checksums and reruns all earlier phases affected by the change. A final integration improvement is not accepted if it regresses the pipeline-disabled baseline.

Two Measurement Layers

Controlled First-Round Comparison

Capture a representative hot selection during pre-roll and the mature selection held at the opening dispatch barrier. Replay each exact selection under every applicable variant. The hot selection proves that merely enabling the pipeline does not invoke the plugin inside grace. The mature selection provides the identical-input plugin-cost comparison. This reports kernel-level overhead but does not claim to represent a complete day’s compaction.

Full Merge-Loop Comparison

Run the actual merge loop for the entire serialized logical schedule, the day-boundary drain, and the separately reported cooldown drain. Do not force a precomputed selection plan. Output sizes may cause later picker decisions to diverge between baseline and plugin variants; that is a real system effect and must be recorded rather than hidden.

Both layers are required. The controlled round answers how much one merge costs. The full loop answers how the feature changes total compaction work over a day.

Timed Region

Start timing only after the generated fixture clone is ready, the table and snapshot are open, the plugin is compiled and loaded, the registry is configured, and profiles are armed. The primary timer starts when the opening dispatch barrier is released and stops after the day-boundary merge-idle barrier. The cooldown timer starts immediately before the two-hour logical advance and stops after the second merge-idle barrier. Every merged introduction must be durable at its phase boundary.

Both timed phases include their publication or trigger notification, policy selection, queueing, core and secondary-index merge I/O, plugin execution, fragment guard work, and merged-part introduction. They exclude fixture generation, directory cloning, plugin loading, result scanning, and checksum verification. Report primary and cooldown profiles and metrics independently; report their sum only as the secondary end-to-end total.

For the controlled first-round layer, timing starts immediately before dispatch and ends after durable introduction.

Metrics

Primary

primary serialized-replay wall-clock seconds, active merge seconds, and publication-to-idle barrier latency;
cooldown wall-clock seconds, reported separately;
process CPU seconds and average effective cores;
compressed input MiB per second;
traces and spans per second;
allocated bytes and allocation count;
peak Go heap and process RSS;
bytes read and written by the process;
merge write amplification: merged output bytes divided by bytes of selected inputs;
total daily compaction amplification: all merge output bytes divided by generated fixture bytes.

Report every resource metric separately for the primary and cooldown phases. Report core and secondary-index bytes independently in addition to their combined totals.

Merge Shape

merge count by lane;
selected part count and bytes per merge;
output part count and bytes per merge;
selection fingerprint in dispatch order;
queue latency and active merge duration;
maximum concurrent merges;
final part count and size histogram;
merge count, selected bytes, output bytes, and duration grouped by sampling classification and reason;
merge queue depth within each epoch, unmerged-part inventory, oldest-part age, publication-to-idle time, and cooldown drain time for the serialized full-loop run.

Pipeline

traces evaluated, retained, dropped, and oversized-bypassed;
traces retained by duration, error, tag, and healthy-hash rules, plus traces reaching each rule;
plugin calls and traces per call;
grace-bypassed merge and trace counts;
staged bytes and peak staged bytes;
fragment-guard candidate parts, Bloom probes, deferrals, and budget exhaustion;
plugin errors, panics, timeouts, and malformed verdicts;
sampled and unsampled merge counts, including every unsampled reason.

The benchmark plugins must export final call, evaluated-trace, and decision checksums so the monitor can reconcile their work with engine counters and the compiler cannot eliminate benchmark-only decision calculations.

Profiles

Collect separate primary and cooldown profiles for each variant rather than combining the phases:

CPU profiles for the primary interval and cooldown drain;
heap-in-use profiles at primary peak and cooldown peak;
allocation profile;
block and mutex profiles;
runtime execution trace for one representative run;
operating-system I/O and peak-RSS samples.

Use one measured operation per process. Run at least five processes per variant, alternating variant order. Report median and p95, not only the fastest run. Record commit, Go version, kernel, filesystem, storage device, GOMAXPROCS, cgroup CPU and memory limits, merge concurrency, and whether the filesystem clone used reflink or a full copy.

Warm-cache and cold-cache results must not be mixed. The default report should use a fresh fixture clone and unchanged host cache policy for every process. An additional warm-cache series is useful for CPU attribution.

Correctness Gates

The performance result is rejected unless all gates pass:

Baseline and always-retain plugin runs end with the same trace, span, and tag counts as the generated input.
Their order-independent logical row checksums equal the input checksum. Physical output bytes need not match.
Every input part selected for a merge is removed exactly once and every output part is introduced exactly once. The same condition applies to every matching secondary-index part.
No trace ID is emitted in more than one logical block inside a merged output part.
The plugin receives exactly one batch entry per trace ID for each merge in which that trace participates.
The plugin’s MinTS and MaxTS equal bounds independently reconstructed from all selected fragments of the trace.
No plugin call occurs for a hot selection. During the measured publication interval, at least ten core merges are classified merge_grace and at least ten are classified executed; every executed input has merge depth of at least one. Cooldown-only executions do not count toward this coverage gate.
There are no plugin failures, timeouts, malformed verdicts, oversized-trace bypasses, or guard-budget failures unless a separate fault scenario intentionally requests them.
The merge loop reaches the defined idle state without leaked in-flight parts or references.
Secondary-index queries over the final output return the same logical rows as the generated fixture for the always-retain variants.
For DeterministicDrop, the final core and every secondary index exclude exactly the confirmed-drop ledger and retain every other expected row. The plugin uses the fixed 35% threshold without a result-selected salt; its observed full-fixture decision count and checksum are frozen before profiling. Every selected ID not confirmed for deletion must reconcile to no evaluation, grace bypass, or fragment-guard retain.
The source downloaded shard and immutable generated fixture remain unchanged.
The monitor event count equals the number of completed core file merges, every event has one sampling classification, and aggregate event totals reconcile with engine pipeline counters.
Every publication occurs only after the preceding epoch-aware merge-idle barrier. Logical time remains fixed from publication until the dispatcher has observed that publication’s snapshot epoch and all recursively eligible merge work has drained.
The primary phase reaches merge-idle at the logical day boundary before the clock advances. Cooldown has independent counters and profiles, and primary plus cooldown reconciles with end-to-end part, row, byte, and merge totals.
Every scheduled part moves exactly once from the staging directory through an atomic same-filesystem rename. Its checksum is unchanged, the external controller never appears in data-node profiles or cgroup counters, and all controller lag and errors are reported separately.
Pipeline-disabled and AlwaysKeep runs have identical ordered selection fingerprints and logical outputs. Dropping variants may diverge, but their first point and subsequent trajectories are recorded.

Comparison and Acceptance

First report absolute values and run-to-run variance for the pipeline-disabled revision under test. Only after that baseline is stable should every plugin variant report absolute values and percentage change from it for wall time, CPU, allocation, peak RSS, read bytes, write bytes, throughput, and amplification. Normalize full-loop results by generated input bytes, traces, and spans because the picker trajectory may differ.

Do not establish a pass/fail regression threshold from the first run. First collect at least five stable baseline and plugin samples and calculate run-to-run variance. A later CI or release gate can use a threshold greater than both the observed variance and an agreed product budget.

Reports should explicitly separate:

the current pipeline-disabled baseline and its repeatability;
trace assembly and plugin overhead from the controlled first-round comparison;
total system impact from the full merge-loop comparison;
AlwaysKeep framework overhead from DeterministicDrop pruning work and real plugin logic;
hot grace-bypass work from mature plugin-evaluated work.

Implementation Plan

Freeze the revision-under-test commit, benchmark binary, pipeline-disabled baseline contract, and artifact hashes.
Add the merge-maturity clock seam plus explicit merge-idle and trigger seams without changing production defaults.
Add the mature and structural source validators, both closure allowlists, shard-local secondary-index validation, and immutable source and observed-write-batch manifests.
Add the hybrid semantic catalog, independent 5,202-template copy selector, timestamp mapper, liaison part encoder and isolated data-node receive fixture builder, whole-fragment batch-template packer, and trace, sampler-ratio, core-byte, and index-byte fixture gates.
Add the minimal AlwaysKeep and trace-ID-hash DeterministicDrop native benchmark plugins.
Add the serialized external publisher, epoch-aware merge-idle barrier, mature-selection dispatch barrier, discovery pass, and full merge-loop harness.
Add logical checksumming and selection/event recording.
Replay exact mature baseline selections offline through the actual sampler, freeze the selection manifest, and report R_merge_verdict alongside the fixed 35% R_calibrated framework case.
Add per-merge sampling classification and resource-delta monitoring.
Add the pinned, resource-limited data-node Docker runner, separately pinned external controller, and data-node-only cgroup v2 collector.
Add pipeline-disabled controlled-round, single-worker attribution, serialized production-concurrency full-loop, and plugin-only unit benchmark entry points.
Add per-variant profiling commands and a machine-readable JSON report.
Run a short pre-roll and mature-selection smoke fixture first, then generate and validate the full 24-hour fixture.
Run and correct the current pipeline-disabled baseline before enabling the plugin.
Iterate through framework profiling and fixes, plugin unit optimization, and final SkyWalking integration in order.

Known Interpretation Risk

Publishing a 24-hour backlog and then merging it all at once makes old small parts eligible for the plugin, which is not the same as steady ingestion. Therefore the primary full-loop run advances through every scheduled publication and drains its picker epoch before moving logical time. A backlog stress mode may still be useful, but it must be labeled separately and must not be used to claim normal two-hour-grace behavior.