Compatibility performance against PR #1412

Completed on 2026-10-18: all 80,000 timed RPCs passed execution and correctness checks across five paired repetitions. All median point estimates meet the G1 budgets. Six workload/size cases pass with uncertainty included; two small QL p50 comparisons remain inconclusive. No metric has a confirmed budget regression. That original campaign remains inconclusive. The separate 20-pair confirmation below passes both uncertain cases under its predefined one-sided method, closing this matrix’s compatibility follow-up. No production code was changed or optimized during this comparison.

Frozen comparison

This compares production revision 012115c56652691d4163f69d0460dfa9b064191f (the reviewed PR #1412 base) with 3e68616a0b265a2591654435f1de0dd2d6353a7d (Catalog B plus the retained-merge-tail fix). The T01 verification-only revision b0a4dc14 has the same production baseline; this comparison does not rerun every ticket against its immediate prerequisite.

Two isolated worktrees use identical benchmark sources: the 16-file T01 harness, service-scoped readiness, raw latency recording, and an explicit 256 MiB measure tracker setting. The overlays change only benchmark files. Production sources, protobuf and dependencies are unchanged in each checkout. Both executables were built before timing with Go 1.26.9, without querymetrics instrumentation.

The narrow matrix preserves the recorded T01 workload:

• Input cardinalities: 1,024 and 100,000, returning 64 and 6,300 rows per query.
• Variants: native, native_rebound, legacy_ql, legacy_ql_rebound.
• One caller, 1,000 timed calls per variant, profiling enabled.
• Configured warmups: 20. The historical inclusive loop executes 21 validated warmup calls; preserved identically on both versions.
• Five paired repetitions; baseline first on odd repetitions and current first on even repetitions. Each version/cardinality runs in a fresh process.
• Every container has four CPUs, 8 GiB RAM, no additional swap, and 4,096 PID limit. The observed cgroup limits are retained for every timed process.
• One fixture timestamp is pinned across the entire campaign. Both versions check independent expected rows and hashes before timing and retain sampled full responses for validation after measurement. All timed calls check row count.

Within a process, the four variants run in their historical order and share one client/liaison/two-data-node cluster. Later variants can observe earlier cache and heap residency. CPU, allocation and profile observations cover that entire process, not isolated planning or server CPU. The raw response-retention protocol is unchanged; adding the recorded latency slice does not add a timed allocation.

The source harness and binary digests are checked before and after each run. No build or other benchmark ran concurrently. This campaign does not reopen the stopped historical T02 investigation or benchmark new unsupported SQL against a fictitious old-version implementation.

Evaluation method

G1 budgets are unchanged: the ratio of current/base median-of-five p50 latency and allocated bytes/query must be at most 1.10; p95 latency at most 1.15. P99, throughput and CPU are reported without introducing new acceptance budgets. Raw samples and all five run values are retained.

Uncertainty is estimated by paired repetition-level bootstrap: 20,000 resamples, seed 1412, with the ratio of medians recomputed for each sample. The 95% intervals are exploratory with only five independent pairs; individual requests are not treated as independent experimental repetitions. A metric is marked pass when its entire interval meets the budget, fail when the entire interval exceeds it, and inconclusive when the interval crosses it. Point-estimate budget checks are reported separately. This avoids turning an unstable comparison into a pass.

Reproduction and artifacts

Local worktrees: /mnt/d/qlb-g1-baseline-012115c5 and /mnt/d/qlb-g1-head-3e68616a. The runner is .omx/compat-g1/run.sh; it invokes prebuilt .omx/g1.test binaries and never builds during measurements. The paired runner takes G1_BASELINE, G1_CURRENT and G1_BASE_NANOS; the latter is saved in .omx/compat-g1/provenance/run-config.json.

.omx/compat-g1/provenance/ contains the frozen harness archive and SHA-256 manifest, both overlay patches, full revisions, binary digests and Go build info. Each r{1..5}/{baseline,current}/ retains merged JSON/Markdown reports, original shards, raw latency samples, CPU/heap profiles, logs and cgroup limit files. The summary is generated by .omx/compat-g1/summarize.py into summary.json.

Results

Values are medians of five runs. Ratios are current/baseline, so values below 1.0 mean less latency or allocation. The p50 interval is the paired-bootstrap 95% interval described above. Every p95 and allocation interval meets its budget.

Input rows	Variant	Base p50 ms	Current p50 ms	p50 ratio [interval]	p95 ratio	Allocation ratio	Verdict
1,024	legacy_ql	1.901	2.001	1.053 [1.016, 1.138]	1.023	0.892	inconclusive
1,024	legacy_ql_rebound	1.880	2.006	1.067 [0.998, 1.109]	1.057	0.892	inconclusive
1,024	native	1.870	1.863	0.996 [0.961, 1.077]	1.018	0.893	pass
1,024	native_rebound	1.906	1.891	0.992 [0.955, 1.055]	1.029	0.895	pass
100,000	legacy_ql	41.708	40.369	0.968 [0.959, 1.012]	0.975	0.901	pass
100,000	legacy_ql_rebound	41.747	40.889	0.979 [0.967, 0.989]	0.984	0.902	pass
100,000	native	41.728	40.443	0.969 [0.955, 0.974]	0.978	0.900	pass
100,000	native_rebound	41.827	40.523	0.969 [0.964, 0.985]	1.000	0.901	pass

The two inconclusive cases are the 1,024-row QL queries. Their p50 medians increase by approximately 0.100 ms (5.3%) and 0.126 ms (6.7%). These estimates fit the 10% budget, but upper interval bounds are 13.8% and 10.9%, respectively. This does not establish a budget regression; it prevents certifying a stable pass under the frozen uncertainty method. No extra repetitions, selective outlier removal, threshold changes or performance fixes were used to force a verdict. The historical T02 investigation remains separate and stopped.

All four 100k-row cases pass. Their p50 medians decrease by 2.1–3.2%, and their allocated bytes/query decrease by approximately 9.8–10.0%. Small-query allocations decrease by approximately 10.5–10.8%. These are complete RPC/process measurements, not claims about isolated planning, storage or operator speed.

CPU, tail latency and throughput

These supporting observations are not additional gates. CPU covers client, liaison and both data nodes; it includes background work occurring during timing. Each cell below lists baseline → current median.

Input rows	Variant	p99 ms	CPU ms/query	Allocated KiB/query	Queries/s
1,024	legacy_ql	3.731 → 3.576	4.840 → 4.980	743.860 → 663.495	502.585 → 479.858
1,024	legacy_ql_rebound	3.289 → 3.515	4.800 → 5.040	742.793 → 662.854	508.780 → 479.798
1,024	native	3.352 → 3.062	4.780 → 4.700	740.863 → 661.321	506.020 → 515.434
1,024	native_rebound	3.414 → 3.373	4.790 → 4.800	740.152 → 662.377	505.834 → 502.413
100,000	legacy_ql	60.701 → 59.010	77.690 → 75.310	26197.341 → 23608.346	22.654 → 23.297
100,000	legacy_ql_rebound	60.887 → 59.381	77.860 → 75.870	26220.566 → 23645.641	22.610 → 23.047
100,000	native	60.053 → 60.304	77.570 → 74.970	26257.870 → 23634.098	22.691 → 23.258
100,000	native_rebound	60.612 → 60.709	77.580 → 75.410	26260.355 → 23672.215	22.714 → 23.210

Artifact verification and scope

All 80 report rows contain 1,000 raw latency samples. Corresponding versions have equal expected row counts and full-result hashes. All 160 CPU/heap profile files are nonempty. All 20 timed processes report Go 1.26.9 and observed cgroup limits of four CPUs, 8 GiB memory and zero extra swap. Driver checksum checks passed before and after every version run. Each full variant validates 21 warmup responses and 11 sampled timed responses exactly; every timed response checks row count. The comparison does not claim that every timed response received a full-tuple check.

Only the existing T01 compatibility matrix was run. The 100k/1m T17 owning queries, spilling, session pagination and new unsupported SQL were not added to this benchmark. Large-query resource feasibility remains documented separately in the million-row fix and probe. The comparison closes the missing PR #1412 measurement work for this matrix, while the two uncertain small-query p50 cases keep overall G1 acceptance open.

Small-query confirmation protocol

A separate confirmation campaign follows the five-pair comparison above; those observations remain unchanged and are excluded from the confirmation estimator. Before timing, the campaign fixes 20 fresh pairs, with ten baseline-first and ten current-first pairs shuffled using seed 14120. There is no early stopping or extension based on observed significance. Only cardinality 1,024 reruns. The unchanged binaries execute the two native variants before the two QL variants to preserve historical cache/heap conditions; native observations are supporting context, not additional confirmation gates. No production or harness rebuild is needed. CPU, memory, profiling, correctness checks and workload remain identical.

Pilot paired log-ratio standard deviations are 0.0515 and 0.0458. A rough normal approximation for 80% power at the observed margin below 1.10 suggests 9 and 15 pairs. Twenty pairs are selected before timing. This approximation is only a sizing diagnostic: it does not establish power for the median-ratio bootstrap, and the five-pair variance estimates themselves are uncertain.

For each QL variant, compute the ratio of current/baseline medians of 20 run metrics. Resample the 20 paired indices together 20,000 times with seed 1412. The empirical 95th percentile is the one-sided 95% upper bound, and the fifth percentile is the separate one-sided 95% lower bound. These are not a two-sided 95% interval. An upper bound at or below the original budget passes; a lower bound above it fails; otherwise the verdict remains inconclusive. Budgets remain 1.10 for p50 and allocation, and 1.15 for p95. All three must pass for a variant to pass; both variants must pass for confirmation acceptance. This all-pass rule is an intersection-union non-inferiority decision, not a claim of simultaneous 95% coverage for every reported bound. Bootstrap coverage is approximate.

The runner, frozen configuration, analysis and their pre-run hashes are retained under .omx/compat-g1-confirmation/. The original campaign and its two-sided verdict remain separately recorded; changing the uncertainty method does not retroactively turn that campaign into a pass.

Confirmation results

All 20 fresh pairs completed without retries or excluded observations. Both QL variants pass every confirmation gate. Together with the six passing cases in the original matrix, this closes the small-query compatibility follow-up under the explicitly revised uncertainty method. The original five-pair verdict and historical T02 acceptance remain unchanged.

Values below are medians of 20 runs. Upper bounds are separate one-sided 95% bootstrap bounds for current/baseline ratios, not simultaneous confidence bounds.

QL variant	Baseline p50 ms	Current p50 ms	p50 ratio	p50 upper bound	p95 ratio / upper bound	Allocation ratio / upper bound	Verdict
legacy_ql	2.2995	2.3450	1.0198	1.0436	1.0029 / 1.0409	0.8911 / 0.8942	pass
legacy_ql_rebound	2.2960	2.3460	1.0218	1.0431	1.0124 / 1.0436	0.8918 / 0.8946	pass

P50 increases are approximately 0.045 and 0.051 ms (2.0% and 2.2%). Their upper slowdown bounds are 4.4% and 4.3%, below the unchanged 10% budget. Allocated bytes per query decrease by approximately 10.9% and 10.8%.

Absolute latency is higher in this campaign than in the original five-pair run (on both versions). The campaigns are not pooled, and this is not evidence of a code optimization: binaries are unchanged. The experiment supports the bounded regression claim for the paired comparisons in this environment; it does not promise identical latency on every host or eliminate systematic measurement bias.

Artifact validation passed for all 40 timed processes, 160 report rows and 160,000 timed RPCs, including 80,000 QL calls. All 320 CPU/heap profiles are nonempty; each report has exactly the expected four variants and 1,000 raw latency samples per variant. Baseline/current expected hashes match, sampled full-response checks pass, and every timed call checks row count. All processes used Go 1.26.9, GOMAXPROCS=4 and the required observed cgroup limits. Protocol, harness and binary digest checks passed. No 100k workload was rerun.

Results and validation are retained in .omx/compat-g1-confirmation/{summary.json,summary.txt,artifact-validation.json}; r{1..20}/{baseline,current}/ retains individual reports, raw samples, profiles, logs and limits. Reproduce using the same environment variables as the original runner with .omx/compat-g1-confirmation/run.sh; choose a fresh output directory because the runner refuses to overwrite existing runs. The merge-stage report configuration contains defaults from its separate process (including three warmups); the frozen runner configuration is authoritative for timed-process settings: 20 configured warmups, executing 21 calls in the historical loop.

Only benchmark evidence and documentation changed. Large-input throughput, spilling and session pagination remain outside this confirmation.