Batch aligned memtable column expansion to reduce region lock contention

[Caideyipi]

Description

An aligned tablet that introduces many measurements previously expanded its TVList and copied full-width accounting, value and bitmap arrays once per missing measurement while holding the DataRegion write lock. Repeated scans and copies made expansion quadratic in schema width, delaying queries waiting for the region read lock.

Reorder existing measurements in a single pass. Only the first missing measurement creates the new-type list and scans the remaining schemas to register missing measurements in insertion order. Extend the TVList in one batch and resize the reordered value and bitmap arrays to the final width once. Writes without schema extension keep one map lookup per measurement and do not create a new-type list. Memory estimates are refreshed once per expansion batch. Existing rows retain implicit nulls, and new value arrays and bitmaps remain lazy. Single-column extension delegates to the batch API.

Regression tests cover mixed existing/new/deleted schemas, duplicate measurement names, preservation of existing-column values and bitmaps before the first new column, no-extension reordering, bitmap alignment with a nonzero tablet offset, historical nulls across array blocks, binary and primitive memory accounting, clone/clear/reuse, WAL recovery, handover, and a 10,000-column tablet followed by another write.

Validation

• mvn spotless:apply -pl iotdb-core/datanode
• mvn test -pl iotdb-core/datanode -am -Dtest=AlignedWritableMemChunkTest,AlignedTVListTest,MemChunkDeserializeTest,MemTableFlushTaskTest,PrimitiveMemTableTest -Dsurefire.failIfNoSpecifiedTests=false — 66 tests passed with no failures, errors or skips; the reactor compiled the DataNode and its dependencies from a fresh worktree.
• git diff --check

Performance

Local microbenchmarks used Oracle JDK 17.0.15, Windows 11, an Intel Core i9-13900H, G1 and a fixed 512 MiB heap. Base is 21d53dd, initial PR is 1f05d5d, and revised is 5f27eb1. Each variant compiles the actual AlignedWritableMemChunk and AlignedTVList sources against the same freshly built reactor dependencies.

No-extension measurements use JMH 1.37: one thread, three independent JVM forks, three 300 ms warmup iterations and five 300 ms measurement iterations per fork. Every measurement already exists, incoming schema objects are distinct and reverse column order, values are dense INT64, and a bitmap array is present with unmarked columns. The schema-reorder benchmark invokes the private method through the same cached MethodHandle and consumes its result. The tablet benchmark times the public writeAlignedTablet call; clearing and prepopulating one row are outside the timed call, so the measured write uses existing value blocks. Values below are means with JMH's 99.9% confidence interval. Allocation is reported for the isolated reorder only; the tablet profiler also counts invocation setup.

No-extension operation	Columns	Rows	Base (µs/op)	Initial PR (µs/op)	Revised (µs/op)
Schema reorder	10	-	0.105 ± 0.018	0.176 ± 0.029	0.117 ± 0.016
Schema reorder	100	-	1.144 ± 0.186	2.033 ± 0.465	1.139 ± 0.142
Schema reorder	1000	-	13.353 ± 1.557	22.187 ± 2.471	13.476 ± 2.471
writeAlignedTablet	10	1	0.523 ± 0.051	0.572 ± 0.083	0.453 ± 0.108
writeAlignedTablet	10	32	0.564 ± 0.144	0.556 ± 0.091	0.443 ± 0.068
writeAlignedTablet	100	1	3.247 ± 0.296	5.604 ± 1.486	3.262 ± 0.386
writeAlignedTablet	100	32	4.218 ± 0.858	5.069 ± 0.940	3.947 ± 0.619
writeAlignedTablet	1000	1	31.928 ± 2.716	42.896 ± 4.491	31.957 ± 2.164
writeAlignedTablet	1000	32	37.621 ± 3.040	51.825 ± 7.318	40.975 ± 5.591

Columns	Base reorder allocation (B/op)	Initial PR (B/op)	Revised (B/op)
10	136	136	136
100	856	856	856
1000	8056	8080	8056

The initial PR's additional schema scan increased isolated reorder time by approximately 66%–78% in these runs. The revised path removes that scan. Revised/base latency confidence intervals overlap for every tested scenario, and revised reorder allocation matches the base at each width. For the 1,000-column/32-row tablet case, the revised point estimate is 8.9% above the base (40.975 ± 5.591 vs 37.621 ± 3.040 µs); these measurements do not establish a statistically distinguishable regression, and they do not prove zero overhead. At 1,000 columns the initial PR also allocated about 24 extra bytes per isolated reorder; lazy initialization removes that allocation.

Expansion measurements write one tablet row after one existing populated INT64 column. Setup and release are excluded. Sparse values supply only the first and last new columns; dense values supply every column. Each JVM warms up with 30 expansions of 256 columns; the reported values are medians of three independent JVMs. Current-thread allocation during writeAlignedTablet is measured with ThreadMXBean, rather than retained heap size. These are local microbenchmarks and do not measure distributed throughput or region-lock wait time.

New INT64 columns	Values	Base (ms)	Initial PR (ms)	Revised (ms)	Base allocated (MiB)	Revised allocated (MiB)
10,000	Sparse	298.12	7.00	6.13	956.22	1.95
10,000	Dense	354.01	18.55	17.19	962.26	7.98

This PR has:

• been self-reviewed, including the existing region locking and query/flush ownership rules.
• added Javadocs and comments explaining the batch expansion and existing-schema path.
• added unit tests covering the new paths and preserved behavior.

Key changed classes: AlignedWritableMemChunk, AlignedTVList.

Remote comparison: parent vs merged commit (2026-10-12)

On 192.168.130.19 (fit19), adding 10,000 INT64 columns reduced the median run-level mean RPC write latency from 175.47 ms to 31.98 ms (5.49x). Concurrent same-region query P99 fell from 138.28 ms to 4.09 ms (97.0% lower). Existing-schema measurements also recorded small overheads and tail variation, detailed below.

Environment and version isolation

• Baseline: direct parent 5c27559f02f525e75ee07a58d06ff8ee54578527; target: merged commit 932c1da25f7e472e84d2a997b376e10b02270c26.
• Intel Core i7-11700 (8 cores / 16 logical CPUs), 32 GB RAM, NVMe /ssd, Ubuntu 20.04.3, Linux 5.15.0-139-generic, Oracle JDK 17.0.9, G1. Existing IoTDB and Redis workloads were stopped before measurement; the CPU governor remained powersave.
• The target distribution was built from a fresh detached worktree using Oracle JDK 17.0.15 on Windows and a full reactor mvn -B -ntp -nsu package -pl distribution -am -DskipTests -Dmaven.test.skip=true -Ddevelocity.off=true. Both variants then compiled their actual AlignedWritableMemChunk and AlignedTVList sources on the test host with javac --release 17, placing the resulting classes in copies of this distribution. These are the only two production source files changed by the commit. SHA-256 checks confirmed that all other jars were identical.
• RPC deployment: 1 ConfigNode + 1 DataNode, 1 SchemaRegion and 1 DataRegion per database, verified with SHOW REGIONS after every run. SimpleConsensus for all three consensus protocols, replication factor 1, WAL ASYNC, automatic schema creation disabled, compaction disabled. ConfigNode heap 1 GiB, DataNode heap 8 GiB / direct memory 2 GiB, client heap 1 GiB. JVM processor limits: 8 for servers, 4 for client and microbenchmarks.

RPC writes and concurrent queries

Three independent clean server starts per variant, alternating parent/target, target/parent, parent/target. Every timed write is the public Session.insertAlignedTablet call over loopback, including client serialization, RPC and server execution; tablet construction is excluded. All values are dense INT64 and schemas are created beforehand.

Existing-schema cases use 32 rows per tablet, 150 warmup writes and 500 measured writes per run. Expansion cases start each device with one populated s0 row, then insert one row containing 1,000 or 10,000 other measurements into that device's existing memchunk. There are 12 / 6 expansion warmups respectively and 24 measured, separate devices per case and run.

During each expansion burst, one separate session continuously executes and fully consumes SELECT count(s0) FROM root.pr18816.anchor, an existing one-row device in the same DataRegion. The reader performs 100 untimed warmup queries. The writer leaves a 10 ms gap after each completed write. Query samples whose RPC starts during the burst are retained. Numbers below are medians of the three run-level means / percentiles, rather than percentiles pooled across different runs; sample counts are totals over the three runs.

Operation	New/existing columns	Parent mean ms	Target mean ms	Parent P50 ms	Target P50 ms	Parent P99 ms	Target P99 ms	Samples parent / target
Existing-schema write (32 rows)	10	0.393	0.393	0.366	0.368	0.700	0.678	1500 / 1500
Existing-schema write (32 rows)	100	0.586	0.570	0.566	0.553	0.895	0.931	1500 / 1500
Existing-schema write (32 rows)	1000	1.108	1.149	1.027	1.040	1.841	2.342	1500 / 1500
Expansion write (1 row)	1000	6.515	5.432	6.132	5.314	13.813	6.720	72 / 72
Expansion write (1 row)	10000	175.468	31.979	169.956	28.644	213.395	68.235	72 / 72
Concurrent count query during expansion	1000	0.927	0.915	0.816	0.875	2.722	1.902	1322 / 1222
Concurrent count query during expansion	10000	2.172	0.642	0.363	0.524	138.280	4.086	6114 / 4683

At 10,000 new columns, the three run-level mean write latencies ranged from 162.31–175.94 ms for the parent and 30.14–37.63 ms for the target. Query P99 ranged from 131.08–141.22 ms and 3.78–4.76 ms respectively. Query P50 was higher for the target (0.524 vs 0.363 ms), so the main query benefit in this workload is the reduction in long stalls. Burst durations and completed query counts differ because write duration differs; this is a fixed-write-count contention experiment, not a fixed-duration throughput benchmark. Query latency includes RPC, planning, queueing, execution and GC effects; individual lock-wait times were not instrumented.

Existing-schema RPC means were similar at 10 and 100 columns. At 1,000 columns, the target's median run-level mean was 3.7% higher (1.149 vs 1.108 ms) and P99 was 27.2% higher (2.342 vs 1.841 ms). P99 varied between 1.43–2.14 ms for the parent and 1.64–2.37 ms for the target. These runs should not be read as evidence of zero overhead or a general steady-state throughput improvement.

In-process expansion cost and allocation

Fixed 2 GiB heap; three independent JVMs per variant in the same alternating order. Each JVM warms up with 120 expansions of 256 columns, alternating 1 / 1,000 historical rows and mixing sparse / dense values. Each measured operation adds the specified number of new columns after one existing INT64 column, then writes one row. Sparse input supplies only the first and last new column; dense input supplies every new column. A bitmap array is present with unmarked entries.

Each case has five measured operations per JVM. Latency is the median within each JVM, then the median across the three JVMs. ThreadMXBean measures current-thread bytes allocated only around writeAlignedTablet; fixture setup, correctness checks, release, explicit GC and the 40 ms pause before measurement are excluded. Primitive-array pools are warm and can reuse value blocks, so these are allocated bytes during the write, not retained heap or guaranteed first-ever dense-write allocations. The sparse cases call the memtable directly: the current Java Session client filters entirely-null columns before RPC.

Historical rows	New INT64 columns	Values	Parent ms	Target ms	Speedup	Parent allocated MiB	Target allocated MiB
1	10	sparse	0.061	0.075	0.81x	0.004	0.002
1	10	dense	0.064	0.079	0.80x	0.004	0.003
1	100	sparse	0.242	0.161	1.51x	0.121	0.019
1	100	dense	0.324	0.269	1.21x	0.130	0.028
1	1000	sparse	7.982	0.945	8.45x	9.788	0.190
1	1000	dense	8.222	1.135	7.25x	9.880	0.281
1	10000	sparse	141.409	6.266	22.57x	956.221	1.945
1	10000	dense	139.393	6.658	20.94x	957.137	2.860
10000	10	sparse	0.064	0.071	0.90x	0.011	0.010
10000	10	dense	0.040	0.082	0.48x	0.017	0.015
10000	100	sparse	0.330	0.236	1.39x	0.182	0.081
10000	100	dense	0.452	0.365	1.24x	0.249	0.148
10000	1000	sparse	8.452	1.729	4.89x	10.385	0.787
10000	1000	dense	9.851	3.097	3.18x	11.071	1.472
10000	10000	sparse	143.885	15.847	9.08x	962.174	7.898
10000	10000	dense	146.734	21.468	6.83x	969.039	14.763

At 10 new columns the target had higher latency point estimates, by about 7–41 µs depending on history and density. The batch path's benefit grows with schema width; the small-width timings use a short manual benchmark and have less precision than the JMH measurements below.

Existing-schema JMH check

JMH 1.37, one thread, three independent forks, three 500 ms warmup iterations and five 500 ms measurement iterations per fork, G1, fixed 512 MiB heap. Incoming schemas are distinct objects in reverse column order, all measurements already exist, and a bitmap array is present with unmarked entries. Isolated reorder invokes the actual private method through a cached MethodHandle and consumes its result. Tablet invocation setup clears and prepopulates one row outside the timed call, so the write uses existing value blocks. Values are means with JMH's 99.9% confidence interval.

Operation	Columns	Rows	Parent µs/op	Target µs/op
reorder	10	-	0.090 ± 0.001	0.093 ± 0.001
reorder	100	-	0.876 ± 0.044	0.928 ± 0.010
reorder	1000	-	9.420 ± 0.099	9.585 ± 0.063
tablet	10	1	0.231 ± 0.005	0.250 ± 0.019
tablet	10	32	0.296 ± 0.007	0.292 ± 0.006
tablet	100	1	1.912 ± 0.052	1.937 ± 0.047
tablet	100	32	2.405 ± 0.075	2.397 ± 0.072
tablet	1000	1	22.076 ± 0.621	22.267 ± 0.530
tablet	1000	32	27.065 ± 0.678	27.174 ± 0.713

Because the first JMH session ran parent before target and showed small reorder differences, an additional three-fork reorder-only session used the reverse variant order with the same settings:

Columns	Parent reorder µs/op	Target reorder µs/op
10	0.093 ± 0.003	0.089 ± 0.001
100	0.845 ± 0.006	0.864 ± 0.011
1000	9.411 ± 0.186	9.423 ± 0.180

The 10-column reorder difference changed sign with order, and the 1,000-column difference reduced to about 0.1% in the reverse check. The 100-column target remained about 2.2% slower in that check, compared with 5.9% in the first session. Thus the remote runs retain evidence of a small reorder cost at this width. Reorder allocation stayed approximately 136 / 856 / 8,056 B/op in both variants at 10 / 100 / 1,000 columns. Full tablet confidence intervals overlapped in all six tested cases; that does not prove zero overhead. Tablet GC-profiler allocation includes invocation setup and is not reported as write-only allocation.

All 480 measured expansion operations validated dimensions, preserved historical values, historical nulls and the first/last new values. All 9,000 measured existing-schema RPC writes and 288 measured expansion RPC writes completed, with count checks after expansion, checked query results and exactly one DataRegion in each run. Raw CSV / JSON, source hashes, jar hashes, configuration, GC logs and harnesses are retained under /ssd/pr18816-perf-20261012 on the host; the local results bundle and report are under tmp-pr18816-comparison/. These measurements cover a single-node tree-model workload.