Skip to content

Conversation

@mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Feb 12, 2026

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Copilot AI review requested due to automatic review settings February 12, 2026 07:45
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the TabletInvertedIndex classes to use fastutil's Long2ObjectOpenHashMap instead of Guava's HashMap and HashBasedTable to reduce memory consumption. The fastutil library provides memory-efficient primitive collections that avoid boxing overhead when using primitive types as keys.

Changes:

  • Replaced HashMap<Long, V> with Long2ObjectOpenHashMap<V> in TabletInvertedIndex and CloudTabletInvertedIndex
  • Replaced HashBasedTable (a nested map structure) with nested Long2ObjectOpenHashMap in LocalTabletInvertedIndex
  • Added fastutil-core 8.5.18 dependency to the project
  • Updated LICENSE file to reflect the new fastutil version

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
fe/fe-core/pom.xml Adds fastutil-core 8.5.18 dependency for memory-efficient primitive collections
dist/LICENSE-dist.txt Updates fastutil version reference from 6.5.6 to 8.5.18
fe/fe-core/src/main/java/org/apache/doris/catalog/TabletInvertedIndex.java Replaces HashMap with Long2ObjectOpenHashMap for tabletMetaMap
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletInvertedIndex.java Replaces HashMap with Long2ObjectOpenHashMap for replicaMetaMap
fe/fe-core/src/main/java/org/apache/doris/catalog/LocalTabletInvertedIndex.java Replaces HashBasedTable with nested Long2ObjectOpenHashMap structures, updates all related methods to handle the new data structure, and adds proper cleanup of empty nested maps

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

<!-- fastutil for memory-efficient primitive collections -->
<dependency>
<groupId>it.unimi.dsi</groupId>
<artifactId>fastutil-core</artifactId>
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a potential mismatch between the artifact declared in pom.xml ("fastutil-core") and the LICENSE file ("fastutil"). The standard fastutil artifact is typically "it.unimi.dsi:fastutil", not "fastutil-core". The "fastutil-core" is a modular variant introduced in fastutil 8.x. Please verify that "fastutil-core" is the correct artifact to use, and if so, ensure the LICENSE file reflects the actual artifact name consistently. If using the standard "fastutil" artifact instead, update the pom.xml accordingly.

Suggested change
<artifactId>fastutil-core</artifactId>
<artifactId>fastutil</artifactId>

Copilot uses AI. Check for mistakes.
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Feb 12, 2026

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 30442 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72ff315832874fa228c26e22eac2ac6fad9d7ffa, data reload: false

------ Round 1 ----------------------------------
q1	17608	4500	4283	4283
q2	2022	354	235	235
q3	10164	1289	782	782
q4	10195	762	302	302
q5	7554	2213	1903	1903
q6	200	177	148	148
q7	909	733	610	610
q8	9281	1386	1207	1207
q9	4635	4610	4651	4610
q10	6802	1935	1569	1569
q11	490	269	235	235
q12	342	380	233	233
q13	17785	4064	3237	3237
q14	234	235	212	212
q15	858	809	788	788
q16	675	695	605	605
q17	687	820	521	521
q18	6559	5676	6084	5676
q19	1119	1119	642	642
q20	586	562	422	422
q21	2783	2079	1961	1961
q22	353	294	261	261
Total cold run time: 101841 ms
Total hot run time: 30442 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4572	4539	4644	4539
q2	274	360	257	257
q3	2398	2899	2527	2527
q4	1599	1826	1421	1421
q5	4767	4524	4530	4524
q6	221	178	137	137
q7	1930	1876	1825	1825
q8	2533	2486	2445	2445
q9	7706	7396	7482	7396
q10	2803	3062	2675	2675
q11	522	439	414	414
q12	707	765	636	636
q13	3828	4556	3548	3548
q14	268	290	267	267
q15	834	781	780	780
q16	653	684	645	645
q17	1074	1323	1305	1305
q18	7575	7519	7462	7462
q19	866	826	825	825
q20	1978	2038	1868	1868
q21	4910	4165	4124	4124
q22	488	455	412	412
Total cold run time: 52506 ms
Total hot run time: 50032 ms


// tablet id -> tablet meta
protected Map<Long, TabletMeta> tabletMetaMap = Maps.newHashMap();
protected Long2ObjectOpenHashMap<TabletMeta> tabletMetaMap = new Long2ObjectOpenHashMap<>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any quantitive comparison with the previous impl.?

@doris-robot
Copy link

TPC-DS: Total hot run time: 189822 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 72ff315832874fa228c26e22eac2ac6fad9d7ffa, data reload: false

query5	4991	636	487	487
query6	329	225	207	207
query7	4230	485	283	283
query8	335	261	240	240
query9	8762	2806	2784	2784
query10	559	386	336	336
query11	17257	17087	16829	16829
query12	186	126	129	126
query13	1271	459	359	359
query14	6445	3268	3032	3032
query14_1	2927	2847	2845	2845
query15	200	198	179	179
query16	978	510	384	384
query17	1068	697	598	598
query18	2602	449	365	365
query19	217	209	192	192
query20	142	129	131	129
query21	235	150	129	129
query22	4846	4981	4894	4894
query23	17424	16901	16623	16623
query23_1	16734	16868	16837	16837
query24	7178	1591	1236	1236
query24_1	1248	1234	1229	1229
query25	563	465	419	419
query26	1238	288	161	161
query27	2740	469	299	299
query28	4523	1889	1887	1887
query29	817	575	487	487
query30	322	261	218	218
query31	897	731	647	647
query32	90	90	79	79
query33	544	372	321	321
query34	1069	931	579	579
query35	647	690	609	609
query36	1086	1109	953	953
query37	141	104	91	91
query38	2997	2918	2882	2882
query39	856	869	845	845
query39_1	810	804	806	804
query40	232	146	129	129
query41	76	71	72	71
query42	115	115	149	115
query43	390	401	353	353
query44	1381	730	729	729
query45	195	190	184	184
query46	906	1008	617	617
query47	2116	2117	2036	2036
query48	323	329	236	236
query49	616	440	351	351
query50	722	277	238	238
query51	4200	4194	4084	4084
query52	107	112	97	97
query53	300	337	290	290
query54	297	282	272	272
query55	88	86	82	82
query56	313	316	303	303
query57	1364	1332	1277	1277
query58	291	284	279	279
query59	2539	2749	2566	2566
query60	346	348	326	326
query61	147	148	146	146
query62	609	573	534	534
query63	312	281	288	281
query64	4904	1228	947	947
query65	4614	4560	4529	4529
query66	1413	450	345	345
query67	16453	16545	16476	16476
query68	2390	1089	741	741
query69	425	322	301	301
query70	1002	980	956	956
query71	344	333	303	303
query72	2913	2782	2570	2570
query73	535	569	330	330
query74	9586	9597	9343	9343
query75	2813	2782	2420	2420
query76	2306	1097	700	700
query77	370	397	314	314
query78	10996	11084	10433	10433
query79	1139	878	626	626
query80	678	578	512	512
query81	529	277	250	250
query82	1337	153	114	114
query83	338	268	244	244
query84	250	134	99	99
query85	857	478	424	424
query86	387	305	334	305
query87	3174	3088	3094	3088
query88	3615	2689	2705	2689
query89	448	390	346	346
query90	1867	185	178	178
query91	174	161	130	130
query92	76	76	72	72
query93	953	886	497	497
query94	455	315	292	292
query95	587	406	321	321
query96	649	538	232	232
query97	2473	2488	2394	2394
query98	227	217	211	211
query99	1013	986	912	912
Total cold run time: 261998 ms
Total hot run time: 189822 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.47 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 72ff315832874fa228c26e22eac2ac6fad9d7ffa, data reload: false

query1	0.05	0.04	0.04
query2	0.10	0.05	0.05
query3	0.25	0.08	0.08
query4	1.60	0.11	0.10
query5	0.26	0.26	0.25
query6	1.18	0.68	0.66
query7	0.04	0.03	0.02
query8	0.05	0.04	0.04
query9	0.58	0.50	0.49
query10	0.56	0.55	0.56
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.64	0.62	0.61
query14	1.07	1.08	1.06
query15	0.90	0.86	0.89
query16	0.37	0.38	0.43
query17	1.14	1.17	1.16
query18	0.22	0.21	0.21
query19	2.05	1.97	2.09
query20	0.02	0.01	0.02
query21	15.40	0.29	0.16
query22	4.99	0.06	0.05
query23	15.90	0.29	0.11
query24	1.53	0.26	0.25
query25	0.09	0.08	0.09
query26	0.14	0.13	0.13
query27	0.06	0.05	0.06
query28	3.03	1.15	0.97
query29	12.60	4.06	3.27
query30	0.28	0.14	0.12
query31	2.81	0.65	0.42
query32	3.24	0.60	0.49
query33	3.17	3.22	3.25
query34	16.45	5.36	4.72
query35	4.81	4.81	4.83
query36	0.65	0.50	0.49
query37	0.11	0.07	0.06
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.20	0.17	0.16
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 97.11 s
Total hot run time: 28.47 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants