Skip to content

MB-62182: Fast merge of vector indexes #336

Open
Thejas-bhat wants to merge 16 commits intomasterfrom
fastmerge
Open

MB-62182: Fast merge of vector indexes #336
Thejas-bhat wants to merge 16 commits intomasterfrom
fastmerge

Conversation

@Thejas-bhat
Copy link
Member

@Thejas-bhat Thejas-bhat commented Jun 17, 2025

  • Introduces a way to merge the vector indexes without having to re-train the merged index.
  • Uses the merge_from API from faiss which merges the underlying inverted lists (based on the identical centroid layout) in a block wise fashion.
  • During the merge process, if the config tells us to train the vector index (during the initial part of index build before the data ingest) the vectors from the indexes are reconstructed + trained on the merged index. If the config specifies us to use the "centroid index" to merge, we will now use that template for the merged index and also use the merge_from API to move all the vectors from the child indexes to the merged one.

@Thejas-bhat Thejas-bhat marked this pull request as ready for review January 15, 2026 20:54
@Thejas-bhat Thejas-bhat changed the title WIP fast merge [WIP] MB-62182: Fast merge of vector indexes Jan 15, 2026
@Thejas-bhat Thejas-bhat changed the title [WIP] MB-62182: Fast merge of vector indexes MB-62182: Fast merge of vector indexes Jan 29, 2026
@Thejas-bhat Thejas-bhat moved this from Todo to In Progress in Fast Merge Jan 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant