feat: support create vector index distributedly

[chenghao-guo]

close #4723

Key Changes:
The distributed index creation leverages the existing IVF framework while adding coordination mechanisms for multi-node execution. The index merger component now handles distributed fragment consolidation and metadata synchronization. This work enables scalable vector index creation for large-scale datasets, significantly reducing index build time.

• Implemented distributed IVF index building infrastructure for parallel index construction across multiple nodes
• Enhanced the index merger component for distributed operations
• For IVF_HNSW part, the HNSW graph is built locally within the shard as a sub-index of the partition; there is no cross-shard graph merging and no cross-shard edges. These are supported but distribution only happens in IVF.
• CPU only, torch accelerator will not be supported and fall back to single node IVF index creation.

Current Status in this PR:
• FLAT/SQ: Should work now, it is under active testing phase, validating distributed performance and accuracy
• PQ (Product Quantization): Currently depends on global training codebook, requiring centralized training before distributed deployment.
• RQ (Residual Quantization): I didn't consider this when I design this PR. Not yet supported in distributed mode maybe, planned for future implementation

Once I finish all the testing phase in my side on performance and recall accuracy, I will mark it ready to review.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 61.06911% with 721 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...lance-index/src/vector/distributed/index_merger.rs	62.15%	484 Missing and 53 partials ⚠️
rust/lance/src/index/vector/ivf.rs	0.00%	133 Missing ⚠️
rust/lance/src/index/vector.rs	81.63%	33 Missing and 3 partials ⚠️
.../lance-index/src/vector/shared/partition_merger.rs	83.82%	8 Missing and 3 partials ⚠️
rust/lance-index/src/vector/hnsw/builder.rs	80.00%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yanghua]

@BubbleCal Thanks for reviewing! I have filed some tickets to track further work: #5621 and #5622.