Support build IVF index distributively in Spark

[fangbo]

This PR lance-format/lance#5117 provides fundamental functions to build IVF index distributively.

[goodqiang]

Upon seeing this issue, is this feature intended to be integrated into Spark? Do you have any expectations regarding this?

[fangbo]

Upon seeing this issue, is this feature intended to be integrated into Spark? Do you have any expectations regarding this?

Yes , I plan to support to create vector index distributively in Spark as soon as possible.

[goodqiang]

Upon seeing this issue, is this feature intended to be integrated into Spark? Do you have any expectations regarding this?

Yes , I plan to support to create vector index distributively in Spark as soon as possible.

I'm not sure if I can participate in this project. I don't have extensive experience with Spark-based distributed IVF, but I'm eager to learn