[EPIC] Shuffle file execs improvement

[milenkovicm]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

shuffle files execs could be improved in few ways

• make better metrics, especially at reader side
• offload I/O operations to dedicated executor
• offload arrow flight server to dedicated executor

Describe the solution you'd like

there is design document at in #660

Describe alternatives you've considered

Additional context

• Investigate use of comet shuffle #1287
• Proposal for more efficient disk-based shuffle mechanism #660

[PhVHoang]

Hi @milenkovicm ,

It seems like offload I/O operations to dedicated executor & offload arrow flight server to dedicated executor are not tackled in the mentioned design doc.

Would it make sense to implement the proposed TaskGroup abstraction first, and then revisit dedicated executors for I/O and Arrow Flight workloads once that foundation is in place?

[milenkovicm]

-offload I/O operations to dedicated executor, executor in this context mens tokio executor, so we have I/O executor for shuffle read.

• offload arrow flight server to dedicated executor is similar, at the moment flight server will run on #[tokio::main] executor which may take all available cpu cores

I'm not sure if TaskGroup will be beneficial in this case, but i'm happy to hear your proposal

[PhVHoang]

Yes, that’s exactly what I meant when I mentioned TaskGroup as well. I brought it up because it was discussed in the related issue (#660).

I agree that having a separate IoExecutor for I/O-heavy workloads makes sense, and the same rationale applies to the Arrow Flight server. Happy to work on the IoExecutor implementation and explore how best to integrate it for shuffle I/O (and potentially Flight as well), if that direction aligns with your expectations.

[milenkovicm]

fine with me. We need to find good way to test for performance regressions