Skip to content

Commit 526f7bb

Browse files
committed
[CELEBORN-2240] Adapt to SPARK-51756 which add a new parameter checksumValue in MapStatus.apply
### What changes were proposed in this pull request? Adapt to SPARK-51756, which changes the MapStatus API used by Celeborn. ### Why are the changes needed? A necessary step to make Celeborn support Spark 4.1. ### Does this PR resolve a correctness bug? No. ### Does this PR introduce _any_ user-facing change? Yes, it makes ### How was this patch tested? Have integrated with Spark 4.1 in the internal test env, and verified by some simple queries. Closes #3570 from pan3793/CELEBORN-2240. Authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
1 parent ac8cffb commit 526f7bb

File tree

1 file changed

+9
-1
lines changed
  • client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn

1 file changed

+9
-1
lines changed

client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkUtils.java

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,9 +82,17 @@ public class SparkUtils {
8282
public static final String FETCH_FAILURE_ERROR_MSG =
8383
"Celeborn FetchFailure appShuffleId/shuffleId: ";
8484

85+
private static final DynMethods.BoundMethod MAP_STATUS_APPLY_METHOD =
86+
DynMethods.builder("apply")
87+
// for SPARK-51756 (4.1.0) and later
88+
.impl(MapStatus$.class, BlockManagerId.class, long[].class, long.class, long.class)
89+
// for Spark 4.0 and earlier
90+
.impl(MapStatus$.class, BlockManagerId.class, long[].class, long.class)
91+
.build(MapStatus$.MODULE$);
92+
8593
public static MapStatus createMapStatus(
8694
BlockManagerId loc, long[] uncompressedSizes, long mapTaskId) {
87-
return MapStatus$.MODULE$.apply(loc, uncompressedSizes, mapTaskId);
95+
return MAP_STATUS_APPLY_METHOD.invoke(loc, uncompressedSizes, mapTaskId, 0L);
8896
}
8997

9098
private static final DynFields.UnboundField<SQLMetric> DATA_SIZE_METRIC_FIELD =

0 commit comments

Comments
 (0)