Skip to content

[SPARK-55831][FOLLOWUP] Fix Maven CI by replacing with dummy SPARK_HOME in test#54651

Open
pan3793 wants to merge 6 commits intoapache:masterfrom
pan3793:SPARK-55831-followup
Open

[SPARK-55831][FOLLOWUP] Fix Maven CI by replacing with dummy SPARK_HOME in test#54651
pan3793 wants to merge 6 commits intoapache:masterfrom
pan3793:SPARK-55831-followup

Conversation

@pan3793
Copy link
Member

@pan3793 pan3793 commented Mar 6, 2026

What changes were proposed in this pull request?

Create a dummy SPARK_HOME for test, so that the YARN test won't depend on assembly package.

Why are the changes needed?

- YARN AM JavaOptions *** FAILED ***
  java.lang.IllegalStateException: Library directory '/home/runner/work/spark/spark/assembly/target/scala-2.13/jars' does not exist; make sure Spark is built.
  at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:230)
  at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:320)
  at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:36)
  at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:711)
  at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:529)
  at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:1042)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:569)
  ...

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass both Maven and SBT CI.

Was this patch authored or co-authored using generative AI tooling?

No.

./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} clean install
if [ "$MODULES_TO_TEST" != "connect" ]; then
if [ "$MODULES_TO_TEST" != *"hive-thriftserver"* ]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we still need to implement more fine-grained control. Some modules have their test sequences prior to the assembly module, and we should ensure that these tests can be conducted without relying on assembly. At the least, we should have a mechanism to inspect, skip, and notify that enabling these tests requires assembly(such as assume).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I switched to a simple approach that makes the test not depend on the assembly package, so we don't need to worry about this case.

@pan3793 pan3793 changed the title [SPARK-55831][FOLLOWUP] Fix Maven CI by retaining assembly for yarn module [SPARK-55831][FOLLOWUP] Fix Maven CI by replacing with dummy SPARK_HOME in test Mar 6, 2026
@pan3793
Copy link
Member Author

pan3793 commented Mar 6, 2026

This reverts commit 51c3096.
@pan3793 pan3793 requested a review from LuciferYang March 6, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants