[MINOR] Bump lance to 4.0.0 and lance-spark to 0.4.0

[rahil-c]

Bumps lance-core from 1.0.2 to 4.0.0 and lance-spark connector from 0.0.15 to 0.4.0. Updates affected import paths and adapts to the LanceArrowUtils.toArrowSchema signature change (drops the errorOnDuplicatedFieldNames parameter).

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[hudi-bot]

CI report:

• 42ea17a Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 53.97%. Comparing base (613fc49) to head (42ea17a).

Files with missing lines	Patch %	Lines
...apache/hudi/io/storage/HoodieSparkLanceWriter.java	0.00%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (613fc49) and HEAD (42ea17a). Click for more details.

HEAD has 27 uploads less than BASE

Flag	BASE (613fc49)	HEAD (42ea17a)
spark-scala-tests	10	0
spark-java-tests	15	0
utilities	1	0
common-and-other-modules	1	0

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #18498       +/-   ##
=============================================
- Coverage     68.84%   53.97%   -14.87%     
+ Complexity    28195    12282    -15913     
=============================================
  Files          2459     1425     -1034     
  Lines        135152    70968    -64184     
  Branches      16379     8004     -8375     
=============================================
- Hits          93039    38306    -54733     
+ Misses        34746    29239     -5507     
+ Partials       7367     3423     -3944     

Flag	Coverage Δ
common-and-other-modules	?
hadoop-mr-java-client	44.88% <ø> (+0.05%)	⬆️
spark-client-hadoop-common	48.47% <0.00%> (ø)
spark-java-tests	?
spark-scala-tests	?
utilities	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...rg/apache/hudi/io/storage/LanceRecordIterator.java	0.00% <ø> (-63.64%)	⬇️
...apache/hudi/io/storage/HoodieSparkLanceWriter.java	0.00% <0.00%> (-96.16%)	⬇️

... and 1807 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.