[SPARK-55864][CORE][TESTS] Add more tests for SHS multiple log directories feature

[sarutak]

What changes were proposed in this pull request?

This PR proposes to add more tests for SHS multiple log directories feature added in SPARK-55793 (#54575).
New tests include:

• directory removed while SHS is running — verifies that removing a log directory at runtime does not crash the scan and apps from remaining directories are still listed
• directory does not exist at startup but created later — verifies that a directory that doesn't exist at startup is picked up on subsequent scans (monthly directory scenario)
• directory temporarily inaccessible then recovers — verifies that apps reappear after a temporarily inaccessible directory is restored
• all directories inaccessible does not crash — verifies graceful handling when all configured directories become unavailable
• config with empty entries between commas — verifies that empty entries in spark.history.fs.logDirectory (e.g., dir1,,dir2) are handled correctly
• logDirectory.names count mismatch falls back to full paths — verifies that when the number of names doesn't match the number of directories, display names fall back to full paths

Why are the changes needed?

For better test coverage.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Confirmed that all new tests passed.

$ build/sbt 'testOnly org.apache.spark.deploy.history.RocksDBBackendFsHistoryProviderSuite'

Was this patch authored or co-authored using generative AI tooling?

Kiro CLI / Opus 4.6