Enabled cfs for BaseIndexFileFormatTestCase.java

[shubhamsrkdev]

Problem

After merging #15295 some nightly tests started failing. @romseygeek reported the following:

TestLucene87StoredFieldsFormatHighCompression > testMergeStability FAILED
--
java.lang.AssertionError: expected:<{fnm=77, fdt=37004, fdx=64, fdm=159}> but was:<{cfs=37366, cfe=150}>
at __randomizedtesting.SeedInfo.seed([DF7E25E7C981E11B:AB3263C8C46BE3AD]:0)
at [email protected]/org.junit.Assert.fail(Assert.java:89)
at [email protected]/org.junit.Assert.failNotEquals(Assert.java:835)
at [email protected]/org.junit.Assert.assertEquals(Assert.java:120)
at [email protected]/org.junit.Assert.assertEquals(Assert.java:146)
at [email protected]/org.apache.lucene.tests.index.BaseIndexFileFormatTestCase.testMergeStability(BaseIndexFileFormatTestCase.java:306)

Solution

There are two solutions I can think of:

• We can modify Lucene87RWCodec to cache compoundFormat like Lucene94Codec:

  public final CompoundFormat compoundFormat() {
    return compoundFormat;
  }

• We can stop disabling CFS explicitly in this test (leave the default as is).

I am going with the second one as we don't want to change existing codecs.

Testing

./gradlew test --tests TestLucene87StoredFieldsFormatHighCompression.testMergeStability -Dtests.seed=DF7E25E7C981E11B -Dtests.nightly=true -Dtests.locale=ig-Latn-NG -Dtests.timezone=Asia/Qostanay -Dtests.asserts=true -Dtests.file.encoding=UTF-8

Passes:

BUILD SUCCESSFUL in 12s
294 actionable tasks: 147 executed, 147 up-to-date
Consider enabling configuration cache to speed up this build: https://docs.gradle.org/9.2.1/userguide/configuration_cache_enabling.html

./gradlew test   -Dtests.nightly=true -Dtests.locale=ig-Latn-NG -Dtests.timezone=Asia/Qostanay -Dtests.asserts=true -Dtests.file.encoding=UTF-8

Passes:

The slowest tests during this run:
  660.88s TestIDVersionPostingsFormat.testGlobalVersions (:lucene:sandbox)
  480.23s TestStressNRTReplication.test (:lucene:replicator)
  347.44s Test2BPostings.test (:lucene:core)
  332.55s TestSortedDvMultiRangeQuery.testDuelWithStandardDisjunction (:lucene:sandbox)
  311.42s TestSimpleTextPostingsFormat.testDocsAndFreqsAndPositionsAndOffsetsAndPayloads (:lucene:codecs)
  238.38s TestBestCompressionLucene80DocValuesFormat.testNumericFieldJumpTables (:lucene:backward-codecs)
  236.20s TestBestSpeedLucene80DocValuesFormat.testNumericFieldJumpTables (:lucene:backward-codecs)
  226.95s TestLucene90DocValuesFormat.testNumericFieldJumpTables (:lucene:core)
  226.79s TestLucene90DocValuesFormatMergeInstance.testNumericFieldJumpTables (:lucene:core)
  162.26s TestDemoParallelLeafReader.testRandomMultipleSchemaGens (:lucene:core)
The slowest suites during this run:
  811.59s TestSimpleTextPostingsFormat (:lucene:codecs)
  661.40s TestIDVersionPostingsFormat (:lucene:sandbox)
  621.32s TestLucene90DocValuesFormat (:lucene:core)
  609.17s TestBestSpeedLucene80DocValuesFormat (:lucene:backward-codecs)
  607.75s TestBestCompressionLucene80DocValuesFormat (:lucene:backward-codecs)
  607.16s TestLucene90DocValuesFormatMergeInstance (:lucene:core)
  480.51s TestStressNRTReplication (:lucene:replicator)
  347.46s Test2BPostings (:lucene:core)
  332.82s TestSortedDvMultiRangeQuery (:lucene:sandbox)
  259.55s TestDemoParallelLeafReader (:lucene:core)

BUILD SUCCESSFUL in 34m 21s
293 actionable tasks: 38 executed, 255 up-to-date
Consider enabling configuration cache to speed up this build: https://docs.gradle.org/9.2.1/userguide/configuration_cache_enabling.html

[stefanvodita]

I'm content with this solution. Thank you for addressing the issue!

[stefanvodita]

On second thought, after seeing #15616, I'm worried we're making a riskier change to the conditions the tests were running under in this PR than in #15616.

[mikemccand]

On second thought, after seeing #15616, I'm worried we're making a riskier change to the conditions the tests were running under in this PR than in #15616.

+1

It looks like this PR is removing the "don't use compound file" from testMergeStability? That does seem a bit dangerous -- does this test somehow rely on this, e.g. need to see individual index files (that would be hidden away inside CFS)? Was that added separately (after the test was originally created, for a specific reason / test failure)? Maybe do some digital archaeology to see why it disabled CFS?

[shubhamsrkdev]

Since this was fixed with #15616 I will close this PR!