Skip to content

Commit 9e0c9da

Browse files
RaduBerindelemire
authored andcommitted
Try smaller segment length in binary fuse build
Some sizes around segment length transitions require many iterations and would work much better with the previous segment length. We add a simple fix that is more robust than tweaking the formula: once every four iterations, we try the previous segment length while keeping the same capacity. Note that in most cases this won't affect the build because it's rare to need more than 1-2 iterations. `TestBinaryFuseBoundarySizes` output (with numTrials=100): ``` binaryfusefilter_test.go:490: size: 2 iterations: 1.02 avg (2 max) binaryfusefilter_test.go:490: size: 8 iterations: 1.08 avg (3 max) binaryfusefilter_test.go:490: size: 24 iterations: 1.08 avg (3 max) binaryfusefilter_test.go:490: size: 27 iterations: 1.03 avg (2 max) binaryfusefilter_test.go:490: size: 55 iterations: 1.02 avg (2 max) binaryfusefilter_test.go:490: size: 91 iterations: 1.02 avg (2 max) binaryfusefilter_test.go:490: size: 120 iterations: 1.04 avg (2 max) binaryfusefilter_test.go:490: size: 303 iterations: 1.04 avg (2 max) binaryfusefilter_test.go:490: size: 349 iterations: 1.01 avg (2 max) binaryfusefilter_test.go:490: size: 1009 iterations: 1.01 avg (2 max) binaryfusefilter_test.go:490: size: 1124 iterations: 1.16 avg (4 max) binaryfusefilter_test.go:490: size: 3361 iterations: 1.03 avg (2 max) binaryfusefilter_test.go:490: size: 3551 iterations: 2.05 avg (6 max) binaryfusefilter_test.go:490: size: 11192 iterations: 1.04 avg (3 max) binaryfusefilter_test.go:490: size: 11521 iterations: 2.10 avg (6 max) binaryfusefilter_test.go:490: size: 37272 iterations: 1.01 avg (2 max) binaryfusefilter_test.go:490: size: 37454 iterations: 2.09 avg (6 max) binaryfusefilter_test.go:490: size: 124117 iterations: 1.03 avg (2 max) binaryfusefilter_test.go:490: size: 126131 iterations: 1.53 avg (4 max) binaryfusefilter_test.go:490: size: 413309 iterations: 1.00 avg (1 max) binaryfusefilter_test.go:490: size: 416077 iterations: 1.50 avg (4 max) binaryfusefilter_test.go:490: size: 1376321 iterations: 1.02 avg (3 max) ```
1 parent 292bf84 commit 9e0c9da

File tree

1 file changed

+20
-0
lines changed

1 file changed

+20
-0
lines changed

binaryfusefilter.go

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,26 @@ func buildBinaryFuse[T Unsigned](b *BinaryFuseBuilder, keys []uint64) (_ BinaryF
9090
// probability (i.e., a cosmic ray corrupts your system).
9191
return BinaryFuse[T]{}, iterations, errors.New("too many iterations")
9292
}
93+
if size > 4 && size < 1_000_000 {
94+
// The segment length is calculated using an empirical formula. For some
95+
// sizes, the segment length is too large and leads to many iterations.
96+
// Once every four iterations, use the previous segment length while
97+
// keeping the same capacity. See TestBinaryFuseBoundarySizes.
98+
switch iterations % 4 {
99+
case 2:
100+
// Switch to smaller segment size.
101+
filter.SegmentLength /= 2
102+
filter.SegmentLengthMask = filter.SegmentLength - 1
103+
filter.SegmentCount = filter.SegmentCount*2 + 2
104+
filter.SegmentCountLength = filter.SegmentCount * filter.SegmentLength
105+
case 3:
106+
// Restore the calculated segment size.
107+
filter.SegmentLength *= 2
108+
filter.SegmentLengthMask = filter.SegmentLength - 1
109+
filter.SegmentCount = filter.SegmentCount/2 - 1
110+
filter.SegmentCountLength = filter.SegmentCount * filter.SegmentLength
111+
}
112+
}
93113

94114
blockBits := 1
95115
for (1 << blockBits) < filter.SegmentCount {

0 commit comments

Comments
 (0)