Add statistics integration tests #20292
Draft
+1,251
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gabotechs
force-pushed
the
add-statistics-tests
branch
from
4cdcdba to
2ee489c
Compare
gabotechs
commented
Feb 11, 2026
Contributor
Author
There was a problem hiding this comment.
I'm not sure if this sizes are acceptable for this project. The new parquet files are the original ones for the TPC-DS benchmark sampled to 1000 rows max.
The total of all the files is 1.8Mb, usually I'd consider this acceptable to be committed directly to a git repository, but let me know if this falls within this project's policy.
gabotechs
commented
Feb 11, 2026
Comment on lines
+34
to
+38
| assert_snapshot!(display, @r" | ||
| row_estimation_accuracy=20% | ||
| byte_estimation_accuracy=8% | ||
| "); | ||
| Ok(()) |
Contributor
Author
There was a problem hiding this comment.
The idea is that further contributions should see this numbers getting increased.
9 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
TODO: create an epic for stats estimation improvements
Rationale for this change
The statistics propagation system is lacking some overall accuracy. There should be many low hanging fruits regarding statistic propagation that if addressed could increase the quality of the physical plans as a results of better estimations.
The idea behind this PR is to add some tests for qualifying further improvements in the statistic propagation system, so that people can incrementally improve it over time.
What changes are included in this PR?
Adds some new integration tests that compare the results estimated through statistics vs what actually happened within the query for the TPC-DS dataset (just that one for now):
ExecutePlan::partition_statistics()on all the nodes, collecting their estimations.It also adds a
Debugimplementation that allows visualizing the comparison in a fine-grained way. This should allow contributors to quickly identify places where stats are being innacurate:Are these changes tested?
This changes are exclusively new integration tests
Are there any user-facing changes?
No