Conversation
xin-zhang2
force-pushed
the
PartitionedOutput1.0-benchmark
branch
from
18b20cf to
216a2fa
Compare
|
|
||
| void run(const RowVectorPtr& vector, int32_t numPartitions) { | ||
| folly::BenchmarkSuspender suspender; | ||
| // roundRobinPartitionFunction(vector, numPartitions, partitions_); |
There was a problem hiding this comment.
Remove the commented out line, or make the distribution a dimension as well
There was a problem hiding this comment.
We can add a non-uniform distribution to test the skew case.
| const auto vectorCopy = std::static_pointer_cast<RowVector>( | ||
| BaseVector::copy(*vector, bm->getPool())); | ||
| suspender.dismiss(); | ||
| bm->run(vectorCopy, 10000); |
There was a problem hiding this comment.
10000 partitions?
#partitions also need to be one test dimension. The value of intcan be {4, 8, 16, 32, 64, 128, 256, 512, 1024}
There was a problem hiding this comment.
Made #partition a dimension, and currently use {4, 16, 64, 256, 1024} as the values.
| rawEndPartitionOffsets[i] = offset; | ||
| } | ||
| endPartitionOffsets_->setSize(numPartitions * sizeof(vector_size_t)); | ||
| } |
There was a problem hiding this comment.
Where did you set beginPartitionOffsets_?
There was a problem hiding this comment.
beginPartitionOffsets_ is set through initializeBeginPartitionOffsets method during the execution of PartitionedVector::create, so we don't set it here.
| types.reserve(numColumns); | ||
|
|
||
| for (int i = 0; i < numColumns; ++i) { | ||
| types.push_back(typeSelection[i % typeSelection.size()]); |
There was a problem hiding this comment.
This causes repeating patterns that optimize caching.
Better shuffle types once.
There was a problem hiding this comment.
Shuffled the types vector before returning the RowType.
| BufferPtr topRowOffsets_; | ||
| BufferPtr beginPartitionOffsets_; | ||
| BufferPtr endPartitionOffsets_; | ||
| BufferPtr swappingBuffer_; |
There was a problem hiding this comment.
Never allocated or sized before use.
There was a problem hiding this comment.
swappingBuffer_ is also sized in partitionFixedWidthValues during the execution of PartitionedVector::create
| folly::runBenchmarks(); | ||
| bm.reset(); | ||
| return 0; | ||
| } |
There was a problem hiding this comment.
Can you please add the run output as a comment at the end of the file?
There was a problem hiding this comment.
Added the output in the comment of this PR.
prestodb-ci
force-pushed
the
main
branch
5 times, most recently
from
3fc79b1 to
d196f5a
Compare
2 participants
Uh oh!
There was an error while loading. Please reload this page.