Add increased rate of parallelism. #35
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thampiotr
reviewed
Mar 17, 2025
| maxPrevious := slices.MaxFunc(p.previous, func(a, b previousDesired) int { | ||
| return cmp.Compare(a.desired, b.desired) | ||
| }) | ||
| // If the most recent previous value was the maximum, that means drift is consistently increasing |
Contributor
There was a problem hiding this comment.
I have mixed feelings about this. Is there a way to calculate the required parallelism using throughput numbers? For example:
- if we're sending 100k samples/s, but WAL is backing up at a rate of 20k samples/s, then required throughput is 120k samples/s.
- From the above we can estimate that current parallelism needs to increase by 20%. Say it's 20 loops -> we increase it to 24.
I worry the current version may not respond well to some traffic patterns and we will need another update here soon, which may in turn break other traffic pattern and it may become a bit of a moving target.
Collaborator
Author
There was a problem hiding this comment.
We can its a bit more complicated. Let me see if I can get something that makes sense.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a burst of traffic comes in the system is to slow to rise, in this case it will double if the previous desired did not catch up.