[improve][misc] Sync commits from Apache into ds-4.17#20
Merged
sandeep-ctds merged 44 commits intods-4.17from Jul 22, 2025
Merged
[improve][misc] Sync commits from Apache into ds-4.17#20sandeep-ctds merged 44 commits intods-4.17from
sandeep-ctds merged 44 commits intods-4.17from
Conversation
--- ### Motivation Change the new ensemble log to the info level. Sometimes, the ensemble may not satisfied with the placement policy. The log should be info level but not a warn level. Because it will fix by the auto recovery later, so this just a information from the ensemble choose. (cherry picked from commit a12943d) (cherry picked from commit a753ace)
* fix entry location compaction * replace entryLocationCompactionEnable with entryLocationCompactionInterval * Add randomCompactionDelay to avoid all the bookies triggering compaction simultaneously * Fix the style issue * Fix the style issue * Fix test --------- Co-authored-by: houbonan <houbonan@didiglobal.com> Co-authored-by: zymap <zhangyong1025.zy@gmail.com> (cherry picked from commit ede1ba9) (cherry picked from commit 138849a)
…ache#4244) ### Motivation If the system property `readonlymode.enabled` is set to true on a ZooKeeper server, read-only mode is enabled. Data can be read from the server in read-only mode even if that server is split from the quorum. https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#Experimental+Options%2FFeatures To connect to the server in read-only mode, the client must also allow read-only mode. The `ZooKeeperClient` class in the bookkeeper repository also has an option called `allowReadOnlyMode`. https://github.com/apache/bookkeeper/blob/15171e1904f7196d8e9f4116ab2aecdf582e0032/bookkeeper-server/src/main/java/org/apache/bookkeeper/zookeeper/ZooKeeperClient.java#L219-L222 However, even if this option is set to true, the connection to the server in read-only mode will actually fail. The cause is in the `ZooKeeperWatcherBase` class. When the `ZooKeeperWatcherBase` class receives the `SyncConnected` event, it releases `clientConnectLatch` and assumes that the connection is complete. https://github.com/apache/bookkeeper/blob/15171e1904f7196d8e9f4116ab2aecdf582e0032/bookkeeper-server/src/main/java/org/apache/bookkeeper/zookeeper/ZooKeeperWatcherBase.java#L128-L144 However, if the server is in read-only mode, it will receive `ConnectedReadOnly` instead of `SyncConnected`. This causes the connection to time out without being completed. ### Changes Modified the switch statement in the `ZooKeeperWatcherBase` class to release `clientConnectLatch` when `ConnectedReadOnly` is received if the `allowReadOnlyMode` option is true. By the way, `allowReadOnlyMode` is never set to true in BookKeeper. So this change would be useless for BookKeeper. However, it is useful for Pulsar. Because Pulsar also uses `ZooKeeperWatcherBase` and needs to be able to connect to ZooKeeper in read-only mode. https://github.com/apache/pulsar/blob/cba1600d0f6a82f1ea194f3214a80f283fe8dc27/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/PulsarZooKeeperClient.java#L242-L244 (cherry picked from commit 4d50a44) (cherry picked from commit 1783677)
…pache#4473) Signed-off-by: ZhangJian He <shoothzj@gmail.com> (cherry picked from commit 7ab29e6) (cherry picked from commit db8e472)
…he#4482) ### Motivation In file RackawareEnsemblePlacementPolicyImpl.java In the log.warn below, we should print out the list of ensemble, instead of the object. The “ensemble” in line-619 should be changed into “ensemble.toList()”. https://github.com/apache/bookkeeper/blob/999cd0f2ab14404be4d6c24e388456dbe56bb1a8/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java#L600C3-L619C47 ### Changes The “ensemble” in line-619 changed into “ensemble.toList()”. (cherry picked from commit 7c41204) (cherry picked from commit 6624d99)
…cute concurrency (apache#4462) ### Motivation | step | `BK client 1` | `BK client 2` | | --- | --- | --- | | 1 | create ledger `1` | | 2 | | open ledger `1` | | 3 | | delete ledger `1` | | 4 | write data to ledger `1` | At the step `4`, the write should fail, but it succeeds. It leads users to assume the data has been written, but it can not be read. You can reproduce the issue by `testWriteAfterDeleted` There is a scenario that will lead to Pulsar loss messages - `broker-2` created a ledger - `broker-2`'s ZK session is expired, which will lead the topic it owned to be assigned to other brokers - `broker-0` owned the topic again - it will delete the last empty ledger - consumers connected to `broker-0` - producers connected to `broker-2` - send messages to the topic - on `broker-2`, the ledger can not be closed due to the ledger metadata has been deleted ### Changes Once the ledger is fenced, it can not be wrote anymore. (cherry picked from commit 47ef48e) (cherry picked from commit e7122c3)
### Motivation CVE-2024-7254 ### Changes Upgrade protobuf to 3.25.5 (cherry picked from commit 0229b5d) (cherry picked from commit ee60d6a)
…mble (apache#4478) (cherry picked from commit 0376bdc) (cherry picked from commit 60d9cd4)
* Upgrade Zookeeper to 3.9.3 to address CVE-2024-51504 * Upgrade curator to 5.7.1 (cherry picked from commit af8baa1) (cherry picked from commit 8ad8e3b)
…pache#4504) (cherry picked from commit 07de650) (cherry picked from commit 13e67f9)
While working on apache#4580, I noticed that the Commons Compress version and the Commons Codec library version aren't compatible. It causes a ClassDefNotFoundError in running tests which were using some specific methods of docker-java. While exploring this, I noticed that Apache Commons library versions haven't been kept up-to-date for a long time and it's better to handle that for 4.18.0 release. - upgrade commons-cli from 1.2 to 1.9.0 - upgrade commons-codec from 1.6 to 1.18.0 - upgrade commons-io from 2.17.0 to 2.19.0 - upgrade commons-lang3 from 3.6 to 3.17.0 - upgrade commons-compress from 1.26.0 to 1.27.0 (cherry picked from commit bcd6b52) (cherry picked from commit fab4780)
(cherry picked from commit 15988a6)
) OpenTelemetry library version is outdated in BookKeeper and conflicting with Pulsar OTel libraries since some libraries have evolved. - upgrade Otel to 1.45.0 - upgrade Otel instrumentation version to 1.33.6 (this was split from main Otel library to a separate project) (cherry picked from commit 1e59ae0) (cherry picked from commit cf0f4e1)
…t after rocksdb has been closed (apache#4581) * Fix the coredump that occurs when calling KeyValueStorageRocksDB.count() (possibly triggered by Prometheus) after RocksDB has been closed(apache#4243) * fix race when count op in process and db gets closed. --------- Co-authored-by: zhaizhibo <zhaizhibo@kuaishou.com> (cherry picked from commit 2831ed3) (cherry picked from commit 9d067f4)
(cherry picked from commit dd8eecc)
… 2.x (apache#4604) * Migration to commons-configuration2 and commons-lang3 * Change test that breaks with commons configuration 2 * Add commons-beanutils dependency * Revert "Change test that breaks with commons configuration 2" This reverts commit 073c267. * Fix ConfigKeyTest * Configure list handling to match commons-configuration 1.x default behavior * Fix the issue in using the API to configure defaults * Configure list delimiter handler also for distributedlog's ConfigurationSubscription * Address issue with system properties being null * Use list delimiter handler in DistributedLogConfiguration too * Fix spotbugs check * Fix ConfigurationTest - system properties are read when ServerConfiguration gets initialized * Fix checkstyle (cherry picked from commit 542ea09)
* Fix check read failed entry memory leak issue. * address the comments. (cherry picked from commit 9b8e566)
…ache#4607) * Fix the data loss issue that caused by the wrong entry log header --- # Motivation We observed numerous errors in the broker that failed to read the ledger from the bookkeeper; although the ledger metadata still exists, it was unable to read from the bookkeeper. After checking the data, we found the ledger located entry log was deleted by the bookkeeper. We have a data loss issue with the bookkeeper. The entry log file was deleted by the Garbage collector because the entry log file wrote a wrong file header. And there is an example that the shows the header is wrong: ``` Failed to get ledgers map index from: 82.log : Not all ledgers were found in ledgers map index. expected: -1932430239 -- found: 0 -- entryLogId: 82 ``` * Add test (cherry picked from commit b7fae1a)
* make recycler static to avoid OOM problem * Fixed missing license headers * Fixed checkstyle * More checkstyle --------- Co-authored-by: fanjianye <fanjianye@bigo.sg> Co-authored-by: Matteo Merli <mmerli@apache.org> (cherry picked from commit 137cc94)
…ache#4557) [fix] Write stuck due to pending add callback by multiple threads (apache#4557) (cherry picked from commit e47926b) (cherry picked from commit 143e5a8)
* Fix branch-4.17 CI * clean code (cherry picked from commit b4a4bd9)
…pache#4586) (cherry picked from commit 7fbb8d8) (cherry picked from commit 05f2d11)
### Motivation & Changes Upgrade Jetty to 9.4.57.v20241219 to address CVE-2024-6763 Jetty 9.4.57.v20241219 contains backported CVE-2024-6763 fix in jetty/jetty.project#12532 although it's not explicitly mentioned and most security scanners don't yet contain the information that it's been addressed in 9.4.57. More details: * jetty/jetty.project#12630 * https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.57.v20241219 Note: The backport is a partial mitigation and Jetty 9.4.57 will continue to be marked as vulnerable. There's a discussion and explanation here: https://gitlab.eclipse.org/security/cve-assignement/-/issues/25#note_2968611 (cherry picked from commit 99eb63a) (cherry picked from commit 7c58be4)
(cherry picked from commit afd4481)
…4-6763 (apache#4600) (apache#4632) (cherry picked from commit 770bd8a)
sandeep-ctds
approved these changes
Jul 22, 2025
sandeep-ctds
left a comment
sandeep-ctds
left a comment
There was a problem hiding this comment.
LGTM!! Need to fix deadlink CI when upstream fixes it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Descriptions of the changes in this PR:
Motivation
(Explain: why you're making that change, what is the problem you're trying to solve)
Changes
(Describe: what changes you have made)
Master Issue: #