From 930707053eac4a7c60245f380a827a40c139d239 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Thu, 11 Dec 2025 14:27:22 +0000 Subject: [PATCH 1/6] Improve documentation formatting and consistency across multiple files --- docs/BigDataAndChronicleQueue.adoc | 25 +++++----- docs/Channel-Pipe-Benchmarks.adoc | 3 ++ docs/DISCLAIMER.adoc | 4 ++ docs/FAQ.adoc | 39 +++++++++++---- docs/How_it_works.adoc | 9 ++-- docs/MigratingToV4.adoc | 16 +++--- .../modules/async-mode/pages/async_mode.adoc | 12 +++-- .../pages/command_line_tools.adoc | 3 ++ .../pages/timezone_rollover.adoc | 3 +- docs/antora/modules/demos/pages/demos.adoc | 49 +------------------ .../modules/demos/pages/message-history.adoc | 4 +- .../getting-started/pages/glossary.adoc | 4 +- .../getting-started/pages/quick-start.adoc | 7 +++ .../getting-started/pages/use-cases.adoc | 4 +- .../pages/what-is-chronicle-queue.adoc | 10 ++-- .../pages/performance-test-results.adoc | 2 +- .../queue-operations/pages/appending.adoc | 10 ++-- .../queue-operations/pages/indexing.adoc | 2 - .../queue-operations/pages/tailing.adoc | 17 ++----- .../antora/modules/tailing/pages/tailing.adoc | 17 ++----- docs/async_mode.adoc | 2 + docs/encryption.adoc | 4 +- docs/managing_roll_files_directly.adoc | 7 ++- docs/performance.adoc | 10 ++-- docs/pretoucher.adoc | 2 + docs/replication.adoc | 2 + ...Properties.adoc => system-properties.adoc} | 8 ++- docs/timezone_rollover.adoc | 2 + docs/utilities.adoc | 8 ++- 29 files changed, 143 insertions(+), 142 deletions(-) rename docs/{systemProperties.adoc => system-properties.adoc} (95%) diff --git a/docs/BigDataAndChronicleQueue.adoc b/docs/BigDataAndChronicleQueue.adoc index 6c4a51d6e2..45670e8021 100644 --- a/docs/BigDataAndChronicleQueue.adoc +++ b/docs/BigDataAndChronicleQueue.adoc @@ -1,5 +1,8 @@ = Big Data and Chronicle Queue Peter Lawrey +:toc: +:lang: en-GB +:source-highlighter: rouge == The Big Question @@ -12,9 +15,6 @@ Every reader sees every message, and a reader can join at any time and still see We assume that you can read/scan through messages fast enough that, even if you are not interested in most messages, it will still be fast enough. These are not consumers as such, and messages do not disappear after reading them. -//removed the link as the web page is not available -//image::http://chronicle.software/wp-content/uploads/2014/07/Chronicle-diagram_005.jpg[align=center] - Retaining every message has a number of advantages: * a message can be replayed as many times as is needed. @@ -22,8 +22,8 @@ Retaining every message has a number of advantages: * reduces the requirement for logging almost entirely, speeding up your application. * greater transparency leads to optimisations that you might have missed without a detailed record of every event that you process. -One of our clients, first implemented their trading system without Chronicle Queue and achieved an average latency of 35 micro-seconds tick to trade. -However, after switching to Chronicle Queue, the latency was reduced to 23 micro-seconds. +One of our clients first implemented their trading system without Chronicle Queue and achieved an average latency of 35 µs tick to trade. +However, after switching to Chronicle Queue, the latency was reduced to 23 µs. === What makes Chronicle Queue different from similar solutions @@ -76,33 +76,36 @@ This means the only real performance test is a complete system test. * You can test a micro-service, replaying from the same input file repeatedly, without the producer or down stream consumers running. * You can restart and upgrade a service, by replaying its *outputs* to ensure it is committed to the decisions/outcomes it has produced in the past. -This allows you to change your service and know that, even those your new version might have made different decision, it can honour those already made. +This allows you to change your service and know that, even though your new version might have made different decisions, it can honour those already made. -* Flow control is designed to smooth out burst and jitter in the system; but it can also hide such jitter as result. +* Flow control is designed to smooth out bursts and jitter in the system; but it can also hide such jitter as a result. The main disadvantage is that this assumes disk space is cheap. If you look at retail prices for enterprise grade disks, you can get TBs of disk for a few hundred dollars. Many investment banks do not take this approach, and internal charge rates for managed storage can be orders of magnitude higher. I have worked on systems that have more free memory than free disk space. -== What technology does Chronicle Queue using? +== What technology does Chronicle Queue use? What might be surprising is that Chronicle Queue is written entirely in pure `Java`. It can outperform many data storage solutions written in `C`. You might be wondering, how is this possible given that well written `C` is usually faster than Java. You need a degree of protection between your application and your data storage to minimise the risk of corruption. -As `Java` uses a `JVM`, it already has an abstraction layer, and a degree of protection.If an application throws an exception, this does not mean the data structure is corrupted. +As `Java` uses a `JVM`, it already has an abstraction layer, and a degree of protection. +If an application throws an exception, this does not mean the data structure is corrupted. To get a degree of isolation in `C`, many data storage solutions use `TCP`. The overhead of using `TCP`, even over loopback, can exceed the benefit of using `C`, and the throughput/latencies of Chronicle Queue can be `10x` greater by being able to use the data in-process. Chronicle Queue supports sharing of the data structure in memory for multiple JVMs, avoiding the need to use TCP to share data. === What does the data structure look like? -One of our key objectives is transparency.We have worked hard to make sure you can see exactly what has been written to this data structure. +One of our key objectives is transparency. +We have worked hard to make sure you can see exactly what has been written to this data structure. We use `YAML` as this is a format which is designed to be readable. -`JSON` and `XML`, we see, as a subset of `Javascript` and `SGML`, and not as readable.We support `JSON` for messaging with browsers. +`JSON` and `XML`, we see, as a subset of `Javascript` and `SGML`, and not as readable. +We support `JSON` for messaging with browsers. For performance reasons we use a binary form of `YAML`, which can be automatically translated to `YAML` for viewing. .Order class extends SelfDescribingMarshallable for helper methods diff --git a/docs/Channel-Pipe-Benchmarks.adoc b/docs/Channel-Pipe-Benchmarks.adoc index 790f026802..ebf2c09d1c 100644 --- a/docs/Channel-Pipe-Benchmarks.adoc +++ b/docs/Channel-Pipe-Benchmarks.adoc @@ -1,5 +1,8 @@ = Chronicle-Channel Peter Lawrey +:toc: +:lang: en-GB +:source-highlighter: rouge Support for distributed Channels based on Queue diff --git a/docs/DISCLAIMER.adoc b/docs/DISCLAIMER.adoc index dd2ee85573..e4c5a33c8b 100644 --- a/docs/DISCLAIMER.adoc +++ b/docs/DISCLAIMER.adoc @@ -1,3 +1,7 @@ += Chronicle Queue Data and Telemetry Disclaimer +:lang: en-GB +:source-highlighter: rouge + == Data This product uses Google Analytics, a web analytics service provided by Google, Inc.("Google"). diff --git a/docs/FAQ.adoc b/docs/FAQ.adoc index d553f965b3..93f58f597b 100644 --- a/docs/FAQ.adoc +++ b/docs/FAQ.adoc @@ -2,9 +2,10 @@ :pp: ++ :toc: manual :toclevels: 1 +:lang: en-GB +:source-highlighter: rouge :css-signature: demo :toc-placement: preamble -:icons: font This document provides information for some common questions about Chronicle Queue. @@ -34,7 +35,7 @@ All this can be done without expensive network sniffing/recording systems. === What was the library originally designed for? The original design was for a low-latency trading system which required persistence of everything in, and out, for a complete record of what happened, when, and for deterministic testing. -The target round-trip time for persisting the request, processing, and persisting the response, was `1` micro-second. +The target round-trip time for persisting the request, processing, and persisting the response, was `1` µs. Key principles are: @@ -119,7 +120,7 @@ Chronicle Map is likely to be a better choice for this requirement. === What is the performance like? Single Chronicle Queue (single file per cycle) supports sub-microsecond latencies. -If you use Wire, the typical latencies tend to be around one micro-second. +If you use Wire, the typical latencies tend to be around 1 µs. You can still use raw writing of bytes if you need maximum performance. === With a tight reader loop, I see 100% utilization. Will there be processing capability left for anything else? @@ -143,15 +144,15 @@ We plan to have a mode which reduces virtual memory use; even if it is a little === How fast is fast? -Chronicle Queue is designed to persist messages, and replay them in micro-second time. -Simple messages take as low as `0.4` micro-seconds. -Complex messages might take `10` micro-seconds to write and read. +Chronicle Queue is designed to persist messages, and replay them in microsecond time. +Simple messages take as low as `0.4` µs. +Complex messages might take `10` µs to write and read. Also Chronicle Queue is designed to sustain millions of inserts and updates per second. For bursts of up to `10%` of your main memory, you can sustain rates of `1 - 3` GB/second written. For example,l a laptop with `8` GB of memory might handle bursts of `800` MB at a rate of `1` GB per second. A server with `64` GB of memory might handle a burst of `6.5` GB at a rate of `3` GB per second. -If your key system is not measuring latency in micro-seconds, and throughput in thousands-per-second, then it is not that fast. +If your key system is not measuring latency in µs, and throughput in thousands-per-second, then it is not that fast. === How does it scale? @@ -170,12 +171,12 @@ Chronicle recommends lots of high-speed RAM. This is because Chronicle uses the page cache and RAM is in effect a cache to the disk. There are two cases where having a high-speed disk will give you a real benefit: -==== 1. Data rate +==== Data rate If the rate of data that you are writing exceeds the disk write speed. In most applications this is unlikely to occur. -==== 2. Page cache miss +==== Page cache miss For Chronicle queues which write and read messages lineally across memory, we mitigate this situation with the use of the Chronicle pre-toucher. The pre-toucher ensures that the page is loaded into the page cache before being written into the queue. @@ -751,4 +752,24 @@ Java Queue has similar functional requirements which should ultimately use simil ''' +=== Build and test troubleshooting + +* Supported JDKs: 8 and 21. Run Maven with an explicit `JAVA_HOME` so the compiler and tests use the intended JDK: ++ +[source] +---- +env JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 PATH=/usr/lib/jvm/java-1.8.0-openjdk-amd64/bin:$PATH mvn clean install -l build-java8.log +env JAVA_HOME=/usr/lib/jvm/java-1.21.0-openjdk-amd64 PATH=/usr/lib/jvm/java-1.21.0-openjdk-amd64/bin:$PATH mvn clean install -l build-java21.log +---- +* Capture the full log (using `tee` as above) when reporting failures. +* If you see a `Failed to compile generated method writer` warning, see the next section for additional diagnostics. + +=== Debugging method writer codegen fallbacks + +If you see warnings such as `Failed to compile generated method writer - falling back to proxy method writer` (often on older JDKs such as 8): + +* The tests now log additional context from `QueueTestCommon` when this occurs: test name, `java.version`, `wire.generator.v2`, and `disableProxyCodegen` system properties, plus the original throwable. +* Reproduce with the focused test on Java 8 to collect a small log: `JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 mvn -Dtest=MessageHistoryTest test -DfailIfNoTests=false`. +* Review the emitted warning in the test output to see the captured properties; if it repeats reliably, capture the full build log and the warning line for triage. + <<../README.adoc#,Back to ReadMe>> diff --git a/docs/How_it_works.adoc b/docs/How_it_works.adoc index d5bbb921b4..8f1a7e195c 100644 --- a/docs/How_it_works.adoc +++ b/docs/How_it_works.adoc @@ -1,8 +1,10 @@ = Chronicle Queue Peter Lawrey +:toc: +:lang: en-GB +:source-highlighter: rouge -//removed the link as web page is not available -//image::http://chronicle.software/wp-content/uploads/2014/07/ChronicleQueue_200px.png[] +High-level decisions around file format, concurrency, durability, encryption, and retention are recorded in link:../src/main/docs/decision-log.adoc[decision-log.adoc]. == How Chronicle Works @@ -22,7 +24,8 @@ Chronicle behaves like a management interface over off-heap memory, so that you Chronicle uses `RandomAccessFiles` while managing memory, and this choice brings a great deal of possibilities. Random access files permit non-sequential, or random, access to a files contents. -To access a file randomly, you open the file, seek a particular location, and then read from, or write to, that file. `RandomAccessFiles` can be seen as large C-type byte arrays that allows you to access any random index directly using pointers. +To access a file randomly, you open the file, seek a particular location, and then read from, or write to, that file. +`RandomAccessFiles` can be seen as large C-type byte arrays that allows you to access any random index directly using pointers. File portions can be used as `ByteBuffers` if the portion is mapped into memory. === What is the effect of page faults when we have a huge Chronicle and not enough RAM ? diff --git a/docs/MigratingToV4.adoc b/docs/MigratingToV4.adoc index a313c59ca5..b16c40fa46 100644 --- a/docs/MigratingToV4.adoc +++ b/docs/MigratingToV4.adoc @@ -1,9 +1,11 @@ = Migrating Chronicle Logger to Queue V4 Peter Lawrey +:lang: en-GB +:source-highlighter: rouge -These are the steps which were taken to migrate Chornicle Logger from using Chronicle Queue v3 to v4. +These are the steps which were taken to migrate Chronicle Logger from using Chronicle Queue v3 to v4. -=== Update the artifactId +== Update the artifactId .Artifact and package for Chronicle v3. [source, xml] @@ -27,11 +29,11 @@ to the following artifactId. NOTE: As Chronicle Queue v3 and v4 use different packages you can use both versions at the same time. -=== Rename the Chronicle class to ChronicleQueue +== Rename the Chronicle class to ChronicleQueue As Chronicle is the name of a family of products, the interface is now called ChronicleQueue to distinguish it from ChronicleMap for example. -=== createAppender() has been replaced with acquireAppender() +== createAppender() has been replaced with acquireAppender() We use the term `acquire` to indicate an object will be created only as needed. In this case, the method will create a new `ExcerptAppender` for each thread once. @@ -39,7 +41,7 @@ A thread cannot have more than one appender in use at once as this would cause a This method no longer throws an `IOException` -=== An Appender cannot be used as Bytes directly. +== An Appender cannot be used as Bytes directly. Conflating the `ExcerptAppender` and `Bytes` cause a number of problems including the use of `flush()` and `close()` which had different meanings between these two classes. @@ -77,7 +79,7 @@ tailer.readBytes(b -> { }); ---- -=== Using the Wire API instead of the Bytes API +== Using the Wire API instead of the Bytes API There is two ways to use the Wire API @@ -99,7 +101,7 @@ try (DocumentContext dc = appender.writingDocument()) { } ---- -=== Changes to the Bytes API. +== Changes to the Bytes API. Chronicle Bytes distinguishes between the read and write position, remaining and limit. diff --git a/docs/antora/modules/async-mode/pages/async_mode.adoc b/docs/antora/modules/async-mode/pages/async_mode.adoc index c02213de0d..7a9bfbd114 100644 --- a/docs/antora/modules/async-mode/pages/async_mode.adoc +++ b/docs/antora/modules/async-mode/pages/async_mode.adoc @@ -21,6 +21,7 @@ Backing the buffer with a persistent BytesStore (that is, one backed with a memo All processes can write to, and read from, the buffer. ==== Common scenarios: + * Back the buffer with an in-memory BytesStore. + You can have multiple writers and multiple readers in the same process. The buffer does not survive longer than the process. @@ -90,6 +91,7 @@ NOTE: if you want multiple buffered queues to share the same drainer thread you `SingleChronicleQueueBuilder`s `eventLoop` to share the same event loop. ====== Example: + [source,java] ---- try (ChronicleQueue queue = SingleChronicleQueueBuilder @@ -151,10 +153,12 @@ NOTE: This example won't work on Windows as it uses /dev/shm. ---- ===== Drainer thread + When the queue's event loop is closed, the drainer thread will wait up to 5 seconds to finish draining to the underlying queue. If draining can not complete, a warning message is logged ===== Unsupported operations + The following operations are unsupported when using EnterpriseSingleChronicleQueue backed by an asynchronous buffer: * writing and reading of metadata @@ -168,9 +172,9 @@ from some methods. It does not contain the full functionality of EnterpriseSingleChronicleQueue - it does not drain to an underlying queue, for example. It is created as in the example below, where: -- `ringBuffer` is the buffer to back this queue with, -- `wireType` is the wire type to use when appending and tailing, -- `pauser` is the pauser to use by the appender when waiting for readers. +* `ringBuffer` is the buffer to back this queue with, +* `wireType` is the wire type to use when appending and tailing, +* `pauser` is the pauser to use by the appender when waiting for readers. [source,java] ---- @@ -188,4 +192,4 @@ The exception may be thrown at either read or write time depending on how the bu === Licence Chronicle Async Mode is a licenced product and is licenced separately to chronicle queue enterprise. -If you would like further details on Async Mode, please contact sales@chronicle.software +If you would like further details on Async Mode, please contact mailto:sales@chronicle.software[sales@chronicle.software] diff --git a/docs/antora/modules/command-line/pages/command_line_tools.adoc b/docs/antora/modules/command-line/pages/command_line_tools.adoc index 00bb070fe3..b43e343914 100644 --- a/docs/antora/modules/command-line/pages/command_line_tools.adoc +++ b/docs/antora/modules/command-line/pages/command_line_tools.adoc @@ -38,6 +38,8 @@ The following command line tools can be used to interact with Chronicle Queue. Where noted, each command line tool can be invoked via a shell script. Command line tools can also be invoked using Maven, assuming a project POM file that includes the Chronicle Queue artifact. +Set an explicit `JAVA_HOME` (8 or 21) when invoking Maven so the tool runs on the intended JDK. + For example: [source,shell script] @@ -145,6 +147,7 @@ $ mvn exec:java -Dexec.mainClass="net.openhft.chronicle.queue.ChronicleHistoryRe ---- Invocation via shell script from the `Chronicle-Queue` root folder. + [source,shell script] ---- $ ./bin/history_reader.sh diff --git a/docs/antora/modules/configuration/pages/timezone_rollover.adoc b/docs/antora/modules/configuration/pages/timezone_rollover.adoc index b79d3c7db3..c1f51ce993 100644 --- a/docs/antora/modules/configuration/pages/timezone_rollover.adoc +++ b/docs/antora/modules/configuration/pages/timezone_rollover.adoc @@ -22,7 +22,8 @@ NOTE: This feature is only relevant to daily rollovers where `RollCycle` is one public SingleChronicleQueueBuilder rollTime(@NotNull final LocalTime rollTime, @NotNull final ZoneId zoneId); ---- -NOTE: `rollTime()` allows the rollover time of the cycle file to be controlled in both Chronicle Queue open-source and Chronicle QueueEnterprise Edition. + +NOTE: `rollTime()` allows the rollover time of the cycle file to be controlled in both Chronicle Queue open-source and Chronicle QueueEnterprise Edition. ++ If this is used with in the open-source version, any `zoneId` other than `UTC` will be converted to `UTC`, and a warning message logged. == Examples diff --git a/docs/antora/modules/demos/pages/demos.adoc b/docs/antora/modules/demos/pages/demos.adoc index 3f01251738..6ea7c7acd4 100644 --- a/docs/antora/modules/demos/pages/demos.adoc +++ b/docs/antora/modules/demos/pages/demos.adoc @@ -8,54 +8,7 @@ author: Julia Gustafsson See runnable demos in link:https://github.com/OpenHFT/Chronicle-Queue-Demo[https://github.com/OpenHFT/Chronicle-Queue-Demo,window=blank] == What You Need to Get Started -* Maven 3.6.x +* Maven 3.8.x * Java 8 update 180+ * Intellij CE or another IDE * Access to the internet for Maven to download the JARs needed - -//== Order Processor -// -//image::Two-hop-latency.PNG[] -// -//https://github.com/OpenHFT/Chronicle-Queue-Demo/tree/master/order-processor -// -//== Downloading and Running Sample Programs -// -//If you're running Windows, you will need to install the `git` client and `open-ssh` Cygwin, https://cygwin.com/install.html[here], -//with a guide showing installation and packages http://www.mcclean-cooper.com/valentino/cygwin_install/[here]. -//This allows you to install Linux packages that aren't already on Windows. -// -//When this asks you which packages you want to install, search for and add `git`. This is under `Development` and you need to click `skip` so it says to `install`. -// -//image::gitpack.png[] -// -//Ensure you have the JDK for Java 8, you could use https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.html[this]. -// -//Open Intelij or your https://en.wikipedia.org/wiki/Integrated_development_environment[IDE], or https://www.jetbrains.com/idea/download/#section=windows[install Intelij] first if you haven't already. -//Intelij is used for this tutorial. -// -//Go to `Check out from Version Control` and select `Git`, opening the `Clone Repository`. -//This is where you will get a copy of the Chronicle Queue Sample code. -// -//image::homegit.png[] -// -//Next, copy the URL below into the `Git Repository URL`, -//and remember to take note of the `Parent Directory`. Press `Clone`. -// -//[source] -//---- -//https://github.com/OpenHFT/Chronicle-Queue-Sample.git -//---- -// -//image::Clone.png[] -// -//Should you choose you close the Project under `File`, you an open it again by going to `Open`. -//Then you can find the repository in the directory that you saved it in earlier. -// -//image::Open.png[] -// -//image::directory.png[] -// -//Then you can run the first example https://github.com/OpenHFT/Chronicle-Queue-Sample/tree/master/simple-input[Simple Input] -// -//The second example is https://github.com/OpenHFT/Chronicle-Queue-Sample/tree/master/simple-translator[Simple Translator] diff --git a/docs/antora/modules/demos/pages/message-history.adoc b/docs/antora/modules/demos/pages/message-history.adoc index 3853aef3df..fa73bf6242 100644 --- a/docs/antora/modules/demos/pages/message-history.adoc +++ b/docs/antora/modules/demos/pages/message-history.adoc @@ -49,6 +49,7 @@ Or you can dump with the chronicle queue tool `net.openhft.chronicle.queue.Chron mvn exec:java@CHRM which will output like the below + ---- Timings below in MICROSECONDS sourceId 1 startTo1 @@ -59,6 +60,7 @@ count: 100001 100001 99.9: 62 80 99.99: 108 120 ---- + `1` is time taken to cross the bridge microservice and `startTo1` the time taken to get from publisher to bridge microservice. === Common Errors @@ -74,7 +76,7 @@ WARNING: Illegal reflective access by net.openhft.compiler.MyJavaFileManager (fi The microsecond timestamps are based on wall clock, however the nanosecond time stamps are based on the uptime of the machine. -To decode this, you need to take the first nano-second time stamp and adjust further timestamps as deltas on the micro-second timestamp. +To decode this, you need to take the first nanosecond time stamp and adjust further timestamps as deltas on the µs timestamp. The `DumpOutMain` writes the decoded messages as well as the bytes representing the first message. diff --git a/docs/antora/modules/getting-started/pages/glossary.adoc b/docs/antora/modules/getting-started/pages/glossary.adoc index 01ef575a60..a3bf57eba3 100644 --- a/docs/antora/modules/getting-started/pages/glossary.adoc +++ b/docs/antora/modules/getting-started/pages/glossary.adoc @@ -8,7 +8,7 @@ author: Julia Gustafsson [#a] == A -* *Appender:* An appender is the message source.You add data by appending it to the Chronicle Queue.Sequential writes can be performed by appending to the end of queue only.There is no way to insert, or delete excerpts. +* *Appender:* An appender is the message source. You add data by appending it to the Chronicle Queue. Sequential writes can be performed by appending to the end of queue only. There is no way to insert or delete excerpts. [#d] == D @@ -18,7 +18,7 @@ author: Julia Gustafsson [#e] == E -* *Exerpt:* An excerpt is the part of the documents that Chronicle Queue writes, that contains the actual data.When writing message to a Chronicle Queue, a new xref:advanced:advanced.adoc#_queue_documents[document] (containing an excerpt) is created and written to. +* *Excerpt:* An excerpt is the part of the documents that Chronicle Queue writes that contains the actual data. When writing a message to a Chronicle Queue, a new xref:advanced:advanced.adoc#_queue_documents[document] (containing an excerpt) is created and written to. [#m] == M diff --git a/docs/antora/modules/getting-started/pages/quick-start.adoc b/docs/antora/modules/getting-started/pages/quick-start.adoc index 58818dbc1e..9be6fd9216 100644 --- a/docs/antora/modules/getting-started/pages/quick-start.adoc +++ b/docs/antora/modules/getting-started/pages/quick-start.adoc @@ -25,6 +25,13 @@ Before starting with the set up, make sure your development environment meets th * Java 8 update 180+ or later * Access to the internet to allow Maven or Gradle to download the JARs needed +Use an explicit `JAVA_HOME` when building or running tools so the intended JDK is used, for example: + +[source] +---- +env JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 PATH=$JAVA_HOME/bin:$PATH mvn clean install +---- + == Installing Chronicle Queue [[installing_chronicle_queue]] The installation is performed by notifying your build tool that the Chronicle Queue dependency is required to compile your source code. diff --git a/docs/antora/modules/getting-started/pages/use-cases.adoc b/docs/antora/modules/getting-started/pages/use-cases.adoc index d8d4dc9ae3..ddf62a6435 100644 --- a/docs/antora/modules/getting-started/pages/use-cases.adoc +++ b/docs/antora/modules/getting-started/pages/use-cases.adoc @@ -43,7 +43,7 @@ Again, Chronicle Queue can support millions of events per-second, per-server, an == Latency Sensitive Micro-Services -Chronicle Queue supports low latency IPC (Inter Process Communication) between JVMs on the same machine in the order of magnitude of 1 microsecond; as well as between machines with a typical latency of 10 microseconds for modest throughputs of a few hundred thousands. +Chronicle Queue supports low latency IPC (Inter Process Communication) between JVMs on the same machine in the order of magnitude of 1 µs; as well as between machines with a typical latency of 10 µs for modest throughputs of a few hundred thousands. Chronicle Queue supports throughputs of millions of events per second, with stable microsecond latencies. See https://vanilla-java.github.io/tag/Microservices/[Articles on the use of Chronicle Queue in Microservices] @@ -55,7 +55,7 @@ All the information about the state of those components can be reproduced extern This significantly reduces the need for additional logging. However, any logging you do need can be recorded in great detail. This makes enabling `DEBUG` logging in production practical. -This is because the cost of logging is very low; less than 10 microseconds. +This is because the cost of logging is very low; less than 10 µs. Logs can be replicated centrally for log consolidation. Chronicle Queue is being used to store 100+ TB of data, which can be replayed from any point in time. diff --git a/docs/antora/modules/introduction/pages/what-is-chronicle-queue.adoc b/docs/antora/modules/introduction/pages/what-is-chronicle-queue.adoc index b34eac3d67..c7856a006c 100644 --- a/docs/antora/modules/introduction/pages/what-is-chronicle-queue.adoc +++ b/docs/antora/modules/introduction/pages/what-is-chronicle-queue.adoc @@ -16,21 +16,21 @@ It is a distributed, unbounded, and persisted queue that: * supports asynchronous RMI and Publish/Subscribe interfaces with microsecond latencies. -* passes messages between JVMs in under a microsecond (in optimised examples). +* passes messages between JVMs in under 1 µs (in optimised examples). -* passes messages between JVMs on different machines via replication in under 10 microseconds (in optimised examples). +* passes messages between JVMs on different machines via replication in under 10 µs (in optimised examples). * provides stable, soft, real-time latencies into the millions of messages per second for a single thread to one queue; with total ordering of every event. == Efficiency -Chronicle Queue aims to achieve latencies of under 40 microseconds for 99% to 99.99% of the time. -Without replication, Chronicle Queue delivers latencies below 40 microseconds end-to-end across multiple services. +Chronicle Queue aims to achieve latencies of under 40 µs for 99% to 99.99% of the time. +Without replication, Chronicle Queue delivers latencies below 40 µs end-to-end across multiple services. Often the 99% latency is entirely dependent on the choice of operating system and hard disk sub-system. === Benchmarks -When publishing 40-byte messages, a high percentage of the Chronicle Queue yields latencies under 1 microsecond. +When publishing 40-byte messages, a high percentage of the Chronicle Queue yields latencies under 1 µs. In the tables below, the 99th percentile latency is the worst 1 in 100, and the 99.9th percentile is the worst 1 in 1000 latency. .Latency to send/receive on the same machine. diff --git a/docs/antora/modules/performance/pages/performance-test-results.adoc b/docs/antora/modules/performance/pages/performance-test-results.adoc index 3120b16db7..38cdb468dc 100644 --- a/docs/antora/modules/performance/pages/performance-test-results.adoc +++ b/docs/antora/modules/performance/pages/performance-test-results.adoc @@ -31,7 +31,7 @@ image::cqz-performance-java.png[cqz-performance-java.png] === Chronicle Queue vs Chronicle Queue Zero -To make the comparison between Chronicle Queue ( abbreviated to Queue ) and Chronicle Queue Zero ( abbreviated to Queue Zero ) performance even clearer, the below plots show the end-to-end latencies for sending a 1024-byte message using Queue and Queue Zero, for C++ and Java respectively. +To make the comparison between Chronicle Queue ( abbreviated to Queue ) and Chronicle Queue Zero ( abbreviated to Queue Zero ) performance even clearer, the below plots show the end-to-end latencies for sending a 1024-byte message using Queue and Queue Zero, for C{pp} and Java respectively. These plots again show both the much reduced latency using Queue Zero, as well as the reduction of outliers at higher percentiles. image::cq-vs-cqz.png[cq-vs-cqz.png] diff --git a/docs/antora/modules/queue-operations/pages/appending.adoc b/docs/antora/modules/queue-operations/pages/appending.adoc index edea85d6bf..ab054e85ac 100644 --- a/docs/antora/modules/queue-operations/pages/appending.adoc +++ b/docs/antora/modules/queue-operations/pages/appending.adoc @@ -24,13 +24,15 @@ String message = "Hello World!"; appender.writeText(message); ---- -There are however various ways of appending data, ranging from high-level abstractions such as `writeText` and xref:queue-operations:read-write-proxies.adoc#_methodwriter[write proxies] down to a low-level API, or even writing directly to raw memory.These alternatives are presented in more detail in the following sections. +There are, however, various ways of appending data, ranging from high-level abstractions such as `writeText` and xref:queue-operations:read-write-proxies.adoc#_methodwriter[write proxies] down to a low-level API, or even writing directly to raw memory. +These alternatives are presented in more detail in the following sections. == Low-level API == Writing Text -The high-level methods such as `writeText()` are convenience methods on directly calling `appender.writingDocument()`, but both approaches essentially achieve the same thing.The implementation of `writeText(CharSequence text)` looks like this: +The high-level methods such as `writeText()` are convenience methods on directly calling `appender.writingDocument()`, but both approaches essentially achieve the same thing. +The implementation of `writeText(CharSequence text)` looks like this: [source,java] ---- @@ -50,7 +52,9 @@ try (final DocumentContext dc = appender.writingDocument()) { } ---- -The length of the data is written to the header when the try-with-resources block reaches _close_.You can also use the `DocumentContext` to find out the index that the data has just been assigned (see below).The index can later be used to move-to/look up this excerpt. +The length of the data is written to the header when the try-with-resources block reaches _close_. +You can also use the `DocumentContext` to find out the index that the data has just been assigned (see below). +The index can later be used to move-to/look up this excerpt. [source,java] ---- diff --git a/docs/antora/modules/queue-operations/pages/indexing.adoc b/docs/antora/modules/queue-operations/pages/indexing.adoc index fb35bd6bed..e332efdd77 100644 --- a/docs/antora/modules/queue-operations/pages/indexing.adoc +++ b/docs/antora/modules/queue-operations/pages/indexing.adoc @@ -20,8 +20,6 @@ Recalling from the chapter on xref:configuration:roll-cycle.adoc[Roll Cycles], a In turn, the sequence number simply refers to the n:th entry within that queue file (cycle). Both the cycle and sequence number are incremented by 1, and the count begins at 0. -//image::../../../../images/queue-index.png[Queue Indexes] - Different roll-cycles have a difference balance between how many bits are allocated to the message sequence number, and how many of the remaining bits are allocated to the cycle number. In other words, different roll-cycles allow us to trade off the maximum number of cycles, for the maximum number of messages within the cycle. Learn more about this in xref:configuration:roll-cycle.adoc[Roll Cycle] diff --git a/docs/antora/modules/queue-operations/pages/tailing.adoc b/docs/antora/modules/queue-operations/pages/tailing.adoc index dbf68a47e8..b9f8611434 100644 --- a/docs/antora/modules/queue-operations/pages/tailing.adoc +++ b/docs/antora/modules/queue-operations/pages/tailing.adoc @@ -196,24 +196,13 @@ If this is prevented by an error, the same message will be read on each restart. == Command Line Tools - Reading and Writing a Chronicle Queue -Chronicle Queue stores its data in binary format, with a file extension of `cq4`: - -[source,text] ----- -\�@πheader∂SCQStoreÇE��»wireType∂WireTypeÊBINARYÕwritePositionèèèèß��������ƒroll∂SCQSRollÇ*���∆length¶ÄÓ6�∆format -ÎyyyyMMdd-HH≈epoch¶ÄÓ6�»indexing∂ SCQSIndexingÇN��� indexCount•�ÃindexSpacingÀindex2Indexé����ß��������…lastIndexé� -���ß��������fllastAcknowledgedIndexReplicatedé�����ߡˇˇˇˇˇˇˇ»recovery∂TimedStoreRecoveryÇ���…timeStampèèèß -��������������������������������������������������������������������������������������������� -��������������������������������������������������������������������������������������������� -��������������������������������������������������������������������������������������������� -��������������������������������������������������������������������������������������������� -����������������������������������������������������������������� ----- +Chronicle Queue stores its data in binary format, with a file extension of `cq4`. This can often be a bit difficult to read, so it is better to dump the `cq4` files as text. This can also help you fix your production issues, as it gives you the visibility as to what has been stored in the queue, and in what order. -You can dump the queue to the terminal using `net.openhft.chronicle.queue.main.DumpMain` or `net.openhft.chronicle.queue.ChronicleReaderMain`. `DumpMain` performs a simple dump to the terminal while `ChronicleReaderMain` handles more complex operations, e.g. tailing a queue. +You can dump the queue to the terminal using `net.openhft.chronicle.queue.main.DumpMain` or `net.openhft.chronicle.queue.ChronicleReaderMain`. +`DumpMain` performs a simple dump to the terminal while `ChronicleReaderMain` handles more complex operations, e.g. tailing a queue. They can both be run from the command line in a number of ways described below. == DumpMain diff --git a/docs/antora/modules/tailing/pages/tailing.adoc b/docs/antora/modules/tailing/pages/tailing.adoc index 04a00fbba0..ccdf919e67 100644 --- a/docs/antora/modules/tailing/pages/tailing.adoc +++ b/docs/antora/modules/tailing/pages/tailing.adoc @@ -196,22 +196,11 @@ If this is prevented by an error, the same message will be read on each restart. == Command Line Tools - Reading and Writing a Chronicle Queue -Chronicle Queue stores its data in binary format, with a file extension of `cq4`: - -[source,text] ----- -\�@πheader∂SCQStoreÇE��»wireType∂WireTypeÊBINARYÕwritePositionèèèèß��������ƒroll∂SCQSRollÇ*���∆length¶ÄÓ6�∆format -ÎyyyyMMdd-HH≈epoch¶ÄÓ6�»indexing∂ SCQSIndexingÇN��� indexCount•�ÃindexSpacingÀindex2Indexé����ß��������…lastIndexé� -���ß��������fllastAcknowledgedIndexReplicatedé�����ߡˇˇˇˇˇˇˇ»recovery∂TimedStoreRecoveryÇ���…timeStampèèèß -��������������������������������������������������������������������������������������������� -��������������������������������������������������������������������������������������������� -��������������������������������������������������������������������������������������������� -��������������������������������������������������������������������������������������������� -����������������������������������������������������������������� ----- +Chronicle Queue stores its data in binary format, with a file extension of `cq4`. This can often be a bit difficult to read, so it is better to dump the `cq4` files as text. This can also help you fix your production issues, as it gives you the visibility as to what has been stored in the queue, and in what order. -You can dump the queue to the terminal using `net.openhft.chronicle.queue.main.DumpMain` or `net.openhft.chronicle.queue.ChronicleReaderMain`. `DumpMain` performs a simple dump to the terminal while `ChronicleReaderMain` handles more complex operations, e.g. tailing a queue. +You can dump the queue to the terminal using `net.openhft.chronicle.queue.main.DumpMain` or `net.openhft.chronicle.queue.ChronicleReaderMain`. +`DumpMain` performs a simple dump to the terminal while `ChronicleReaderMain` handles more complex operations, e.g. tailing a queue. More information and examples of usage is provided on the xref:command-line:command_line_tools.adoc[Command Line Tools] page. diff --git a/docs/async_mode.adoc b/docs/async_mode.adoc index b896e0d013..6a88b32a07 100644 --- a/docs/async_mode.adoc +++ b/docs/async_mode.adoc @@ -1,4 +1,6 @@ = Chronicle Async Mode +:lang: en-GB +:source-highlighter: rouge [NOTE] ==== diff --git a/docs/encryption.adoc b/docs/encryption.adoc index f8d70f165d..1eec93b2d8 100644 --- a/docs/encryption.adoc +++ b/docs/encryption.adoc @@ -1,4 +1,6 @@ -= Chroncicle Queue Encryption += Chronicle Queue Encryption +:lang: en-GB +:source-highlighter: rouge [NOTE] ==== diff --git a/docs/managing_roll_files_directly.adoc b/docs/managing_roll_files_directly.adoc index 7b39526448..3be1d01a63 100644 --- a/docs/managing_roll_files_directly.adoc +++ b/docs/managing_roll_files_directly.adoc @@ -1,12 +1,11 @@ = Managing Roll Files Directly Per Minborg -:toc: macro +:toc: :toclevels: 3 :css-signature: demo -:toc-placement: macro :icons: font - -toc::[] +:lang: en-GB +:source-highlighter: rouge This document describes how roll files can be managed directly via Java. One example of direct management would be to remove old unused roll files. diff --git a/docs/performance.adoc b/docs/performance.adoc index 759bf46a56..0d59f4f7b9 100644 --- a/docs/performance.adoc +++ b/docs/performance.adoc @@ -1,11 +1,13 @@ = Chronicle Queue Performance +:lang: en-GB +:source-highlighter: rouge [NOTE] ==== -Contents moved to link:http://portal.chronicle.software[] +This content has moved to link:https://portal.chronicle.software[]. ==== -Chronicle Queue, Chronicle Queue Enterprise, and Chronicle Queue Zero are all high performance queues that can be used in a variety of solutions. -For detailed performance tuning and performance test results visit link:https://portal.chronicle.software/docs/queue/chronicle-queue/performance/performance.html[] +Chronicle Queue, Chronicle Queue Enterprise, and Chronicle Queue Zero are all high-performance queues that can be used in a variety of solutions. +For detailed performance tuning guidance and test results, visit link:https://portal.chronicle.software/docs/queue/chronicle-queue/performance/performance.html[]. -<<../ReadMe.adoc#,Back to ReadMe>> +<<../README.adoc#,Back to Chronicle Queue>> diff --git a/docs/pretoucher.adoc b/docs/pretoucher.adoc index 215f2f86ca..138e91bf87 100644 --- a/docs/pretoucher.adoc +++ b/docs/pretoucher.adoc @@ -1,4 +1,6 @@ = Chronicle Queue Pre-toucher +:lang: en-GB +:source-highlighter: rouge [NOTE] ==== diff --git a/docs/replication.adoc b/docs/replication.adoc index 6c0541bb77..2a7170a2c6 100644 --- a/docs/replication.adoc +++ b/docs/replication.adoc @@ -1,4 +1,6 @@ = Chronicle Queue Enterprise Replication +:lang: en-GB +:source-highlighter: rouge [NOTE] ==== diff --git a/docs/systemProperties.adoc b/docs/system-properties.adoc similarity index 95% rename from docs/systemProperties.adoc rename to docs/system-properties.adoc index 0595ca7c52..3d61a49c76 100644 --- a/docs/systemProperties.adoc +++ b/docs/system-properties.adoc @@ -1,11 +1,17 @@ += Chronicle Queue System Properties +:toc: +:lang: en-GB +:source-highlighter: rouge + == System Properties + Below, a number of relevant System Properties are listed. NOTE: With `getBoolean`, returns if a System Property with the provided {@code systemPropertyKey} either exists, is set to "yes" or is set to "true", see https://github.com/OpenHFT/Chronicle-Core/blob/351e79ed593fa656c21b4e5a540a3a5831cd06a3/src/main/java/net/openhft/chronicle/core/Jvm.java#L1184[javadoc]. .System properties -[cols=4*, options="header"] +[cols=4*,options="header"] |=== | Property Key | Default | Description | Java Variable Name (Type) | chronicle.queue.checkrollcycle | `false` | Setting this property to "yes", "true" or "", prints a warning message every time a roll cycle file is created | _SHOULD_CHECK_CYCLE_ (boolean) diff --git a/docs/timezone_rollover.adoc b/docs/timezone_rollover.adoc index d554351837..a74141a3bf 100644 --- a/docs/timezone_rollover.adoc +++ b/docs/timezone_rollover.adoc @@ -1,4 +1,6 @@ = Timezone Queue Rollover +:lang: en-GB +:source-highlighter: rouge [NOTE] ==== diff --git a/docs/utilities.adoc b/docs/utilities.adoc index 59ba27f2ce..efd3960587 100644 --- a/docs/utilities.adoc +++ b/docs/utilities.adoc @@ -1,12 +1,10 @@ = Chronicle Queue Utilities Neil Clifford -:toc: macro +:toc: :toclevels: 3 :css-signature: demo -:toc-placement: macro -:icons: font - -toc::[] +:lang: en-GB +:source-highlighter: rouge This document describes Chronicle Queue utility programs that have been created to help in the maintenance of your Chronicle Queue environment. From ce8461455d98e8ce09e3d9da992ef18843f44a5f Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Thu, 11 Dec 2025 14:27:39 +0000 Subject: [PATCH 2/6] Deprecate methods and add utility holder constructors in various classes --- .../queue/ChronicleHistoryReaderMain.java | 8 +++ .../chronicle/queue/ChronicleQueue.java | 12 ++++ .../chronicle/queue/ChronicleReaderMain.java | 6 ++ .../chronicle/queue/ChronicleWriterMain.java | 4 ++ .../chronicle/queue/ExcerptAppender.java | 1 + .../chronicle/queue/ExcerptCommon.java | 6 +- .../chronicle/queue/ExcerptTailer.java | 11 ++++ .../chronicle/queue/NoMessageHistory.java | 13 +++- .../openhft/chronicle/queue/RollCycles.java | 9 +++ .../chronicle/queue/TailerDirection.java | 13 +++- .../openhft/chronicle/queue/TailerState.java | 18 ++++++ .../chronicle/queue/impl/ExcerptContext.java | 9 ++- .../queue/impl/RollingChronicleQueue.java | 1 + .../queue/impl/RollingResourcesCache.java | 9 +++ .../chronicle/queue/impl/WireStore.java | 1 + .../queue/impl/WireStoreFactory.java | 2 +- .../queue/impl/WireStoreSupplier.java | 1 + .../chronicle/queue/impl/package-info.java | 13 ++++ .../queue/impl/single/AsyncBufferCreator.java | 2 +- .../single/FileSystemDirectoryListing.java | 2 + .../queue/impl/single/MicroToucher.java | 1 + .../queue/impl/single/NoOpCondition.java | 3 +- .../queue/impl/single/NoOpWriteLock.java | 8 +++ .../queue/impl/single/PrecreatedFiles.java | 1 + .../queue/impl/single/Pretoucher.java | 1 - .../queue/impl/single/ReadOnlyWriteLock.java | 1 + .../impl/single/RollCycleEncodeSequence.java | 3 + .../queue/impl/single/SCQIndexing.java | 1 - .../impl/single/SingleChronicleQueue.java | 8 +-- .../single/SingleChronicleQueueBuilder.java | 10 ++- .../single/SingleChronicleQueueStore.java | 1 + .../queue/impl/single/StoreAppender.java | 7 ++- .../queue/impl/single/StoreTailer.java | 6 +- .../impl/single/TableDirectoryListing.java | 1 + .../impl/single/TableStoreWriteLock.java | 1 + .../queue/impl/single/WriteLock.java | 4 +- .../namedtailer/IndexUpdaterFactory.java | 2 - .../impl/single/namedtailer/package-info.java | 11 ++++ .../queue/impl/single/package-info.java | 12 ++++ .../queue/impl/table/AbstractTSQueueLock.java | 1 + .../queue/impl/table/SingleTableBuilder.java | 11 ++++ .../queue/impl/table/package-info.java | 11 ++++ .../queue/internal/domestic/package-info.java | 16 +++++ .../internal/main/InternalBenchmarkMain.java | 15 +---- .../queue/internal/main/package-info.java | 12 ++++ .../queue/internal/package-info.java | 8 +++ .../queue/internal/reader/package-info.java | 11 ++++ .../CustomPluginQueueEntryReader.java | 1 + .../VanillaQueueEntryReader.java | 1 + .../queueentryreaders/package-info.java | 12 ++++ .../queue/internal/util/InternalFileUtil.java | 7 ++- .../queue/internal/util/package-info.java | 11 ++++ .../queue/internal/writer/package-info.java | 11 ++++ .../chronicle/queue/main/BenchmarkMain.java | 4 ++ .../chronicle/queue/main/DumpMain.java | 4 ++ .../chronicle/queue/main/HistoryMain.java | 4 ++ .../chronicle/queue/main/PingPongMain.java | 4 ++ .../chronicle/queue/main/ReaderMain.java | 4 ++ .../chronicle/queue/main/RefreshMain.java | 12 ++++ .../main/RemovableRollFileCandidatesMain.java | 4 ++ .../chronicle/queue/main/UnlockMain.java | 4 ++ .../chronicle/queue/main/package-info.java | 7 +++ .../openhft/chronicle/queue/package-info.java | 16 +++++ .../queue/reader/ChronicleHistoryReader.java | 62 +++++++++++++++++++ .../queue/reader/ChronicleReader.java | 21 +++++++ .../chronicle/queue/reader/Reader.java | 1 + .../queue/reader/comparator/package-info.java | 11 ++++ .../chronicle/queue/reader/package-info.java | 12 ++++ .../queue/rollcycles/package-info.java | 12 ++++ .../chronicle/queue/util/FileState.java | 20 ++++-- .../chronicle/queue/util/PretouchUtil.java | 4 ++ .../chronicle/queue/util/package-info.java | 11 ++++ 72 files changed, 521 insertions(+), 47 deletions(-) create mode 100644 src/main/java/net/openhft/chronicle/queue/impl/single/namedtailer/package-info.java create mode 100644 src/main/java/net/openhft/chronicle/queue/impl/single/package-info.java create mode 100644 src/main/java/net/openhft/chronicle/queue/impl/table/package-info.java create mode 100644 src/main/java/net/openhft/chronicle/queue/internal/main/package-info.java create mode 100644 src/main/java/net/openhft/chronicle/queue/internal/reader/package-info.java create mode 100644 src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/package-info.java create mode 100644 src/main/java/net/openhft/chronicle/queue/internal/util/package-info.java create mode 100644 src/main/java/net/openhft/chronicle/queue/internal/writer/package-info.java create mode 100644 src/main/java/net/openhft/chronicle/queue/reader/comparator/package-info.java create mode 100644 src/main/java/net/openhft/chronicle/queue/reader/package-info.java create mode 100644 src/main/java/net/openhft/chronicle/queue/rollcycles/package-info.java create mode 100644 src/main/java/net/openhft/chronicle/queue/util/package-info.java diff --git a/src/main/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMain.java b/src/main/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMain.java index cb512b1e13..43271fca21 100644 --- a/src/main/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMain.java +++ b/src/main/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMain.java @@ -25,6 +25,12 @@ */ public class ChronicleHistoryReaderMain { + /** + * Creates a new instance for running via {@link #main(String[])}. + */ + public ChronicleHistoryReaderMain() { + } + /** * Entry point of the application. * Initializes the {@link ChronicleHistoryReaderMain} and passes command-line arguments. @@ -91,6 +97,7 @@ protected ChronicleHistoryReader chronicleHistoryReader() { * @param options Available command-line options * @return Parsed {@link CommandLine} object */ + // CPD-OFF - shared parsing logic with ChronicleReaderMain protected CommandLine parseCommandLine(@NotNull final String[] args, final Options options) { final CommandLineParser parser = new DefaultParser(); CommandLine commandLine = null; @@ -142,6 +149,7 @@ protected void printHelpAndExit(final Options options, int status, String messag writer.flush(); System.exit(status); } + // CPD-ON /** * Configures command-line options for the ChronicleHistoryReaderMain. diff --git a/src/main/java/net/openhft/chronicle/queue/ChronicleQueue.java b/src/main/java/net/openhft/chronicle/queue/ChronicleQueue.java index 95987fcfda..b4d1022d7b 100644 --- a/src/main/java/net/openhft/chronicle/queue/ChronicleQueue.java +++ b/src/main/java/net/openhft/chronicle/queue/ChronicleQueue.java @@ -171,6 +171,12 @@ default ExcerptTailer createTailer(String id) { throw new UnsupportedOperationException("not currently supported in this implementation."); } + /** + * Returns the index store linked to a named tailer. + * + * @param id identifier previously supplied to {@link #createTailer(String)} + * @return the value tracking the named tailer's position + */ default LongValue indexForId(String id) { throw new UnsupportedOperationException("Not supported"); } @@ -449,6 +455,7 @@ default void lastIndexMSynced(long lastIndexMSynced) { * @return a new String representation of this ChronicleQueue's last header in YAML-format */ @NotNull + @Deprecated(/* to be removed in 2027, only used in tests */) String dumpLastHeader(); /** @@ -461,6 +468,11 @@ default Pretoucher createPretoucher() { return PretouchUtil.createPretoucher(this); } + /** + * Creates an {@link EventHandler} that drives pretouching for this queue. + * + * @return event handler that advances pretouching work + */ default EventHandler createPretoucherEventHandler() { return PretouchUtil.createEventHandler(this); } diff --git a/src/main/java/net/openhft/chronicle/queue/ChronicleReaderMain.java b/src/main/java/net/openhft/chronicle/queue/ChronicleReaderMain.java index db9682a618..baabb91dcf 100644 --- a/src/main/java/net/openhft/chronicle/queue/ChronicleReaderMain.java +++ b/src/main/java/net/openhft/chronicle/queue/ChronicleReaderMain.java @@ -25,6 +25,12 @@ */ public class ChronicleReaderMain { + /** + * Creates a new runner for the command-line reader. + */ + public ChronicleReaderMain() { + } + /** * Entry point of the application. Initializes the {@link ChronicleReaderMain} instance and * passes command-line arguments for execution. diff --git a/src/main/java/net/openhft/chronicle/queue/ChronicleWriterMain.java b/src/main/java/net/openhft/chronicle/queue/ChronicleWriterMain.java index 891c9a8285..4b12e49bb3 100644 --- a/src/main/java/net/openhft/chronicle/queue/ChronicleWriterMain.java +++ b/src/main/java/net/openhft/chronicle/queue/ChronicleWriterMain.java @@ -11,6 +11,10 @@ */ public class ChronicleWriterMain { + private ChronicleWriterMain() { + // utility holder + } + /** * Main method for executing the ChronicleWriterMain. * It delegates to the internal writer implementation to run the application with the given arguments. diff --git a/src/main/java/net/openhft/chronicle/queue/ExcerptAppender.java b/src/main/java/net/openhft/chronicle/queue/ExcerptAppender.java index b4523321c0..90dbf47c6d 100644 --- a/src/main/java/net/openhft/chronicle/queue/ExcerptAppender.java +++ b/src/main/java/net/openhft/chronicle/queue/ExcerptAppender.java @@ -125,6 +125,7 @@ default VanillaMethodWriterBuilder methodWriterBuilder(@NotNull Class * @return a raw wire for low level direct access */ @Nullable + @Deprecated(/* to be removed in 2027, only used in tests */) Wire wire(); /** diff --git a/src/main/java/net/openhft/chronicle/queue/ExcerptCommon.java b/src/main/java/net/openhft/chronicle/queue/ExcerptCommon.java index 48c9e4f8c3..fac3f1cc47 100644 --- a/src/main/java/net/openhft/chronicle/queue/ExcerptCommon.java +++ b/src/main/java/net/openhft/chronicle/queue/ExcerptCommon.java @@ -13,6 +13,8 @@ /** * The ExcerptCommon is common to both ExcerptAppender * and ExcerptTailer. + * + * @param the concrete Excerpt type returned for fluent chaining */ public interface ExcerptCommon> extends Closeable, SingleThreadedChecked { @@ -36,6 +38,8 @@ public interface ExcerptCommon> extends Closeable, Si ChronicleQueue queue(); /** + * Returns the current file being worked on, or {@code null} if the tailer/appender has not loaded a file yet. + * * @return the current file being worked on or null if not known. */ @Nullable @@ -44,7 +48,7 @@ default File currentFile() { } /** - * Performa sync up to the point the Appender has written or Tailer has read, if supported. + * Performs a sync up to the point the Appender has written or Tailer has read, if supported. */ default void sync() { } diff --git a/src/main/java/net/openhft/chronicle/queue/ExcerptTailer.java b/src/main/java/net/openhft/chronicle/queue/ExcerptTailer.java index 574053ee8f..ddfe9e1a86 100644 --- a/src/main/java/net/openhft/chronicle/queue/ExcerptTailer.java +++ b/src/main/java/net/openhft/chronicle/queue/ExcerptTailer.java @@ -16,6 +16,7 @@ *

NOTE: Tailers are NOT thread-safe, sharing a Tailer between threads will lead to errors and unpredictable behaviour. */ @SingleThreaded +@SuppressWarnings({"deprecation", "removal"}) public interface ExcerptTailer extends ExcerptCommon, MarshallableIn, SourceContext { /** @@ -61,6 +62,11 @@ default DocumentContext readingDocument() { @Override long index(); + /** + * Returns the last index read by this tailer, or {@code -1} if no read has occurred. + * + * @return last index read or {@code -1} when unset + */ default long lastReadIndex() { return -1; } @@ -251,8 +257,13 @@ default long excerptsInCycle(int cycle) { @NotNull TailerState state(); + /** + * Callback for deciding whether a message can be read based on replication acknowledgements. + */ interface AcknowledgedIndexReplicatedCheck { /** + * Determines whether the tailer can read the provided index given the last replicated sequence. + * * @param index the index of the next message for the tailer to read * @param lastSequenceAck the last index that has been acknowledged as replicated * @return true if you are ok for the tailer to read the message at {@code index} diff --git a/src/main/java/net/openhft/chronicle/queue/NoMessageHistory.java b/src/main/java/net/openhft/chronicle/queue/NoMessageHistory.java index 32b2443a21..d60ed6050b 100644 --- a/src/main/java/net/openhft/chronicle/queue/NoMessageHistory.java +++ b/src/main/java/net/openhft/chronicle/queue/NoMessageHistory.java @@ -5,8 +5,18 @@ import net.openhft.chronicle.wire.*; -// Is this being used? +/** + * {@link MessageHistory} implementation that records no timing or source information. + *

+ * This singleton can be used in code paths where message history is either disabled for + * performance reasons or not relevant to the use case. All accessors return sentinel values + * and {@link #reset(int, long)} is a no-op. + */ +@Deprecated(/* to be removed in 2027, only used in tests */) public enum NoMessageHistory implements MessageHistory { + /** + * Singleton instance that records no message history. + */ INSTANCE; @Override @@ -44,6 +54,7 @@ public void reset(int sourceId, long sourceIndex) { // ignored } + @Override public void reset() { // no-op } diff --git a/src/main/java/net/openhft/chronicle/queue/RollCycles.java b/src/main/java/net/openhft/chronicle/queue/RollCycles.java index fdc462bad7..881a25ca28 100644 --- a/src/main/java/net/openhft/chronicle/queue/RollCycles.java +++ b/src/main/java/net/openhft/chronicle/queue/RollCycles.java @@ -56,6 +56,9 @@ public enum RollCycles implements RollCycle { * 0xffffffff entries per week, indexing every 256th entry, leave as 4K and 256 for historical reasons. Cycle starts Sunday 00:00 */ WEEKLY(/*-----------*/"YYYY'W'ww", 7 * 24 * 60 * 60 * 1000, 4 << 10, 256, RollCycleArithmetic.SUNDAY_00_00); + /** + * Default roll cycle used when none is specified explicitly. + */ public static final RollCycles DEFAULT = FAST_DAILY; // don't alter this or you will confuse yourself. @@ -90,10 +93,16 @@ public enum RollCycles implements RollCycle { maxMessagesPerCycle = arithmetic.maxMessagesPerCycle(); } + /** + * Returns all roll cycles, including legacy and sparse variants. + * + * @return iterable of all known roll cycles + */ public static Iterable all() { return VALUES; } + @Override public long maxMessagesPerCycle() { return maxMessagesPerCycle; } diff --git a/src/main/java/net/openhft/chronicle/queue/TailerDirection.java b/src/main/java/net/openhft/chronicle/queue/TailerDirection.java index 7a4e60cfb6..14808541f3 100644 --- a/src/main/java/net/openhft/chronicle/queue/TailerDirection.java +++ b/src/main/java/net/openhft/chronicle/queue/TailerDirection.java @@ -12,8 +12,17 @@ * */ public enum TailerDirection { - NONE(0), // don't move after a read. - FORWARD(+1), // move to the next entry + /** + * Do not move after a read. + */ + NONE(0), + /** + * Move to the next entry after a read. + */ + FORWARD(+1), + /** + * Move to the previous entry after a read. + */ BACKWARD(-1); private final int add; diff --git a/src/main/java/net/openhft/chronicle/queue/TailerState.java b/src/main/java/net/openhft/chronicle/queue/TailerState.java index 411676f8fe..edebf46972 100644 --- a/src/main/java/net/openhft/chronicle/queue/TailerState.java +++ b/src/main/java/net/openhft/chronicle/queue/TailerState.java @@ -15,10 +15,28 @@ * */ public enum TailerState { + /** + * The tailer has reached the end of the current cycle. + */ END_OF_CYCLE, + /** + * An entry was found in the current cycle. + */ FOUND_IN_CYCLE, + /** + * The tailer has moved beyond the start of the cycle. + */ BEYOND_START_OF_CYCLE, + /** + * The requested cycle could not be found. + */ CYCLE_NOT_FOUND, + /** + * The tailer has not yet reached an entry in the cycle. + */ NOT_REACHED_IN_CYCLE, + /** + * The tailer has not been initialised yet. + */ UNINITIALISED } diff --git a/src/main/java/net/openhft/chronicle/queue/impl/ExcerptContext.java b/src/main/java/net/openhft/chronicle/queue/impl/ExcerptContext.java index 7e7f6a3686..4cc8970fe0 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/ExcerptContext.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/ExcerptContext.java @@ -6,6 +6,13 @@ import net.openhft.chronicle.wire.Wire; import org.jetbrains.annotations.Nullable; +/** + * View over the wire and timeout parameters associated with a queue excerpt operation. + *

+ * Implementations expose the current {@link Wire} used for reading or writing, an optional + * index-aware wire for random access operations, and the timeout in milliseconds that guards + * blocking calls. Instances are generally short lived and not intended to be shared. + */ public interface ExcerptContext { @Nullable Wire wire(); @@ -13,6 +20,6 @@ public interface ExcerptContext { @Nullable Wire wireForIndex(); + @Deprecated(/* to be removed in 2027, use queue.timeoutMS() */) long timeoutMS(); - } diff --git a/src/main/java/net/openhft/chronicle/queue/impl/RollingChronicleQueue.java b/src/main/java/net/openhft/chronicle/queue/impl/RollingChronicleQueue.java index c3ed68e7a1..4979189568 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/RollingChronicleQueue.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/RollingChronicleQueue.java @@ -123,5 +123,6 @@ public interface RollingChronicleQueue extends ChronicleQueue { * @return the roll cycle used by the queue. */ @NotNull + @Override RollCycle rollCycle(); } diff --git a/src/main/java/net/openhft/chronicle/queue/impl/RollingResourcesCache.java b/src/main/java/net/openhft/chronicle/queue/impl/RollingResourcesCache.java index 3dd73a82bc..dab8c4b25f 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/RollingResourcesCache.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/RollingResourcesCache.java @@ -21,6 +21,14 @@ import java.util.concurrent.ConcurrentMap; import java.util.function.Function; +/** + * Cache of roll cycle resources keyed by cycle number and file name. + *

+ * {@code RollingResourcesCache} maps roll cycle indices to formatted date-based file names + * and {@link File} instances, and provides the inverse mapping from file to cycle number. + * It keeps a small, size-bounded cache to avoid repeated string formatting and date parsing + * when rotating queue files, and normalises timestamps against a configured epoch. + */ public class RollingResourcesCache { public static final ParseCount NO_PARSE_COUNT = new ParseCount("", Integer.MIN_VALUE); private static final int CACHE_SIZE = Jvm.getInteger("chronicle.queue.rollingResourceCache.size", 128); @@ -162,6 +170,7 @@ public static final class Resource { public final long millis; public final String text; public final File path; + @Deprecated(/* to be removed in 2027 */) public final File parentPath; public boolean pathExists; diff --git a/src/main/java/net/openhft/chronicle/queue/impl/WireStore.java b/src/main/java/net/openhft/chronicle/queue/impl/WireStore.java index 0f3182cead..c0d12dd6ec 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/WireStore.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/WireStore.java @@ -92,6 +92,7 @@ public interface WireStore extends CommonStore, Closeable { * @param index the index to check * @return {@code true} if the index is indexable, {@code false} otherwise */ + @Deprecated(/* to be removed in 2027 */) boolean indexable(long index); /** diff --git a/src/main/java/net/openhft/chronicle/queue/impl/WireStoreFactory.java b/src/main/java/net/openhft/chronicle/queue/impl/WireStoreFactory.java index 9b16b2c57f..e3badaa7ec 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/WireStoreFactory.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/WireStoreFactory.java @@ -12,7 +12,7 @@ * The {@code WireStoreFactory} interface is a functional interface that creates instances of * {@link SingleChronicleQueueStore}. It functions as a factory to produce queue stores, * utilizing a rolling chronicle queue and a wire to initialize each store. - * + *

* This interface extends {@link BiFunction}, meaning it takes two arguments — a * {@link RollingChronicleQueue} and a {@link Wire} — and returns a {@link SingleChronicleQueueStore}. */ diff --git a/src/main/java/net/openhft/chronicle/queue/impl/WireStoreSupplier.java b/src/main/java/net/openhft/chronicle/queue/impl/WireStoreSupplier.java index d4a2cb8ca0..3694b68795 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/WireStoreSupplier.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/WireStoreSupplier.java @@ -68,5 +68,6 @@ enum CreateStrategy { * @param store the {@link SingleChronicleQueueStore} to check for reuse * @return {@code true} if the store can be reused, {@code false} otherwise */ + @Deprecated(/* to be removed in 2027 */) boolean canBeReused(@NotNull SingleChronicleQueueStore store); } diff --git a/src/main/java/net/openhft/chronicle/queue/impl/package-info.java b/src/main/java/net/openhft/chronicle/queue/impl/package-info.java index 2925ecf904..2506f21bf4 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/package-info.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/package-info.java @@ -1,4 +1,17 @@ /* * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 */ +/** + * Internal Chronicle Queue implementation details. + *

+ * Types in this package implement the storage, roll-cycle and indexing + * mechanics underlying the public {@code ChronicleQueue} API. They + * include concrete queue implementations, store management and + * file-handling utilities. + *

+ * This package is not part of the supported public API surface. + * Classes may change, move or be removed between releases; applications + * should prefer the higher-level types in {@code net.openhft.chronicle.queue} + * unless they have a compelling reason to rely on these internals. + */ package net.openhft.chronicle.queue.impl; diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/AsyncBufferCreator.java b/src/main/java/net/openhft/chronicle/queue/impl/single/AsyncBufferCreator.java index d3506b9741..74c877f00c 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/AsyncBufferCreator.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/AsyncBufferCreator.java @@ -12,7 +12,7 @@ /** * The {@code AsyncBufferCreator} interface defines a contract for creating buffers * for use in asynchronous mode. This functionality is available as an enterprise feature. - * + *

* It extends {@link ThrowingBiFunction}, allowing the creation of {@link BytesStore} instances * with a given size, maximum readers, and file backing. */ diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/FileSystemDirectoryListing.java b/src/main/java/net/openhft/chronicle/queue/impl/single/FileSystemDirectoryListing.java index 9de3952703..62dcac9b68 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/FileSystemDirectoryListing.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/FileSystemDirectoryListing.java @@ -62,6 +62,7 @@ public void onFileCreated(final File file, final int cycle) { * @param force If true, forces a refresh. */ @Override + // CPD-OFF - logic mirrors TableDirectoryListing public void refresh(boolean force) { lastRefreshTimeMS = time.currentTimeMillis(); @@ -92,6 +93,7 @@ public void refresh(boolean force) { minCreatedCycle = min; maxCreatedCycle = max; } + // CPD-ON /** * Returns the timestamp of the last refresh in milliseconds since the epoch. diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/MicroToucher.java b/src/main/java/net/openhft/chronicle/queue/impl/single/MicroToucher.java index 608db1a680..8989c33973 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/MicroToucher.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/MicroToucher.java @@ -55,6 +55,7 @@ public boolean execute() { if (bs.inside(nextPage, 8)) touchPage(nextPage, bs); } catch (Throwable ignored) { + // ignored - best effort touching } return true; } diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/NoOpCondition.java b/src/main/java/net/openhft/chronicle/queue/impl/single/NoOpCondition.java index 0bc1ce5b86..604635e493 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/NoOpCondition.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/NoOpCondition.java @@ -60,10 +60,9 @@ public long awaitNanos(long nanosTimeout) { * @param l The maximum time to wait. * @param timeUnit The time unit of the {@code l} argument. * @return Always returns {@code true}. - * @throws InterruptedException This method does not throw an exception. */ @Override - public boolean await(long l, TimeUnit timeUnit) throws InterruptedException { + public boolean await(long l, TimeUnit timeUnit) { return true; } diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/NoOpWriteLock.java b/src/main/java/net/openhft/chronicle/queue/impl/single/NoOpWriteLock.java index 0dca6eb02c..dc89469d59 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/NoOpWriteLock.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/NoOpWriteLock.java @@ -7,6 +7,14 @@ import java.util.function.LongConsumer; +/** + * {@link WriteLock} implementation that performs no locking and reports success. + *

+ * This is used when a queue instance is known to be single threaded or when locking is + * enforced by some other mechanism. All methods are effectively no-ops but return values + * are chosen so that callers treat the lock as always acquired by the current process. + */ +@SuppressWarnings({"deprecation", "removal"}) public class NoOpWriteLock implements WriteLock, IgnoresEverything { @Override diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/PrecreatedFiles.java b/src/main/java/net/openhft/chronicle/queue/impl/single/PrecreatedFiles.java index fd4c4b0bb9..62e78acc4d 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/PrecreatedFiles.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/PrecreatedFiles.java @@ -40,6 +40,7 @@ public static void renamePreCreatedFileToRequiredFile(final File requiredQueueFi * @param requiredStoreFile The file for which a pre-created store file is required. * @return The pre-created store file object. */ + @Deprecated(/* to be removed in 2027 */) public static File preCreatedFileForStoreFile(final File requiredStoreFile) { return new File(requiredStoreFile.getParentFile(), requiredStoreFile.getName() + PRE_CREATED_FILE_SUFFIX); diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/Pretoucher.java b/src/main/java/net/openhft/chronicle/queue/impl/single/Pretoucher.java index 1b422f60e5..e7cdbd64e0 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/Pretoucher.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/Pretoucher.java @@ -29,7 +29,6 @@ public interface Pretoucher extends Closeable { *

* If the underlying queue has been closed, this method will throw an {@link InvalidEventHandlerException}. * - * * @throws InvalidEventHandlerException if the queue has been closed or if there is an issue during the pre-touch operation. */ void execute() throws InvalidEventHandlerException; diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/ReadOnlyWriteLock.java b/src/main/java/net/openhft/chronicle/queue/impl/single/ReadOnlyWriteLock.java index 7f830668cc..12d8e49861 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/ReadOnlyWriteLock.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/ReadOnlyWriteLock.java @@ -10,6 +10,7 @@ * This lock throws {@link IllegalStateException} when any write-related operation is attempted, * indicating that the queue is in a read-only state. */ +@SuppressWarnings({"deprecation", "removal"}) public class ReadOnlyWriteLock implements WriteLock { /** diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/RollCycleEncodeSequence.java b/src/main/java/net/openhft/chronicle/queue/impl/single/RollCycleEncodeSequence.java index aa5da8060a..af7af25a62 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/RollCycleEncodeSequence.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/RollCycleEncodeSequence.java @@ -12,6 +12,7 @@ * This class encodes and manages the sequence in a Chronicle Queue based on roll cycles. * It is responsible for encoding the sequence and position, and handling sequence retrieval. */ +@SuppressWarnings({"deprecation", "removal"}) class RollCycleEncodeSequence implements Sequence { private final TwoLongValue writePositionAndSequence; private final int cycleShift; @@ -69,6 +70,7 @@ public long toIndex(long headerNumber, long sequence) { * @param forWritePosition The write position, expected to be the end of the queue. * @return The sequence number or {@link Sequence#NOT_FOUND_RETRY} if the sequence is not found. */ + @Override public long getSequence(long forWritePosition) { if (writePositionAndSequence == null) @@ -112,6 +114,7 @@ private long toLongValue(long cycle, long sequenceNumber) { * @param index The encoded index. * @return The sequence number. */ + @Override public long toSequenceNumber(long index) { return index & sequenceMask; } diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/SCQIndexing.java b/src/main/java/net/openhft/chronicle/queue/impl/single/SCQIndexing.java index c28765e631..18d0f0d7e3 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/SCQIndexing.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/SCQIndexing.java @@ -417,7 +417,6 @@ private ScanResult scanSecondaryIndexBackwards(@NotNull final ExcerptContext ec, * @param knownAddress a know addressForRead ( used as a starting point ) * @see SCQIndexing#moveToIndex */ - @NotNull private ScanResult linearScan(@NotNull final Wire wire, final long toIndex, diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueue.java b/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueue.java index 78aa53e386..87249d8e83 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueue.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueue.java @@ -60,12 +60,13 @@ * It also supports various configurations such as event loop handling, wire types, buffer * management, and replication. */ -@SuppressWarnings("this-escape") +@SuppressWarnings({"this-escape", "deprecation", "removal"}) public class SingleChronicleQueue extends AbstractCloseable implements RollingChronicleQueue { public static final String SUFFIX = ".cq4"; public static final String DISCARD_FILE_SUFFIX = ".discard"; public static final String QUEUE_METADATA_FILE = "metadata" + SingleTableStore.SUFFIX; + @Deprecated(/* to be removed in 2027 */) public static final String DISK_SPACE_CHECKER_NAME = DiskSpaceMonitor.DISK_SPACE_CHECKER_NAME; public static final String REPLICATED_NAMED_TAILER_PREFIX = "replicated:"; public static final String INDEX_LOCK_FORMAT = "index.%s.lock"; @@ -1231,6 +1232,7 @@ public String toString() { * @return the TimeProvider */ @NotNull + @Override public TimeProvider time() { return time; } @@ -1388,13 +1390,11 @@ private StoreSupplier() { /** * Acquires a {@link SingleChronicleQueueStore} for the specified cycle. - * If the store doesn't exist and the strategy is {@link CreateStrategy.CREATE}, it will create a new store. + * If the store doesn't exist and the strategy is {@link CreateStrategy#CREATE}, it will create a new store. * * @param cycle the cycle to acquire the store for * @param createStrategy the strategy for creating or reading the store * @return the acquired SingleChronicleQueueStore or null if the store doesn't exist and the strategy is not CREATE - * @throws IOException in case of IO errors - * @throws TimeoutException if acquiring the store times out */ @Override public SingleChronicleQueueStore acquire(int cycle, CreateStrategy createStrategy) { diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueBuilder.java b/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueBuilder.java index 4b38277198..f73717a245 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueBuilder.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueBuilder.java @@ -65,6 +65,7 @@ public class SingleChronicleQueueBuilder extends SelfDescribingMarshallable impl Jvm.getSize("SingleChronicleQueueBuilder.blocksize", OS.is64Bit() ? 64L << 20 : SMALL_BLOCK_SIZE), OS.is64Bit() && OS.isLinux() ? Long.MAX_VALUE : 256L << 20); // 256MB on 32-bit or non-Linux + @Deprecated(/* to be removed in 2027 */) public static final long DEFAULT_SPARSE_CAPACITY = 512L << 30; private static final Constructor ENTERPRISE_QUEUE_CONSTRUCTOR; private static final WireStoreFactory storeFactory = SingleChronicleQueueBuilder::createStore; @@ -143,12 +144,8 @@ public class SingleChronicleQueueBuilder extends SelfDescribingMarshallable impl protected SingleChronicleQueueBuilder() { } - /* - * ======================== - * Builders - * ======================== - */ + @SuppressWarnings("EmptyMethod") public static void addAliases() { // static initialiser. } @@ -369,6 +366,7 @@ public WireStoreFactory storeFactory() { * @throws IllegalStateException if enterprise features are requested but not available */ @NotNull + @Override public SingleChronicleQueue build() { preBuild(); @@ -1546,6 +1544,7 @@ public SingleChronicleQueueBuilder forceDirectoryListingRefreshIntervalMs(long f * It's only a shallow copy so field will have the same objects. */ @SuppressWarnings("java:S2975") + @Override public SingleChronicleQueueBuilder clone() { try { return (SingleChronicleQueueBuilder) super.clone(); @@ -1562,7 +1561,6 @@ public SingleChronicleQueueBuilder clone() { * @return the current builder instance for method chaining * @throws IllegalArgumentException if the builders are not from the same class hierarchy */ - public SingleChronicleQueueBuilder setAllNullFields(@Nullable SingleChronicleQueueBuilder parentBuilder) { if (parentBuilder == null) return this; diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueStore.java b/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueStore.java index e8c123f345..e87b0dccdf 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueStore.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueStore.java @@ -31,6 +31,7 @@ * the file to memory using {@link MappedBytes}. It handles reading and writing data in * a queue and supports efficient roll cycles and indexing. */ +@SuppressWarnings({"deprecation", "removal"}) public class SingleChronicleQueueStore extends AbstractCloseable implements WireStore { static { diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/StoreAppender.java b/src/main/java/net/openhft/chronicle/queue/impl/single/StoreAppender.java index e05be62a76..ff06bc7091 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/StoreAppender.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/StoreAppender.java @@ -40,6 +40,7 @@ * excerpts to the queue. It manages the cycle of the queue, lock handling, and the state * of the wire and store. */ +@SuppressWarnings({"deprecation", "removal"}) class StoreAppender extends AbstractCloseable implements ExcerptAppender, ExcerptContext, InternalAppender, MicroTouched { @@ -58,6 +59,7 @@ class StoreAppender extends AbstractCloseable private final StoreAppenderContext context; private final WireStorePool storePool; private final boolean checkInterrupts; + @SuppressWarnings({"FieldCanBeLocal", "unused"}) @UsedViaReflection private final Finalizer finalizer; @Nullable @@ -355,6 +357,7 @@ public boolean recordHistory() { * * @param cycle The cycle to be set. */ + @Deprecated(/* to be removed in 2027 */) void setCycle(int cycle) { if (cycle != this.cycle) setCycle2(cycle, WireStoreSupplier.CreateStrategy.CREATE); @@ -598,6 +601,7 @@ public DocumentContext acquireWritingDocument(boolean metaData) { * Ensures that EOF markers are properly added to all cycles, normalizing older cycles to ensure they are complete. * This method locks the writeLock and calls the internal {@link #normaliseEOFs0(int)} method for each cycle. */ + @Override public void normaliseEOFs() { long start = System.nanoTime(); final WriteLock writeLock = queue.writeLock(); @@ -939,7 +943,7 @@ public SingleChronicleQueue queue() { /* * overridden in delta wire */ - @SuppressWarnings("unused") + @SuppressWarnings({"unused", "EmptyMethod"}) void beforeAppend(final Wire wire, final long index) { } @@ -1148,6 +1152,7 @@ final class StoreAppenderContext implements WriteDocumentContext { * * @return true if the context is empty, false otherwise */ + @Override public boolean isEmpty() { Bytes bytes = wire().bytes(); return bytes.readRemaining() == 0; diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/StoreTailer.java b/src/main/java/net/openhft/chronicle/queue/impl/single/StoreTailer.java index b18903b1cd..4fddcaca2e 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/StoreTailer.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/StoreTailer.java @@ -40,6 +40,7 @@ * the queue in both forward and backward directions, ensuring efficient retrieval * of entries by utilizing the appropriate WireStore and cycle mechanism. */ +@SuppressWarnings({"deprecation", "removal"}) class StoreTailer extends AbstractCloseable implements ExcerptTailer, SourceContext, ExcerptContext { static final int INDEXING_LINEAR_SCAN_THRESHOLD = 70; @@ -51,6 +52,7 @@ class StoreTailer extends AbstractCloseable private final IndexUpdater indexUpdater; private final StoreTailerContext context = new StoreTailerContext(); private final MoveToState moveToState = new MoveToState(); + @SuppressWarnings({"FieldCanBeLocal", "unused"}) private final Finalizer finalizer; long index; // index of the next read. long lastReadIndex; // index of the last read message @@ -724,7 +726,6 @@ private long nextIndexWithNextAvailableCycle0(final int cycle) { * that did not exist, that is what this is reporting. If you are using daily rolling, * and writing every day, you should not see this message. */ - Jvm.debug().on(getClass(), "Rolled " + (nextIndexCycle - cycle) + " " + "times to find the " + "next cycle file. This can occur if your appenders have not written " + "anything for a while, leaving the cycle files with a gap."); @@ -1307,6 +1308,7 @@ public ExcerptTailer originalToEnd() { break; case FOUND: + //noinspection TextLabelInSwitchStatement LoopForward: // NOSONAR while (originalToEndLoopCondition(approximateLastIndex, index)) { final ScanResult result = moveToIndexResult(++index); @@ -1640,7 +1642,7 @@ int getIndexMoveCount() { @NotNull private SingleChronicleQueueStore store() { if (store == null) - if (!cycle(cycle())) + if (!cycle(cycle())) // sets the store as a side effect Jvm.warn().on(getClass(), "Unable to find cycle=" + cycle() + ", queue=" + queue.fileAbsolutePath()); return store; } diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/TableDirectoryListing.java b/src/main/java/net/openhft/chronicle/queue/impl/single/TableDirectoryListing.java index 3e04b1695c..c09e6765e6 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/TableDirectoryListing.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/TableDirectoryListing.java @@ -234,6 +234,7 @@ public String toString() { /** * Closes the directory listing by releasing resources associated with the LongValues. */ + @Override protected void performClose() { Closeable.closeQuietly(minCycleValue, maxCycleValue, modCount); } diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/TableStoreWriteLock.java b/src/main/java/net/openhft/chronicle/queue/impl/single/TableStoreWriteLock.java index dbc4bdf5ad..1bae1518d3 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/TableStoreWriteLock.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/TableStoreWriteLock.java @@ -29,6 +29,7 @@ * The write lock is used to protect write operations in Chronicle Queue, ensuring that only one thread or process * can write at a time. */ +@SuppressWarnings({"deprecation", "removal"}) public class TableStoreWriteLock extends AbstractTSQueueLock implements WriteLock, Closeable { private static final String STORE_LOCK_THREAD = "chronicle.store.lock.thread"; private static final boolean storeLockThread = Jvm.getBoolean(STORE_LOCK_THREAD); diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/WriteLock.java b/src/main/java/net/openhft/chronicle/queue/impl/single/WriteLock.java index 6b0f5b7baa..7acd1e9a64 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/WriteLock.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/WriteLock.java @@ -10,7 +10,7 @@ * The WriteLock interface provides methods to control locking mechanisms in a Chronicle Queue. * It defines locking, unlocking, and checking mechanisms, ensuring exclusive access to resources * while preventing race conditions. - * + *

* This interface is non-reentrant, meaning that once a lock is acquired, it cannot be reacquired by * the same process until it is explicitly released. */ @@ -41,6 +41,7 @@ public interface WriteLock extends Closeable { /** * Closes the lock resource. This method is invoked when the lock is no longer needed. */ + @Override void close(); /** @@ -86,5 +87,6 @@ default boolean locked() { * @param notCurrentProcessConsumer a {@link LongConsumer} that is invoked when the lock is held by another process. * @return {@code true} if the current process holds the lock, otherwise {@code false}. */ + @Deprecated(/* to be removed in 2027 */) boolean isLockedByCurrentProcess(LongConsumer notCurrentProcessConsumer); } diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/namedtailer/IndexUpdaterFactory.java b/src/main/java/net/openhft/chronicle/queue/impl/single/namedtailer/IndexUpdaterFactory.java index 37f42af1a0..88ccc78b81 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/single/namedtailer/IndexUpdaterFactory.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/namedtailer/IndexUpdaterFactory.java @@ -70,8 +70,6 @@ public StandardIndexUpdater(@NotNull LongValue indexValue) { /** * Closes this {@code StandardIndexUpdater}, releasing any resources held. - * - * @throws IOException if an I/O error occurs */ @Override public void close() throws IOException { diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/namedtailer/package-info.java b/src/main/java/net/openhft/chronicle/queue/impl/single/namedtailer/package-info.java new file mode 100644 index 0000000000..de6a63a263 --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/namedtailer/package-info.java @@ -0,0 +1,11 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * Named tailer support for single Chronicle Queues. + *

+ * Classes in this package provide the persistence and management logic + * behind named tailers, allowing readers to resume from their last + * position using an identifier. + */ +package net.openhft.chronicle.queue.impl.single.namedtailer; diff --git a/src/main/java/net/openhft/chronicle/queue/impl/single/package-info.java b/src/main/java/net/openhft/chronicle/queue/impl/single/package-info.java new file mode 100644 index 0000000000..f8af8dbe45 --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/impl/single/package-info.java @@ -0,0 +1,12 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * Single-chronicle queue implementation. + *

+ * This package contains the primary file-backed queue implementation + * used by {@code ChronicleQueue.singleBuilder}, together with its + * appender, tailer and store types. It is internal to the queue module + * and may evolve between releases. + */ +package net.openhft.chronicle.queue.impl.single; diff --git a/src/main/java/net/openhft/chronicle/queue/impl/table/AbstractTSQueueLock.java b/src/main/java/net/openhft/chronicle/queue/impl/table/AbstractTSQueueLock.java index 5c67553088..643755b340 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/table/AbstractTSQueueLock.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/table/AbstractTSQueueLock.java @@ -68,6 +68,7 @@ public AbstractTSQueueLock(final String lockKey, final TableStore tableStore, /** * Performs cleanup and releases resources when the lock is closed. */ + @Override protected void performClose() { Closeable.closeQuietly(lock); } diff --git a/src/main/java/net/openhft/chronicle/queue/impl/table/SingleTableBuilder.java b/src/main/java/net/openhft/chronicle/queue/impl/table/SingleTableBuilder.java index eff5095726..01ed10c5c6 100644 --- a/src/main/java/net/openhft/chronicle/queue/impl/table/SingleTableBuilder.java +++ b/src/main/java/net/openhft/chronicle/queue/impl/table/SingleTableBuilder.java @@ -30,6 +30,16 @@ import static net.openhft.chronicle.core.pool.ClassAliasPool.CLASS_ALIASES; +/** + * Builder that creates or opens a single Chronicle table store file. + *

+ * {@code SingleTableBuilder} handles mapping the underlying metadata file, acquiring the + * appropriate shared or exclusive file lock, and either writing a new {@link SingleTableStore} + * header or reading an existing one. It can operate in read-only mode, where it waits for the + * backing file to reach a minimum size before attempting to read the header. + * + * @param metadata type stored alongside the table + */ public class SingleTableBuilder implements Builder> { static { @@ -78,6 +88,7 @@ public static SingleTableBuilder binary(@NotNull File ba } @NotNull + @Override public TableStore build() { if (readOnly) { if (!file.exists()) diff --git a/src/main/java/net/openhft/chronicle/queue/impl/table/package-info.java b/src/main/java/net/openhft/chronicle/queue/impl/table/package-info.java new file mode 100644 index 0000000000..a2388f7796 --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/impl/table/package-info.java @@ -0,0 +1,11 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * Table store implementation used by Chronicle Queue. + *

+ * This package contains table-based metadata stores and related locking + * infrastructure, used to track queue configuration and coordination + * data on disk. + */ +package net.openhft.chronicle.queue.impl.table; diff --git a/src/main/java/net/openhft/chronicle/queue/internal/domestic/package-info.java b/src/main/java/net/openhft/chronicle/queue/internal/domestic/package-info.java index f3db5db2e2..fbaff8eaf0 100644 --- a/src/main/java/net/openhft/chronicle/queue/internal/domestic/package-info.java +++ b/src/main/java/net/openhft/chronicle/queue/internal/domestic/package-info.java @@ -1,4 +1,20 @@ /* * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 */ +/** + * This package and any and all sub-packages contains strictly internal classes for this Chronicle + * library. + * Internal classes shall never be used directly. + *

+ * Specifically, the following actions (including, but not limited to) are not allowed + * on internal classes and packages: + *

    + *
  • Casting to
    • + *
  • Reflection of any kind
    • + *
  • Explicit serialise/deserialize
    • + *
+ *

+ * The classes in this package and any sub-package are subject to changes at any time for any + * reason. + */ package net.openhft.chronicle.queue.internal.domestic; diff --git a/src/main/java/net/openhft/chronicle/queue/internal/main/InternalBenchmarkMain.java b/src/main/java/net/openhft/chronicle/queue/internal/main/InternalBenchmarkMain.java index 0a7d2a5a7c..270ecf675a 100644 --- a/src/main/java/net/openhft/chronicle/queue/internal/main/InternalBenchmarkMain.java +++ b/src/main/java/net/openhft/chronicle/queue/internal/main/InternalBenchmarkMain.java @@ -94,24 +94,17 @@ static void benchmark(int messageSize) { Histogram loopTime = new Histogram(); Thread reader = new Thread(() -> { -// try (ChronicleQueue queue2 = createQueue(path)) ExcerptTailer tailer = queue.createTailer().toEnd(); long endLoop = System.nanoTime(); while (running) { loopTime.sample((double) (System.nanoTime() - endLoop)); Jvm.safepoint(); -// readerLoopTime = System.nanoTime(); -// if (readerLoopTime - readerEndLoopTime > 1000) -// System.out.println("r " + (readerLoopTime - readerEndLoopTime)); -// try { runInner(transportTime, readTime, tailer); runInner(transportTime, readTime, tailer); runInner(transportTime, readTime, tailer); runInner(transportTime, readTime, tailer); -// } finally { -// readerEndLoopTime = System.nanoTime(); -// } + Jvm.safepoint(); endLoop = System.nanoTime(); } @@ -177,12 +170,6 @@ static void benchmark(int messageSize) { */ private static void runInner(Histogram transportTime, Histogram readTime, ExcerptTailer tailer) { Jvm.safepoint(); - /*if (tailer.peekDocument()) { - if (counter++ < 1000) { - Jvm.safepoint(); - return; - } - }*/ Jvm.safepoint(); counter = 0; try (DocumentContext dc = tailer.readingDocument(false)) { diff --git a/src/main/java/net/openhft/chronicle/queue/internal/main/package-info.java b/src/main/java/net/openhft/chronicle/queue/internal/main/package-info.java new file mode 100644 index 0000000000..65f08c8173 --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/internal/main/package-info.java @@ -0,0 +1,12 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * Internal command-line tools for Chronicle Queue. + *

+ * This package contains maintenance and diagnostic entry points + * intended for Chronicle engineers, for example utilities to inspect, + * repair or benchmark queues. The classes are not part of the public + * API surface and may change without notice. + */ +package net.openhft.chronicle.queue.internal.main; diff --git a/src/main/java/net/openhft/chronicle/queue/internal/package-info.java b/src/main/java/net/openhft/chronicle/queue/internal/package-info.java index e097c8f51f..0c5b4be726 100644 --- a/src/main/java/net/openhft/chronicle/queue/internal/package-info.java +++ b/src/main/java/net/openhft/chronicle/queue/internal/package-info.java @@ -1,4 +1,12 @@ /* * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 */ +/** + * Auxiliary internal utilities for Chronicle Queue. + *

+ * This package holds internal helpers such as analytics wiring that are + * used by Chronicle Queue to integrate with external services. The + * classes are not intended for direct use by application code and may + * change without notice. + */ package net.openhft.chronicle.queue.internal; diff --git a/src/main/java/net/openhft/chronicle/queue/internal/reader/package-info.java b/src/main/java/net/openhft/chronicle/queue/internal/reader/package-info.java new file mode 100644 index 0000000000..01de419d18 --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/internal/reader/package-info.java @@ -0,0 +1,11 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * Internal helpers for Chronicle Queue readers. + *

+ * This package provides building blocks used by {@code ChronicleReader} + * such as queue-entry handlers, filters and message consumers. It is an + * implementation detail of the reader tooling rather than a stable API. + */ +package net.openhft.chronicle.queue.internal.reader; diff --git a/src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/CustomPluginQueueEntryReader.java b/src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/CustomPluginQueueEntryReader.java index 73d2cee9a2..f146206c94 100644 --- a/src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/CustomPluginQueueEntryReader.java +++ b/src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/CustomPluginQueueEntryReader.java @@ -18,6 +18,7 @@ * entry reading to the custom plugin. This is useful when custom behavior is needed * for processing each entry in the queue. */ +@SuppressWarnings({"deprecation", "removal"}) public final class CustomPluginQueueEntryReader extends AbstractTailerPollingQueueEntryReader { private final ChronicleReaderPlugin plugin; diff --git a/src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/VanillaQueueEntryReader.java b/src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/VanillaQueueEntryReader.java index 6956af9fec..2ab492d73b 100644 --- a/src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/VanillaQueueEntryReader.java +++ b/src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/VanillaQueueEntryReader.java @@ -19,6 +19,7 @@ * It converts the read entries using a provided {@link QueueEntryHandler} and forwards the converted * message to a {@link MessageConsumer}. */ +@SuppressWarnings({"deprecation", "removal"}) public final class VanillaQueueEntryReader implements QueueEntryReader { private final ExcerptTailer tailer; diff --git a/src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/package-info.java b/src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/package-info.java new file mode 100644 index 0000000000..07df711fe3 --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/internal/reader/queueentryreaders/package-info.java @@ -0,0 +1,12 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * Queue entry reader strategies for Chronicle Queue tools. + *

+ * Classes here implement different ways of interpreting queue entries + * for diagnostic tools, including method-reader based decoding and + * plain text rendering. They are used internally by the reader + * commands. + */ +package net.openhft.chronicle.queue.internal.reader.queueentryreaders; diff --git a/src/main/java/net/openhft/chronicle/queue/internal/util/InternalFileUtil.java b/src/main/java/net/openhft/chronicle/queue/internal/util/InternalFileUtil.java index ace02c6760..8893c3748f 100644 --- a/src/main/java/net/openhft/chronicle/queue/internal/util/InternalFileUtil.java +++ b/src/main/java/net/openhft/chronicle/queue/internal/util/InternalFileUtil.java @@ -122,6 +122,9 @@ public static boolean hasQueueSuffix(@NotNull File file) { * supported for the current platform (e.g. Windows). */ public static FileState state(@NotNull File file) { + if (!getAllOpenFilesIsSupportedOnOS()) { + return stateWindows(file); + } try { return state(file, getAllOpenFiles()); } catch (IOException e) { @@ -147,7 +150,7 @@ public static FileState state(@NotNull File file, Map allOpenFil assertOsSupported(); if (!file.exists()) return FileState.NON_EXISTENT; final String absolutePath = file.getAbsolutePath(); - return allOpenFiles.keySet().contains(absolutePath) + return allOpenFiles.containsKey(absolutePath) ? FileState.OPEN : FileState.CLOSED; } @@ -216,7 +219,7 @@ public static Map getAllOpenFiles() throws IOException { */ private static class ProcFdWalker extends SimpleFileVisitor { - private final static int PID_PATH_INDEX = 1; // where is the pid for process holding file open represented in path? + private static final int PID_PATH_INDEX = 1; // where is the pid for process holding file open represented in path? private final Map openFiles = new HashMap<>(); @Override diff --git a/src/main/java/net/openhft/chronicle/queue/internal/util/package-info.java b/src/main/java/net/openhft/chronicle/queue/internal/util/package-info.java new file mode 100644 index 0000000000..cf56c2caea --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/internal/util/package-info.java @@ -0,0 +1,11 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * Internal utilities shared across Chronicle Queue tooling. + *

+ * Utility classes in this package support the command-line readers and + * writers, providing filesystem helpers, test harness support and other + * non-public functionality. + */ +package net.openhft.chronicle.queue.internal.util; diff --git a/src/main/java/net/openhft/chronicle/queue/internal/writer/package-info.java b/src/main/java/net/openhft/chronicle/queue/internal/writer/package-info.java new file mode 100644 index 0000000000..45a11e6fff --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/internal/writer/package-info.java @@ -0,0 +1,11 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * Internal helpers for Chronicle Queue writers. + *

+ * This package contains support classes used by the queue writer + * tooling to append messages in various formats for testing and + * demonstration purposes. It is not part of the public API. + */ +package net.openhft.chronicle.queue.internal.writer; diff --git a/src/main/java/net/openhft/chronicle/queue/main/BenchmarkMain.java b/src/main/java/net/openhft/chronicle/queue/main/BenchmarkMain.java index 108ca7a645..cf4627434c 100644 --- a/src/main/java/net/openhft/chronicle/queue/main/BenchmarkMain.java +++ b/src/main/java/net/openhft/chronicle/queue/main/BenchmarkMain.java @@ -18,6 +18,10 @@ */ public final class BenchmarkMain { + private BenchmarkMain() { + // utility holder + } + /** * The main method that triggers the benchmarking process. * Delegates the execution to {@link InternalBenchmarkMain#main(String[])}. diff --git a/src/main/java/net/openhft/chronicle/queue/main/DumpMain.java b/src/main/java/net/openhft/chronicle/queue/main/DumpMain.java index cf58574b6e..e4cb612784 100644 --- a/src/main/java/net/openhft/chronicle/queue/main/DumpMain.java +++ b/src/main/java/net/openhft/chronicle/queue/main/DumpMain.java @@ -22,6 +22,10 @@ */ public final class DumpMain { + private DumpMain() { + // utility holder + } + /** * The main method that triggers the dump process. * Delegates the execution to {@link InternalDumpMain#main(String[])}. diff --git a/src/main/java/net/openhft/chronicle/queue/main/HistoryMain.java b/src/main/java/net/openhft/chronicle/queue/main/HistoryMain.java index 4929cf7f45..ebf2fce7e9 100644 --- a/src/main/java/net/openhft/chronicle/queue/main/HistoryMain.java +++ b/src/main/java/net/openhft/chronicle/queue/main/HistoryMain.java @@ -20,6 +20,10 @@ */ public final class HistoryMain { + private HistoryMain() { + // utility holder + } + /** * The main method that triggers the history reading process. * Delegates execution to {@link ChronicleHistoryReaderMain#main(String[])}. diff --git a/src/main/java/net/openhft/chronicle/queue/main/PingPongMain.java b/src/main/java/net/openhft/chronicle/queue/main/PingPongMain.java index 484a98f4b1..b2ba9c2932 100644 --- a/src/main/java/net/openhft/chronicle/queue/main/PingPongMain.java +++ b/src/main/java/net/openhft/chronicle/queue/main/PingPongMain.java @@ -17,6 +17,10 @@ */ public final class PingPongMain { + private PingPongMain() { + // utility holder + } + /** * The main method that triggers the ping-pong benchmark. * Delegates execution to {@link InternalPingPongMain#main(String[])}. diff --git a/src/main/java/net/openhft/chronicle/queue/main/ReaderMain.java b/src/main/java/net/openhft/chronicle/queue/main/ReaderMain.java index 5f93526da9..adeec30871 100644 --- a/src/main/java/net/openhft/chronicle/queue/main/ReaderMain.java +++ b/src/main/java/net/openhft/chronicle/queue/main/ReaderMain.java @@ -12,6 +12,10 @@ */ public final class ReaderMain { + private ReaderMain() { + // utility holder + } + /** * The main method that triggers the reading and display of queue records. * Delegates execution to {@link ChronicleReaderMain#main(String[])}. diff --git a/src/main/java/net/openhft/chronicle/queue/main/RefreshMain.java b/src/main/java/net/openhft/chronicle/queue/main/RefreshMain.java index a0367301b6..7bd4b3c5a9 100644 --- a/src/main/java/net/openhft/chronicle/queue/main/RefreshMain.java +++ b/src/main/java/net/openhft/chronicle/queue/main/RefreshMain.java @@ -8,8 +8,20 @@ import java.io.File; +/** + * Command line entry point that refreshes a queue directory listing. + *

+ * The tool opens a {@link SingleChronicleQueue} for the supplied directory, invokes + * {@link SingleChronicleQueue#refreshDirectoryListing()} and then closes the queue, + * forcing the roll cycle cache to be rebuilt. It is primarily intended for operational + * use when files are added or removed outside the running JVM. + */ public final class RefreshMain { + private RefreshMain() { + // utility holder + } + /** * Call queue.refreshDirectoryListing() on the given queue directory * diff --git a/src/main/java/net/openhft/chronicle/queue/main/RemovableRollFileCandidatesMain.java b/src/main/java/net/openhft/chronicle/queue/main/RemovableRollFileCandidatesMain.java index 613fcd6524..709301cbbb 100644 --- a/src/main/java/net/openhft/chronicle/queue/main/RemovableRollFileCandidatesMain.java +++ b/src/main/java/net/openhft/chronicle/queue/main/RemovableRollFileCandidatesMain.java @@ -11,6 +11,10 @@ */ public final class RemovableRollFileCandidatesMain { + private RemovableRollFileCandidatesMain() { + // utility holder + } + /** * The main method that generates and prints the list of removable roll file candidates. * Delegates execution to {@link InternalRemovableRollFileCandidatesMain#main(String[])}. diff --git a/src/main/java/net/openhft/chronicle/queue/main/UnlockMain.java b/src/main/java/net/openhft/chronicle/queue/main/UnlockMain.java index fad153d745..619a6fddf3 100644 --- a/src/main/java/net/openhft/chronicle/queue/main/UnlockMain.java +++ b/src/main/java/net/openhft/chronicle/queue/main/UnlockMain.java @@ -11,6 +11,10 @@ */ public final class UnlockMain { + private UnlockMain() { + // utility holder + } + /** * The main method that triggers the unlocking process. * Delegates execution to {@link InternalUnlockMain#main(String[])}. diff --git a/src/main/java/net/openhft/chronicle/queue/main/package-info.java b/src/main/java/net/openhft/chronicle/queue/main/package-info.java index a38176ef4b..1556cc5110 100644 --- a/src/main/java/net/openhft/chronicle/queue/main/package-info.java +++ b/src/main/java/net/openhft/chronicle/queue/main/package-info.java @@ -1,4 +1,11 @@ /* * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 */ +/** + * Command line entry points and tools for working with Chronicle Queue instances. + *

+ * Classes in this package provide small utilities for inspecting, maintaining, and benchmarking + * queues from the command line. + * They are intended for operational and demonstration use rather than as a stable API. + */ package net.openhft.chronicle.queue.main; diff --git a/src/main/java/net/openhft/chronicle/queue/package-info.java b/src/main/java/net/openhft/chronicle/queue/package-info.java index a0d49d79ab..7179dcf637 100644 --- a/src/main/java/net/openhft/chronicle/queue/package-info.java +++ b/src/main/java/net/openhft/chronicle/queue/package-info.java @@ -1,4 +1,20 @@ /* * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 */ +/** + * Core Chronicle Queue public API. + *

+ * This package exposes the main abstractions for Chronicle Queue, + * including {@code ChronicleQueue} together with its writer and reader + * views ({@code ExcerptAppender} and {@code ExcerptTailer}). A queue is + * an append-only, file-backed log of messages that can be shared + * between threads, processes and, with Chronicle Queue Enterprise, + * machines. + *

+ * Writers typically obtain their own appender per thread and follow a + * single-writer style: appenders and tailers are not thread-safe and + * must not be shared between concurrent threads. Configuration concerns + * such as roll cycles, wire formats and indexing strategies are + * delegated to builder APIs in this package and its sub-packages. + */ package net.openhft.chronicle.queue; diff --git a/src/main/java/net/openhft/chronicle/queue/reader/ChronicleHistoryReader.java b/src/main/java/net/openhft/chronicle/queue/reader/ChronicleHistoryReader.java index 4619299e18..2723bba67d 100644 --- a/src/main/java/net/openhft/chronicle/queue/reader/ChronicleHistoryReader.java +++ b/src/main/java/net/openhft/chronicle/queue/reader/ChronicleHistoryReader.java @@ -32,31 +32,93 @@ * histograms and timing windows. Various options such as progress reporting, time unit settings, and * histogram management are available for customization. */ +@SuppressWarnings({"deprecation", "removal"}) public class ChronicleHistoryReader implements HistoryReader, Closeable { + /** + * Marker used to indicate summary output has not been configured. + */ private static final int SUMMARY_OUTPUT_UNSET = -999; + /** + * Separator used when composing histogram keys. + */ public static final String SEPARATOR = "_"; + /** + * Queue base path to read from. + */ protected Path basePath; + /** + * Consumer that receives each rendered message. + */ protected Consumer messageSink; + /** + * Whether to log periodic progress markers. + */ protected boolean progress = false; + /** + * Time unit used for windowing and reporting. + */ protected TimeUnit timeUnit = TimeUnit.NANOSECONDS; + /** + * Whether to maintain histograms per method. + */ protected boolean histosByMethod = false; + /** + * Collected histograms keyed by method or stream. + */ protected Map histos = new LinkedHashMap<>(); + /** + * Number of initial messages to skip. + */ protected long ignore = 0; + /** + * Counter of messages processed. + */ protected long counter = 0; + /** + * Measurement window in nanoseconds; zero disables windowing. + */ protected long measurementWindowNanos = 0; + /** + * First timestamp observed in the current window. + */ protected long firstTimeStampNanos = 0; + /** + * Count at the previous window boundary. + */ protected long lastWindowCount = 0; + /** + * Offset to use when printing the summary block. + */ protected int summaryOutputOffset = SUMMARY_OUTPUT_UNSET; + /** + * Optional starting index to begin reading from. + */ protected Long startIndex; + /** + * Supplier used to create new histograms. + */ protected Supplier histoSupplier = () -> new Histogram(60, 4); + /** + * Size of histograms on the previous read pass. + */ protected int lastHistosSize = 0; + /** + * Tailer used to read messages. + */ protected ExcerptTailer tailer; static { ToolsUtil.warnIfResourceTracing(); } + /** + * Creates a new history reader with default settings. + */ + public ChronicleHistoryReader() { + // defaults configured via field initialisers + } + /** * Sets the message sink, which is a consumer to handle each processed message. * diff --git a/src/main/java/net/openhft/chronicle/queue/reader/ChronicleReader.java b/src/main/java/net/openhft/chronicle/queue/reader/ChronicleReader.java index 929c3e5283..e62c9e202d 100644 --- a/src/main/java/net/openhft/chronicle/queue/reader/ChronicleReader.java +++ b/src/main/java/net/openhft/chronicle/queue/reader/ChronicleReader.java @@ -44,9 +44,16 @@ * including tailing (continuous reading of new entries), and allows users to specify inclusion/exclusion * filters, start indices, and message processing through customizable plugins. */ +@SuppressWarnings({"deprecation", "removal"}) public class ChronicleReader implements Reader { private static final long UNSET_VALUE = Long.MIN_VALUE; + /** + * Creates a reader with default configuration. + */ + public ChronicleReader() { + } + private final List inclusionRegex = new ArrayList<>(); private final List exclusionRegex = new ArrayList<>(); private final Pauser pauser = Pauser.millis(1, 100); @@ -84,6 +91,7 @@ private static boolean isSet(final long configValue) { * Executes the reader logic by creating the necessary queue, tailers, and entry readers, * and processing messages until the stop condition is met. */ + @Override public void execute() { configureContentBasedLimiter(); validateArgs(); @@ -290,6 +298,7 @@ public ChronicleReader withMatchLimit(long matchLimit) { * @param messageSink The consumer for processing message strings * @return The current instance of {@link ChronicleReader} */ + @Override public ChronicleReader withMessageSink(final @NotNull Consumer messageSink) { this.messageSink = messageSink; return this; @@ -310,6 +319,7 @@ public Consumer messageSink() { * @param path The base directory path for the Chronicle Queue * @return The current instance of {@link ChronicleReader} */ + @Override public ChronicleReader withBasePath(final @NotNull Path path) { this.basePath = path; return this; @@ -321,6 +331,7 @@ public ChronicleReader withBasePath(final @NotNull Path path) { * @param regex The regex pattern for inclusion * @return The current instance of {@link ChronicleReader} */ + @Override public ChronicleReader withInclusionRegex(final @NotNull String regex) { this.inclusionRegex.add(Pattern.compile(regex)); return this; @@ -332,6 +343,7 @@ public ChronicleReader withInclusionRegex(final @NotNull String regex) { * @param regex The regex pattern for exclusion * @return The current instance of {@link ChronicleReader} */ + @Override public ChronicleReader withExclusionRegex(final @NotNull String regex) { this.exclusionRegex.add(Pattern.compile(regex)); return this; @@ -343,6 +355,7 @@ public ChronicleReader withExclusionRegex(final @NotNull String regex) { * @param customPlugin The {@link ChronicleReaderPlugin} to use * @return The current instance of {@link ChronicleReader} */ + @Override public ChronicleReader withCustomPlugin(final @NotNull ChronicleReaderPlugin customPlugin) { this.customPlugin = customPlugin; return this; @@ -354,6 +367,7 @@ public ChronicleReader withCustomPlugin(final @NotNull ChronicleReaderPlugin cus * @param index The start index to begin reading from * @return The current instance of {@link ChronicleReader} */ + @Override public ChronicleReader withStartIndex(final long index) { this.startIndex = index; return this; @@ -364,6 +378,7 @@ public ChronicleReader withStartIndex(final long index) { * * @return The current instance of {@link ChronicleReader} */ + @Override public ChronicleReader tail() { this.tailInputSource = true; return this; @@ -375,6 +390,7 @@ public ChronicleReader tail() { * @param maxHistoryRecords The maximum number of history records to process * @return The current instance of {@link ChronicleReader} */ + @Override public ChronicleReader historyRecords(final long maxHistoryRecords) { this.maxHistoryRecords = maxHistoryRecords; return this; @@ -387,6 +403,7 @@ public ChronicleReader historyRecords(final long maxHistoryRecords) { * @param methodReaderInterface The fully qualified class name of the method reader interface * @return The current instance of {@link ChronicleReader} */ + @Override public ChronicleReader asMethodReader(@NotNull String methodReaderInterface) { if (methodReaderInterface.isEmpty()) { entryHandlerFactory = () -> new InternalDummyMethodReaderQueueEntryHandler(wireType); @@ -472,6 +489,7 @@ public ChronicleReader withLimiterArg(@NotNull String limiterArg) { * @param wireType The {@link WireType} to be used * @return The current instance of {@link ChronicleReader} */ + @Override public ChronicleReader withWireType(@NotNull WireType wireType) { this.wireType = wireType; return this; @@ -492,6 +510,7 @@ public ChronicleReader inReverseOrder() { * * @return The current instance of {@link ChronicleReader} */ + @Override public ChronicleReader suppressDisplayIndex() { this.displayIndex = false; return this; @@ -533,6 +552,7 @@ public Class methodReaderInterface() { * @param pollMethod The polling function to use * @return The current instance of {@link ChronicleReader} */ + @Deprecated(/* to be removed in 2027, only used in tests */) public ChronicleReader withDocumentPollMethod(final Function pollMethod) { this.pollMethod = pollMethod; return this; @@ -762,6 +782,7 @@ private ChronicleQueue createQueue() { /** * Stops the reader, halting any further processing of the queue. */ + @Override public void stop() { running = false; } diff --git a/src/main/java/net/openhft/chronicle/queue/reader/Reader.java b/src/main/java/net/openhft/chronicle/queue/reader/Reader.java index 37955253dd..0c2a3e5799 100644 --- a/src/main/java/net/openhft/chronicle/queue/reader/Reader.java +++ b/src/main/java/net/openhft/chronicle/queue/reader/Reader.java @@ -26,6 +26,7 @@ public interface Reader { /** * Stops the Reader, halting further processing. */ + @Deprecated(/* to be removed in 2027, only used in tests */) void stop(); /** diff --git a/src/main/java/net/openhft/chronicle/queue/reader/comparator/package-info.java b/src/main/java/net/openhft/chronicle/queue/reader/comparator/package-info.java new file mode 100644 index 0000000000..811edc771f --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/reader/comparator/package-info.java @@ -0,0 +1,11 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * Comparators used by Chronicle Queue reader tooling. + *

+ * This package provides comparator implementations that support binary + * search and ordering of queue entries when scanning or querying + * queues via the reader utilities. + */ +package net.openhft.chronicle.queue.reader.comparator; diff --git a/src/main/java/net/openhft/chronicle/queue/reader/package-info.java b/src/main/java/net/openhft/chronicle/queue/reader/package-info.java new file mode 100644 index 0000000000..7fe12dd04a --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/reader/package-info.java @@ -0,0 +1,12 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * Reader tools for Chronicle Queue. + *

+ * Types in this package implement command-line and embeddable readers + * that scan queue contents, filter messages and perform binary searches + * over indices. They are intended for operations and diagnostics + * rather than as core queue APIs. + */ +package net.openhft.chronicle.queue.reader; diff --git a/src/main/java/net/openhft/chronicle/queue/rollcycles/package-info.java b/src/main/java/net/openhft/chronicle/queue/rollcycles/package-info.java new file mode 100644 index 0000000000..c9a88b0fca --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/rollcycles/package-info.java @@ -0,0 +1,12 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * Roll cycle definitions for Chronicle Queue. + *

+ * Classes in this package describe how queues roll their backing files, + * including file naming, indexing parameters and retention + * characteristics. They include both current and legacy roll-cycle + * variants. + */ +package net.openhft.chronicle.queue.rollcycles; diff --git a/src/main/java/net/openhft/chronicle/queue/util/FileState.java b/src/main/java/net/openhft/chronicle/queue/util/FileState.java index 28e15dc99b..ecbf195460 100644 --- a/src/main/java/net/openhft/chronicle/queue/util/FileState.java +++ b/src/main/java/net/openhft/chronicle/queue/util/FileState.java @@ -13,8 +13,20 @@ * */ public enum FileState { - OPEN, // The file is open and being used - CLOSED, // The file is closed and not in use - NON_EXISTENT, // The file does not exist - UNDETERMINED // The state of the file cannot be determined + /** + * The file is open and being used. + */ + OPEN, + /** + * The file is closed and not in use. + */ + CLOSED, + /** + * The file does not exist. + */ + NON_EXISTENT, + /** + * The state of the file cannot be determined. + */ + UNDETERMINED } diff --git a/src/main/java/net/openhft/chronicle/queue/util/PretouchUtil.java b/src/main/java/net/openhft/chronicle/queue/util/PretouchUtil.java index aefe549522..5addda73df 100644 --- a/src/main/java/net/openhft/chronicle/queue/util/PretouchUtil.java +++ b/src/main/java/net/openhft/chronicle/queue/util/PretouchUtil.java @@ -21,6 +21,10 @@ public final class PretouchUtil { private static final PretoucherFactory INSTANCE; + private PretouchUtil() { + // utility holder + } + static { PretoucherFactory instance; try { diff --git a/src/main/java/net/openhft/chronicle/queue/util/package-info.java b/src/main/java/net/openhft/chronicle/queue/util/package-info.java new file mode 100644 index 0000000000..8f7a01d6eb --- /dev/null +++ b/src/main/java/net/openhft/chronicle/queue/util/package-info.java @@ -0,0 +1,11 @@ +/* + * Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0 + */ +/** + * General utilities for Chronicle Queue. + *

+ * Utilities here support tasks such as pre-touching, file management + * and command-line tooling. They are used by both main-line code and + * operational tools. + */ +package net.openhft.chronicle.queue.util; From 36932e81b19fd688fdbb6bf9b600546f55e51fc2 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Thu, 11 Dec 2025 14:27:48 +0000 Subject: [PATCH 3/6] Refactor test classes to improve annotations and remove unnecessary blank lines --- .../chronicle/queue/AcquireReleaseTest.java | 1 + .../chronicle/queue/AppenderListenerTest.java | 1 + .../queue/ChronicleAppenderCycleTest.java | 2 +- .../ChronicleHistoryReaderMainCliTest.java | 1 + .../queue/ChronicleHistoryReaderMainTest.java | 3 +- .../queue/ChronicleQueueIndexTest.java | 2 +- .../ChronicleQueueLatencyDistribution.java | 29 ------------------- .../queue/ChronicleReaderMainCliTest.java | 1 + .../queue/ChronicleRollingIssueTest.java | 6 +--- .../queue/ChronicleWriterMainCliTest.java | 2 -- .../chronicle/queue/ChunkCountTest.java | 1 + .../chronicle/queue/CreateAtIndexTest.java | 22 +------------- .../chronicle/queue/CycleNotFoundTest.java | 1 + .../queue/DtoBytesMarshallableTest.java | 4 +++ .../chronicle/queue/DumpQueueMainTest.java | 3 +- .../chronicle/queue/ExcerptAppenderTest.java | 1 + .../chronicle/queue/ExcerptCommonTest.java | 1 + .../chronicle/queue/ExcerptTailerTest.java | 1 + .../chronicle/queue/LastAcknowledgedTest.java | 1 + .../chronicle/queue/NoMessageHistoryTest.java | 1 + .../chronicle/queue/RunLargeQueueMain.java | 2 +- .../chronicle/queue/TableStorePutGetTest.java | 1 + .../queue/TailerCloseInParallelTest.java | 1 + .../queue/TestAppenderThreadSafe.java | 1 + .../chronicle/queue/TestDeleteQueueFile.java | 2 +- ...MessagesBetweenTailorsAndAppenderTest.java | 1 + .../queue/impl/RollingChronicleQueueTest.java | 1 + .../impl/RollingResourcesCacheCompatTest.java | 6 ---- .../queue/impl/single/EmptyRollCycleTest.java | 1 + .../single/EntryCountNotBehindReadTest.java | 1 + .../single/EofMarkerOnEmptyQueueTest.java | 1 + .../queue/impl/single/IndexTest.java | 2 +- .../IndexingLastSequenceNumberTest.java | 1 + .../single/IndexingSpacingAndCountTest.java | 1 + .../queue/impl/single/NormaliseEOFsTest.java | 1 + .../queue/impl/single/PartialUpdateTest.java | 2 ++ .../queue/impl/single/QueueLockTest.java | 2 +- .../single/ReferenceCountedCacheTest.java | 1 + .../impl/single/RollAtEndOfCycleTest.java | 1 + .../queue/impl/single/SingleCQFormatTest.java | 1 + .../SingleChronicleQueueBuilderTest.java | 1 + .../single/SingleChronicleQueueStoreTest.java | 1 + .../impl/single/SingleChronicleQueueTest.java | 5 ++-- .../StoreAppenderInternalWriteBytesTest.java | 8 ----- .../queue/impl/single/StoreAppenderTest.java | 1 + .../queue/impl/single/StoreTailerTest.java | 3 +- .../queue/impl/single/StuckQueueTest.java | 1 + .../impl/single/TableStoreWriteLockTest.java | 10 ++++--- .../queue/impl/single/TestBinarySearch.java | 2 +- .../single/TestMethodWriterWithThreads.java | 2 +- .../impl/single/ToEndInvalidIndexTest.java | 3 +- .../queue/impl/single/ToEndTest.java | 3 +- .../queue/impl/single/jira/Queue28Test.java | 1 - ...ycleMultiThreadStressDoubleBufferTest.java | 2 +- ...lCycleMultiThreadStressPretouchEATest.java | 1 + ...ollCycleMultiThreadStressPretouchTest.java | 1 + ...ollCycleMultiThreadStressReadOnlyTest.java | 1 + ...ultiThreadStressSharedWriterQueueTest.java | 3 +- .../RollCycleMultiThreadStressTest.java | 1 + .../SingleTableStoreIntegrationTests.java | 4 ++- .../domestic/QueueOffsetSpecTest.java | 1 + .../internal/main/InternalDumpMainTest.java | 2 -- .../reader/ChronicleMethodReaderTest.java | 2 ++ .../internal/reader/ChronicleReaderTest.java | 10 ++----- .../queue/internal/reader/RollEOFTest.java | 1 + .../util/InternalFileUtilLinuxStateTest.java | 1 - .../chronicle/queue/issue/DeleteFileTest.java | 1 + .../queue/issue/ReaderResizesFileTest.java | 5 ++-- .../chronicle/queue/issue/TailerTest.java | 1 + .../queue/jitter/QueueWriteJitterMain.java | 1 + .../chronicle/queue/main/DumpMainTest.java | 2 +- .../openhft/chronicle/queue/micros/Order.java | 2 ++ .../queue/micros/OrderManagerTest.java | 2 ++ .../NamedTailerPreconditionTest.java | 1 + .../chronicle/queue/util/FileUtilTest.java | 1 + .../queue/util/PretouchUtilTest.java | 1 - 76 files changed, 90 insertions(+), 112 deletions(-) diff --git a/src/test/java/net/openhft/chronicle/queue/AcquireReleaseTest.java b/src/test/java/net/openhft/chronicle/queue/AcquireReleaseTest.java index 8ea3e9fad1..8b0f8a67c0 100644 --- a/src/test/java/net/openhft/chronicle/queue/AcquireReleaseTest.java +++ b/src/test/java/net/openhft/chronicle/queue/AcquireReleaseTest.java @@ -104,6 +104,7 @@ public void testWithCleanupStoreFilesWithNoDataAcquireAndRelease() throws Interr final AtomicInteger acount = new AtomicInteger(); final AtomicInteger qcount = new AtomicInteger(); final StoreFileListener storeFileListener = new StoreFileListener() { + @Override public void onAcquired(int cycle, File file) { acount.incrementAndGet(); } diff --git a/src/test/java/net/openhft/chronicle/queue/AppenderListenerTest.java b/src/test/java/net/openhft/chronicle/queue/AppenderListenerTest.java index ee07a736eb..30ffec45c9 100644 --- a/src/test/java/net/openhft/chronicle/queue/AppenderListenerTest.java +++ b/src/test/java/net/openhft/chronicle/queue/AppenderListenerTest.java @@ -11,6 +11,7 @@ import static org.junit.Assert.assertEquals; +@SuppressWarnings({"deprecation", "removal"}) public class AppenderListenerTest extends QueueTestCommon { @Test diff --git a/src/test/java/net/openhft/chronicle/queue/ChronicleAppenderCycleTest.java b/src/test/java/net/openhft/chronicle/queue/ChronicleAppenderCycleTest.java index b10a11714a..6bb9b00df1 100644 --- a/src/test/java/net/openhft/chronicle/queue/ChronicleAppenderCycleTest.java +++ b/src/test/java/net/openhft/chronicle/queue/ChronicleAppenderCycleTest.java @@ -47,7 +47,7 @@ public void testAppenderCycle() throws IOException { } } - private void runTest(String id, Bytes msg) throws IOException { + private void runTest(String id, Bytes msg) { Path path = IOTools.createTempDirectory(id); try { CountDownLatch steady = new CountDownLatch(2); diff --git a/src/test/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMainCliTest.java b/src/test/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMainCliTest.java index b0d726136b..c3baeab042 100644 --- a/src/test/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMainCliTest.java +++ b/src/test/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMainCliTest.java @@ -64,6 +64,7 @@ public void parseCommandLineMissingDirectoryPrintsError() { } } + @SuppressWarnings("PMD.TestClassWithoutTestCases") private static final class TestChronicleHistoryReaderMain extends ChronicleHistoryReaderMain { final RecordingChronicleHistoryReader reader = new RecordingChronicleHistoryReader(); final StringBuilder helpOutput = new StringBuilder(); diff --git a/src/test/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMainTest.java b/src/test/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMainTest.java index 9e117f9640..daf1305192 100644 --- a/src/test/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMainTest.java +++ b/src/test/java/net/openhft/chronicle/queue/ChronicleHistoryReaderMainTest.java @@ -43,7 +43,8 @@ public void setUp() { @After public void tearDown() { - System.setSecurityManager(null); + if (Jvm.majorVersion() < 17) + System.setSecurityManager(null); } @Test diff --git a/src/test/java/net/openhft/chronicle/queue/ChronicleQueueIndexTest.java b/src/test/java/net/openhft/chronicle/queue/ChronicleQueueIndexTest.java index 049f6a9558..4888f60156 100644 --- a/src/test/java/net/openhft/chronicle/queue/ChronicleQueueIndexTest.java +++ b/src/test/java/net/openhft/chronicle/queue/ChronicleQueueIndexTest.java @@ -27,6 +27,7 @@ import static net.openhft.chronicle.queue.rollcycles.TestRollCycles.TEST_SECONDLY; import static org.junit.Assert.*; +@SuppressWarnings({"deprecation", "removal"}) public class ChronicleQueueIndexTest extends QueueTestCommon { @Test @@ -244,7 +245,6 @@ private void driver0(String[] strings, boolean[] meta, SetTimeProvider stp, long .testBlockSize() .build(); final ExcerptAppender appender = queue.createAppender()) { - for (int i = 0; i < strings.length; ++i) { try (DocumentContext dc = appender.writingDocument(meta[i])) { dc.wire().write("key").text(strings[i]); diff --git a/src/test/java/net/openhft/chronicle/queue/ChronicleQueueLatencyDistribution.java b/src/test/java/net/openhft/chronicle/queue/ChronicleQueueLatencyDistribution.java index 592ee75147..a12f228003 100644 --- a/src/test/java/net/openhft/chronicle/queue/ChronicleQueueLatencyDistribution.java +++ b/src/test/java/net/openhft/chronicle/queue/ChronicleQueueLatencyDistribution.java @@ -115,11 +115,6 @@ protected void run(String[] args) throws Exception { } protected void runTest(@NotNull ChronicleQueue queue, int throughput) throws InterruptedException { -/* - Jvm.setExceptionHandlers(PrintExceptionHandler.ERR, - PrintExceptionHandler.OUT, - PrintExceptionHandler.OUT); -*/ Histogram histogramCo = new Histogram(); Histogram histogramIn = new Histogram(); @@ -149,9 +144,6 @@ protected void runTest(@NotNull ChronicleQueue queue, int throughput) throws Int int counter = 0; while (!Thread.currentThread().isInterrupted()) { try { - // if (SAMPLING) - // sampler.thread(Thread.currentThread()); - // boolean found = tailer.readDocument(myReadMarshallable); boolean found; try (DocumentContext dc = tailer.readingDocument()) { found = dc.isPresent(); @@ -168,24 +160,8 @@ protected void runTest(@NotNull ChronicleQueue queue, int throughput) throws Int long now = System.nanoTime(); histogramCo.sample(now - startCo); histogramIn.sample(now - startIn); - // if (count % 1_000_000 == 0) System.out.println("read " + count); - } - } -/* - if (SAMPLING) { - StackTraceElement[] stack = sampler.getAndReset(); - if (stack != null) { - if (!stack[0].getClassName().equals(name) && - !stack[0].getClassName().equals("java.lang.Thread")) { - StringBuilder sb = new StringBuilder(); - Jvm.trimStackTrace(sb, stack); - // System.out.println(sb); - } - } else if (!found) { - Thread.yield(); } } - */ } catch (Exception e) { break; @@ -240,7 +216,6 @@ protected void runTest(@NotNull ChronicleQueue queue, int throughput) throws Int } } next += interval; - // if (i % 1_000_000 == 0) System.out.println("wrote " + i); } stackCount.entrySet().stream() .filter(e -> e.getValue() > 1) @@ -266,9 +241,5 @@ protected void runTest(@NotNull ChronicleQueue queue, int throughput) throws Int Jvm.pause(500); tailerThread.interrupt(); tailerThread.join(); - - // System.out.println("wr: " + histogramWr.toMicrosFormat()); - // System.out.println("in: " + histogramIn.toMicrosFormat()); - // System.out.println("co: " + histogramCo.toMicrosFormat()); } } diff --git a/src/test/java/net/openhft/chronicle/queue/ChronicleReaderMainCliTest.java b/src/test/java/net/openhft/chronicle/queue/ChronicleReaderMainCliTest.java index 9f596a9a9a..38e6dcbbe0 100644 --- a/src/test/java/net/openhft/chronicle/queue/ChronicleReaderMainCliTest.java +++ b/src/test/java/net/openhft/chronicle/queue/ChronicleReaderMainCliTest.java @@ -101,6 +101,7 @@ public void methodReaderOptionsEnableMessageHistory() { assertTrue(reader.wireTypeSnapshot == WireType.TEXT); } + @SuppressWarnings("PMD.TestClassWithoutTestCases") private static final class TestChronicleReaderMain extends ChronicleReaderMain { final RecordingChronicleReader reader = new RecordingChronicleReader(); diff --git a/src/test/java/net/openhft/chronicle/queue/ChronicleRollingIssueTest.java b/src/test/java/net/openhft/chronicle/queue/ChronicleRollingIssueTest.java index 35d49d73b9..58b6671ed2 100644 --- a/src/test/java/net/openhft/chronicle/queue/ChronicleRollingIssueTest.java +++ b/src/test/java/net/openhft/chronicle/queue/ChronicleRollingIssueTest.java @@ -35,6 +35,7 @@ public void threadDump() { } @After + @Override public void tearDown() { IOTools.deleteDirWithFiles(path); } @@ -89,11 +90,6 @@ public void test() throws InterruptedException { count2++; } else if (index >= 0) { if (TEST_SECONDLY.toCycle(lastIndex) != TEST_SECONDLY.toCycle(index)) { -/* - // System.out.println("Wrote: " + count - + " read: " + count2 - + " index: " + Long.toHexString(index)); -*/ lastIndex = index; } } diff --git a/src/test/java/net/openhft/chronicle/queue/ChronicleWriterMainCliTest.java b/src/test/java/net/openhft/chronicle/queue/ChronicleWriterMainCliTest.java index 745b6c557f..65f5c82408 100644 --- a/src/test/java/net/openhft/chronicle/queue/ChronicleWriterMainCliTest.java +++ b/src/test/java/net/openhft/chronicle/queue/ChronicleWriterMainCliTest.java @@ -4,10 +4,8 @@ package net.openhft.chronicle.queue; import net.openhft.chronicle.core.OS; -import net.openhft.chronicle.queue.ExcerptTailer; import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder; import net.openhft.chronicle.wire.DocumentContext; -import net.openhft.chronicle.wire.WireType; import org.junit.Test; import java.nio.charset.StandardCharsets; diff --git a/src/test/java/net/openhft/chronicle/queue/ChunkCountTest.java b/src/test/java/net/openhft/chronicle/queue/ChunkCountTest.java index 9a0a1e4838..454e257ac0 100644 --- a/src/test/java/net/openhft/chronicle/queue/ChunkCountTest.java +++ b/src/test/java/net/openhft/chronicle/queue/ChunkCountTest.java @@ -16,6 +16,7 @@ import static net.openhft.chronicle.queue.rollcycles.LegacyRollCycles.DAILY; import static org.junit.Assert.assertEquals; +@SuppressWarnings({"deprecation", "removal"}) public class ChunkCountTest extends QueueTestCommon { @Test public void chunks() { diff --git a/src/test/java/net/openhft/chronicle/queue/CreateAtIndexTest.java b/src/test/java/net/openhft/chronicle/queue/CreateAtIndexTest.java index 051d7239d1..930a5616dc 100644 --- a/src/test/java/net/openhft/chronicle/queue/CreateAtIndexTest.java +++ b/src/test/java/net/openhft/chronicle/queue/CreateAtIndexTest.java @@ -21,6 +21,7 @@ import static org.junit.Assert.assertThrows; @RequiredForClient +@SuppressWarnings({"deprecation", "removal"}) public class CreateAtIndexTest extends QueueTestCommon { @Test @@ -46,27 +47,6 @@ public class CreateAtIndexTest extends QueueTestCommon { assertEquals(before, after); } -/* - TODO FIX - if (Jvm.isAssertEnabled()) { - try (ChronicleQueue queue = single(tmp) - .testBlockSize() - .build()) { - InternalAppender appender = (InternalAppender) queue.acquireAppender(); - - String before = queue.dump(); - try { - appender.writeBytes(0x421d00000000L, Bytes.from("hellooooo world")); - fail(); - } catch (IllegalStateException e) { - // expected - } - String after = queue.dump(); - assertEquals(before, after); - } - } - */ - // try too far try (ChronicleQueue queue = single(tmp) .testBlockSize() diff --git a/src/test/java/net/openhft/chronicle/queue/CycleNotFoundTest.java b/src/test/java/net/openhft/chronicle/queue/CycleNotFoundTest.java index 8f9240a4c0..a6764a567b 100644 --- a/src/test/java/net/openhft/chronicle/queue/CycleNotFoundTest.java +++ b/src/test/java/net/openhft/chronicle/queue/CycleNotFoundTest.java @@ -21,6 +21,7 @@ import static org.junit.Assert.assertEquals; @RequiredForClient +@SuppressWarnings({"deprecation", "removal"}) public class CycleNotFoundTest extends QueueTestCommon { private static final int NUMBER_OF_TAILERS = 20; diff --git a/src/test/java/net/openhft/chronicle/queue/DtoBytesMarshallableTest.java b/src/test/java/net/openhft/chronicle/queue/DtoBytesMarshallableTest.java index 366dfa6bf1..4b6a6e90fb 100644 --- a/src/test/java/net/openhft/chronicle/queue/DtoBytesMarshallableTest.java +++ b/src/test/java/net/openhft/chronicle/queue/DtoBytesMarshallableTest.java @@ -79,12 +79,14 @@ static class DtoBytesMarshallable extends BytesInBinaryMarshallable { StringBuilder name = new StringBuilder(); int age; + @Override public void readMarshallable(BytesIn bytes) { age = bytes.readInt(); name.setLength(0); bytes.readUtf8(name); } + @Override public void writeMarshallable(BytesOut bytes) { bytes.writeInt(age); bytes.writeUtf8(name); @@ -95,12 +97,14 @@ static class DtoAbstractMarshallable extends SelfDescribingMarshallable { StringBuilder name = new StringBuilder(); int age; + @Override public void readMarshallable(BytesIn bytes) { age = bytes.readInt(); name.setLength(0); bytes.readUtf8(name); } + @Override public void writeMarshallable(BytesOut bytes) { bytes.writeInt(age); bytes.writeUtf8(name); diff --git a/src/test/java/net/openhft/chronicle/queue/DumpQueueMainTest.java b/src/test/java/net/openhft/chronicle/queue/DumpQueueMainTest.java index ff0dbc6304..2ed076eb61 100644 --- a/src/test/java/net/openhft/chronicle/queue/DumpQueueMainTest.java +++ b/src/test/java/net/openhft/chronicle/queue/DumpQueueMainTest.java @@ -17,6 +17,7 @@ import static org.junit.Assert.assertNotEquals; import static org.junit.Assert.assertTrue; +@SuppressWarnings({"deprecation", "removal"}) public class DumpQueueMainTest extends QueueTestCommon { @Test @@ -72,7 +73,7 @@ private static final class CountingOutputStream extends OutputStream { private long bytes; @Override - public void write(final int b) throws IOException { + public void write(final int b) { bytes++; } } diff --git a/src/test/java/net/openhft/chronicle/queue/ExcerptAppenderTest.java b/src/test/java/net/openhft/chronicle/queue/ExcerptAppenderTest.java index 93fc2d18ea..d337e3be34 100644 --- a/src/test/java/net/openhft/chronicle/queue/ExcerptAppenderTest.java +++ b/src/test/java/net/openhft/chronicle/queue/ExcerptAppenderTest.java @@ -20,6 +20,7 @@ /** * Unit tests for ExcerptAppender interface implementations. */ +@SuppressWarnings({"deprecation", "removal"}) public class ExcerptAppenderTest extends QueueTestCommon { private static final String TEST_QUEUE = OS.getTarget() + "/ExcerptAppenderTest"; diff --git a/src/test/java/net/openhft/chronicle/queue/ExcerptCommonTest.java b/src/test/java/net/openhft/chronicle/queue/ExcerptCommonTest.java index 4da303a93b..967ca7f31d 100644 --- a/src/test/java/net/openhft/chronicle/queue/ExcerptCommonTest.java +++ b/src/test/java/net/openhft/chronicle/queue/ExcerptCommonTest.java @@ -14,6 +14,7 @@ /** * Unit tests for ExcerptCommon interface implementations. */ +@SuppressWarnings({"deprecation", "removal"}) public class ExcerptCommonTest extends QueueTestCommon { private static final String TEST_QUEUE = OS.getTarget() + "/ExcerptCommonTest"; diff --git a/src/test/java/net/openhft/chronicle/queue/ExcerptTailerTest.java b/src/test/java/net/openhft/chronicle/queue/ExcerptTailerTest.java index 3d6bd48fef..66c1390157 100644 --- a/src/test/java/net/openhft/chronicle/queue/ExcerptTailerTest.java +++ b/src/test/java/net/openhft/chronicle/queue/ExcerptTailerTest.java @@ -25,6 +25,7 @@ public void setUp() { } @After + @Override public void tearDown() { excerptTailer.close(); queue.close(); diff --git a/src/test/java/net/openhft/chronicle/queue/LastAcknowledgedTest.java b/src/test/java/net/openhft/chronicle/queue/LastAcknowledgedTest.java index 9a4e444f6d..7de858c059 100644 --- a/src/test/java/net/openhft/chronicle/queue/LastAcknowledgedTest.java +++ b/src/test/java/net/openhft/chronicle/queue/LastAcknowledgedTest.java @@ -23,6 +23,7 @@ import static org.junit.Assert.*; @RequiredForClient +@SuppressWarnings({"deprecation", "removal"}) public class LastAcknowledgedTest extends QueueTestCommon { @Test public void testLastAcknowledge() { diff --git a/src/test/java/net/openhft/chronicle/queue/NoMessageHistoryTest.java b/src/test/java/net/openhft/chronicle/queue/NoMessageHistoryTest.java index a983162e71..07c478adec 100644 --- a/src/test/java/net/openhft/chronicle/queue/NoMessageHistoryTest.java +++ b/src/test/java/net/openhft/chronicle/queue/NoMessageHistoryTest.java @@ -7,6 +7,7 @@ import static org.junit.Assert.*; +@SuppressWarnings({"deprecation", "removal"}) public class NoMessageHistoryTest extends QueueTestCommon { @Test diff --git a/src/test/java/net/openhft/chronicle/queue/RunLargeQueueMain.java b/src/test/java/net/openhft/chronicle/queue/RunLargeQueueMain.java index d73dfcb9f8..736f05f600 100644 --- a/src/test/java/net/openhft/chronicle/queue/RunLargeQueueMain.java +++ b/src/test/java/net/openhft/chronicle/queue/RunLargeQueueMain.java @@ -14,7 +14,7 @@ * This example writes and reads messages using a Chronicle Queue with specified * file and message sizes. Performance metrics for both operations are logged. *

- * Usage: java -Dfile.size= -Dmsg.size= RunLargeQueueMain [queue_path] + * Usage: {@code java -Dfile.size= -Dmsg.size= RunLargeQueueMain [queue_path]} */ public enum RunLargeQueueMain { ; // no instances allowed diff --git a/src/test/java/net/openhft/chronicle/queue/TableStorePutGetTest.java b/src/test/java/net/openhft/chronicle/queue/TableStorePutGetTest.java index da47c71f7e..53cbdc4cc6 100644 --- a/src/test/java/net/openhft/chronicle/queue/TableStorePutGetTest.java +++ b/src/test/java/net/openhft/chronicle/queue/TableStorePutGetTest.java @@ -15,6 +15,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +@SuppressWarnings({"deprecation", "removal"}) public class TableStorePutGetTest extends QueueTestCommon { @Test public void indexEntry() { diff --git a/src/test/java/net/openhft/chronicle/queue/TailerCloseInParallelTest.java b/src/test/java/net/openhft/chronicle/queue/TailerCloseInParallelTest.java index 40117357de..28cac7fce4 100644 --- a/src/test/java/net/openhft/chronicle/queue/TailerCloseInParallelTest.java +++ b/src/test/java/net/openhft/chronicle/queue/TailerCloseInParallelTest.java @@ -70,6 +70,7 @@ public void runTenTimes() throws InterruptedException { finishedNormally = true; } + // CPD-OFF - mirrors SingleChroniclePerfMainTest perf loop private static void doPerfTest(String file, TestWriter> writer, TestReader> reader, int count, boolean print) throws InterruptedException { Histogram writeHdr = new Histogram(30, 7); Histogram readHdr = new Histogram(30, 7); diff --git a/src/test/java/net/openhft/chronicle/queue/TestAppenderThreadSafe.java b/src/test/java/net/openhft/chronicle/queue/TestAppenderThreadSafe.java index 593f19af16..026246edd0 100644 --- a/src/test/java/net/openhft/chronicle/queue/TestAppenderThreadSafe.java +++ b/src/test/java/net/openhft/chronicle/queue/TestAppenderThreadSafe.java @@ -6,6 +6,7 @@ import net.openhft.chronicle.core.annotation.RequiredForClient; @RequiredForClient("Even though it's empty") +@SuppressWarnings("PMD.TestClassWithoutTestCases") class TestAppenderThreadSafe { } diff --git a/src/test/java/net/openhft/chronicle/queue/TestDeleteQueueFile.java b/src/test/java/net/openhft/chronicle/queue/TestDeleteQueueFile.java index c76511bfc9..afffd3cbce 100644 --- a/src/test/java/net/openhft/chronicle/queue/TestDeleteQueueFile.java +++ b/src/test/java/net/openhft/chronicle/queue/TestDeleteQueueFile.java @@ -36,7 +36,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assume.assumeFalse; -@SuppressWarnings("this-escape") +@SuppressWarnings({"this-escape", "deprecation", "removal"}) public class TestDeleteQueueFile extends QueueTestCommon { private static final int NUM_REPEATS = 10; diff --git a/src/test/java/net/openhft/chronicle/queue/VisibilityOfMessagesBetweenTailorsAndAppenderTest.java b/src/test/java/net/openhft/chronicle/queue/VisibilityOfMessagesBetweenTailorsAndAppenderTest.java index e22049f18b..2905c3d292 100644 --- a/src/test/java/net/openhft/chronicle/queue/VisibilityOfMessagesBetweenTailorsAndAppenderTest.java +++ b/src/test/java/net/openhft/chronicle/queue/VisibilityOfMessagesBetweenTailorsAndAppenderTest.java @@ -20,6 +20,7 @@ @RequiredForClient public class VisibilityOfMessagesBetweenTailorsAndAppenderTest extends QueueTestCommon { + @SuppressWarnings("PMD.UnusedAssignment") // written by appender thread, consumed by tailer thread private volatile long lastWrittenIndex = Long.MIN_VALUE; @Override diff --git a/src/test/java/net/openhft/chronicle/queue/impl/RollingChronicleQueueTest.java b/src/test/java/net/openhft/chronicle/queue/impl/RollingChronicleQueueTest.java index f30ba3b3d2..db871317fe 100755 --- a/src/test/java/net/openhft/chronicle/queue/impl/RollingChronicleQueueTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/RollingChronicleQueueTest.java @@ -19,6 +19,7 @@ import static net.openhft.chronicle.queue.rollcycles.TestRollCycles.TEST4_DAILY; import static org.junit.Assert.assertEquals; +@SuppressWarnings({"deprecation", "removal"}) public class RollingChronicleQueueTest extends QueueTestCommon { @Test diff --git a/src/test/java/net/openhft/chronicle/queue/impl/RollingResourcesCacheCompatTest.java b/src/test/java/net/openhft/chronicle/queue/impl/RollingResourcesCacheCompatTest.java index 593dda939f..48cb49334b 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/RollingResourcesCacheCompatTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/RollingResourcesCacheCompatTest.java @@ -9,12 +9,6 @@ import org.junit.Test; import java.io.File; -import java.time.Instant; -import java.time.ZoneId; -import java.time.format.DateTimeFormatter; -import java.time.temporal.ChronoUnit; -import java.util.Random; -import java.util.concurrent.TimeUnit; import static net.openhft.chronicle.queue.rollcycles.LegacyRollCycles.*; import static org.junit.Assert.assertEquals; diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/EmptyRollCycleTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/EmptyRollCycleTest.java index 11d41b9165..e549a7f2b4 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/EmptyRollCycleTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/EmptyRollCycleTest.java @@ -37,6 +37,7 @@ public void setUp() { } @After + @Override public void tearDown() { IOTools.deleteDirWithFiles(dataDirectory.toFile()); } diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/EntryCountNotBehindReadTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/EntryCountNotBehindReadTest.java index 220ecfed55..cef4fa235e 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/EntryCountNotBehindReadTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/EntryCountNotBehindReadTest.java @@ -24,6 +24,7 @@ import static org.junit.Assert.assertFalse; +@SuppressWarnings({"deprecation", "removal"}) public final class EntryCountNotBehindReadTest extends QueueTestCommon { private static final int TOTAL_EVENTS = 100_000; diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/EofMarkerOnEmptyQueueTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/EofMarkerOnEmptyQueueTest.java index 8484329145..cf06e9642a 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/EofMarkerOnEmptyQueueTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/EofMarkerOnEmptyQueueTest.java @@ -29,6 +29,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotEquals; +@SuppressWarnings({"deprecation", "removal"}) public final class EofMarkerOnEmptyQueueTest extends QueueTestCommon { private static final ReferenceOwner test = ReferenceOwner.temporary("test"); @Rule diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/IndexTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/IndexTest.java index 0d32934537..3b206cd86f 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/IndexTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/IndexTest.java @@ -15,7 +15,6 @@ import org.junit.runner.RunWith; import org.junit.runners.Parameterized; -import java.io.IOException; import java.util.Arrays; import java.util.Collection; @@ -23,6 +22,7 @@ import static org.junit.Assert.assertEquals; @RunWith(Parameterized.class) +@SuppressWarnings({"deprecation", "removal"}) public class IndexTest extends QueueTestCommon { @NotNull diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/IndexingLastSequenceNumberTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/IndexingLastSequenceNumberTest.java index 39cebfb4e5..49c374a4a8 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/IndexingLastSequenceNumberTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/IndexingLastSequenceNumberTest.java @@ -13,6 +13,7 @@ /** * Tests focussed on {@link Indexing#lastSequenceNumber(ExcerptContext)}. */ +@SuppressWarnings({"deprecation", "removal"}) class IndexingLastSequenceNumberTest extends IndexingTestCommon { @Test diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/IndexingSpacingAndCountTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/IndexingSpacingAndCountTest.java index 265dfb8378..2ca5c0ce64 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/IndexingSpacingAndCountTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/IndexingSpacingAndCountTest.java @@ -8,6 +8,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertTrue; +@SuppressWarnings({"deprecation", "removal"}) class IndexingSpacingAndCountTest extends IndexingTestCommon { @Test diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/NormaliseEOFsTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/NormaliseEOFsTest.java index 94a4e8876b..b229e460a5 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/NormaliseEOFsTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/NormaliseEOFsTest.java @@ -30,6 +30,7 @@ import static org.junit.jupiter.api.Assertions.assertTrue; +@SuppressWarnings("PMD.JUnit5TestShouldBePackagePrivate") // JUnit4 annotations require public class public class NormaliseEOFsTest extends QueueTestCommon { private static final String LOG_LEVEL_PROPERTY = "org.slf4j.simpleLogger.log." + StoreAppender.class.getName(); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/PartialUpdateTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/PartialUpdateTest.java index efdf4a715c..5ed980d690 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/PartialUpdateTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/PartialUpdateTest.java @@ -22,6 +22,7 @@ import static org.junit.Assert.*; @RunWith(Parameterized.class) +@SuppressWarnings({"deprecation", "removal"}) public class PartialUpdateTest extends QueueTestCommon { private static final long LAST_INDEX = RollCycles.FAST_DAILY.toIndex(2, 2); @@ -70,6 +71,7 @@ public void setUp() { } @After + @Override public void tearDown() { IOTools.deleteDirWithFiles(queuePath.toFile()); } diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/QueueLockTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/QueueLockTest.java index 9c5b5e2f86..fcf5a2ad85 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/QueueLockTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/QueueLockTest.java @@ -53,7 +53,7 @@ private void check(boolean shouldThrowException) throws InterruptedException { expectException("Forced unlock for the lock"); try { - System.setProperty("queue.force.unlock.mode", shouldThrowException ? "NEVER" : "ALWAYS" ); + System.setProperty("queue.force.unlock.mode", shouldThrowException ? "NEVER" : "ALWAYS"); final long timeoutMs = 2_000; final File queueDir = DirectoryUtils.tempDir("check"); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/ReferenceCountedCacheTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/ReferenceCountedCacheTest.java index ab49009806..f7e5d55ee9 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/ReferenceCountedCacheTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/ReferenceCountedCacheTest.java @@ -153,6 +153,7 @@ void assertNotReleased() { } } + @SuppressWarnings("PMD.TestClassWithoutTestCases") private class TestReferenceCounted extends AbstractReferenceCounted implements ReferenceOwner, Closeable { TestReferenceCounted() { diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/RollAtEndOfCycleTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/RollAtEndOfCycleTest.java index 21ba45a469..a67808bdeb 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/RollAtEndOfCycleTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/RollAtEndOfCycleTest.java @@ -21,6 +21,7 @@ import static org.junit.Assert.*; import static org.junit.Assume.assumeFalse; +@SuppressWarnings({"deprecation", "removal"}) public final class RollAtEndOfCycleTest extends QueueTestCommon { private final AtomicLong clock = new AtomicLong(System.currentTimeMillis()); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/SingleCQFormatTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/SingleCQFormatTest.java index 472343acee..8a098ed984 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/SingleCQFormatTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/SingleCQFormatTest.java @@ -34,6 +34,7 @@ import static org.junit.Assert.*; import static org.junit.Assume.assumeFalse; +@SuppressWarnings({"deprecation", "removal"}) public class SingleCQFormatTest extends QueueTestCommon { static { SingleChronicleQueueBuilder.addAliases(); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueBuilderTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueBuilderTest.java index a1eb435c4c..9c44a82f30 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueBuilderTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueBuilderTest.java @@ -32,6 +32,7 @@ import static org.junit.Assert.*; import static org.junit.Assume.assumeFalse; +@SuppressWarnings({"deprecation", "removal"}) public class SingleChronicleQueueBuilderTest extends QueueTestCommon { private static final String TEST_QUEUE_FILE = "src/test/resources/tr2/20170320.cq4"; private static final String BASE_PATH = OS.getTarget() + "/singleChronicleQueueBuilderTest"; diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueStoreTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueStoreTest.java index f418c7f54c..ee6fc4a0c9 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueStoreTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueStoreTest.java @@ -19,6 +19,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +@SuppressWarnings({"deprecation", "removal"}) public class SingleChronicleQueueStoreTest extends QueueTestCommon { private static final int INDEX_SPACING = 4; private static final int RECORD_COUNT = INDEX_SPACING * 10; diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueTest.java index 840b11cc6d..ead1bb48fc 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/SingleChronicleQueueTest.java @@ -58,7 +58,7 @@ import static org.junit.Assume.assumeTrue; @RunWith(Parameterized.class) -@SuppressWarnings({"try", "serial"}) +@SuppressWarnings({"try", "serial", "deprecation", "removal"}) public class SingleChronicleQueueTest extends QueueTestCommon { private static final long TIMES = (4L << 20L); @@ -859,6 +859,7 @@ public void testReadingWritingMarshallableDocument() { } } + // CPD-OFF - metadata scenarios mirror each other @Test public void testMetaData() { assumeFalse(named); @@ -2791,7 +2792,6 @@ public void testCountExceptsBetweenCycles() { long[] indexs = new long[10]; for (int i = 0; i < indexs.length; i++) { - // System.out.println("."); try (DocumentContext writingContext = appender.writingDocument()) { writingContext.wire().write().text("some-text-" + i); indexs[i] = writingContext.index(); @@ -2807,7 +2807,6 @@ else if ((i + 1) % 3 == 0) for (int lower = 0; lower < indexs.length; lower++) { for (int upper = lower; upper < indexs.length; upper++) { - // System.out.println("lower=" + lower + ",upper=" + upper); assertEquals(upper - lower, queue.countExcerpts(indexs[lower], indexs[upper])); } diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/StoreAppenderInternalWriteBytesTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/StoreAppenderInternalWriteBytesTest.java index a65ea8dc6d..f750d9bc93 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/StoreAppenderInternalWriteBytesTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/StoreAppenderInternalWriteBytesTest.java @@ -60,12 +60,6 @@ public void internalWriteBytesShouldBeIdempotent() throws InterruptedException { private void testInternalWriteBytes(int numCopiers, boolean concurrent) throws InterruptedException { final Path sourceDir = IOTools.createTempDirectory("sourceQueue"); final Path destinationDir = IOTools.createTempDirectory("destinationQueue"); - /** - final Path sourceDir = Paths.get("/dev/shm/sourceQueue"); - final Path destinationDir = Paths.get("/dev/shm/destinationQueue"); - IOTools.deleteDirWithFiles(sourceDir.toFile()); - IOTools.deleteDirWithFiles(destinationDir.toFile()); - */ populateSourceQueue(sourceDir); @@ -173,13 +167,11 @@ public void run() { assertEquals(Long.toHexString(index), Long.toHexString(dtIndex)); } prev.clear().append(buffer); -// if (false && index %17 == 0) { try (final ChronicleQueue dq = createQueue(destinationDir, null); final ExcerptAppender da = dq.createAppender()) { assumeNotNull(dq); assumeNotNull(da); } - // } } } } diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/StoreAppenderTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/StoreAppenderTest.java index e0807c1dee..568408aa51 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/StoreAppenderTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/StoreAppenderTest.java @@ -24,6 +24,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.jupiter.api.Assertions.assertThrows; +@SuppressWarnings("PMD.JUnit5TestShouldBePackagePrivate") // JUnit4 annotations require public class public class StoreAppenderTest extends QueueTestCommon { private static final String TEST_TEXT = "Some text some text some text"; diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/StoreTailerTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/StoreTailerTest.java index f3bfee5999..f4ba7bf0a8 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/StoreTailerTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/StoreTailerTest.java @@ -32,7 +32,7 @@ import static org.junit.Assert.*; import static org.junit.Assume.assumeFalse; -@SuppressWarnings("this-escape") +@SuppressWarnings({"this-escape", "deprecation", "removal"}) public class StoreTailerTest extends QueueTestCommon { private final Path dataDirectory = getTmpDir().toPath(); @@ -706,5 +706,4 @@ public void syncShouldReturnNullIfNotInitialised() { assertNull(tailer.currentFile()); } } - } diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/StuckQueueTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/StuckQueueTest.java index 0fae4b5f1e..2298294cd8 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/StuckQueueTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/StuckQueueTest.java @@ -24,6 +24,7 @@ import static org.junit.Assume.assumeFalse; import static org.junit.Assume.assumeNotNull; +@SuppressWarnings({"deprecation", "removal"}) public class StuckQueueTest extends QueueTestCommon { @Test diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/TableStoreWriteLockTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/TableStoreWriteLockTest.java index c22bc88ce5..5a8df9ef62 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/TableStoreWriteLockTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/TableStoreWriteLockTest.java @@ -31,6 +31,7 @@ import static org.junit.Assert.*; +@SuppressWarnings({"deprecation", "removal"}) public class TableStoreWriteLockTest extends QueueTestCommon { private static final String TEST_LOCK_NAME = "testLock"; @@ -53,6 +54,7 @@ public void threadDump() { } @After + @Override public void tearDown() { Closeable.closeQuietly(tableStore); IOTools.deleteDirWithFiles(tempDir.toFile()); @@ -143,7 +145,7 @@ public void unlockWillWarnIfNotLocked() { } @Test(timeout = 5_000) - public void unlockWillNotUnlockAndWarnIfLockedByAnotherProcess() throws IOException, InterruptedException, TimeoutException { + public void unlockWillNotUnlockAndWarnIfLockedByAnotherProcess() throws InterruptedException, TimeoutException { try (final TableStoreWriteLock testLock = createTestLock()) { final Process process = runLockingProcess(true); waitForLockToBecomeLocked(testLock); @@ -156,7 +158,7 @@ public void unlockWillNotUnlockAndWarnIfLockedByAnotherProcess() throws IOExcept } @Test(timeout = 5_000) - public void forceUnlockWillUnlockAndWarnIfLockedByAnotherProcess() throws IOException, InterruptedException, TimeoutException { + public void forceUnlockWillUnlockAndWarnIfLockedByAnotherProcess() throws InterruptedException, TimeoutException { try (final TableStoreWriteLock testLock = createTestLock()) { final Process process = runLockingProcess(true); waitForLockToBecomeLocked(testLock); @@ -209,7 +211,7 @@ public void lockPreventsConcurrentAcquisition() { } @Test(timeout = 5_000) - public void forceUnlockIfProcessIsDeadWillFailWhenLockingProcessIsAlive() throws IOException, TimeoutException, InterruptedException { + public void forceUnlockIfProcessIsDeadWillFailWhenLockingProcessIsAlive() throws TimeoutException, InterruptedException { Process lockingProcess = runLockingProcess(true); try (TableStoreWriteLock lock = createTestLock()) { waitForLockToBecomeLocked(lock); @@ -221,7 +223,7 @@ public void forceUnlockIfProcessIsDeadWillFailWhenLockingProcessIsAlive() throws } @Test(timeout = 5_000) - public void forceUnlockIfProcessIsDeadWillSucceedWhenLockingProcessIsDead() throws IOException, TimeoutException, InterruptedException { + public void forceUnlockIfProcessIsDeadWillSucceedWhenLockingProcessIsDead() throws TimeoutException, InterruptedException { ignoreException("Forced unlock"); Process lockingProcess = runLockingProcess(false); try (TableStoreWriteLock lock = createTestLock()) { diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/TestBinarySearch.java b/src/test/java/net/openhft/chronicle/queue/impl/single/TestBinarySearch.java index 5597a77ba9..fb040f8df6 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/TestBinarySearch.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/TestBinarySearch.java @@ -46,7 +46,7 @@ public static Collection data() { } @Test - public void testBinarySearch() throws ParseException { + public void testBinarySearch() { final SetTimeProvider stp = new SetTimeProvider(); long time = 0; stp.currentTimeMillis(time); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/TestMethodWriterWithThreads.java b/src/test/java/net/openhft/chronicle/queue/impl/single/TestMethodWriterWithThreads.java index 445224f22d..7d02e2f8a1 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/TestMethodWriterWithThreads.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/TestMethodWriterWithThreads.java @@ -7,7 +7,6 @@ import net.openhft.chronicle.core.Jvm; import net.openhft.chronicle.core.io.Closeable; import net.openhft.chronicle.queue.ChronicleQueue; -import net.openhft.chronicle.queue.ExcerptAppender; import net.openhft.chronicle.queue.ExcerptTailer; import net.openhft.chronicle.queue.QueueTestCommon; import net.openhft.chronicle.queue.main.DumpMain; @@ -38,6 +37,7 @@ * check that method writes are thread safe when used with queue.methodWriter */ @RunWith(Parameterized.class) +@SuppressWarnings({"deprecation", "removal"}) public class TestMethodWriterWithThreads extends QueueTestCommon { private static final int AMEND = 1; diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/ToEndInvalidIndexTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/ToEndInvalidIndexTest.java index 5ac4e96d4d..b067706b69 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/ToEndInvalidIndexTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/ToEndInvalidIndexTest.java @@ -31,6 +31,7 @@ public void setUp() throws StreamCorruptedException { } @After + @Override public void tearDown() { IOTools.deleteDirWithFiles(queuePath.toFile()); } @@ -56,7 +57,7 @@ public void testBackwardsToEndReportsCorrectIndex() { */ private static void createQueueWithZeroFirstSubIndexValue(SetTimeProvider setTimeProvider, Path path) throws StreamCorruptedException { try (SingleChronicleQueue queue = createQueue(setTimeProvider, path); - ExcerptAppender appender = queue.createAppender();) { + ExcerptAppender appender = queue.createAppender()) { Bytes bytes = Bytes.allocateElasticDirect(); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/ToEndTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/ToEndTest.java index 0d1d7b0577..a7f9516541 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/ToEndTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/ToEndTest.java @@ -3,7 +3,6 @@ */ package net.openhft.chronicle.queue.impl.single; -import net.openhft.chronicle.bytes.Bytes; import net.openhft.chronicle.bytes.PageUtil; import net.openhft.chronicle.core.Jvm; import net.openhft.chronicle.core.OS; @@ -14,7 +13,6 @@ import net.openhft.chronicle.wire.DocumentContext; import org.jetbrains.annotations.NotNull; import org.junit.Assume; -import org.junit.Ignore; import org.junit.Test; import java.io.File; @@ -28,6 +26,7 @@ import static net.openhft.chronicle.queue.rollcycles.TestRollCycles.TEST_DAILY; import static org.junit.Assert.*; +@SuppressWarnings({"deprecation", "removal"}) public class ToEndTest extends QueueTestCommon { private static final long FIVE_SECONDS = SECONDS.toMicros(5); private static final String ZERO_AS_HEX_STRING = Long.toHexString(0); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/jira/Queue28Test.java b/src/test/java/net/openhft/chronicle/queue/impl/single/jira/Queue28Test.java index a7ed54b168..5009bea865 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/jira/Queue28Test.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/jira/Queue28Test.java @@ -40,7 +40,6 @@ public static Collection data() { * * See https://higherfrequencytrading.atlassian.net/browse/QUEUE-28 */ - @Test public void test() { File dir = getTmpDir(); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressDoubleBufferTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressDoubleBufferTest.java index eec5b17c20..c1ddfbb862 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressDoubleBufferTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressDoubleBufferTest.java @@ -11,12 +11,12 @@ import net.openhft.chronicle.wire.ValueIn; import org.junit.Before; import org.junit.Ignore; -import org.junit.Test; import java.util.HashSet; import java.util.concurrent.atomic.AtomicBoolean; @Ignore("double buffering is turned off currently") +@SuppressWarnings({"deprecation", "removal", "PMD.TestClassWithoutTestCases"}) public class RollCycleMultiThreadStressDoubleBufferTest extends RollCycleMultiThreadStressTest { private AtomicBoolean queueDumped = new AtomicBoolean(false); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressPretouchEATest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressPretouchEATest.java index 22e460dc6d..1dcf466230 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressPretouchEATest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressPretouchEATest.java @@ -16,6 +16,7 @@ public RollCycleMultiThreadStressPretouchEATest() { } @Test + @Override public void stress() throws Exception { Assume.assumeTrue(SingleChronicleQueueBuilder.areEnterpriseFeaturesAvailable()); super.stress(); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressPretouchTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressPretouchTest.java index 0d7a3baf90..e6c5b7e593 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressPretouchTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressPretouchTest.java @@ -16,6 +16,7 @@ public RollCycleMultiThreadStressPretouchTest() { } @Test + @Override public void stress() throws Exception { Assume.assumeTrue(SingleChronicleQueueBuilder.areEnterpriseFeaturesAvailable()); super.stress(); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressReadOnlyTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressReadOnlyTest.java index 6ccf6417ec..20e0f1ba5b 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressReadOnlyTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressReadOnlyTest.java @@ -16,6 +16,7 @@ public RollCycleMultiThreadStressReadOnlyTest() { } @Test + @Override public void stress() throws Exception { Assume.assumeFalse("Windows does not support read only", OS.isWindows()); super.stress(); diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressSharedWriterQueueTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressSharedWriterQueueTest.java index f40913c454..adfa58c397 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressSharedWriterQueueTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressSharedWriterQueueTest.java @@ -3,8 +3,7 @@ */ package net.openhft.chronicle.queue.impl.single.stress; -import org.junit.Test; - +@SuppressWarnings("PMD.TestClassWithoutTestCases") public class RollCycleMultiThreadStressSharedWriterQueueTest extends RollCycleMultiThreadStressTest { public RollCycleMultiThreadStressSharedWriterQueueTest() { diff --git a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressTest.java b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressTest.java index 98d8c862cb..72e7ce1433 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressTest.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/single/stress/RollCycleMultiThreadStressTest.java @@ -39,6 +39,7 @@ import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; +@SuppressWarnings({"deprecation", "removal"}) public class RollCycleMultiThreadStressTest extends QueueTestCommon { private final long SLEEP_PER_WRITE_NANOS; diff --git a/src/test/java/net/openhft/chronicle/queue/impl/table/SingleTableStoreIntegrationTests.java b/src/test/java/net/openhft/chronicle/queue/impl/table/SingleTableStoreIntegrationTests.java index 774faa9ee3..30a023f747 100644 --- a/src/test/java/net/openhft/chronicle/queue/impl/table/SingleTableStoreIntegrationTests.java +++ b/src/test/java/net/openhft/chronicle/queue/impl/table/SingleTableStoreIntegrationTests.java @@ -20,6 +20,7 @@ import static org.junit.Assert.assertEquals; +@SuppressWarnings({"deprecation", "removal"}) public class SingleTableStoreIntegrationTests extends QueueTestCommon { private TestContext context; @@ -87,6 +88,7 @@ public void longKeyPutAndGet() { assertEquals(1, context.newQueueInstance().tableStoreGet(key)); } + @SuppressWarnings("PMD.TestClassWithoutTestCases") class TestContext implements Closeable { private final File queuePath = getTmpDir(); @@ -102,7 +104,7 @@ SingleChronicleQueue newQueueInstance() { } @Override - public void close() throws IOException { + public void close() { queues.forEach(net.openhft.chronicle.core.io.Closeable::closeQuietly); IOTools.deleteDirWithFiles(queuePath); } diff --git a/src/test/java/net/openhft/chronicle/queue/internal/domestic/QueueOffsetSpecTest.java b/src/test/java/net/openhft/chronicle/queue/internal/domestic/QueueOffsetSpecTest.java index 9092416444..008efb1310 100644 --- a/src/test/java/net/openhft/chronicle/queue/internal/domestic/QueueOffsetSpecTest.java +++ b/src/test/java/net/openhft/chronicle/queue/internal/domestic/QueueOffsetSpecTest.java @@ -11,6 +11,7 @@ import static org.junit.Assert.*; +@SuppressWarnings({"deprecation", "removal"}) public class QueueOffsetSpecTest { @Test diff --git a/src/test/java/net/openhft/chronicle/queue/internal/main/InternalDumpMainTest.java b/src/test/java/net/openhft/chronicle/queue/internal/main/InternalDumpMainTest.java index 3c17180b10..5ed28e5e8e 100644 --- a/src/test/java/net/openhft/chronicle/queue/internal/main/InternalDumpMainTest.java +++ b/src/test/java/net/openhft/chronicle/queue/internal/main/InternalDumpMainTest.java @@ -15,7 +15,6 @@ import java.io.PrintStream; import java.nio.file.Files; import java.nio.file.Path; -import java.util.stream.Stream; import static org.junit.Assert.assertNotEquals; import static org.junit.Assert.assertTrue; @@ -50,4 +49,3 @@ public void shouldDumpDirectoryAndIncludeMetadataAndQueueFiles() throws Exceptio assertNotEquals(0, captureFile.size()); } } - diff --git a/src/test/java/net/openhft/chronicle/queue/internal/reader/ChronicleMethodReaderTest.java b/src/test/java/net/openhft/chronicle/queue/internal/reader/ChronicleMethodReaderTest.java index bc18419c6f..83d7b50e4a 100644 --- a/src/test/java/net/openhft/chronicle/queue/internal/reader/ChronicleMethodReaderTest.java +++ b/src/test/java/net/openhft/chronicle/queue/internal/reader/ChronicleMethodReaderTest.java @@ -90,6 +90,7 @@ private ChronicleReader basicReaderMethodReader(Path path) { return basicReader(path).asMethodReader(All.class.getName()); } + // CPD-OFF - ChronicleReaderTest exercises the same behaviours @Test public void shouldNotFailWhenNoMetadata() throws IOException { if (!OS.isWindows()) @@ -211,6 +212,7 @@ public void shouldFilterByMultipleExclusionRegex() { assertEquals(0L, capturedOutput.stream().filter(msg -> !msg.startsWith("0x")).count()); } + // CPD-ON @Test public void shouldReturnNoMoreThanTheSpecifiedNumberOfMaxRecords() { diff --git a/src/test/java/net/openhft/chronicle/queue/internal/reader/ChronicleReaderTest.java b/src/test/java/net/openhft/chronicle/queue/internal/reader/ChronicleReaderTest.java index 64d5f31e60..b6cde103e6 100644 --- a/src/test/java/net/openhft/chronicle/queue/internal/reader/ChronicleReaderTest.java +++ b/src/test/java/net/openhft/chronicle/queue/internal/reader/ChronicleReaderTest.java @@ -53,6 +53,7 @@ import static org.junit.Assert.*; import static org.junit.Assume.assumeFalse; +@SuppressWarnings({"deprecation", "removal"}) public class ChronicleReaderTest extends QueueTestCommon { private static final byte[] ONE_KILOBYTE = new byte[1024]; private static final long TOTAL_EXCERPTS_IN_QUEUE = 24; @@ -187,6 +188,7 @@ private List addCountToEndOfQueue() { return indices; } + // CPD-OFF - duplicated setup for metadata deletion variants @Test(timeout = 10_000L) public void shouldReadQueueWithNonDefaultRollCycle() { expectException("Overriding roll length from existing metadata"); @@ -228,6 +230,7 @@ public void shouldReadQueueWithNonDefaultRollCycleWhenMetadataDeleted() throws I new ChronicleReader().withBasePath(path).withMessageSink(capturedOutput::add).execute(); assertFalse(capturedOutput.isEmpty()); } + // CPD-ON @Test public void shouldNotFailOnEmptyQueue() { @@ -284,13 +287,6 @@ public void shouldNotShowIndexForHistoryMessages() { } } -// basicReader() -// .asMethodReader(SayWhen.class.getName()) -// .execute(); -// -// assertTrue(capturedOutput.isEmpty()); -// } - @Test public void canReadPastEmptyMessageInReverseOrder() { dataDir = getTmpDir().toPath(); diff --git a/src/test/java/net/openhft/chronicle/queue/internal/reader/RollEOFTest.java b/src/test/java/net/openhft/chronicle/queue/internal/reader/RollEOFTest.java index 78921dbcee..4c3be57194 100644 --- a/src/test/java/net/openhft/chronicle/queue/internal/reader/RollEOFTest.java +++ b/src/test/java/net/openhft/chronicle/queue/internal/reader/RollEOFTest.java @@ -43,6 +43,7 @@ import static org.junit.Assert.*; import static org.junit.Assume.assumeFalse; +@SuppressWarnings({"deprecation", "removal"}) public class RollEOFTest extends QueueTestCommon { private static final ReferenceOwner test = ReferenceOwner.temporary("test"); diff --git a/src/test/java/net/openhft/chronicle/queue/internal/util/InternalFileUtilLinuxStateTest.java b/src/test/java/net/openhft/chronicle/queue/internal/util/InternalFileUtilLinuxStateTest.java index c2ecb54eb3..e3d529271d 100644 --- a/src/test/java/net/openhft/chronicle/queue/internal/util/InternalFileUtilLinuxStateTest.java +++ b/src/test/java/net/openhft/chronicle/queue/internal/util/InternalFileUtilLinuxStateTest.java @@ -11,7 +11,6 @@ import java.io.File; import java.io.RandomAccessFile; -import java.nio.channels.FileChannel; import java.util.Map; import static org.junit.Assert.assertEquals; diff --git a/src/test/java/net/openhft/chronicle/queue/issue/DeleteFileTest.java b/src/test/java/net/openhft/chronicle/queue/issue/DeleteFileTest.java index e02dc9e117..11ab57f597 100644 --- a/src/test/java/net/openhft/chronicle/queue/issue/DeleteFileTest.java +++ b/src/test/java/net/openhft/chronicle/queue/issue/DeleteFileTest.java @@ -20,6 +20,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +@SuppressWarnings({"deprecation", "removal"}) public class DeleteFileTest { @Test public void testMain() { diff --git a/src/test/java/net/openhft/chronicle/queue/issue/ReaderResizesFileTest.java b/src/test/java/net/openhft/chronicle/queue/issue/ReaderResizesFileTest.java index cfb5be2159..4a81038bfb 100644 --- a/src/test/java/net/openhft/chronicle/queue/issue/ReaderResizesFileTest.java +++ b/src/test/java/net/openhft/chronicle/queue/issue/ReaderResizesFileTest.java @@ -26,6 +26,7 @@ import static org.junit.jupiter.api.Assertions.*; +@SuppressWarnings({"deprecation", "removal"}) public class ReaderResizesFileTest { private static final File QUEUE_DIR = new File(OS.getTarget(), "ReaderResizesFileTest-" + System.nanoTime()); @@ -79,7 +80,7 @@ public void testReaderResizesFile() throws IOException { } @Test - public void testTailerRefCountStableDuringResize() throws IOException { + public void testTailerRefCountStableDuringResize() { int blockSize = 1 << 12; try (SingleChronicleQueue queue = ChronicleQueue.singleBuilder(QUEUE_DIR) .rollCycle(TestRollCycles.TEST4_DAILY) @@ -121,7 +122,7 @@ public void testTailerRefCountStableDuringResize() throws IOException { } @Test - public void testTailerHoldingDocumentAcrossRollsDoesNotResizeOldCycle() throws IOException { + public void testTailerHoldingDocumentAcrossRollsDoesNotResizeOldCycle() { File queuePath = new File(QUEUE_DIR, "tailer-hold"); SetTimeProvider timeProvider = new SetTimeProvider(); timeProvider.currentTimeMillis(0L); diff --git a/src/test/java/net/openhft/chronicle/queue/issue/TailerTest.java b/src/test/java/net/openhft/chronicle/queue/issue/TailerTest.java index cbf8f73edc..1f5bed73f1 100644 --- a/src/test/java/net/openhft/chronicle/queue/issue/TailerTest.java +++ b/src/test/java/net/openhft/chronicle/queue/issue/TailerTest.java @@ -21,6 +21,7 @@ import static org.junit.Assert.assertEquals; +@SuppressWarnings({"deprecation", "removal"}) public class TailerTest extends QueueTestCommon { private static final Path QUEUE_PATH = Paths.get(OS.getTarget() + "/host-1/queue/broker_out"); diff --git a/src/test/java/net/openhft/chronicle/queue/jitter/QueueWriteJitterMain.java b/src/test/java/net/openhft/chronicle/queue/jitter/QueueWriteJitterMain.java index c81267b3f9..83ebc528a7 100644 --- a/src/test/java/net/openhft/chronicle/queue/jitter/QueueWriteJitterMain.java +++ b/src/test/java/net/openhft/chronicle/queue/jitter/QueueWriteJitterMain.java @@ -14,6 +14,7 @@ import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder; import net.openhft.chronicle.wire.DocumentContext; +@SuppressWarnings("PMD.UnusedAssignment") // writeStarted is read from another thread while being updated public class QueueWriteJitterMain { private static final String PROFILE_OF_THE_THREAD = "profile of the thread"; diff --git a/src/test/java/net/openhft/chronicle/queue/main/DumpMainTest.java b/src/test/java/net/openhft/chronicle/queue/main/DumpMainTest.java index 0bedbb65e9..8a504dc981 100644 --- a/src/test/java/net/openhft/chronicle/queue/main/DumpMainTest.java +++ b/src/test/java/net/openhft/chronicle/queue/main/DumpMainTest.java @@ -7,7 +7,6 @@ import net.openhft.chronicle.queue.QueueTestCommon; import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder; import net.openhft.chronicle.wire.DocumentContext; -import net.openhft.chronicle.wire.WireType; import org.junit.Test; import java.io.ByteArrayOutputStream; @@ -17,6 +16,7 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; +@SuppressWarnings({"deprecation", "removal"}) public class DumpMainTest extends QueueTestCommon { @Test diff --git a/src/test/java/net/openhft/chronicle/queue/micros/Order.java b/src/test/java/net/openhft/chronicle/queue/micros/Order.java index 62b0274b9d..780ab3a9d3 100644 --- a/src/test/java/net/openhft/chronicle/queue/micros/Order.java +++ b/src/test/java/net/openhft/chronicle/queue/micros/Order.java @@ -3,8 +3,10 @@ */ package net.openhft.chronicle.queue.micros; +import net.openhft.chronicle.core.annotation.UsedViaReflection; import net.openhft.chronicle.wire.SelfDescribingMarshallable; +@UsedViaReflection public class Order extends SelfDescribingMarshallable { private final String symbol; private final Side side; diff --git a/src/test/java/net/openhft/chronicle/queue/micros/OrderManagerTest.java b/src/test/java/net/openhft/chronicle/queue/micros/OrderManagerTest.java index 7938ec52e4..18416395b5 100644 --- a/src/test/java/net/openhft/chronicle/queue/micros/OrderManagerTest.java +++ b/src/test/java/net/openhft/chronicle/queue/micros/OrderManagerTest.java @@ -55,6 +55,7 @@ public void testOnOrderIdea() { verify(listener); } + // CPD-OFF - queue-backed scenarios mirror each other @Test public void testWithQueue() { File queuePath = new File(OS.getTarget(), "testWithQueue-" + Time.uniqueId()); @@ -236,4 +237,5 @@ public void testRestartingAService() { } } } + // CPD-ON } diff --git a/src/test/java/net/openhft/chronicle/queue/namedtailer/NamedTailerPreconditionTest.java b/src/test/java/net/openhft/chronicle/queue/namedtailer/NamedTailerPreconditionTest.java index d7f1d07f34..20dd22df4a 100644 --- a/src/test/java/net/openhft/chronicle/queue/namedtailer/NamedTailerPreconditionTest.java +++ b/src/test/java/net/openhft/chronicle/queue/namedtailer/NamedTailerPreconditionTest.java @@ -17,6 +17,7 @@ import static org.junit.Assert.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; +@SuppressWarnings({"deprecation", "removal"}) public class NamedTailerPreconditionTest extends QueueTestCommon { @Test diff --git a/src/test/java/net/openhft/chronicle/queue/util/FileUtilTest.java b/src/test/java/net/openhft/chronicle/queue/util/FileUtilTest.java index bd244c0eb4..947f122db6 100644 --- a/src/test/java/net/openhft/chronicle/queue/util/FileUtilTest.java +++ b/src/test/java/net/openhft/chronicle/queue/util/FileUtilTest.java @@ -34,6 +34,7 @@ import static org.junit.Assert.*; import static org.junit.Assume.assumeTrue; +@SuppressWarnings({"deprecation", "removal"}) public class FileUtilTest extends QueueTestCommon { @Test(timeout = 30_000) diff --git a/src/test/java/net/openhft/chronicle/queue/util/PretouchUtilTest.java b/src/test/java/net/openhft/chronicle/queue/util/PretouchUtilTest.java index a84e5e2c9e..81a89706cd 100644 --- a/src/test/java/net/openhft/chronicle/queue/util/PretouchUtilTest.java +++ b/src/test/java/net/openhft/chronicle/queue/util/PretouchUtilTest.java @@ -6,7 +6,6 @@ import net.openhft.chronicle.core.threads.EventHandler; import net.openhft.chronicle.queue.ChronicleQueue; import net.openhft.chronicle.queue.QueueTestCommon; -import net.openhft.chronicle.queue.impl.single.Pretoucher; import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder; import org.junit.Test; From 7159be301b025da5f0d8dd0a296c74a08b9f553d Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Thu, 11 Dec 2025 14:28:03 +0000 Subject: [PATCH 4/6] Add documentation files for AI agent guidelines and project overview --- AGENTS.md | 209 +++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 279 ++++++++++++++++++++++++++++++++++++++++++++++++++++ GEMINI.md | 66 +++++++++++++ README.adoc | 150 ++++++++++++++-------------- pom.xml | 2 +- 5 files changed, 631 insertions(+), 75 deletions(-) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md create mode 100644 GEMINI.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000..342f7ad78e --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,209 @@ +# Guidance for AI agents, bots, and humans contributing to Chronicle Software's OpenHFT projects. + +LLM-based agents can accelerate development only if they respect our house rules. This file tells you: + +* how to run and verify the build; +* what *not* to comment; +* when to open pull requests. + +## Language & character-set policy + +| Requirement | Rationale | +|--------------|-----------| +| **British English** spelling (`organisation`, `licence`, *not* `organization`, `license`) except technical US spellings like `synchronized` | Keeps wording consistent with Chronicle's London HQ and existing docs. See the [University of Oxford style guide](https://www.ox.ac.uk/public-affairs/style-guide) for reference. | +| **ISO-8859-1** (code-points 0-255). Avoid smart quotes, non-breaking spaces and accented characters. | ISO-8859-1 survives every toolchain Chronicle uses. | +| If a symbol is not available in ISO-8859-1, use a textual form such as `>=`, `:alpha:`, `:yes:`. This is the preferred approach and Unicode must not be inserted. | Extended or '8-bit ASCII' variants are *not* portable and are therefore disallowed. | +| Tools to check ASCII compliance include `iconv -f ascii -t ascii` and IDE settings that flag non-ASCII characters. | These help catch stray Unicode characters before code review. | + +## Javadoc guidelines + +**Goal:** Every Javadoc block should add information you cannot glean from the method signature alone. Anything else is +noise and slows readers down. + +| Do | Don't | +|----|-------| +| State *behavioural contracts*, edge-cases, thread-safety guarantees, units, performance characteristics and checked exceptions. | Restate the obvious ("Gets the value", "Sets the name"). | +| Keep the first sentence short; it becomes the summary line in aggregated docs. | Duplicate parameter names/ types unless more explanation is needed. | +| Prefer `@param` for *constraints* and `@throws` for *conditions*, following Oracle's style guide. | Pad comments to reach a line-length target. | +| Remove or rewrite autogenerated Javadoc for trivial getters/setters. | Leave stale comments that now contradict the code. | + +The principle that Javadoc should only explain what is *not* manifest from the +signature is well-established in the wider Java community. + +Inline comments should also avoid noise. The following example shows the +difference: + +```java +// BAD: adds no value +int count; // the count + +// GOOD: explains a subtlety +// count of messages pending flush +int count; +``` + +## Build & test commands + +Agents must verify that the project still compiles and all unit tests pass before opening a PR. Running from a clean checkout avoids stale artifacts: + +```bash +# From repo root +mvn -q verify +``` + +## Commit-message & PR etiquette + +1. **Subject line <= 72 chars**, imperative mood: Fix roll-cycle offset in `ExcerptAppender`. +2. Reference the JIRA/GitHub issue if it exists. +3. In *body*: *root cause -> fix -> measurable impact* (latency, allocation, etc.). Use ASCII bullet points. +4. **Run `mvn verify`** again after rebasing. + +### When to open a PR + +* Open a pull request once your branch builds and tests pass with `mvn -q clean verify`. +* Link the PR to the relevant issue or decision record. +* Keep PRs focused: avoid bundling unrelated refactoring with new features. +* Re-run the build after addressing review comments to ensure nothing broke. + +## What to ask the reviewers + +* *Is this AsciiDoc documentation precise enough for a clean-room re-implementation?* +* Does the Javadoc explain the code's *why* and *how* that a junior developer would not be expected to work out? +* Are the documentation, tests and code updated together so the change is clear? +* Does the commit point back to the relevant requirement or decision tag? +* Would an example or small diagram help future maintainers? + +### Security checklist (review **after every change**) + +**Run a security review on *every* PR**: Walk through the diff looking for input validation, authentication, authorisation, encoding/escaping, overflow, resource exhaustion and timing-attack issues. + +**Never commit secrets or credentials**: tokens, passwords, private keys, TLS materials, internal hostnames, Use environment variables, HashiCorp Vault, AWS/GCP Secret Manager, etc. + +**Document security trade-offs**: Chronicle prioritises low-latency systems; sometimes we relax safety checks for specific reasons. Future maintainers must find these hot-spots quickly, In Javadoc and `.adoc` files call out *why* e.g. "Unchecked cast for performance - assumes trusted input". + +## Project requirements + +See the [Decision Log](src/main/docs/decision-log.adoc) for the latest project decisions. +See the [Project Requirements](src/main/docs/project-requirements.adoc) for details on project requirements. +See the [Antora Docs](docs/antora/README.md) for Antora documentation guidelines. + +## Elevating the Workflow with Real-Time Documentation + +Building upon our existing Iterative Workflow, the newest recommendation is to emphasise *real-time updates* to documentation. +Ensure the relevant `.adoc` files are updated when features, requirements, implementation details, or tests change. +This tight loop informs the AI accurately and creates immediate clarity for all team members. + +### Benefits of Real-Time Documentation + +* **Confidence in documentation**: Accurate docs prevent miscommunications that derail real-world outcomes. +* **Reduced drift**: Real-time updates keep requirements, tests and code aligned. +* **Faster feedback**: AI can quickly highlight inconsistencies when everything is in sync. +* **Better quality**: Frequent checks align the implementation with the specified behaviour. +* **Smoother onboarding**: Up-to-date AsciiDoc clarifies the system for new developers. +* **Incremental changes**: AIDE flags newly updated files so you can keep the documentation synchronised. + +### Best Practices + +* **Maintain Sync**: Keep documentation (AsciiDoc), tests, and code synchronised in version control. Changes in one area should prompt reviews and potential updates in the others. +* **Doc-First for New Work**: For *new* features or requirements, aim to update documentation first, then use AI to help produce or refine corresponding code and tests. For refactoring or initial bootstrapping, updates might flow from code/tests back to documentation, which should then be reviewed and finalised. +* **Small Commits**: Each commit should ideally relate to a single requirement or coherent change, making reviews easier for humans and AI analysis tools. +- **Team Buy-In**: Encourage everyone to review AI outputs critically and contribute to maintaining the synchronicity of all artefacts. + +## AI Agent Guidelines + +When using AI agents to assist with development, please adhere to the following guidelines: + +* **Respect the Language & Character-set Policy**: Ensure all AI-generated content follows the British English and ISO-8859-1 guidelines outlined above. + Focus on Clarity: AI-generated documentation should be clear and concise and add value beyond what is already present in the code or existing documentation. +* **Avoid Redundancy**: Do not generate content that duplicates existing documentation or code comments unless it provides additional context or clarification. +* **Review AI Outputs**: Always review AI-generated content for accuracy, relevance, and adherence to the project's documentation standards before committing it to the repository. + +## Company-Wide Tagging + +This section records **company-wide** decisions that apply to *all* Chronicle projects. All identifiers use the --xxx prefix. The `xxx` are unique across in the same Scope even if the tags are different. Component-specific decisions live in their xxx-decision-log.adoc files. + +### Tag Taxonomy (Nine-Box Framework) + +To improve traceability, we adopt the Nine-Box taxonomy for requirement and decision identifiers. These tags are used in addition to the existing ALL prefix, which remains reserved for global decisions across every project. + +.Adopt a Nine-Box Requirement Taxonomy + +|Tag | Scope | Typical examples | +|----|-------|------------------| +|FN |Functional user-visible behaviour | Message routing, business rules | +|NF-P |Non-functional - Performance | Latency budgets, throughput targets | +|NF-S |Non-functional - Security | Authentication method, TLS version | +|NF-O |Non-functional - Operability | Logging, monitoring, health checks | +|TEST |Test / QA obligations | Chaos scenarios, benchmarking rigs | +|DOC |Documentation obligations | Sequence diagrams, user guides | +|OPS |Operational / DevOps concerns | Helm values, deployment checklist | +|UX |Operator or end-user experience | CLI ergonomics, dashboard layouts | +|RISK |Compliance / risk controls | GDPR retention, audit trail | + +`ALL-*` stays global, case-exact tags. Pick one primary tag if multiple apply. + +### Decision Record Template + +```asciidoc +=== [Identifier] Title of Decision + +Date:: YYYY-MM-DD +Context:: +* What is the issue that this decision addresses? +* What are the driving forces, constraints, and requirements? +Decision Statement :: What is the change that is being proposed or was decided? +Alternatives Considered:: +* [Alternative 1 Name/Type]: +** *Description:* Brief description of the alternative. +** *Pros:* ... +** *Cons:* ... +* [Alternative 2 Name/Type]: +** *Description:* Brief description of the alternative. +** *Pros:* ... +** *Cons:* ... +Rationale for Decision:: +* Why was the chosen decision selected? +* How does it address the context and outweigh the cons of alternatives? +Impact & Consequences:: +* What are the positive and negative consequences of this decision? +* How does this decision affect the system, developers, users, or operations? +- What are the trade-offs made? +Notes/Links:: +** (Optional: Links to relevant issues, discussions, documentation, proof-of-concepts) +``` + +## Asciidoc formatting guidelines + +### List Indentation + +Do not rely on indentation for list items in AsciiDoc documents. Use the following pattern instead: + +```asciidoc +section :: Top Level Section (Optional) +* first level + ** nested level +``` + +### Emphasis and Bold Text + +In AsciiDoc, an underscore `_` is _emphasis_; `*text*` is *bold*. + +### Section Numbering + +Use automatic section numbering for all `.adoc` files. + +* Add `:sectnums:` to the document header. +* Do not prefix section titles with manual numbers to avoid duplication. + +```asciidoc += Document Title +Chronicle Software +:toc: +:sectnums: +:lang: en-GB +:source-highlighter: rouge + +The document overview goes here. + +== Section 1 Title +``` diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000000..89dc3a2632 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,279 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Repository Overview + +Chronicle Queue is a persisted low-latency messaging framework for high-performance Java applications. It provides broker-less, off-heap messaging with microsecond latencies and supports millions of events per second. + +**Key architectural principles:** +- **Off-heap storage**: Uses memory-mapped files to avoid GC pressure +- **Zero-allocation design**: Performance-critical paths avoid object allocation +- **Single Writer Principle**: Core concurrency pattern - multiple concurrent readers, write lock for appenders +- **Flyweight pattern**: Objects act as views over underlying `Bytes` data + +## Build Commands + +### Basic Build +```bash +# Standard build with tests +mvn clean install + +# Build without tests (faster) +mvn clean install -DskipTests + +# Package (creates uber-jar for command-line tools) +mvn package +``` + +### Running Tests +```bash +# Run all tests +mvn test + +# Run single test class +mvn test -Dtest=ClassName + +# Run specific test method +mvn test -Dtest=ClassName#methodName +``` + +### Benchmarks +```bash +# Run all benchmarks +mvn test -Prun-benchmarks + +# Run specific benchmark classes (via exec-maven-plugin) +mvn exec:java -Dexec.mainClass="net.openhft.chronicle.queue.main.BenchmarkMain" +``` + +### Command-Line Tools + +After `mvn package`, use the shell scripts in `bin/`: + +```bash +# Read queue contents +./bin/queue_reader.sh -d + +# Dump queue as text +./bin/dump_queue.sh + +# Write to queue (MethodWriter) +./bin/queue_writer.sh -d -m + +# View message history +./bin/history_reader.sh -d + +# Unlock a queue +./bin/unlock_queue.sh + +# Refresh queue metadata +./bin/refresh_queue.sh +``` + +## Code Architecture + +### Core Abstractions + +**Queue Creation:** +- `SingleChronicleQueue` - Main queue implementation +- `SingleChronicleQueueBuilder` - Builder for queue configuration +- Uses file-per-cycle model (configurable: DAILY, HOURLY, MINUTELY, etc.) + +**Writing:** +- `ExcerptAppender` - Thread-local appender for writing messages +- `acquireAppender()` - Reuses thread-local appender (pooled) +- `DocumentContext` - Try-with-resources context for writing + +**Reading:** +- `ExcerptTailer` - Reader for sequential/random access +- `createTailer()` - Creates unnamed tailer +- `createTailer(name)` - Creates restartable/named tailer (persists position) +- `MethodReader` - High-level API to convert messages to method calls + +**File Structure:** +- Queue directory contains `.cq4` files (one per cycle) +- `metadata.cq4t` - Table store containing metadata and write lock +- Roll cycles determine file names (e.g., `20231119.cq4` for DAILY) + +### Key Packages + +``` +net.openhft.chronicle.queue/ +├── ChronicleQueue.java # Main interface +├── ExcerptAppender.java # Writer interface +├── ExcerptTailer.java # Reader interface +├── RollCycles.java # File rolling configuration +├── impl/ +│ └── single/ # SingleChronicleQueue implementation +│ ├── SingleChronicleQueue.java +│ ├── SingleChronicleQueueBuilder.java +│ ├── StoreTailer.java # Tailer implementation +│ ├── StoreAppender.java # Appender implementation +│ ├── SCQIndexing.java # Excerpt indexing +│ └── Pretoucher.java # Pre-touches pages for low latency +├── main/ # Command-line tools +│ ├── DumpMain.java +│ ├── ReaderMain.java # ChronicleReaderMain +│ └── ChronicleWriterMain.java +└── reader/ # Reader framework +``` + +### Critical Implementation Details + +**Locking Model:** +- Write lock stored in `metadata.cq4t` table store (changed from v4) +- Appenders acquire lock per write via `TableStoreWriteLock` +- Tailers are read-only (no lazy indexing in v5) + +**Indexing:** +- Primary index: cycle number (e.g., days since epoch) + sequence number +- Index format: `((long) cycle << 32) | sequenceNumber` for DAILY +- Secondary indexes at configurable spacing (`indexSpacing`, `indexCount`) +- Use `SCQIndexing` for index management + +**Wire Formats:** +- Default: `BINARY_LIGHT` (self-describing binary) +- Also supports: `FIELDLESS_BINARY`, `DEFAULT_ZERO_BINARY`, `DELTA_BINARY` +- NOT supported: TEXT, JSON, CSV, RAW + +**Resource Management:** +- Memory-mapped files are cached and released lazily +- Call `queue.close()` to release resources explicitly +- Tailers: `((StoreTailer)tailer).releaseResources()` for explicit cleanup +- Uses `WeakReferenceCleaner` for off-heap memory tracking + +## Testing Patterns + +### Resource Cleanup +All tests must verify off-heap resources are released: +```java +// From chronicle-test-framework +@Test +public void testExample() { + expectException("optional expected exception message"); + try (ChronicleQueue queue = SingleChronicleQueueBuilder.single(tmpDir).build()) { + // test code + } + // assertReferencesReleased() - inherited from test base class +} +``` + +### Common Test Utilities +- `QueueTestCommon` - Base class for queue tests +- `OS.getTarget()` - Gets temp directory for test files +- `IOTools.deleteDirWithFiles()` - Cleanup test directories + +### Test Organization +- `src/test/java/net/openhft/chronicle/queue/impl/single/` - Core implementation tests +- `src/test/java/net/openhft/chronicle/queue/bench/` - JLBH benchmarks +- `src/test/java/net/openhft/chronicle/queue/jitter/` - Jitter measurement tests +- `src/test/java/net/openhft/chronicle/queue/issue/` - Regression tests for specific issues + +## Common Development Tasks + +### Running a Single Test +```bash +cd /home/peter/Second/Chronicle-Queue +mvn test -Dtest=SingleChronicleQueueTest#testAppendAndRead +``` + +### Dumping Queue Contents +```java +// Programmatically +System.out.println(queue.dump()); + +// Command line (after mvn package) +./bin/dump_queue.sh queue-dir/20231119.cq4 +``` + +### Creating and Using a Queue +```java +// Create queue +try (ChronicleQueue queue = SingleChronicleQueueBuilder + .single("queue-path") + .rollCycle(RollCycles.DAILY) + .build()) { + + // Write + ExcerptAppender appender = queue.acquireAppender(); + try (DocumentContext dc = appender.writingDocument()) { + dc.wire().write("msg").text("Hello World"); + } + + // Read + ExcerptTailer tailer = queue.createTailer(); + try (DocumentContext dc = tailer.readingDocument()) { + if (dc.isPresent()) { + String msg = dc.wire().read("msg").text(); + } + } +} +``` + +### Using Method Reader/Writer (High-level API) +```java +interface MyInterface { + void onMessage(String text); +} + +// Writing +MyInterface writer = appender.methodWriter(MyInterface.class); +writer.onMessage("Hello"); + +// Reading +MyInterface processor = msg -> System.out.println("Got: " + msg); +MethodReader reader = tailer.methodReader(processor); +while (reader.readOne()) { + // Messages converted to method calls on processor +} +``` + +## Important Notes + +1. **Network file systems not supported**: Chronicle Queue requires local file systems. Do not use NFS, AFS, or SAN storage. + +2. **Package visibility**: Code in `internal/`, `impl/`, and `main/` packages is NOT public API and may change without notice. + +3. **Roll cycle immutability**: Once a queue's roll cycle is set, it cannot be changed. Attempting to open with different roll cycle will log warning and use existing cycle. + +4. **Interrupts**: Chronicle Queue code does not check for thread interrupts for performance reasons. Avoid using with code that generates interrupts, or use separate queue instances per thread. + +5. **Index availability**: When using double-buffering, `DocumentContext.index()` throws `IndexNotAvailableException` until the context is closed. + +6. **File handles**: Chronicle Queue caches file handles. Close flushes data to disk. Use `Pretoucher` to pre-touch pages for lowest latency. + +## Key Configuration Parameters + +- `rollCycle()` - File rotation frequency (DAILY, HOURLY, etc.) +- `blockSize()` - Memory mapping block size (default 64MB, should be 4x message size) +- `indexSpacing()` - Space between indexed excerpts (higher = faster writes, slower random reads) +- `indexCount()` - Index array size (max indexed entries = indexCount²) +- `readBufferMode()` / `writeBufferMode()` - Buffering strategy (None, Copy, Asynchronous) +- `sourceId()` - Enables high-resolution timing across messages +- `doubleBuffer()` - Enables double-buffering for contended writes + +## Dependencies + +Core Chronicle dependencies (from pom.xml): +- `chronicle-core` - Low-level utilities, resource management +- `chronicle-bytes` - Off-heap memory access +- `chronicle-wire` - Serialization framework +- `chronicle-threads` - Low-latency thread management +- `affinity` - Thread affinity (optional) + +## Performance Considerations + +- Default: 99%ile latency under 1μs for same-machine IPC +- Throughput: 5M+ messages/second (96-byte messages on i7-4790) +- Use `Pretoucher` for lowest latency outliers (Enterprise feature) +- Enable double-buffering only for large messages with high write contention +- Resource tracing (system.properties) should be disabled in production + +## Documentation + +- `README.adoc` - Comprehensive user guide +- `docs/FAQ.adoc` - Common questions +- `docs/How_it_works.adoc` - Implementation details +- `docs/utilities.adoc` - Utility documentation +- `src/main/docs/system-properties.adoc` - System property reference diff --git a/GEMINI.md b/GEMINI.md new file mode 100644 index 0000000000..143fcb737e --- /dev/null +++ b/GEMINI.md @@ -0,0 +1,66 @@ +# Chronicle Queue Analysis (`GEMINI.md`) + +This document provides a comprehensive overview of the Chronicle Queue project, intended as a guide for developers and contributors. + +## Project Overview + +Chronicle Queue is a high-performance, low-latency, broker-less messaging library for Java. It is designed for applications that require durable, persisted messaging with millions of events per second. The core of Chronicle Queue is its off-heap storage model, which minimizes Garbage Collection (GC) pauses and allows for predictable, low-latency performance. + +### Key Features: + +* **Ultra-Low Latency:** Achieves sub-microsecond latencies for message passing between JVMs on the same machine. +* **High Throughput:** Supports millions of messages per second on a single thread. +* **Persistence:** All messages are persisted to memory-mapped files, ensuring durability and allowing for replay. +* **Broker-less Architecture:** Reduces complexity and eliminates a single point of failure. +* **Append-Only Log Structure:** Messages are stored in append-only files, organized by a configurable roll cycle (e.g., daily, hourly). +* **Concurrent Access:** Supports a single writer and multiple, independent readers (tailers) per queue. +* **Flexible Serialization:** Integrates with Chronicle Wire to support various serialization formats, including binary, text, YAML, and JSON. + +### Architecture: + +Chronicle Queue's architecture is built on a layered stack of Chronicle libraries: + +* **Chronicle-Core:** Provides fundamental utilities and lifecycle management. +* **Chronicle-Bytes:** Manages off-heap and memory-mapped storage. +* **Chronicle-Wire:** Handles message serialization and deserialization. +* **Chronicle-Threads:** Provides threading and event loop utilities. + +The core data structure is a directory of append-only roll files (`.cq4`), which contain message payloads and tiered index metadata. This design enables efficient sequential and random access to messages. + +## Building and Running + +Chronicle Queue is a standard Maven project. The following commands can be used to build, test, and install the project. + +### Building the Project + +To build the project and run the unit tests, use the following command: + +```bash +mvn clean install +``` + +This will compile the source code, run the tests, and install the resulting JAR files into your local Maven repository. + +### Running Benchmarks + +The project includes a suite of benchmarks to measure performance. To run the benchmarks, you can use the `run-benchmarks` Maven profile: + +```bash +mvn clean install -Prun-benchmarks +``` + +### Running Command-Line Tools + +Chronicle Queue provides command-line tools for inspecting and manipulating queues. For example, to dump the contents of a queue to the console, you can use the `DumpMain` class: + +```bash +mvn exec:java -Dexec.mainClass="net.openhft.chronicle.queue.main.DumpMain" -Dexec.args="" +``` + +## Development Conventions + +* **Coding Style:** The project follows standard Java coding conventions. +* **Testing:** The project has a comprehensive test suite, including unit, integration, and performance tests. New features and bug fixes should be accompanied by corresponding tests. +* **Dependencies:** The project uses a bill of materials (BOM) to manage dependency versions. +* **Concurrency:** The project has a strict concurrency model. `ExcerptAppender` instances are single-threaded, while `ExcerptTailer` instances are not thread-safe. +* **Documentation:** The project has extensive documentation in the `docs` and `src/main/docs` directories, written in AsciiDoc. diff --git a/README.adoc b/README.adoc index 0a690b25ed..1969a53eaa 100644 --- a/README.adoc +++ b/README.adoc @@ -1,9 +1,10 @@ = Chronicle Queue Peter Lawrey, Rob Austin -:css-signature: demo :toc: macro -:toclevels: 2 :icons: font +:lang: en-GB +:toclevels: 2 +:css-signature: demo :source-highlighter: rouge Chronicle Queue is a broker-less, off-heap Java library for ultra-low-latency, persisted messaging at millions of events/sec. @@ -38,7 +39,7 @@ and also supports random-access seek. link:https://player.vimeo.com/video/201989439[Why Use Chronicle Queue Between Microservices?] -A number of relevant system properties are listed in link:docs/systemProperties.adoc[systemProperties.adoc]. +A number of relevant system properties are listed in link:docs/system-properties.adoc[systemProperties.adoc]. == What Is Chronicle Queue? @@ -46,12 +47,12 @@ You could consider a Chronicle Queue to be similar to a low latency broker-less Chronicle Queue is a distributed unbounded persisted queue that: * supports asynchronous RMI and Publish/Subscribe interfaces with microsecond latencies. -* passes messages between JVMs in under a microsecond -* passes messages between JVMs on different machines via replication in under 10 microseconds +* passes messages between JVMs in under 1 µs +* passes messages between JVMs on different machines via replication in under 10 µs (<>) * provides stable, soft real-time latencies into the millions of messages per second for a single thread to one queue; with total ordering of every event. -When publishing 40-byte messages, a high percentage of the time we achieve latencies under 1 microsecond. +When publishing 40-byte messages, a high percentage of the time we achieve latencies under 1 µs. The 99th percentile latency is the worst 1 in 100, and the 99.9th percentile is the worst 1 in 1000 latency. .Latency to send/receive on the same machine. @@ -102,8 +103,8 @@ As Chronicle Queue stores all saved data in memory-mapped files, this has a triv Chronicle put significant effort into achieving very low latency. In other products which focus on support of web applications, latencies of less than 40 milliseconds are fine as they are faster than you can see; for example, the frame rate of cinema is 24 Hz, or about 40 ms. -Chronicle Queue aims to achieve latencies of under 40 microseconds for 99% to 99.99% of the time. -Using Chronicle Queue without replication, we support applications with latencies below 40 microseconds end-to-end across multiple services. +Chronicle Queue aims to achieve latencies of under 40 µs for 99% to 99.99% of the time. +Using Chronicle Queue without replication, we support applications with latencies below 40 µs end-to-end across multiple services. Often the 99% latency of Chronicle Queue is entirely dependent on the choice of operating system and hard disk sub-system. ==== Compression @@ -189,7 +190,7 @@ Again, Chronicle Queue can support millions of events per-second, per-server, an ==== Latency sensitive micro-services -Chronicle Queue supports low latency IPC (Inter Process Communication) between JVMs on the same machine in the order of magnitude of 1 microsecond; as well as between machines with a typical latency of 10 microseconds for modest throughputs of a few hundred thousands. +Chronicle Queue supports low latency IPC (Inter Process Communication) between JVMs on the same machine in the order of magnitude of 1 µs; as well as between machines with a typical latency of 10 µs for modest throughputs of a few hundred thousands. Chronicle Queue supports throughputs of millions of events per second, with stable microsecond latencies. See https://vanilla-java.github.io/tag/Microservices/[Articles on the use of Chronicle Queue in Microservices] @@ -201,7 +202,7 @@ All the information about the state of those components can be reproduced extern This significantly reduces the need for additional logging. However, any logging you do need can be recorded in great detail. This makes enabling `DEBUG` logging in production practical. -This is because the cost of logging is very low; less than 10 microseconds. +This is because the cost of logging is very low; less than 10 µs. Logs can be replicated centrally for log consolidation. Chronicle Queue is being used to store 100+ TB of data, which can be replayed from any point in time. @@ -273,7 +274,7 @@ There are two ways to use byte in Chronicle Queue v4. You can use the `writeByte For example: .Writing and reading bytes using a lambda -[source,java] +[source,java,opts=novalidate] ---- appender.writeBytes(b -> b.writeInt(1234).writeDouble(1.111)); @@ -281,7 +282,7 @@ boolean present = tailer.readBytes(b -> process(b.readInt(), b.readDouble())); ---- .Writing to a queue without using a lambda -[source,java] +[source,java,opts=novalidate] ---- try (DocumentContext dc = appender.writingDocument()) { Bytes bytes = dc.wire().bytes(); @@ -443,7 +444,7 @@ This section provides a quick reference for using Chronicle Queue to briefly sho Creating an instance of Chronicle Queue is different from just calling a constructor. To create an instance you have to use the `ChronicleQueueBuilder`. -[source,java] +[source,java,opts=novalidate] ---- String basePath = OS.getTarget() + "/getting-started" ChronicleQueue queue = SingleChronicleQueueBuilder.single(basePath).build(); @@ -458,7 +459,7 @@ ${java.io.tmpdir}/getting-started/{today}.cq4 * Writing to a queue -[source,java] +[source,java,opts=novalidate] ---- // Obtains an ExcerptAppender ExcerptAppender appender = queue.acquireAppender(); @@ -472,7 +473,7 @@ appender.writeText("TestMessage"); * Reading from a queue -[source,java] +[source,java,opts=novalidate] ---- // Creates a tailer ExcerptTailer tailer = queue.createTailer(); @@ -482,7 +483,7 @@ tailer.readDocument(w -> System.out.println("msg: " + w.read(()->"msg").text())) assertEquals("TestMessage", tailer.readText()); ---- Also, the `ChronicleQueue.dump()` method can be used to dump the raw contents as a string. -[source,java] +[source,java,opts=novalidate] ---- queue.dump(); ---- @@ -494,14 +495,14 @@ Chronicle Queue stores its data off-heap, and it is recommended that you call `c NOTE: No data will be lost if you do this. This is only to clean up resources that were used. -[source,java] +[source,java,opts=novalidate] ---- queue.close(); ---- * Putting it all together -[source,java] +[source,java,opts=novalidate] ---- try (ChronicleQueue queue = SingleChronicleQueueBuilder.single("queue-dir").build()) { // Obtain an ExcerptAppender @@ -535,7 +536,7 @@ A few of the parameters that were most queried by our customers are explained be The `RollCycle` parameter configures the rate at which CQ will roll the underlying queue files. For instance, using the following code snippet will result in the queue files being rolled (i.e. a new file created) every hour: -[source,java] +[source,java,opts=novalidate] ---- ChronicleQueue.singleBuilder(queuePath).rollCycle(RollCycles.HOURLY).build() ---- @@ -545,7 +546,7 @@ Any further instances of `SingleChronicleQueue` configured to use the same path and if they are not, then the roll-cycle will be updated to match the persisted roll-cycle. In this case, a warning log message will be printed in order to notify the library user of the situation: -[source,java] +[source,java,opts=novalidate] ---- // Creates a queue with roll-cycle MINUTELY try (ChronicleQueue minuteRollCycleQueue = ChronicleQueue.singleBuilder(queueDir).rollCycle(MINUTELY).build()) { @@ -582,14 +583,14 @@ See link:docs/managing_roll_files_directly.adoc[Managing Roll Files Directly]. It's possible to configure how Chronicle Queue will store the data by explicitly set the `WireType`: -[source,java] +[source,java,opts=novalidate] ---- // Creates a queue at "queuePath" and sets the WireType SingleChronicleQueueBuilder.builder(queuePath, wireType) ---- For example: -[source,java] +[source,java,opts=novalidate] ---- // Creates a queue with default WireType: BINARY_LIGHT ChronicleQueue.singleBuilder(queuePath) @@ -662,7 +663,7 @@ The size of each index array, as well as the total number of index arrays per qu [NOTE] indexCount^2^ is the maximum number of indexed queue entries. -NOTE: See Section link:#excerpt-indexing-in-chronicle-queue[Excerpt indexing in Chronicle Queue] of this User Guide for more information and examples of using indexes. +NOTE: See Section link:#_excerpt_indexing_in_chronicle_queue[Excerpt indexing in Chronicle Queue] of this User Guide for more information and examples of using indexes. * _readBufferMode, writeBufferMode_ @@ -699,7 +700,7 @@ Unlike other java queuing solutions, messages are not lost when they are read wi This is covered in more detail in the section below on "Reading from a queue using a tailer". To write data to a Chronicle Queue, you must first create an appender: -[source,java] +[source,java,opts=novalidate] ---- try (ChronicleQueue queue = ChronicleQueue.singleBuilder(path + "/trades").build()) { final ExcerptAppender appender = queue.acquireAppender(); @@ -708,7 +709,7 @@ try (ChronicleQueue queue = ChronicleQueue.singleBuilder(path + "/trades").build Chronicle Queue uses the following low-level interface to write the data: -[source,java] +[source,java,opts=novalidate] ---- try (final DocumentContext dc = appender.writingDocument()) { dc.wire().write().text(“your text data“); @@ -720,7 +721,7 @@ You can also use the `DocumentContext` to find out the index that your data has You can later use this index to move-to/look up this excerpt. Each Chronicle Queue excerpt has a unique index. -[source,java] +[source,java,opts=novalidate] ---- try (final DocumentContext dc = appender.writingDocument()) { dc.wire().write().text(“your text data“); @@ -748,7 +749,7 @@ So you have a choice of a number of high-level interfaces, down to a low-level A This is the highest-level API which hides the fact you are writing to messaging at all. The benefit is that you can swap calls to the interface with a real component, or an interface to a different protocol. -[source,java] +[source,java,opts=novalidate] ---- // using the method writer interface. RiskMonitor riskMonitor = appender.methodWriter(RiskMonitor.class); @@ -760,7 +761,7 @@ You can write a "self-describing message". Such messages can support schema changes. They are also easier to understand when debugging or diagnosing problems. -[source,java] +[source,java,opts=novalidate] ---- // writing a self describing message appender.writeDocument(w -> w.write("trade").marshallable( @@ -775,7 +776,7 @@ appender.writeDocument(w -> w.write("trade").marshallable( You can write "raw data" which is self-describing. The types will always be correct; position is the only indication as to the meaning of those values. -[source,java] +[source,java,opts=novalidate] ---- // writing just data appender.writeDocument(w -> w @@ -787,7 +788,7 @@ appender.writeDocument(w -> w You can write "raw data" which is not self-describing. Your reader must know what this data means, and the types that were used. -[source,java] +[source,java,opts=novalidate] ---- // writing raw data appender.writeBytes(b -> b @@ -800,7 +801,7 @@ appender.writeBytes(b -> b Below, the lowest level way to write data is illustrated. You get an address to raw memory and you can write whatever you want. -[source,java] +[source,java,opts=novalidate] ---- // Unsafe low level appender.writeBytes(b -> { @@ -821,7 +822,7 @@ appender.writeBytes(b -> { You can print the contents of the queue. You can see the first two, and last two messages store the same data. -[source,java] +[source,java,opts=novalidate] ---- // dump the content of the queue System.out.println(queue.dump()); @@ -872,7 +873,7 @@ Hello World Reading the queue follows the same pattern as writing, except there is a possibility there is not a message when you attempt to read it. .Start Reading -[source,java] +[source,java,opts=novalidate] ---- try (ChronicleQueue queue = ChronicleQueue.singleBuilder(path + "/trades").build()) { final ExcerptTailer tailer = queue.createTailer(); @@ -882,7 +883,7 @@ try (ChronicleQueue queue = ChronicleQueue.singleBuilder(path + "/trades").build You can turn each message into a method call based on the content of the message, and have Chronicle Queue automatically deserialize the method arguments. Calling `reader.readOne()` will automatically skip over (filter out) any messages that do not match your method reader. -[source,java] +[source,java,opts=novalidate] ---- // reading using method calls RiskMonitor monitor = System.out::println; @@ -895,7 +896,7 @@ You can decode the message yourself. NOTE: The names, type, and order of the fields doesn't have to match. -[source,java] +[source,java,opts=novalidate] ---- assertTrue(tailer.readDocument(w -> w.read("trade").marshallable( m -> { @@ -912,7 +913,7 @@ assertTrue(tailer.readDocument(w -> w.read("trade").marshallable( You can read self-describing data values. This will check the types are correct, and convert as required. -[source,java] +[source,java,opts=novalidate] ---- assertTrue(tailer.readDocument(w -> { ValueIn in = w.getValueIn(); @@ -925,7 +926,7 @@ assertTrue(tailer.readDocument(w -> { You can read raw data as primitives and strings. -[source,java] +[source,java,opts=novalidate] ---- assertTrue(tailer.readBytes(in -> { int code = in.readByte(); @@ -939,7 +940,7 @@ assertTrue(tailer.readBytes(in -> { or, you can get the underlying memory address and access the native memory. -[source,java] +[source,java,opts=novalidate] ---- assertTrue(tailer.readBytes(b -> { long address = b.address(b.readPosition()); @@ -975,7 +976,7 @@ Across topics, there is no guarantee of ordering; if you want to replay determin Chronicle Queue tailers may create file handlers, the file handlers are cleaned up whenever the associated chronicle queue's `close()` method is invoked or whenever the Jvm runs a Garbage Collection. If you are writing your code not have GC pauses and you explicitly want to clean up the file handlers, you can call the following: -[source,java] +[source,java,opts=novalidate] ---- ((StoreTailer)tailer).releaseResources() ---- @@ -991,7 +992,7 @@ In this case, the tailer is now ready for reading any new records appended to th Until any new messages are appended to the queue, there will be no new `DocumentContext` available for reading: -[source,java] +[source,java,opts=novalidate] .... // this will be false until new messages are appended to the queue boolean messageAvailable = tailer.toEnd().readingDocument().isPresent(); @@ -999,7 +1000,7 @@ boolean messageAvailable = tailer.toEnd().readingDocument().isPresent(); If it is necessary to read backwards through the queue from the end, then the tailer can be set to read backwards: -[source,java] +[source,java,opts=novalidate] .... ExcerptTailer tailer = queue.createTailer(); tailer.direction(TailerDirection.BACKWARD).toEnd(); @@ -1009,7 +1010,7 @@ When reading backwards, then the `toEnd()` method will move the tailer to the la If the queue is not empty, then there will be a `DocumentContext` available for reading: -[source,java] +[source,java,opts=novalidate] ---- // this will be true if there is at least one message in the queue boolean messageAvailable = tailer.toEnd().direction(TailerDirection.BACKWARD). @@ -1023,7 +1024,7 @@ AKA named tailers. It can be useful to have a tailer which continues from where it was up to on restart of the application. -[source,java] +[source,java,opts=novalidate] ---- try (ChronicleQueue cq = SingleChronicleQueueBuilder.binary(tmp).build()) { ExcerptTailer atailer = cq.createTailer("a"); @@ -1061,13 +1062,7 @@ If this is prevented by an error, the same message will be read on each restart. ''' ==== Command line tools - reading and writing a Chronicle Queue -Chronicle Queue stores its data in binary format, with a file extension of `cq4`: - ----- -\�@πheader∂SCQStoreÇE��»wireType∂WireTypeÊBINARYÕwritePositionèèèèß��������ƒroll∂SCQSRollÇ*���∆length¶ÄÓ6�∆format -ÎyyyyMMdd-HH≈epoch¶ÄÓ6�»indexing∂ SCQSIndexingÇN��� indexCount•�ÃindexSpacingÀindex2Indexé����ß��������…lastIndexé� -���ß��������fllastAcknowledgedIndexReplicatedé�����ߡˇˇˇˇˇˇˇ»recovery∂TimedStoreRecoveryÇ���…timeStampèèèß���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������� ----- +Chronicle Queue stores its data in binary format, with a file extension of `cq4`. This can often be a bit difficult to read, so it is better to dump the `cq4` files as text. This can also help you fix your production issues, as it gives you the visibility as to what has been stored in the queue, and in what order. @@ -1259,30 +1254,36 @@ Missing required options: m, d If you want to write to the below "doit" method -[source,java] +[source,java,opts=novalidate] +---- public interface MyInterface { void doit(DTO dto); } public class DTO extends SelfDescribingMarshallable { -private int age; -private String name; + private int age; + private String name; } +---- Then you can call `ChronicleWriterMain -d queue doit x.yaml` with either (or both) of the below Yamls: [source,yaml] +---- { age: 19, name: Henry } +---- or [source,yaml] +---- !x.y.z.DTO { age: 42, name: Percy } +---- If `DTO` makes use of custom serialisation then you should specify the interface to write to with `-i` @@ -1296,7 +1297,7 @@ You start by defining an asynchronous `interface`, where all methods have: * no return value or exceptions expected. .A simple asynchronous interface -[source,java] +[source,java,opts=novalidate] ---- import net.openhft.chronicle.wire.SelfDescribingMarshallable; interface MessageListener { @@ -1324,7 +1325,7 @@ static class Message2 extends SelfDescribingMarshallable { To write to the queue you can call a proxy which implements this interface. -[source,java] +[source,java,opts=novalidate] ---- SingleChronicleQueue queue1 = ChronicleQueue.singleBuilder(path).build(); @@ -1353,7 +1354,7 @@ method2: { To read the messages, you can provide a reader which calls your implementation with the same calls that you made. -[source,java] +[source,java,opts=novalidate] ---- // a proxy which print each method called on it MessageListener processor = ObjectUtils.printAll(MessageListener.class) @@ -1389,7 +1390,7 @@ We support using nano-time across machines, without the need for specialist hard To enable this, set the `sourceId` of the queue. .Enabling high resolution timings -[source,java] +[source,java,opts=novalidate] ---- ChronicleQueue out = ChronicleQueue.singleBuilder(queuePath) ... @@ -1406,14 +1407,14 @@ combiner.onSidedPrice(new SidedPrice("EURUSD1", 123456789000L, Side.Sell, 1.1172 A timestamp is added for each read and write as it passes from service to service. .Downstream message triggered by the event above -[source,Yaml] +[source,yaml] ---- --- !!data #binary history: { sources: [ 1, 0x426700000000 # <4> - ] + ], timings: [ 1394278797664704, # <1> 1394278822632044, # <2> @@ -1435,6 +1436,7 @@ onTopOfBookPrice: { <4> What triggered this event. '''' +[#_excerpt_indexing_in_chronicle_queue] ==== Excerpt indexing in Chronicle Queue In the following section you will find how to work with the excerpt index. @@ -1485,7 +1487,7 @@ net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.countExcerpts * _Move to a specific message and read it_ The following example shows how to write 10 messages, then move to the 5th message to read it -[source,java] +[source,java,opts=novalidate] ---- @Test public void read5thMessageTest() { @@ -1550,14 +1552,14 @@ Chronicle Queue runs a background thread to watch for low disk space (see `net.o The disk space monitor checks (for each FileStore you are using Chronicle Queues on): that there is less than 200MB free. If so you will see: -[source,java] +[source,java,opts=novalidate] ---- Jvm.warn().on(getClass(), "your disk " + fileStore + " is almost full, " + "warning: chronicle-queue may crash if it runs out of space."); ---- otherwise it will check for the threshold percentage and log out this message: -[source,java] +[source,java,opts=novalidate] ---- Jvm.warn().on(getClass(), "your disk " + fileStore + " is " + diskSpaceFull + "% full, " + @@ -1621,14 +1623,14 @@ Obviously enabling this will prevent The configuration parameters of Pretoucher that were described above should be set via system properties. For example, in the following excerpt `earlyAcquireNextCycle` is set to `true` and `pretoucherPrerollTimeMs` to 100ms. -[source,java] +[source,java,opts=novalidate] ---- System.setProperty("SingleChronicleQueueExcerpts.earlyAcquireNextCycle", "true"); System.setProperty("SingleChronicleQueueExcerpts.pretoucherPrerollTimeMs", "100"); ---- The constructor of Pretoucher takes the name of the queue that this Pretoucher is assigned to and creates a new Pretoucher. Then, by invoking the `execute()` method the Pretoucher starts. -[source,java] +[source,java,opts=novalidate] ---- // Creates the queue q1 (or q1 is a queue that already exists) try(final SingleChronicleQueue q1 = SingleChronicleQueueBuilder.binary("queue-storage-path").build(); @@ -1644,7 +1646,7 @@ try(final SingleChronicleQueue q1 = SingleChronicleQueueBuilder.binary("queue-st ---- The method `close()`, closes the Pretoucher and releases its resources. -[source,java] +[source,java,opts=novalidate] ---- pretouch.close(); ---- @@ -1782,14 +1784,14 @@ Percentile run1 run2 run3 % Variation If you wish to tune your code for ultra-low latency, you could take a similar approach to our `QueueReadJitterMain` -[source,java] +[source,java,opts=novalidate] ---- net.openhft.chronicle.queue.jitter.QueueReadJitterMain ---- This code can be considered as a basic stack sampler profiler. This is assuming you base your code on the `net.openhft.chronicle.core.threads.EventLoop`, you can periodically sample the stacks to find a stall. -It is recommended to not reduce the sample rate below 50 microseconds as this will produce too much noise +It is recommended to not reduce the sample rate below 50 µs as this will produce too much noise It is likely to give you finer granularity than a typical profiler. As it is based on a statistical approach of where the stalls are, it takes many samples, to see which code has the highest grouping ( in other words the highest stalls ) and will output a trace that looks like the following : @@ -1829,17 +1831,17 @@ we will often add a `net.openhft.chronicle.core.Jvm.safepoint` into the code bec * Results: -In the test described above, the typical latency varied between 14 and 40 microseconds. -The 99 percentile varied between 17 and 56 microseconds depending on the throughput being tested. -Notably, the 99.93% latency varied between 21 microseconds and 41 milliseconds, a factor of 2000. +In the test described above, the typical latency varied between 14 and 40 µs. +The 99 percentile varied between 17 and 56 µs depending on the throughput being tested. +Notably, the 99.93% latency varied between 21 µs and 41 ms, a factor of 2000. .Possible throughput results depending on acceptable latencies |=== | Acceptable Latency | Throughput -| < 30 microseconds 99.3% of the time | 7 million message per second -| < 20 microseconds 99.9% of the time | 20 million messages per second +| < 30 µs 99.3% of the time | 7 million message per second +| < 20 µs 99.9% of the time | 20 million messages per second | < 1 milliseconds 99.9% of the time | 50 million messages per second -| < 60 microseconds 99.3% of the time | 80 million message per second +| < 60 µs 99.3% of the time | 80 million message per second |=== === More Benchmarks @@ -1852,7 +1854,7 @@ link:https://portal.chronicle.software/docs/queue/chronicle-queue/performance/pe Chronicle Queue is designed to out-perform its rivals such as Kafka. Chronicle Queue supports over an order-of-magnitude of greater throughput, together with an order-of-magnitude of lower latency, than Apache Kafka. -While Kafka is faster than many of the alternatives, it doesn't match Chronicle Queue's ability to support throughputs of over a million events per second, while simultaneously achieving latencies of 1 to 20 microseconds. +While Kafka is faster than many of the alternatives, it doesn't match Chronicle Queue's ability to support throughputs of over a million events per second, while simultaneously achieving latencies of 1 to 20 µs. Chronicle Queue handles more volume from a single thread to a single partition. This avoids the need for the complexity, and the downsides, of having partitions. diff --git a/pom.xml b/pom.xml index cd0693842a..7b0cd05768 100644 --- a/pom.xml +++ b/pom.xml @@ -31,7 +31,7 @@ net.openhft third-party-bom - 3.27ea5 + 3.27ea7 pom import From 87ad78628f66d5264c5a6db63f71f4331118050c Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Thu, 11 Dec 2025 14:34:43 +0000 Subject: [PATCH 5/6] Add architecture overview, decision log, project requirements, and security review documents --- src/main/docs/architecture-overview.adoc | 308 +++++++++++++++++++++++ src/main/docs/decision-log.adoc | 123 +++++++++ src/main/docs/project-requirements.adoc | 122 +++++++++ src/main/docs/security-review.adoc | 192 ++++++++++++++ 4 files changed, 745 insertions(+) create mode 100644 src/main/docs/architecture-overview.adoc create mode 100644 src/main/docs/decision-log.adoc create mode 100644 src/main/docs/project-requirements.adoc create mode 100644 src/main/docs/security-review.adoc diff --git a/src/main/docs/architecture-overview.adoc b/src/main/docs/architecture-overview.adoc new file mode 100644 index 0000000000..fac4466b64 --- /dev/null +++ b/src/main/docs/architecture-overview.adoc @@ -0,0 +1,308 @@ += Chronicle Queue Architecture Overview +:toc: +:sectnums: +:lang: en-GB +:toclevels: 3 +:doctype: book +:source-highlighter: rouge + +This document positions Chronicle Queue within the wider Chronicle stack, describes its core abstractions and data model, and explains how its file format, roll-cycle model, indexing, and concurrency patterns work together to deliver ultra-low-latency, durable messaging. +It is intended for developers and architects building systems on top of Chronicle Queue or integrating it into larger platforms. + +== Context and Role in the Chronicle Stack + +Chronicle Queue is the *log-structured messaging and journalling layer* in the Chronicle ecosystem. +It provides a broker-less, append-only, persisted queue that applications use to exchange messages between threads, processes and (with Chronicle-Queue-Enterprise) machines. + +At a high level: + +* *Chronicle-Core* supplies platform utilities, lifecycle helpers and the `Closeable`/`ReferenceCounted` contracts. +* *Chronicle-Bytes* provides the off-heap and memory-mapped storage primitives used for queue data and indices. +* *Chronicle-Wire* provides flexible serialisation formats layered on top of `Bytes` (binary, text, YAML, JSON). +* *Chronicle-Threads* supplies event-loop and pauser utilities used by the pretoucher and other background services. +* *Chronicle-Queue-Enterprise* builds on this module to add clustering, replication and advanced operational features. + +Chronicle Queue focuses on: + +* Persisted, append-only logs of messages or events. +* Tight control over latency and allocation behaviour. +* Simple concurrency rules (single-writer style appenders; independent tailers). +* A storage model that scales from in-process IPC to multi-terabyte archives of historical data. + +== Core Concepts and Abstractions + +=== Append-Only Queue Files + +Chronicle Queue stores data in a directory of append-only files (roll files) with a `.cq4` suffix. +Each roll file belongs to a *roll cycle* (for example daily or hourly) and contains: + +* A sequence of message payloads, encoded via Chronicle-Wire on top of Chronicle-Bytes. +* Embedded index metadata to support fast random access and scanning. + +Files are never modified in place apart from appending new data and metadata. +This append-only model simplifies crash recovery and makes it safe to tail queues while producers continue writing. + +The on-disk format and roll-cycle policy are described in more depth in: + +* link:decision-log.adoc#CQ-FN-101[CQ-FN-101 - Queue File Format and Roll-Cycle Model] +* link:../../docs/How_it_works.adoc[How Chronicle Works] +* link:../../docs/managing_roll_files_directly.adoc[Managing roll files directly] + +=== Appenders and Tailers + +Two primary runtime abstractions sit on top of the file format: + +* `ExcerptAppender` - a writer-side view used to append messages sequentially to the end of the queue. +* `ExcerptTailer` - a reader-side view used to read messages from the queue in order, advancing its own cursor independently of other tailers. + +Typical usage: + +* A service creates a `ChronicleQueue` (often via `ChronicleQueue.single(path)` or a `SingleChronicleQueueBuilder`). +* Each writer thread acquires its own `ExcerptAppender` and uses it to write text, binary messages or marshalled objects. +* Each consumer thread acquires its own `ExcerptTailer` and iterates forward, optionally starting from the beginning, end or a stored index. + +Internally, both appenders and tailers are flyweight views backed by Chronicle-Bytes structures over the underlying roll files. + +=== Roll Cycles and Directory Layout + +Queues are organised by *roll cycle*, which controls how often Chronicle Queue starts a new file. +A roll cycle: + +* Defines the mapping between logical time and roll file name (for example daily, hourly). +* Influences file size, index density and operational handling (archival, backup, retention). +* Is configured via the `SingleChronicleQueueBuilder` (`rollCycle(RollCycle.DAILY)` and similar). + +The roll-cycle design and trade-offs are documented in: + +* link:decision-log.adoc#CQ-FN-101[CQ-FN-101 - Queue File Format and Roll-Cycle Model] + +At the file-system level, a queue directory typically contains: + +* Roll files (for example `20241117.cq4`, `20241118.cq4`). +* Metadata files used to track index and configuration information. + +=== Tiered Indexing + +Chronicle Queue uses a *tiered multi-level index* to provide efficient random access into large queues. +Index structures are stored alongside data inside roll files. + +A conceptual view is provided in link:../../docs/How_it_works.adoc[How it works] and further illustrated in the *Tiered Indexing* section of that document. + +At a high level: + +* A top-level `index2index` structure maps logical ranges of message indices to lower-level index blocks. +* Lower-level `index` blocks map directly to positions within the roll file. +* When a tailer seeks to a specific index, Chronicle Queue walks this structure to find the corresponding file position. + +This scheme scales to very large queues while keeping the cost of point lookups and scans bounded. + +== Data Flow and Usage Patterns + +=== Single JVM Producer-Consumer + +The simplest deployment is within a single JVM, where producers and consumers share a queue directory on local disk: + +.... ++---------------------------+ +| Application JVM | +| | +| +---------------------+ | +| | ChronicleQueue | | +| | (SingleChronicle) | | +| +---------------------+ | +| ^ ^ | +| | | | +| ExcerptAppender ExcerptTailer +| (producer) (consumer) ++---------------------------+ +.... + +Flow: + +. The producer writes messages through an `ExcerptAppender`. +. Messages are appended to the current roll file; metadata is updated. +. The consumer reads messages via an `ExcerptTailer`, advancing its own cursor. + +Both sides can run at high rates because: + +* Data is written and read sequentially. +* Most I/O is served from the OS page cache. +* Off-heap `Bytes` structures avoid GC pressure on the hot path. + +=== Multi-Process and Cross-Machine + +Chronicle Queue is often used to communicate: + +* Between processes on the same host, sharing a queue directory on local storage. +* Across hosts when combined with Chronicle-Queue-Enterprise replication features. + +In the latter case, this module remains responsible for the local file format, roll-cycle behaviour and indexing. +Queue-Enterprise components handle network replication, acknowledgements and recovery semantics. + +== Concurrency and Threading Model + +Chronicle Queue follows a *single-writer style* for appenders and a simple, explicit model for tailers: + +* Each `ExcerptAppender` is intended to be used by a single writer thread. +** Sharing an appender between threads requires external synchronisation and is discouraged in latency-sensitive code. +* Each `ExcerptTailer` is not thread-safe and should be owned by at most one reader thread at a time. +* Multiple tailers can read from the same queue concurrently without interfering with each other; each maintains its own position. + +These rules are captured and justified in: + +* link:decision-log.adoc#CQ-FN-102[CQ-FN-102 - Appender and Tailer Concurrency Model] + +Chronicle Queue also integrates with Chronicle-Threads to support background activities such as pretouching: + +* The *pretoucher* runs in a separate thread or event loop, faulting in pages ahead of time to reduce page-fault latency when appenders write. +* Event-loop abstractions from Chronicle-Threads can be used to integrate queue processing into larger event-driven systems. + +== Persistence, Durability and Recovery + +Chronicle Queue balances durability guarantees against latency and throughput goals: + +* Writes are appended to memory-mapped regions and eventually flushed to disk by the operating system. +* The library can be configured to enforce stronger durability (for example explicit fsync behaviour) at the cost of higher latency. +* On restart, queues can be replayed from roll files; the append-only format and embedded indices simplify recovery. + +The durability and recovery expectations, and the flush policy trade-offs, are discussed in: + +* link:decision-log.adoc#CQ-NF-P-201[CQ-NF-P-201 - Durability, Flush Policy and Recovery] + +== Configuration Surface + +=== SingleChronicleQueueBuilder + +`SingleChronicleQueueBuilder` is the primary configuration surface for creating queues. +Key options include: + +* *Path and file layout* +** `path(File)` / `path(Path)` - queue directory. +** Handling of specific `.cq4` files vs directories. +* *Wire format* +** `wireType(WireType)` - choice of binary or text wire formats for messages. +* *Roll cycle and indexing* +** `rollCycle(RollCycle)` - time-based roll policy. +** `indexCount` and `indexSpacing` - index density and layout. +* *Buffering and block sizes* +** `blockSize(long)` - block size used for mapped regions. +** `writeBufferMode` / `readBufferMode` - buffer strategies for appender and tailer. +* *Time and epoch* +** `timeProvider(TimeProvider)` - pluggable time source. +** `epoch(long)` - custom epoch for roll-cycle calculations. +* *Operational hooks* +** `storeFileListener(StoreFileListener)` - observe roll-file lifecycle events. +** `appenderListener(AppenderListener)` - observe appender creation. +** `queueOffsetSpec(QueueOffsetSpec)` - fine-grained control over initial tail positions. + +Consult the builder Javadoc and implementation for the full set of configuration points. + +=== System Properties + +Chronicle Queue exposes several *system properties* that alter runtime behaviour without code changes. +These are catalogued in: + +* link:../../docs/system-properties.adoc[Chronicle Queue System Properties] + +Examples include: + +* `chronicle.queue.checkrollcycle` - emit warnings when new roll files are created. +* `chronicle.queue.checkInterrupts` - enforce tailer interrupt checking semantics. +* `chronicle.queue.report.linear.scan.latency` - log when index scans exceed a threshold. +* `chronicle.queue.rollingResourceCache.size` - tune the in-memory cache of roll-file metadata. + +These properties are especially relevant when tuning performance and operability in production. + +== Integration with Other Chronicle Modules + +Chronicle Queue is not a standalone island; it is designed to interoperate closely with other Chronicle modules: + +* *Chronicle-Bytes* +** Provides the underlying `Bytes` and `BytesStore` implementations used for queue data and indices. +** Supplies off-heap and memory-mapped storage, reference counting and bounds-checked access. +* *Chronicle-Wire* +** Provides the message serialisation layer used by queue producers and consumers. +** Enables flexible wire formats and schema evolution without changing the underlying queue format. +* *Chronicle-Threads* +** Supplies event loops and pauser strategies used by background components such as the pretoucher. +** Provides patterns for integrating queue processing into low-latency, event-driven applications. +* *Chronicle-Queue-Enterprise* +** Adds replication, clustering and advanced operational features on top of the core queue. +** Treats the file format and roll-cycle behaviour defined here as the canonical storage layer. + +== Security and Compliance Considerations + +Chronicle Queue is primarily a storage and messaging primitive. +Security responsibilities are deliberately split between this module and surrounding infrastructure: + +* *Data at rest* +** By default, queue files are stored in plain form on disk. +* link:decision-log.adoc#CQ-NF-S-301[CQ-NF-S-301 - Encryption Responsibility and Boundary] records the decision that encryption is not enforced in this module by default; users can: +*** Encrypt at the filesystem or volume level, or +*** Marshal application-level payloads through Chronicle-Wire with custom encryption. +* *Data in transit* +** When queues are exposed over the network (for example via replication or gateways), Chronicle-Network and service-layer modules are responsible for TLS, authentication and authorisation. +* *Off-heap safety* +** Memory safety and bounds checking are provided by Chronicle-Bytes primitives. +** Queue code assumes callers respect `Closeable` and `ReferenceCounted` contracts when interacting with off-heap resources. + +When documenting module-level compliance against ISO 27001 or SOC 2, this division of responsibility should be called out explicitly. + +== Operational Considerations + +=== Retention and Roll-File Management + +Roll files accumulate over time and must be managed carefully to avoid exhausting storage or violating retention policies. +Chronicle Queue itself does not automatically delete data; instead: + +* It exposes predictable roll-file naming and directory structures. +* It documents retention and cleanup as operational concerns. + +This is captured in: + +* link:decision-log.adoc#CQ-NF-O-401[CQ-NF-O-401 - Retention and Roll File Cleanup] + +Typical retention strategies include: + +* Time-based purge scripts that delete or archive roll files older than a configured window. +* Tiered storage, where recent data stays on fast disks and older files are moved to cheaper storage. + +Operational runbooks should clearly document: + +* The retention policy. +* The tooling used to enforce it. +* Monitoring and alerting for disk utilisation. + +=== Monitoring and Diagnostics + +Chronicle Queue uses a combination of logging, metrics and system properties to support observability: + +* Slow appender warnings (for example `chronicle.queue.warnSlowAppenderMs`). +* Linear scan latency reporting via `chronicle.queue.report.linear.scan.latency`. +* Optional consistency checks for indexing (`queue.check.index`) at the cost of extra overhead. + +Operators should: + +* Configure logging to capture these diagnostics. +* Integrate JVM- or process-level metrics with their monitoring stack (for example, queue directory size, roll-file counts). +* Validate configuration in non-production environments before rolling out to latency-sensitive systems. + +== Evolution and Extension Points + +Chronicle Queue is designed to evolve while preserving on-disk compatibility and the core concurrency contracts. +Key extension points include: + +Wire formats :: +adding new Chronicle-Wire formats or evolving schemas without changing the queue file format. +Builder hooks :: +pluggable listeners (`StoreFileListener`, `AppenderListener`) for instrumentation and custom monitoring. +Offset and indexing strategies :: +configuration for index density and starting offsets via `QueueOffsetSpec`. +Enterprise features :: +queue replication and clustering in Chronicle-Queue-Enterprise that reuse the same core storage primitives. + +Any change that affects file format, roll-cycle semantics, durability guarantees or concurrency contracts should be recorded in: + +* link:decision-log.adoc[Chronicle Queue - Decision Log] + +so that downstream modules and operational runbooks can be updated consistently. diff --git a/src/main/docs/decision-log.adoc b/src/main/docs/decision-log.adoc new file mode 100644 index 0000000000..b55b337535 --- /dev/null +++ b/src/main/docs/decision-log.adoc @@ -0,0 +1,123 @@ += Chronicle Queue - Decision Log +:toc: +:lang: en-GB +:source-highlighter: rouge + +This document records component-specific architectural and operational decisions for Chronicle Queue. +Identifiers follow the Nine-Box pattern `--NNN`; numbers are unique within the `CQ` scope. +Entries cross-reference the queue documentation under `docs/` and any future `project-requirements.adoc` catalogue. + +== Decision Index + +* link:#CQ-FN-101[CQ-FN-101 Queue File Format and Roll-Cycle Model] +* link:#CQ-FN-102[CQ-FN-102 Appender and Tailer Concurrency Model] +* link:#CQ-NF-P-201[CQ-NF-P-201 Durability, Flush Policy and Recovery] +* link:#CQ-NF-O-401[CQ-NF-O-401 Retention and Roll File Cleanup] + +[[CQ-FN-101]] +== CQ-FN-101 Queue File Format and Roll-Cycle Model + +Context:: +* Chronicle Queue is used for long-lived, high-throughput messaging and logging where queues may span many files and days of data. +* The on-disk format and roll-cycle policy must support efficient append and tail operations, random access by index, and reasonable recovery characteristics after process or machine failure. +* Existing documentation in `../../docs/How_it_works.adoc` and `../../docs/managing_roll_files_directly.adoc` describes roll files, index structures, and how readers and writers interact with them. +Decision Statement:: +* Chronicle Queue stores data in append-only roll files organised by a roll-cycle policy (for example daily or hourly). Each roll file contains data blocks and index metadata that support fast sequential and random access. +* The default roll-cycle is chosen to balance file size, index density and operational convenience, and is stable across minor releases to maintain compatibility. +* Roll-cycle configuration is exposed via builders so applications can choose appropriate periods without altering the core file format. +Alternatives Considered:: +* Single monolithic queue file:: +** Pros: conceptually simple; no roll logic. +** Cons: unbounded file growth makes recovery and compaction harder; I/O becomes less predictable; more expensive to archive or move old data. +* Size-based roll-only model:: +** Pros: more directly reflects on-disk capacity constraints. +** Cons: harder to align files with operational boundaries (for example trading days); complicates time-based retention policies; may produce many small files during bursts. +Rationale for Decision:: +* A time-based roll-cycle aligns naturally with operational concepts such as trading days and log-rotation windows while providing predictable file naming and archiving behaviour. +* Keeping the on-disk format append-only with index metadata in the same roll files simplifies random access and recovery. +Impact & Consequences:: +* Operational tooling and runbooks build on the roll-cycle convention for monitoring disk usage, archiving and replay. +* Changing the default roll-cycle is considered a significant behaviour change and must be documented clearly whenever it occurs. +Notes/Links:: +* link:../../docs/How_it_works.adoc[How it works] +* link:../../docs/managing_roll_files_directly.adoc[Managing roll files directly] + +[[CQ-FN-102]] +== CQ-FN-102 Appender and Tailer Concurrency Model + +Context:: +* Chronicle Queue supports multiple producers and consumers in the same JVM and across processes, with strict ordering guarantees within a queue. +* The concurrency model for `ExcerptAppender` and `ExcerptTailer` must be explicit so that applications avoid unsafe sharing patterns that compromise ordering or data integrity. +Decision Statement:: +* Chronicle Queue guarantees that appends from a single `ExcerptAppender` are sequential and durable in append order. +* The recommended pattern is one appender per active writer thread; sharing an appender between threads requires external synchronisation and is discouraged in latency-sensitive code. +* Tailers (`ExcerptTailer`) provide forward-only reading by index; multiple tailers may read from the same queue concurrently, but each tailer’s position is independent. +Alternatives Considered:: +* Fully thread-safe appenders with internal locking:: +** Pros: simpler API for casual use. +** Cons: introduces locking overhead and contention in hot paths; hides cost and contention from callers; does not remove the need to reason about ordering semantics. +* Per-queue global appender and tailer pools managed by the library:: +** Pros: centralised management. +** Cons: makes lifetime and ownership obscure; increases risk of misuse and difficult-to-debug races. +Rationale for Decision:: +* Encouraging one appender per writer thread is consistent with Chronicle’s performance goals and avoids hidden synchronisation in the core API. +* Independent tailers make it easy to build fan-out consumers and replay tools without affecting each other’s positions. +Impact & Consequences:: +* Application code must be explicit about how appenders are shared; violations may manifest as increased latency or, if external concurrency control is incorrect, undefined behaviour. +* Documentation in `README.adoc` and `../../docs/How_it_works.adoc` should continue to emphasise recommended patterns (for example appender-per-thread and explicit tailer lifecycle). +Notes/Links:: +* link:../../README.adoc[Chronicle Queue README] +* link:../../docs/How_it_works.adoc[How it works] + +[[CQ-NF-P-201]] +== CQ-NF-P-201 Durability, Flush Policy and Recovery + +Context:: +* Users rely on Chronicle Queue for durable logging and message persistence and need clarity on when data is considered committed. +* Flushing every write to disk maximises durability but can severely impact throughput and latency, especially on spinning disks or busy volumes. +* Recovery behaviour after process or machine failure must be predictable and documented. +Decision Statement:: +* Chronicle Queue treats messages as committed once they are appended to the memory-mapped region and the corresponding index entry is written; operating system policies govern when data reaches physical media. +* The API and configuration allow callers to choose between more aggressive flushing (for example explicit fsync or metadata updates) and higher throughput with batched flushes. +* On restart, Chronicle Queue rebuilds any necessary in-memory state from the on-disk roll files and indexes, treating incomplete trailing writes conservatively. +Alternatives Considered:: +* Always fsync on every append:: +** Pros: strongest durability semantics within a single host. +** Cons: unacceptable performance overhead for many real-world Chronicle deployments; latency spikes tied to storage behaviour. +* Never expose explicit flush or fsync options:: +** Pros: simpler surface area. +** Cons: prevents latency-insensitive applications from opting into stronger durability; makes trade-offs implicit rather than explicit. +Rationale for Decision:: +* Delegating fine-grained flush behaviour to configuration keeps Chronicle Queue flexible enough for both low-latency and durability-focused use cases. +* A clear statement of when messages are considered committed helps users design correct upstream and downstream failure handling. +Impact & Consequences:: +* Operational runbooks must describe the chosen flush and durability profile for each deployment. +* Tests and examples should cover common configurations (for example high-throughput versus fsync-heavy modes) and document their trade-offs. +Notes/Links:: +* link:../../docs/performance.adoc[Performance] +* link:../../docs/How_it_works.adoc[How it works] + +[[CQ-NF-O-401]] +== CQ-NF-O-401 Retention and Roll File Cleanup + +Context:: +* Long-running queues can generate large numbers of roll files; without retention policies, disks will eventually fill. +* Operators need predictable ways to manage retention windows and reclaim space without violating application correctness. +Decision Statement:: +* Chronicle Queue treats retention and cleanup as operational concerns layered on top of the append-only file format. +* The core library exposes the information needed to implement retention (for example roll file naming, directory structure and indices), while higher-level tools or operational scripts perform actual deletion and archival. +Alternatives Considered:: +* Built-in automatic deletion of old roll files in the core library:: +** Pros: no external tooling required for simple use cases. +** Cons: dangerous for regulated or audit-sensitive workloads; hard to encode all possible retention policies; risk of accidental data loss. +* No documented guidance on retention:: +** Pros: smallest code and doc footprint. +** Cons: increases operational risk; makes it harder for users to design safe retention strategies. +Rationale for Decision:: +* Delegating deletion and archival to explicit operational processes reduces the chance of unexpected data loss and allows retention policies to be tailored to each deployment. +Impact & Consequences:: +* Operational documentation and examples should include retention patterns (for example time-based purge scripts) and note the risks involved. +* Monitoring should track disk usage and roll file counts to detect misconfigured or missing retention. +Notes/Links:: +* link:../../docs/How_it_works.adoc[How it works] +* link:../../docs/FAQ.adoc[FAQ] diff --git a/src/main/docs/project-requirements.adoc b/src/main/docs/project-requirements.adoc new file mode 100644 index 0000000000..771cc7bca2 --- /dev/null +++ b/src/main/docs/project-requirements.adoc @@ -0,0 +1,122 @@ += Chronicle Queue Project Requirements +:toc: +:sectnums: +:lang: en-GB +:doctype: book +:source-highlighter: rouge + +// Legend +// Requirement identifiers follow the company Nine-Box taxonomy: +// --NNN where +// Scope = QUEUE (Chronicle Queue) +// Tag = FN | NF-P | NF-S | NF-O | TEST | DOC | OPS | RISK +// NNN = sequential number with three digits +// All text is British English and ISO-8859-1 only, per AGENTS.md. + +== Scope + +Chronicle Queue is the log-structured messaging and journalling layer for the Chronicle stack. +These requirements define the core behavioural contracts, performance envelopes and operational obligations that Chronicle Queue must meet to be accepted into the Chronicle Software code-base. + +They are complementary to: + +* link:architecture-overview.adoc[architecture-overview.adoc] - describes the design and data flows. +* link:decision-log.adoc[decision-log.adoc] - records key architectural and operational decisions (`CQ-*` identifiers). +* link:../../docs/How_it_works.adoc[How it works] and link:../../docs/FAQ.adoc[FAQ.adoc] - provide additional narrative detail. + +== Functional Requirements + +[cols="1,5,4",options="header"] +|=== +|ID |Description |Verification +|[[QUEUE-FN-001]]QUEUE-FN-001 |Provide an append-only queue abstraction that durably persists all successfully appended messages to disk-backed roll files in index order, without reordering or silent loss under normal operation. |Unit and integration tests append and tail messages across roll boundaries, asserting ordering and durability; crash-recovery tests reopen queues and verify no acknowledged messages are lost. +|[[QUEUE-FN-002]]QUEUE-FN-002 |Expose forward-only reader views (`ExcerptTailer`) that can read messages sequentially from the start, end or a stored index, with each tailer maintaining an independent position. |Tests create multiple tailers over the same queue and verify that advancing one tailer does not affect the position or behaviour of others; position persistence tests reopen named tailers and continue from the previous index. +|[[QUEUE-FN-003]]QUEUE-FN-003 |Support random access by logical index, allowing a tailer to seek to a given index and resume sequential reading from there using the embedded tiered index structures. |Indexing tests write batches of messages, record selected indices and then seek back to each index, asserting that the expected message is returned within bounded time. +|[[QUEUE-FN-004]]QUEUE-FN-004 |Provide builder-based configuration (`SingleChronicleQueueBuilder`) for core queue properties including path, wire type, roll cycle, block size and index density (index count/spacing). |Builder tests construct queues with different combinations of configuration options and verify that resulting directories, roll cycles and index layouts match expectations; misconfiguration tests assert that invalid combinations are rejected. +|[[QUEUE-FN-005]]QUEUE-FN-005 |Integrate with Chronicle-Wire to allow messages to be written and read as marshalled objects, method calls or text/binary payloads over the same underlying queue files. |End-to-end tests use `MethodWriter`/`MethodReader` over a queue to send and receive strongly-typed messages, asserting correctness across restart; additional tests cover text and binary payloads with the same file format. +|=== + +== Non-Functional Requirements - Performance + +[cols="1,5,4",options="header"] +|=== +|ID |Requirement |Benchmark Target +|[[QUEUE-NF-P-001]]QUEUE-NF-P-001 |For simple messages on the same host, Chronicle Queue should support sub-microsecond median latency for append-plus-tail operations under typical hardware conditions. |Micro-benchmarks (and JLBH harnesses where applicable) measure append + read latency for representative message sizes and report median and tail latencies; results are documented in `performance.adoc` and kept within agreed budgets. +|[[QUEUE-NF-P-002]]QUEUE-NF-P-002 |Support sustained throughput in the millions of messages per second per writer thread on modern x86-64 servers without excessive tail latency growth, assuming adequate storage bandwidth. |Throughput benchmarks run under sustained load with one and multiple writer threads, capturing messages-per-second and latency percentiles; results compared against historical baselines and documented in performance reports. +|[[QUEUE-NF-P-003]]QUEUE-NF-P-003 |Maintain predictable seek and scan times when using the tiered index to jump to arbitrary indices within roll files, avoiding unbounded linear scans for typical access patterns. |Index benchmarks measure time to seek to random indices across large queues; regression tests assert that seek times remain within agreed thresholds as queue size grows. +|=== + +== Non-Functional Requirements - Security + +[cols="1,5,4",options="header"] +|=== +|ID |Requirement |Verification +|[[QUEUE-NF-S-001]]QUEUE-NF-S-001 |Ensure that queue file operations rely on Chronicle-Bytes bounds-checked accessors and avoid unchecked direct memory writes that could corrupt adjacent data or metadata. |Code reviews confirm that write paths use safe `Bytes` APIs; fuzz and stress tests append malformed or extreme data and verify queue integrity and recoverability. +|[[QUEUE-NF-S-002]]QUEUE-NF-S-002 |Document the encryption responsibility boundary: Chronicle Queue must not silently encrypt or decrypt data at rest; instead, it must allow callers to layer encryption via filesystem tools or Marshallable payloads and clearly describe this in documentation. |`decision-log.adoc` entry `CQ-NF-S-301` describes the boundary; security review and FAQ sections cross-reference this requirement; tests validate that enabling encryption at surrounding layers does not break queue semantics. +|=== + +== Non-Functional Requirements - Operability + +[cols="1,5,4",options="header"] +|=== +|ID |Requirement |Verification +|[[QUEUE-NF-O-001]]QUEUE-NF-O-001 |Expose configuration and tuning knobs via documented system properties for key behaviours such as roll-cycle checks, interrupt handling, scan latency reporting and cache sizes. |`../../docs/system-properties.adoc` lists all relevant properties with defaults and descriptions; unit tests and integration tests confirm that toggling properties affects behaviour as expected. +|[[QUEUE-NF-O-002]]QUEUE-NF-O-002 |Support time-based roll cycles and predictable roll-file naming so that operational tooling can implement retention, archival and backup without modifying the core library. |`decision-log.adoc` entry `CQ-FN-101` documents the roll-cycle model; operational runbooks demonstrate retention scripts or tooling; tests cover behaviour across roll boundaries and after manual removal of old roll files. +|[[QUEUE-NF-O-003]]QUEUE-NF-O-003 |Provide sufficient diagnostics (logging, metrics and optional consistency checks) to detect slow appenders, expensive scans and indexing anomalies without impacting hot-path latency when disabled. |System-property-driven features such as slow appender warnings and linear scan reporting are covered by tests that deliberately trigger warnings; performance tests compare overhead with features enabled vs disabled. +|=== + +== Traceability and Future Work + +This document captures the initial functional and non-functional requirement set for Chronicle Queue. +Each requirement includes a brief description of how it is verified. + +=== Requirements-to-tests mapping + +The table below links each currently documented requirement identifier to representative unit or integration tests from the Chronicle Queue test suite (as packaged in `chronicle-queue-*-tests.jar`). +It is not exhaustive but provides reviewers with concrete entry points for verification. + +[cols="1,3,4",options="header"] +|=== +|Requirement ID |Representative tests |Notes + +|QUEUE-FN-001 +|`ExcerptAppenderTest`, `ChronicleQueueTwoThreadsTest` +|Exercise append-only behaviour, durability across reopen and ordering guarantees when writing and tailing messages, including multi-threaded writer scenarios. + +|QUEUE-FN-002 +|`TailerDirectionTest`, `VisibilityOfMessagesBetweenTailorsAndAppenderTest` +|Cover forward and backward tailing behaviour and verify that independent tailers maintain their own positions over the same queue data. + +|QUEUE-FN-003 +|`ChronicleQueueIndexTest`, `CreateAtIndexTest` +|Validate tiered index creation and random access by index, including seeking to stored indices and reading back the expected messages. + +|QUEUE-FN-004 +|`RollCyclesTest`, `RollCycleDefaultingTest`, `ChronicleAppenderCycleTest` +|Exercise builder configuration for roll cycles and related properties, checking that roll file naming, cycling and defaulting behaviour match expectations. + +|QUEUE-FN-005 +|`ChronicleQueueMethodsWithoutParametersTest`, `DtoBytesMarshallableTest`, `ProxyTest` +|Verify Chronicle Wire integration using method writers/readers and marshalled DTOs over queue files, including proxy-based service interfaces. + +|QUEUE-NF-P-001, QUEUE-NF-P-002, QUEUE-NF-P-003 +|`SingleChroniclePerfMainTest`, `RareAppenderLatencyTest`, `MoveToCycleMultiThreadedStressTest` +|Provide micro-benchmarks and stress tests that measure append-plus-tail latency, sustained throughput and index seek characteristics under load. + +|QUEUE-NF-S-001, QUEUE-NF-S-002 +|`IncompleteMessageTest`, `ReaderResizesFileTest`, `DeleteFileTest` +|Exercise behaviour in the presence of truncated data, file resizing and deletion scenarios, helping to validate bounds checking and recovery expectations. + +|QUEUE-NF-O-001, QUEUE-NF-O-002, QUEUE-NF-O-003 +|`DumpQueueMainTest`, `ChronicleReaderMainTest`, `DumpMainTest`, `InternalDumpMainTest` +|Cover operational tooling and diagnostics, ensuring that queue contents can be inspected, dumped and scanned using documented command-line entry points. +|=== + +For the requirements explicitly listed in this document (the `QUEUE-FN-*` and `QUEUE-NF-*` identifiers above), there is at least one representative test or benchmark, so functional and non-functional traceability for this set is effectively 100 per cent. +When new requirements are added, corresponding entries should be appended to this table. + +Future work should: + +* expand the requirements catalogue as new behaviours are formalised; +* continue to link `QUEUE-*` identifiers from architecture and decision documents (for example in `architecture-overview.adoc` and `decision-log.adoc`); +* add explicit `TEST` and `OPS` requirements describing regression suites and operational runbooks once they are stabilised, and keep the mapping between key `QUEUE-*` requirements and representative test classes in sync with the evolving test suite. diff --git a/src/main/docs/security-review.adoc b/src/main/docs/security-review.adoc new file mode 100644 index 0000000000..b30a4eab84 --- /dev/null +++ b/src/main/docs/security-review.adoc @@ -0,0 +1,192 @@ += Chronicle Queue Security Review +:toc: +:sectnums: +:lang: en-GB +:toclevels: 2 +:doctype: book +:source-highlighter: rouge + +This document summarises the main security considerations for Chronicle Queue. +It complements the security-related requirements in `project-requirements.adoc` (`QUEUE-NF-S-_`, `QUEUE-NF-O-_`) and the architectural decisions recorded in `decision-log.adoc` (for example `CQ-NF-S-301` and `CQ-NF-O-401`). + +== Trust Zone and Responsibilities + +Chronicle Queue operates in the _Core (Zone B)_ as defined in `Chronicle-Quality-Rules/src/main/docs/architectural-standards.adoc`. +In this role it assumes that: + +* untrusted inputs have been validated at the edge (for example by Chronicle-Network and Chronicle-Wire); +* authentication and authorisation decisions are enforced by services and applications built on top of the queue; +* this module is responsible for preserving integrity, ordering and durability of messages once accepted. + +The remainder of this document describes how Chronicle Queue fulfils those Core responsibilities and where it deliberately relies on the wider platform (for example for encryption, identity and access management). + +== Scope and Threat Model + +Chronicle Queue is a low-level, log-structured messaging and journalling component. +It: + +* runs in-process as a library linked into applications; +* stores messages on local or shared file systems using memory-mapped files; +* exposes no network listeners or authentication mechanisms by itself. + +Chronicle Queue assumes: + +* the running process is trusted and operates in a controlled environment; +* file-system permissions and network access are managed by the host platform and surrounding infrastructure; +* untrusted input is validated at system boundaries (for example REST endpoints, message brokers) before reaching queue writers. + +Chronicle Queue does *not* provide: + +* transport-level encryption (for example TLS); +* application-level authentication or authorisation; +* application-specific input validation for message payloads. + +Those capabilities belong to higher-level modules such as Chronicle-Network, Chronicle-Services and the caller’s own code. + +The primary security risks within Chronicle Queue itself are: + +* data corruption or loss due to misuse of roll files or sharing patterns; +* exposure of sensitive data at rest if encryption and access controls are not configured externally; +* resource exhaustion through unbounded queue growth or misconfigured roll/retention policies; +* misuse of off-heap or memory-mapped storage in the underlying Chronicle-Bytes layer. + +== Data at Rest and Encryption Boundary (QUEUE-NF-S-002, CQ-NF-S-301) + +Chronicle Queue stores messages in plain form in `.cq4` roll files. +Requirement `QUEUE-NF-S-002` as follows: + +* Chronicle Queue does not transparently encrypt or decrypt queue files by default. + +Implications: + +* Sensitive data written to queues must be treated as sensitive on disk; deployments should: +** use OS-level permissions to restrict access to queue directories; +** use encrypted volumes where regulatory requirements dictate; +** avoid placing queues on shared or world-readable file systems. +* Adding transparent encryption inside Chronicle Queue would introduce latency and complexity and is therefore explicitly not the default behaviour. + +When integrating Chronicle Queue into security-sensitive systems: + +* document which queues contain sensitive data and how the underlying volumes are protected; +* ensure that any application-level encryption or signing is applied before messages reach the queue; +* avoid relying on queue file names or directory structures as the sole means of access control. + +Chronicle Queue does not currently apply cryptographic hashing to queue contents or roll files. +Where deployments require additional integrity guarantees (for example signed payloads or checksummed archives), those concerns should be addressed at the application or storage layer. +Cryptographic keys must be managed by external key-management systems or application code; they are not embedded in Chronicle Queue itself and queue files should not be used as a long-term store for raw keys. + +== Off-Heap and Memory-Mapped Storage (QUEUE-NF-S-001) + +Chronicle Queue is built on top of Chronicle-Bytes and uses off-heap and memory-mapped storage for both data and indices. +Requirement `QUEUE-NF-S-001` mandates that: + +* queue code uses bounds-checked `Bytes` APIs rather than unchecked pointer arithmetic; +* new code paths that manipulate off-heap or mapped memory are implemented in Chronicle-Bytes where possible and reuse its safety mechanisms; +* queue-specific logic does not introduce unchecked casts or raw address manipulation beyond what Chronicle-Bytes already exposes. + +Mitigations: + +* Bounds checking and memory-safety concerns are primarily handled in Chronicle-Bytes (`CB-NF-S-*` requirements and `security-review.adoc` for that module). +* Queue tests that exercise roll boundaries, index structures and crash-recovery scenarios help to detect corruption caused by misuse. +* SpotBugs and other static-analysis tools are used at the repository level to detect unsafe patterns. + +When modifying queue internals: + +* prefer high-level `Bytes` operations and existing helper methods; +* avoid caching raw addresses or direct buffer references unless strictly necessary for performance; +* ensure new code paths are covered by tests that exercise extreme sizes, roll boundaries and restart scenarios. + +== Queue Sharing, Roll Cycles and Retention (QUEUE-NF-O-002, CQ-FN-101, CQ-NF-O-401) + +Chronicle Queue is designed around time-based roll cycles and append-only roll files (see `CQ-FN-101` and `architecture-overview.adoc`). +From a security and compliance perspective: + +* long-lived queues can accumulate large numbers of roll files containing historical data; +* deletion or alteration of roll files has direct impact on auditability and forensics; +* misconfigured retention policies can either: +** retain data longer than required (risking regulatory non-compliance), or +** delete data too aggressively (risking loss of evidence or operational insight). + +Requirement `QUEUE-NF-O-002` and decision `CQ-NF-O-401` state that: + +* retention and roll-file cleanup are explicit operational responsibilities; +* Chronicle Queue exposes predictable naming and directory layout so scripts and tooling can act safely; +* the core library avoids automatic deletion of data to reduce the risk of unexpected loss. + +Recommended practices: + +* document a clear retention policy per queue (for example 30 days of intraday trading data); +* implement audited scripts or services that: +** archive or export roll files to longer-term storage; +** delete files only after policy preconditions are met; +* monitor disk utilisation and roll-file counts and alert when thresholds are approached. + +== Logging, Diagnostics and Sensitive Data + +Chronicle Queue exposes diagnostics via: + +* system properties such as `chronicle.queue.warnSlowAppenderMs` and `chronicle.queue.report.linear.scan.latency`; +* optional consistency checks and index validation; +* operational logging around roll events and exceptions. + +These facilities are intended for trusted environments. +Potential issues: + +* message contents or derived fields may appear in logs if application code logs them directly, or if debug logging is enabled at higher layers; +* misconfigured log levels can expose more operational detail than necessary. + +Mitigations: + +* treat queue contents as sensitive and restrict log verbosity in production, particularly for payloads that include client identifiers or financial data; +* ensure logging configuration is part of security reviews and operational runbooks; +* prefer structured logging that redacts or truncates sensitive fields where appropriate. + +== Secure Configuration and Dependencies (QUEUE-NF-O-001) + +Requirement `QUEUE-NF-O-001` calls for documented configuration knobs, particularly system properties that alter behaviour in ways relevant to security and operability. + +Key aspects: + +* `system-properties.adoc` documents the supported properties, their defaults and their effects; +* configuration handling is performed using standard Java mechanisms (`System.getProperty` etc.), with no dynamic code loading based on untrusted values; +* queue code does not fetch remote configurations or execute scripts; it assumes configuration is provided by trusted deployment tooling. + +Dependency considerations: + +* Chronicle Queue depends on other Chronicle libraries (Core, Bytes, Wire, Threads) and a small set of external libraries; +* dependency management and vulnerability scanning (for example CVE checks) are handled at the repository or build level, not in this module alone. + +When changing configuration surfaces or adding new dependencies: + +* document any new system properties or environment options in `system-properties.adoc` and, where appropriate, in `project-requirements.adoc`; +* avoid introducing dependencies that bring in heavy reflection or deserialisation frameworks without a clear need; +* ensure the build pipeline continues to run static analysis and dependency checks. + +== Vulnerability Management and Static Analysis + +Chronicle Queue participates in the wider Chronicle vulnerability-management process described in the root-level compliance and quality playbooks. +In particular: + +* static analysis (Checkstyle and SpotBugs) is wired via the shared quality configuration from `root-parent-pom` and `Chronicle-Quality-Rules`; +* Java 21 quality builds for this module are required to be clean for both tools before changes are merged; +* dependency vulnerability scanning (for example CVE checks) is performed at the repository level and tracked in central compliance artefacts rather than in this document alone. + +Module-specific follow-ups (for example reviewing new SpotBugs findings or analysing security-related GitHub issues) are recorded in `TODO.md` and summarised in `TODO_STATUS.md`. +Any future suppressions of static-analysis findings in this module should be justified with a short comment and, where relevant, a note in this security review. + +== Summary and Future Work + +Chronicle Queue’s security posture is characterised by: + +* a clear separation of concerns between: +** low-level storage and indexing (this module), +** serialisation and schema handling (Chronicle-Wire), +** networking and authentication (Chronicle-Network and services), +* explicit documentation of the encryption responsibility boundary and retention responsibilities; +* reliance on Chronicle-Bytes for safe off-heap and memory-mapped access. + +Further improvements may include: + +* a more formal threat model for deployments that expose queues to partially untrusted producers; +* additional fuzz and stress testing around index structures and roll-cycle transitions; +* richer guidance for combining Chronicle Queue with Chronicle-Network and external security controls in regulated environments. From b392996c8ebada1ff464a7bc88fb00ad3dbb2af1 Mon Sep 17 00:00:00 2001 From: Peter Lawrey Date: Thu, 11 Dec 2025 14:39:51 +0000 Subject: [PATCH 6/6] revert two changes --- pom.xml | 2 +- .../chronicle/queue/internal/util/InternalFileUtil.java | 7 ++----- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/pom.xml b/pom.xml index 7b0cd05768..cd0693842a 100644 --- a/pom.xml +++ b/pom.xml @@ -31,7 +31,7 @@ net.openhft third-party-bom - 3.27ea7 + 3.27ea5 pom import diff --git a/src/main/java/net/openhft/chronicle/queue/internal/util/InternalFileUtil.java b/src/main/java/net/openhft/chronicle/queue/internal/util/InternalFileUtil.java index 8893c3748f..ace02c6760 100644 --- a/src/main/java/net/openhft/chronicle/queue/internal/util/InternalFileUtil.java +++ b/src/main/java/net/openhft/chronicle/queue/internal/util/InternalFileUtil.java @@ -122,9 +122,6 @@ public static boolean hasQueueSuffix(@NotNull File file) { * supported for the current platform (e.g. Windows). */ public static FileState state(@NotNull File file) { - if (!getAllOpenFilesIsSupportedOnOS()) { - return stateWindows(file); - } try { return state(file, getAllOpenFiles()); } catch (IOException e) { @@ -150,7 +147,7 @@ public static FileState state(@NotNull File file, Map allOpenFil assertOsSupported(); if (!file.exists()) return FileState.NON_EXISTENT; final String absolutePath = file.getAbsolutePath(); - return allOpenFiles.containsKey(absolutePath) + return allOpenFiles.keySet().contains(absolutePath) ? FileState.OPEN : FileState.CLOSED; } @@ -219,7 +216,7 @@ public static Map getAllOpenFiles() throws IOException { */ private static class ProcFdWalker extends SimpleFileVisitor { - private static final int PID_PATH_INDEX = 1; // where is the pid for process holding file open represented in path? + private final static int PID_PATH_INDEX = 1; // where is the pid for process holding file open represented in path? private final Map openFiles = new HashMap<>(); @Override