Add ETag and If-None-Match support to GET /node/ledger-chunk/{chunk_name} (#7653)

Copilot · achamayou · Copilot · web-flow · commit e6ca9ed6dd74 · 2026-02-13T18:48:10.000Z
Co-authored-by: copilot-swe-agent[bot] &lt;198982749+Copilot@users.noreply.github.com&gt;
Co-authored-by: achamayou &lt;4016369+achamayou@users.noreply.github.com&gt;
Co-authored-by: Amaury Chamayou &lt;amchamay@microsoft.com&gt;
Co-authored-by: Amaury Chamayou &lt;amaury@xargs.fr&gt;
Co-authored-by: Copilot &lt;175728472+Copilot@users.noreply.github.com&gt;
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 - `GET` and `HEAD` `/node/ledger-chunk?since={seqno}` and `/node/ledger-chunk/{chunk_name}` endpoints, gated by the `LedgerChunkDownload` RPC interface operator feature. See [documentation](https://microsoft.github.io/CCF/main/operations/ledger_snapshot.html#download-endpoints) for more detail.
 - `GET` and `HEAD` `/node/ledger-chunk/{chunk_name}` and `/node/snapshot/{snapshot_name}` now support the `Want-Repr-Digest` request header and return the `Repr-Digest` response header accordingly (RFC 9530). Supported algorithms are `sha-256`, `sha-384`, and `sha-512`. If no supported algorithm is requested, the server defaults to `sha-256` (#7650).
+- `ETag` and `If-None-Match` support on `GET /node/ledger-chunk/{chunk_name}`, using SHA-256 by default for the `ETag` response header. Clients can supply `If-None-Match` with `sha-256`, `sha-384`, or `sha-512` digest ETags to avoid re-downloading unchanged content (#7652).
 
 ### Changed
 
diff --git a/doc/operations/ledger_snapshot.rst b/doc/operations/ledger_snapshot.rst
@@ -50,26 +50,12 @@ Download Endpoints
 In order to faciliate long term backup of the ledger files (also called chunks), nodes can enable HTTP endpoints that allow a client to download committed ledger files.
 The `LedgerChunkDownload` feature must be added to `enabled_operator_features` on the relevant `rpc_interfaces` entries in the node configuration.
 
-1. :http:GET:`/node/ledger-chunk/{chunk_name}` and :http:HEAD:`/node/ledger-chunk/{chunk_name}`
-
-These endpoints allow downloading a specific ledger chunk by name, where `<chunk-name>` is of the form `ledger_<start_seqno>-<end_seqno>.committed`.
-They support the HTTP `Range` header for partial downloads, and the `HEAD` method for clients to query metadata such as the total size without downloading the full chunk.
-They also populate the `x-ms-ccf-ledger-chunk-name` response header with the name of the chunk being served.
-
-These endpoints also support the ``Want-Repr-Digest`` request header (`RFC 9530 <https://www.rfc-editor.org/rfc/rfc9530>`_).
-When set, the response will include a ``Repr-Digest`` header containing the digest of the full representation of the file.
-Supported algorithms are ``sha-256``, ``sha-384``, and ``sha-512``. If the header contains only unsupported or invalid algorithms, the server defaults to ``sha-256`` (as permitted by `RFC 9530 Appendix C.2 <https://www.rfc-editor.org/rfc/rfc9530#appendix-C.2>`_).
-For example, a client sending ``Want-Repr-Digest: sha-256=1`` will receive a header such as ``Repr-Digest: sha-256=:AEGPTgUMw5e96wxZuDtpfm23RBU3nFwtgY5fw4NYORo=:`` in the response.
-This allows clients to verify the integrity of downloaded files and avoid re-downloading files they already hold by comparing digests.
-
-.. note:: The ``Want-Repr-Digest`` / ``Repr-Digest`` support also applies to the snapshot download endpoints (``/node/snapshot/{snapshot_name}``).
-
-2. :http:GET:`/node/ledger-chunk` and :http:HEAD:`/node/ledger-chunk`, both taking a `seqno` query parameter.
+1. :http:GET:`/node/ledger-chunk` and :http:HEAD:`/node/ledger-chunk`, both taking a `seqno` query parameter.
 
 These endpoints can be used by a client to download the next ledger chunk including a given sequence number `<seqno>`.
-The redirects to the appropriate chunk if it exists, using the previous set of endpoints, or returns a `404 Not Found` response if no such chunk is available.
+They redirect to the appropriate chunk if it exists, using the endpoints described below, or return a `404 Not Found` response if no such chunk is available.
 
-In the usual case, a downloading client will first hit a Backup, and will eventually want to download files recent enough that only the primary can provide them:
+In the typical case, a requesting client will first hit a Backup, and will eventually work its way to chunks recent enough that only the primary can provide them:
 
 .. mermaid::
 
@@ -120,6 +106,56 @@ then the following sequence can occur:
         Backup->>-Client: 308 Location: https://primary/node/ledger-chunk?since=51
         Client->>+Primary: GET /node/ledger-chunk?since=101
 
+2. :http:GET:`/node/ledger-chunk/{chunk_name}` and :http:HEAD:`/node/ledger-chunk/{chunk_name}`
+
+These endpoints allow downloading a specific ledger chunk by name, where `<chunk-name>` is of the form `ledger_<start_seqno>-<end_seqno>.committed`.
+They support the HTTP `Range` header for partial downloads, and the `HEAD` method for clients to query metadata such as the total size without downloading the full chunk.
+They also populate the `x-ms-ccf-ledger-chunk-name` response header with the name of the chunk being served.
+
+Want-Repr-Digest and Repr-Digest
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These endpoints also support the ``Want-Repr-Digest`` request header (`RFC 9530 <https://www.rfc-editor.org/rfc/rfc9530>`_).
+When set, the response will include a ``Repr-Digest`` header containing the digest of the full representation of the file.
+Supported algorithms are ``sha-256``, ``sha-384``, and ``sha-512``. If the header contains only unsupported or invalid algorithms, the server defaults to ``sha-256`` (as permitted by `RFC 9530 Appendix C.2 <https://www.rfc-editor.org/rfc/rfc9530#appendix-C.2>`_).
+For example, a client sending ``Want-Repr-Digest: sha-256=1`` will receive a header such as ``Repr-Digest: sha-256=:AEGPTgUMw5e96wxZuDtpfm23RBU3nFwtgY5fw4NYORo=:`` in the response.
+This allows clients to verify the integrity of downloaded files and avoid re-downloading files they already hold by comparing digests.
+
+.. note:: The ``Want-Repr-Digest`` / ``Repr-Digest`` support also applies to the snapshot download endpoints (:http:GET:`/node/snapshot/{snapshot_name}` and :http:HEAD:`/node/snapshot/{snapshot_name}`).
+
+ETag and If-None-Match
+^^^^^^^^^^^^^^^^^^^^^^
+
+``GET /node/ledger-chunk/{chunk_name}`` supports ``ETag`` and ``If-None-Match`` headers, allowing clients to atomically check whether a chunk (or a range of a chunk) has changed and re-download it in a single request, without needing a separate metadata query first.
+Every successful ``GET`` response includes an ``ETag`` header whose value uses the `RFC 9530 <https://www.rfc-editor.org/rfc/rfc9530>`_ digest format: ``"sha-256=:<base64_digest>:"``, where ``<base64_digest>`` is the base64-encoded SHA-256 digest of the returned content (which may be a sub-range when the ``Range`` header is used).
+
+.. note:: ETag values must be surrounded by double quotes, as per `RFC 7232 <https://www.rfc-editor.org/rfc/rfc7232#section-2.3>`_.
+
+Clients can send an ``If-None-Match`` request header containing one or more ETags. If the content matches any of the provided ETags, the server responds with ``304 Not Modified`` instead of re-sending the body. The supported digest algorithms are ``sha-256``, ``sha-384``, and ``sha-512``.
+When the client already holds a chunk and wants to check if it has changed, it sends the previously received ETag in ``If-None-Match``. If the content has not changed, the server responds with ``304 Not Modified``, saving bandwidth:
+
+.. note:: The node currently defaults to ``sha-256`` for ETags, but clients can also send other supported digest algorithms in the ``If-None-Match`` header, and the server will use the first supported one it finds. For example, if the client sends ``If-None-Match: "sha-384=:AAAA...=:", "sha-256=:47DEQpj8HBSa+/TImW...=:"``, the server will use the ``sha-384`` digest if it supports it, and fall back to ``sha-256`` otherwise.
+
+.. mermaid::
+
+    sequenceDiagram
+        Note over Client: Client already has chunk with known ETag
+        Client->>+Node: GET /node/ledger-chunk/ledger_1-100.committed<br/>If-None-Match: "sha-256=:47DEQpj8HBSa+/TImW...=:"
+        Note over Node: Computes digest, matches If-None-Match
+        Node->>-Client: 304 Not Modified<br/>ETag: "sha-256=:47DEQpj8HBSa+/TImW...=:"
+        Note over Client: No body transferred, client keeps existing copy
+
+If the ``If-None-Match`` ETag does not match the current content (e.g. the client has an outdated copy, or is checking a different chunk), the server returns the full content as a fresh download:
+
+.. mermaid::
+
+    sequenceDiagram
+        Note over Client: Client sends an ETag that does not match
+        Client->>+Node: GET /node/ledger-chunk/ledger_1-100.committed<br/>If-None-Match: "sha-256=:AAAA...=:"
+        Note over Node: Computes digest, does not match If-None-Match
+        Node->>-Client: 200 OK<br/>ETag: "sha-256=:47DEQpj8HBSa+/TImW...=:"<br/><Chunk Contents>
+        Note over Client: Client stores chunk and new ETag for future requests
+
 Snapshots
 ---------
 
diff --git a/include/ccf/http_consts.h b/include/ccf/http_consts.h
@@ -21,6 +21,8 @@ namespace ccf
       static constexpr auto HOST = "host";
       static constexpr auto LOCATION = "location";
       static constexpr auto RANGE = "range";
+      static constexpr auto ETAG = "etag";
+      static constexpr auto IF_NONE_MATCH = "if-none-match";
       static constexpr auto REPR_DIGEST = "repr-digest";
       static constexpr auto RETRY_AFTER = "retry-after";
       static constexpr auto TRAILER = "trailer";
diff --git a/include/ccf/http_etag.h b/include/ccf/http_etag.h
@@ -3,12 +3,28 @@
 #pragma once
 
 #define FMT_HEADER_ONLY
+#include <exception>
 #include <regex>
 #include <set>
 #include <string>
 
 namespace ccf::http
 {
+  /** Exception thrown when the Matcher encounters an invalid ETag header. */
+  class MatcherError : public std::exception
+  {
+  private:
+    std::string msg;
+
+  public:
+    MatcherError(std::string msg_) : msg(std::move(msg_)) {}
+
+    [[nodiscard]] const char* what() const noexcept override
+    {
+      return msg.c_str();
+    }
+  };
+
   /** Utility class to resolve If-Match and If-None-Match as described
    * in https://www.rfc-editor.org/rfc/rfc9110#field.if-match
    */
@@ -33,7 +49,7 @@ namespace ccf::http
         return;
       }
 
-      std::regex etag_rx(R"(\"([0-9a-f]+)\",?\s*)");
+      std::regex etag_rx(R"(\"([^\"]+)\",?\s*)");
       auto etags_begin =
         std::sregex_iterator(match_header.begin(), match_header.end(), etag_rx);
       auto etags_end = std::sregex_iterator();
@@ -43,7 +59,7 @@ namespace ccf::http
       {
         if (i->position() != last_matched)
         {
-          throw std::runtime_error("Invalid If-Match header");
+          throw MatcherError("Invalid If-Match header");
         }
         const std::smatch& match = *i;
         if_etags.insert(match[1].str());
@@ -54,7 +70,7 @@ namespace ccf::http
 
       if (last_matched != last_index_in_header || if_etags.empty())
       {
-        throw std::runtime_error("Invalid If-Match header");
+        throw MatcherError("Invalid If-Match header");
       }
     }
 
diff --git a/samples/apps/logging/logging.cpp b/samples/apps/logging/logging.cpp
@@ -843,25 +843,45 @@ namespace loggingapp
             // side-effect.
             if (match_headers.if_match.has_value())
             {
-              ccf::http::Matcher matcher(match_headers.if_match.value());
-              if (!matcher.matches(etag))
+              try
+              {
+                ccf::http::Matcher matcher(match_headers.if_match.value());
+                if (!matcher.matches(etag))
+                {
+                  return ccf::make_error(
+                    HTTP_STATUS_PRECONDITION_FAILED,
+                    ccf::errors::PreconditionFailed,
+                    "Resource has changed.");
+                }
+              }
+              catch (const ccf::http::MatcherError& e)
               {
                 return ccf::make_error(
-                  HTTP_STATUS_PRECONDITION_FAILED,
-                  ccf::errors::PreconditionFailed,
-                  "Resource has changed.");
+                  HTTP_STATUS_BAD_REQUEST,
+                  ccf::errors::InvalidHeaderValue,
+                  e.what());
               }
             }
 
             if (match_headers.if_none_match.has_value())
             {
-              ccf::http::Matcher matcher(match_headers.if_none_match.value());
-              if (matcher.matches(etag))
+              try
+              {
+                ccf::http::Matcher matcher(match_headers.if_none_match.value());
+                if (matcher.matches(etag))
+                {
+                  return ccf::make_error(
+                    HTTP_STATUS_PRECONDITION_FAILED,
+                    ccf::errors::PreconditionFailed,
+                    "Resource has changed.");
+                }
+              }
+              catch (const ccf::http::MatcherError& e)
               {
                 return ccf::make_error(
-                  HTTP_STATUS_PRECONDITION_FAILED,
-                  ccf::errors::PreconditionFailed,
-                  "Resource has changed.");
+                  HTTP_STATUS_BAD_REQUEST,
+                  ccf::errors::InvalidHeaderValue,
+                  e.what());
               }
             }
           }
@@ -935,23 +955,43 @@ namespace loggingapp
 
           if (match_headers.if_match.has_value())
           {
-            ccf::http::Matcher matcher(match_headers.if_match.value());
-            if (!matcher.matches(etag))
+            try
+            {
+              ccf::http::Matcher matcher(match_headers.if_match.value());
+              if (!matcher.matches(etag))
+              {
+                return ccf::make_error(
+                  HTTP_STATUS_PRECONDITION_FAILED,
+                  ccf::errors::PreconditionFailed,
+                  "Resource has changed.");
+              }
+            }
+            catch (const ccf::http::MatcherError& e)
             {
               return ccf::make_error(
-                HTTP_STATUS_PRECONDITION_FAILED,
-                ccf::errors::PreconditionFailed,
-                "Resource has changed.");
+                HTTP_STATUS_BAD_REQUEST,
+                ccf::errors::InvalidHeaderValue,
+                e.what());
             }
           }
 
           // On a GET, If-None-Match passing returns 304 Not Modified
           if (match_headers.if_none_match.has_value())
           {
-            ccf::http::Matcher matcher(match_headers.if_none_match.value());
-            if (matcher.matches(etag))
+            try
+            {
+              ccf::http::Matcher matcher(match_headers.if_none_match.value());
+              if (matcher.matches(etag))
+              {
+                return ccf::make_redirect(HTTP_STATUS_NOT_MODIFIED);
+              }
+            }
+            catch (const ccf::http::MatcherError& e)
             {
-              return ccf::make_redirect(HTTP_STATUS_NOT_MODIFIED);
+              return ccf::make_error(
+                HTTP_STATUS_BAD_REQUEST,
+                ccf::errors::InvalidHeaderValue,
+                e.what());
             }
           }
 
@@ -1030,22 +1070,42 @@ namespace loggingapp
 
             if (match_headers.if_match.has_value())
             {
-              ccf::http::Matcher matcher(match_headers.if_match.value());
-              if (!matcher.matches(etag))
+              try
+              {
+                ccf::http::Matcher matcher(match_headers.if_match.value());
+                if (!matcher.matches(etag))
+                {
+                  return ccf::make_error(
+                    HTTP_STATUS_PRECONDITION_FAILED,
+                    ccf::errors::PreconditionFailed,
+                    "Resource has changed.");
+                }
+              }
+              catch (const ccf::http::MatcherError& e)
               {
                 return ccf::make_error(
-                  HTTP_STATUS_PRECONDITION_FAILED,
-                  ccf::errors::PreconditionFailed,
-                  "Resource has changed.");
+                  HTTP_STATUS_BAD_REQUEST,
+                  ccf::errors::InvalidHeaderValue,
+                  e.what());
               }
             }
 
             if (match_headers.if_none_match.has_value())
             {
-              ccf::http::Matcher matcher(match_headers.if_none_match.value());
-              if (matcher.matches(etag))
+              try
               {
-                return ccf::make_redirect(HTTP_STATUS_NOT_MODIFIED);
+                ccf::http::Matcher matcher(match_headers.if_none_match.value());
+                if (matcher.matches(etag))
+                {
+                  return ccf::make_redirect(HTTP_STATUS_NOT_MODIFIED);
+                }
+              }
+              catch (const ccf::http::MatcherError& e)
+              {
+                return ccf::make_error(
+                  HTTP_STATUS_BAD_REQUEST,
+                  ccf::errors::InvalidHeaderValue,
+                  e.what());
               }
             }
           }
diff --git a/src/http/test/http_etag_test.cpp b/src/http/test/http_etag_test.cpp
@@ -39,37 +39,69 @@ TEST_CASE("If-Match: \"abc\", \"def\"")
 TEST_CASE("If-Match invalid inputs")
 {
   REQUIRE_THROWS_AS_MESSAGE(
-    ccf::http::Matcher im(""), std::runtime_error, "Invalid If-Match header");
+    ccf::http::Matcher im(""),
+    ccf::http::MatcherError,
+    "Invalid If-Match header");
   REQUIRE_THROWS_AS_MESSAGE(
     ccf::http::Matcher im("not etags"),
-    std::runtime_error,
+    ccf::http::MatcherError,
     "Invalid If-Match header");
   REQUIRE_THROWS_AS_MESSAGE(
     ccf::http::Matcher im("\"abc\", not etags"),
-    std::runtime_error,
+    ccf::http::MatcherError,
     "Invalid If-Match header");
   REQUIRE_THROWS_AS_MESSAGE(
     ccf::http::Matcher im("not etags, \"abc\""),
-    std::runtime_error,
+    ccf::http::MatcherError,
     "Invalid If-Match header");
   REQUIRE_THROWS_AS_MESSAGE(
     ccf::http::Matcher im("W/\"abc\""),
-    std::runtime_error,
+    ccf::http::MatcherError,
     "Invalid If-Match header");
   REQUIRE_THROWS_AS_MESSAGE(
     ccf::http::Matcher im("W/\"abc\", \"def\""),
-    std::runtime_error,
+    ccf::http::MatcherError,
     "Invalid If-Match header");
   REQUIRE_THROWS_AS_MESSAGE(
     ccf::http::Matcher im("\"abc\", \"def"),
-    std::runtime_error,
+    ccf::http::MatcherError,
     "Invalid If-Match header");
   REQUIRE_THROWS_AS_MESSAGE(
     ccf::http::Matcher im("\"abc\",, \"def\""),
-    std::runtime_error,
+    ccf::http::MatcherError,
     "Invalid If-Match header");
   REQUIRE_THROWS_AS_MESSAGE(
     ccf::http::Matcher im(",\"abc\""),
-    std::runtime_error,
+    ccf::http::MatcherError,
     "Invalid If-Match header");
+}
+
+TEST_CASE("If-None-Match with RFC 9530 digest ETag format")
+{
+  // Single sha-256 ETag in RFC 9530 structured field format
+  {
+    ccf::http::Matcher im(
+      "\"sha-256=:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=:\"");
+    REQUIRE(!im.is_any());
+    REQUIRE(
+      im.matches("sha-256=:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=:"));
+    REQUIRE(!im.matches("sha-256=:AAAA:"));
+  }
+
+  // Multiple algorithm ETags
+  {
+    ccf::http::Matcher im(
+      "\"sha-256=:abc=:\", \"sha-384=:def=:\", \"sha-512=:ghi=:\"");
+    REQUIRE(im.matches("sha-256=:abc=:"));
+    REQUIRE(im.matches("sha-384=:def=:"));
+    REQUIRE(im.matches("sha-512=:ghi=:"));
+    REQUIRE(!im.matches("sha-256=:000=:"));
+  }
+
+  // Wildcard still works
+  {
+    ccf::http::Matcher im("*");
+    REQUIRE(im.is_any());
+    REQUIRE(im.matches("sha-256=:anything:"));
+  }
 }
diff --git a/src/node/rpc/file_serving_handlers.h b/src/node/rpc/file_serving_handlers.h
diff --git a/tests/e2e_operations.py b/tests/e2e_operations.py
