feat(bom): map and createreleases consider PURL qualifiers

This adds support for PURLs with qualifiers, introducing the following
semantics: Only qualifiers specified in the BOM are compared.  If
entries are found where all of them match, only those are returned.

Author: gernot-h

ChangeLog.md

@@ -8,8 +8,8 @@
 ## NEXT
 
 * `bom map`: The options `--dbx` and `-all` were replaced by `--matchmode`.
-* `bom map --matchmode full-search` allows full search for the best possible
-  matches and report all of them, see the discussion in `Readme_Mapping.md`.
+* `bom map`: new `--matchmode` options `full-search` (report all best matches) and
+  `qualifier-match` (consider PackageURL qualifiers). See `Readme_Mapping.md`.
 
 ## 2.9.1
 

Readme_Mapping.md

@@ -45,20 +45,36 @@ and version etc.
 
 ## Notes on id mapping / PackageURL mapping
 
-CaPyCli supports mapping releases by the PackageURL. As encoding of a
-PackageURL is not unique (some characters *may* use URL encoding, qualifiers
+CaPyCli supports mapping **releases** by the PackageURL. As encoding of a
+PackageURL is not unique (some characters may be percent-encoded, qualifiers
 can be given in random order etc.), we can't just do a string comparison, but
 instead *all* SW360 releases with PackageURLs (using external id `package-url`)
 are retrieved and decoded. When your input BOM specifies a `purl` field, then
 the PackageURL is compared field by field (type, namespace, name, version) for
 a `FULL_MATCH_BY_ID`.
 
-Also, components will be mapped by PackageURL and if a match is found, the
+Also, **components** will be mapped by PackageURL and if a match is found, the
 `capycli:componentId` property will be added to the output BOM item. Components
 can be identified directly by their external id `package-url` or as fallback
 also by the `package-url`s of their releases.
 
-PackageURL subpath and qualifiers are currently ignored during PURL matching.
+PackageURL **qualifiers** (like `?distro=alpine-3.21&package-id=3a23`) will be
+considered when using `bom map --matchmode qualifier-match`. In some cases,
+qualifiers are essential for correct mapping, but many scanners also include
+non-essential qualifiers in their SBOMs. And the distinction might be
+challenging: while `distro` is crucial for correct mapping of Alpine packages
+(same package release can have different patches in different Alpine releases),
+but for Debian, `distro` is unnecessary since package versions are already
+unique. So we use the following rules to balance accuracy and practicality:
+
+* Only the qualifiers specified in the input BOM are considered during matching,
+  qualifiers only present in SW360 releases are ignored. So you can control
+  matching by removing the unwanted qualifiers in your SBOM.
+* If one or more SW360 releases are found where *all* qualifiers specified in the
+  input BOM match, *only* these releases are added to the output BOM. Otherwise,
+  qualifiers will be ignored, so all release matches will be added.
+
+PackageURL subpath is currently ignored during PURL matching.
 
 ## Example 1: Very Simple, Full Match
 

capycli/bom/create_components.py

@@ -399,7 +399,7 @@ def update_release(self, cx_comp: Component, release_data: Dict[str, Any]) -> No
                         bom_purl = packageurl.PackageURL.from_string(
                             data["externalIds"][repository_type])
                         sw360_purls = PurlUtils.get_purl_list_from_sw360_object(release_data)
-                        id_match = PurlUtils.contains(sw360_purls, bom_purl)
+                        id_match = PurlUtils.contains(sw360_purls, bom_purl, compare_qualifiers=True)
                     except ValueError:
                         pass
                     if not id_match:

capycli/bom/map_bom.py

@@ -60,6 +60,7 @@ def __init__(self) -> None:
         self.purl_service: Optional[PurlService] = None
         self.no_match_by_name_only = True
         self.full_search = False
+        self.qualifier_match = False
 
     def is_id_match(self, release: Dict[str, Any], component: Component) -> bool:
         """Determines whether this release is a match via identifier for the specified SBOM item"""
@@ -761,7 +762,9 @@ def map_bom_commons(self, component: Component) -> MapResult:
         # search release and component by purl which is independent of the component cache.
         if component.purl:
             result.component_hrefs = self.external_id_svc.search_components_by_purl(component.purl)
-            result.release_hrefs = self.external_id_svc.search_releases_by_purl(component.purl)
+            r = self.external_id_svc.search_releases_by_purl(component.purl, self.qualifier_match)
+            result.release_hrefs = r["hrefs"]
+            result.release_hrefs_results = r["results"]
 
         return result
 
@@ -849,6 +852,7 @@ def show_help(self) -> None:
         print("    --matchmode MATCHMODE matching mode, comma separated list of:")
         print("                          full-search = report best matches, don't abort on first match (recommended)")
         print("                          all-versions = also report matches for name, but different version")
+        print("                          qualifier-match = consider qualifiers for PURL matching")
         print("                          ignore-debian = ignore Debian revision in version comparison, so SBOM")
         print("                                          version 3.1 will match SW360 version 3.1-3.debian")
         print("    -all                  deprecated, please use --matchmode all-versions")
@@ -904,6 +908,9 @@ def run(self, args: Any) -> None:
         if "full-search" in args.matchmode:
             self.full_search = True
 
+        if "qualifier-match" in args.matchmode:
+            self.qualifier_match = True
+
         print_text("Loading SBOM file", args.inputfile)
         try:
             sbom = CaPyCliBom.read_sbom(args.inputfile)

capycli/common/capycli_bom_support.py

@@ -60,6 +60,7 @@ class CycloneDxSupport():
     CDX_PROP_COMPONENT_ID = "capycli:componentId"
     CDX_PROP_FILENAME = "siemens:filename"
     CDX_PROP_MAPRESULT = "capycli:mapResult"
+    CDX_PROP_MAPRESULT_BY_ID = "capycli:mapResultById"
     CDX_PROP_SW360_HREF = "capycli:sw360Href"
     CDX_PROP_SW360_URL = "capycli:sw360Url"
     CDX_PROP_REL_STATE = "capycli:releaseMainlineState"

capycli/common/map_result.py

@@ -7,12 +7,20 @@
 # -------------------------------------------------------------------------------
 
 from typing import Any, List, Optional
+from enum import Enum
 
 from cyclonedx.model.component import Component
 
 from capycli.common.capycli_bom_support import CycloneDxSupport
 
 
+class MapResultByIdQualifiers(Enum):
+    FULL_MATCH = "qualifiers-full-match"
+    IGNORED = "qualifiers-ignored"
+    UNKNOWN = "qualifiers-unknown-match"
+    NO_QUALIFIER_MAPPING = ""
+
+
 class MapResult:
     """Result of mapping a SBOM item to the list of releases"""
 
@@ -50,8 +58,22 @@ def __init__(self, component: Optional[Component] = None) -> None:
         self.result: str = MapResult.NO_MATCH
         self._component_hrefs: List[str] = []
         self._release_hrefs: List[str] = []
+        self._release_hrefs_results: List[str] = []
         self.releases: List[Any] = []
 
+    @property
+    def release_hrefs_results(self) -> list[str]:
+        return self._release_hrefs_results
+
+    @release_hrefs_results.setter
+    def release_hrefs_results(self, value: list[str]) -> None:
+        self._release_hrefs_results = value
+        if not self.input_component or not value:
+            return
+        CycloneDxSupport.update_or_set_property(
+            self.input_component, CycloneDxSupport.CDX_PROP_MAPRESULT_BY_ID,
+            " ".join(value))
+
     @property
     def component_hrefs(self) -> List[str]:
         return self._component_hrefs

capycli/common/purl_service.py

@@ -60,7 +60,10 @@ def build_purl_cache(self, purl_types: Any = tuple(), no_warnings: bool = True)
                     if purl_types and purl.type not in purl_types:
                         continue
                     if not no_warnings:
-                        for e in self.purl_cache.get_by_version(purl):
+                        already_in_cache = self.purl_cache.get_by_version(purl)
+                        _, already_in_cache = PurlStore.filter_by_qualifiers(
+                            already_in_cache, purl)
+                        for e in already_in_cache:
                             if e["purl"] == purl:
                                 print_yellow("-> Multiple entries for purl:", purl)
                                 print_yellow(
@@ -78,14 +81,18 @@ def build_purl_cache(self, purl_types: Any = tuple(), no_warnings: bool = True)
                         print_yellow("-> Ignoring invalid purl entry in", entry["_links"]["self"]["href"])
                         print_yellow(purl_string)
 
-    def search_releases_by_purl(self, purl: packageurl.PackageURL) -> List[str]:
+    def search_releases_by_purl(self, purl: packageurl.PackageURL, qualifier_match: bool = False) -> Dict[str, Any]:
         """Get SW360 releases by Package URL using the purl cache
 
-        :return: list of release urls
+        :return: tuple of release hrefs and list of notes about mapping
         """
         self.build_purl_cache((purl.type,))
 
         result = self.purl_cache.get_by_version(purl)
+        if qualifier_match:
+            qualifier_result, result = PurlStore.filter_by_qualifiers(result, purl)
+        else:
+            qualifier_result = None
         unique_hrefs = {r["href"] for r in result}
 
         if len(unique_hrefs) > 1:
@@ -94,7 +101,12 @@ def search_releases_by_purl(self, purl: packageurl.PackageURL) -> List[str]:
                 print_yellow("      Candidate", self.client.get_id_from_href(r["href"]),
                              "has purl", r["purl"])
 
-        return list(unique_hrefs)
+        search_result = {
+            "hrefs": list(unique_hrefs),
+            # can be extended with more details in the future
+            "results": [qualifier_result.value] if (qualifier_result and qualifier_result.value) else []
+        }
+        return search_result
 
     def search_components_by_purl(self, purl: packageurl.PackageURL) -> List[str]:
         """

capycli/common/purl_store.py

@@ -6,9 +6,10 @@
 # SPDX-License-Identifier: MIT
 # -------------------------------------------------------------------------------
 
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Tuple
 
 from packageurl import PackageURL
+from capycli.common.map_result import MapResultByIdQualifiers
 
 
 class PurlStore:
@@ -84,3 +85,27 @@ def get_by_version(self, purl: PackageURL) -> List[Dict[str, Any]]:
             return entries[purl.version]
 
         return []
+
+    @staticmethod
+    def filter_by_qualifiers(entries: List[Dict[str, Any]], purl: PackageURL) -> Tuple[MapResultByIdQualifiers,
+                                                                                       List[Dict[str, Any]]]:
+        """
+        Filter entries based on the qualifiers in the given PackageURL and return the match type.
+
+        :param entries: A list of entries to filter as returned by get_by_version.
+        :param purl: The PackageURL object containing qualifiers to match.
+        :return: A tuple (qualifier_result, list of entries)
+        """
+        if not purl.qualifiers or len(entries) == 0:
+            return MapResultByIdQualifiers.NO_QUALIFIER_MAPPING, entries
+
+        assert isinstance(purl.qualifiers, dict)
+        qualifiers_items = purl.qualifiers.items()
+        filtered_entries = [
+            entry for entry in entries
+            if all(entry["purl"].qualifiers.get(key) == value for key, value in qualifiers_items)
+        ]
+
+        if filtered_entries:
+            return MapResultByIdQualifiers.FULL_MATCH, filtered_entries
+        return MapResultByIdQualifiers.IGNORED, entries

capycli/common/purl_utils.py

@@ -56,17 +56,23 @@ def parse_purls_from_external_id(purl_entries: Any) -> list:  # type: ignore
         return []
 
     @staticmethod
-    def contains(purls: list, search_purl: packageurl.PackageURL) -> bool:  # type: ignore
+    def contains(purls: list, search_purl: packageurl.PackageURL,  # type: ignore
+                 compare_qualifiers: bool = False) -> bool:
         """
         Search the given PackageURL in the provided list
         Important: The matching is only based on type, namespace, name and version.
-        We do not consider qualifiers and subpath.
+        If `compare_qualifiers` is set, the qualifiers present in the search_purl are also checked.
+        We do not consider other qualifiers and subpath.
         """
         for entry in purls:
             if (entry.type == search_purl.type
                     and entry.namespace == search_purl.namespace
                     and entry.name == search_purl.name
                     and entry.version == search_purl.version):
+                if compare_qualifiers and isinstance(search_purl.qualifiers, dict):
+                    for key, value in search_purl.qualifiers.items():
+                        if key not in entry.qualifiers or entry.qualifiers[key] != value:
+                            return False
                 return True
         return False
 

tests/test_bom_map2.py

@@ -243,6 +243,108 @@ def test_map_bom_item_purl_release_conflict(self) -> None:
         else:
             assert False, "Unexpected release id"
 
+    @responses.activate
+    def test_map_bom_item_purl_release_w_qualifiers(self) -> None:
+        """test bom mapping: search for releases by PURL with qualifiers
+        """
+        if not self.app.client:
+            return
+
+        self.app.purl_service = PurlService(self.app.client, cache={'maven': {
+            'com.fasterxml.jackson.core': {'jackson-core': {
+                None: [{
+                    "purl": PackageURL("maven", "com.fasterxml.jackson.core", "jackson-core"),
+                    "href": SW360_BASE_URL + "components/a035"}],
+                "2.18.0": [
+                    {"purl": PackageURL("maven", "com.fasterxml.jackson.core", "jackson-core", version="2.18.0",
+                                        qualifiers={"classifier": "sources"}),
+                     "href": SW360_BASE_URL + "releases/1234"},
+                    {"purl": PackageURL("maven", "com.fasterxml.jackson.core", "jackson-core", version="2.18.0",
+                                        qualifiers={"classifier": "javadoc"}),
+                     "href": SW360_BASE_URL + "releases/1235"},
+                    {"purl": PackageURL("maven", "com.fasterxml.jackson.core", "jackson-core", version="2.18.0",
+                                        qualifiers={"classifier": "sources", "packaging": "jar"}),
+                     "href": SW360_BASE_URL + "releases/1236"}]}}}})
+
+        self.app.releases = [{"Id": "1234", "ComponentId": "a035",
+                              "Name": "Jackson Core", "Version": "2.18.0",
+                              "ExternalIds": {
+                                  "package-url": "pkg:maven/com.fasterxml.jackson.core/[email protected]"
+                                                 "?classifier=sources"}},
+                             {"Id": "1235", "ComponentId": "a034",
+                              "Name": "com.fasterxml.jackson.core:jackson-core", "Version": "2.18.0_javadoc",
+                              "ExternalIds": {
+                                  "package-url": "pkg:maven/com.fasterxml.jackson.core/[email protected]"
+                                                 "?classifier=javadoc"}},
+                             {"Id": "1236", "ComponentId": "a034",
+                              "Name": "com.fasterxml.jackson.core:jackson-core", "Version": "2.18.0_jar",
+                              "ExternalIds": {
+                                  "package-url": "pkg:maven/com.fasterxml.jackson.core/[email protected]"
+                                                 "?classifier=sources&packaging=jar"}}]
+
+        bomitem = Component(
+            name="jackson-core",
+            version="2.18.0",
+            purl=PackageURL.from_string("pkg:maven/com.fasterxml.jackson.core/[email protected]"
+                                        "?classifier=sources"))
+
+        self.app.full_search = True
+
+        # 3 matches when ignoring qualifiers
+        res = self.app.map_bom_item(bomitem, check_similar=False, result_required=False)
+        assert res.result == MapResult.FULL_MATCH_BY_ID
+        assert len(res.releases) == 3
+
+        self.app.qualifier_match = True
+
+        # 2 matches for classifier=sources
+        bomitem.properties.clear()  # resert properties to remove results from previous mapping
+        res = self.app.map_bom_item(bomitem, check_similar=False, result_required=False)
+        assert res.result == MapResult.FULL_MATCH_BY_ID
+        assert len(res.releases) == 2
+
+        if res.releases[0]["Sw360Id"] == "1234":
+            assert res.releases[0]["ComponentId"] == "a035"
+            assert res.releases[1]["Sw360Id"] == "1236"
+            assert res.releases[1]["ComponentId"] == "a034"
+        elif res.releases[0]["Sw360Id"] == "1236":
+            assert res.releases[0]["ComponentId"] == "a034"
+            assert res.releases[1]["Sw360Id"] == "1234"
+            assert res.releases[1]["ComponentId"] == "a035"
+        else:
+            assert False, "Unexpected release id"
+        assert res.input_component is not None
+        assert (
+            CycloneDxSupport.get_property(res.input_component, CycloneDxSupport.CDX_PROP_MAPRESULT_BY_ID).value
+            == "qualifiers-full-match")
+
+        self.app.qualifier_match = False
+
+        # bomitem has unknown qualifier -> all PURL version matches returned
+        assert bomitem.purl is not None
+        assert type(bomitem.purl.qualifiers) is dict
+        bomitem.purl.qualifiers["themorequalifiers"] = "thebetter"
+        bomitem.properties.clear()  # resert properties to remove results from previous mapping
+        res = self.app.map_bom_item(bomitem, check_similar=False, result_required=False)
+        assert res.result == MapResult.FULL_MATCH_BY_ID
+        assert len(res.releases) == 3
+        all_results = [r["Sw360Id"] for r in res.releases]
+        assert all_results == ["1234", "1235", "1236"]
+        assert res.input_component is not None
+        assert (
+            CycloneDxSupport.get_property(res.input_component, CycloneDxSupport.CDX_PROP_MAPRESULT_BY_ID)
+            is None)
+
+        self.app.qualifier_match = True
+
+        bomitem.properties.clear()  # resert properties to remove results from previous mapping
+        res = self.app.map_bom_item(bomitem, check_similar=False, result_required=False)
+        assert len(res.releases) == 3
+        assert res.input_component is not None
+        assert (
+            CycloneDxSupport.get_property(res.input_component, CycloneDxSupport.CDX_PROP_MAPRESULT_BY_ID).value
+            == "qualifiers-ignored")
+
     @responses.activate
     def test_map_bom_item_mixed_match(self) -> None:
         bomitem = Component(