From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Mon, 04 Mar 2024 18:28:00 +0100 Received: from metis.whiteo.stw.pengutronix.de ([2a0a:edc0:2:b01:1d::104]) by lore.white.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1rhC6a-008XmQ-31 for lore@lore.pengutronix.de; Mon, 04 Mar 2024 18:28:00 +0100 Received: from localhost ([127.0.0.1] helo=metis.whiteo.stw.pengutronix.de) by metis.whiteo.stw.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1rhC6a-0002HZ-Cp; Mon, 04 Mar 2024 18:28:00 +0100 Received: from drehscheibe.grey.stw.pengutronix.de ([2a0a:edc0:0:c01:1d::a2]) by metis.whiteo.stw.pengutronix.de with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rhC6I-0002Gc-H8; Mon, 04 Mar 2024 18:27:42 +0100 Received: from [2a0a:edc0:2:b01:1d::c5] (helo=pty.whiteo.stw.pengutronix.de) by drehscheibe.grey.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rhC6H-004Ogp-W9; Mon, 04 Mar 2024 18:27:42 +0100 Received: from mol by pty.whiteo.stw.pengutronix.de with local (Exim 4.96) (envelope-from ) id 1rhC6H-001yBF-2w; Mon, 04 Mar 2024 18:27:41 +0100 Date: Mon, 4 Mar 2024 18:27:41 +0100 From: Michael Olbrich To: Simon Falsig Message-ID: Mail-Followup-To: Simon Falsig , ptxdist@pengutronix.de References: <20240219165617.70971-1-sfalsig@verity.net> <20240219165617.70971-3-sfalsig@verity.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240219165617.70971-3-sfalsig@verity.net> X-Sent-From: Pengutronix Hildesheim X-URL: http://www.pengutronix.de/ X-Accept-Language: de,en X-Accept-Content-Type: text/plain Subject: Re: [ptxdist] [PATCH 3/3] RFC: sbom_report: Add support X-BeenThere: ptxdist@pengutronix.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: PTXdist Development Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ptxdist@pengutronix.de Cc: ptxdist@pengutronix.de Sender: "ptxdist" X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: ptxdist-bounces@pengutronix.de X-SA-Exim-Scanned: No (on metis.whiteo.stw.pengutronix.de); SAEximRunCond expanded to false On Mon, Feb 19, 2024 at 05:56:17PM +0100, Simon Falsig wrote: > This provides support for building SBOMs in CycloneDX format. > > A target is added alongside the other reports, that (based on the > fast-bsp-report) extracts name, version, cpe and license of each target > package, and puts these into a final sbom-report in CycloneDX/JSON > format. > > This requires a working Python3 setup with the cyclonedx-python-lib > package installed. > --- > bin/ptxdist | 3 +- > rules/host-system-python3.in | 3 ++ > rules/host-system-python3.make | 6 ++++ > rules/post/ptxd_make_report.make | 16 +++++++-- > rules/sbom-report.in | 9 +++++ > scripts/lib/ptxd_make_report.sh | 16 +++++++++ > scripts/lib/ptxd_make_sbom_report.py | 52 ++++++++++++++++++++++++++++ > 7 files changed, 102 insertions(+), 3 deletions(-) > create mode 100644 rules/sbom-report.in > create mode 100644 scripts/lib/ptxd_make_sbom_report.py > > diff --git a/bin/ptxdist b/bin/ptxdist > index 77cad673b..9e62d2bc0 100755 > --- a/bin/ptxdist > +++ b/bin/ptxdist > @@ -792,6 +792,7 @@ Misc: > full-bsp-report generate a yaml file that describes the BSP and > all packages. More data but will build all > packages if necessary. > + sbom-report generate a CycloneDX json SBOM So I'm not so sure if I want a separate command for this. I think it would be better to build this with the images. > print print the contents of a variable, in the way > it is known by "make" > printnext assumes that the contents of is another > @@ -1841,7 +1842,7 @@ EOF > ptxd_make_log export_src EXPORTDIR="${1}" > exit > ;; > - fast-bsp-report|full-bsp-report) > + fast-bsp-report|full-bsp-report|sbom-report) > check_premake_compiler && > ptxd_make_log "${cmd}" > exit > diff --git a/rules/host-system-python3.in b/rules/host-system-python3.in > index 25201e93f..c158a295f 100644 > --- a/rules/host-system-python3.in > +++ b/rules/host-system-python3.in > @@ -35,4 +35,7 @@ config HOST_SYSTEM_PYTHON3_PYELFTOOLS > config HOST_SYSTEM_PYTHON3_PYYAML > bool > > +config HOST_SYSTEM_PYTHON3_CYCLONEDX > + bool > + > endif > diff --git a/rules/host-system-python3.make b/rules/host-system-python3.make > index 3688cf09d..f4979a453 100644 > --- a/rules/host-system-python3.make > +++ b/rules/host-system-python3.make > @@ -88,6 +88,12 @@ ifdef PTXCONF_HOST_SYSTEM_PYTHON3_PYYAML > ptxd_bailout "Python pyyaml module not found! \ > Please install python3-yaml (debian)"; > endif > +ifdef PTXCONF_HOST_SYSTEM_PYTHON3_CYCLONEDX > + @echo "Checking for Python cyclonedx-python-lib ..." > + @$(SYSTEMPYTHON3) -c 'import cyclonedx.factory' 2>/dev/null || \ > + ptxd_bailout "Python cyclonedx-python-lib module not found! \ > + Please install cyclonedx-python-lib (pip)"; > +endif > > @$(call touch) > This part (or whatever replaces this) should be a separate patch. > diff --git a/rules/post/ptxd_make_report.make b/rules/post/ptxd_make_report.make > index eecd2a577..529b1c78e 100644 > --- a/rules/post/ptxd_make_report.make > +++ b/rules/post/ptxd_make_report.make > @@ -10,7 +10,10 @@ ptx/report-env = \ > $(image/env) \ > ptx_report_target="$(strip $(1))" \ > ptx_packages_selected="$(filter-out $(IMAGE_PACKAGES),$(PTX_PACKAGES_SELECTED))" \ > - ptx_image_packages="$(IMAGE_PACKAGES)" > + ptx_image_packages="$(IMAGE_PACKAGES)" \ > + ptx_target_packages="$(PACKAGES)" \ This should not be needed. See below. > + ptx_system_python3="$(SYSTEMPYTHON3)" > + Just the normal she-bang should to the right thing. > > PHONY += full-bsp-report > full-bsp-report: $(RELEASEDIR)/full-bsp-report.yaml > @@ -26,13 +29,22 @@ $(RELEASEDIR)/full-bsp-report.yaml: \ > @$(call ptx/report-env, $@) ptxd_make_full_bsp_report > @$(call finish) > > + > PHONY += fast-bsp-report > fast-bsp-report: $(RELEASEDIR)/fast-bsp-report.yaml > > - Unrelated changes. > $(RELEASEDIR)/fast-bsp-report.yaml: $(addprefix $(STATEDIR)/,$(addsuffix .fast-report,$(PTX_PACKAGES_SELECTED))) > @$(call targetinfo) > @$(call ptx/report-env, $@) ptxd_make_fast_bsp_report > @$(call finish) > > + > +PHONY += sbom-report > +sbom-report: $(RELEASEDIR)/sbom-report.json > + > +$(RELEASEDIR)/sbom-report.json: $(STATEDIR)/host-system-python3.install $(addprefix $(STATEDIR)/,$(addsuffix .fast-report,$(PACKAGES))) > + @$(call targetinfo) > + @$(call ptx/report-env, $@) ptxd_make_sbom_report > + @$(call finish) > + So I don't think this should be one target like this. The old license documents work like this. But it's not very flexible. What you typically want is basically an SBOM for an image with all its dependencies or maybe one for the whole BSP (including all host packages). The implementation should be flexible enough to do this. I've done that for the 'new' license documents in scripts/generate-report.py and I am currently extending that to create an SPDX SBOM. And the input is always the full-bsp-report.yaml (or maybe fast-bsp-report.yaml) but never the individual fragments. > # vim: syntax=make > diff --git a/rules/sbom-report.in b/rules/sbom-report.in > new file mode 100644 > index 000000000..311ad5a23 > --- /dev/null > +++ b/rules/sbom-report.in > @@ -0,0 +1,9 @@ > +## SECTION=ptxdist_options > + > +config ENABLE_SBOM_REPORT > + bool > + select HOST_SYSTEM_PYTHON3 > + select HOST_SYSTEM_PYTHON3_CYCLONEDX > + prompt "enable sbom report generation" > + help > + This enables the generation of an SBOM report, through ptxdist sbom-report. > diff --git a/scripts/lib/ptxd_make_report.sh b/scripts/lib/ptxd_make_report.sh > index 885f07828..f2e92e6be 100644 > --- a/scripts/lib/ptxd_make_report.sh > +++ b/scripts/lib/ptxd_make_report.sh > @@ -146,3 +146,19 @@ ptxd_make_fast_bsp_report() { > } > export -f ptxd_make_fast_bsp_report > > +ptxd_make_sbom_report() { > + local -a ptxd_reply > + local pkg_lic pkg > + > + ptxd_make_layer_init || return > + > + echo "Generating $(ptxd_print_path "${ptx_report_target}") ..." > + echo > + > + mkdir -p "$(dirname "${ptx_report_target}")" && > + "${ptx_system_python3}" ${PTXDIST_LIB_DIR}/ptxd_make_sbom_report.py "${ptx_report_dir}/fast/" ${ptx_target_packages} > ${PTXDIST_TEMPDIR}/sbom-report && > + mv "${PTXDIST_TEMPDIR}/sbom-report" "${ptx_report_target}" || > + ptxd_bailout "failed to create SBOM report" > +} > +export -f ptxd_make_sbom_report > + > diff --git a/scripts/lib/ptxd_make_sbom_report.py b/scripts/lib/ptxd_make_sbom_report.py > new file mode 100644 > index 000000000..a9ec36697 > --- /dev/null > +++ b/scripts/lib/ptxd_make_sbom_report.py > @@ -0,0 +1,52 @@ > +from cyclonedx.factory.license import LicenseFactory > +from cyclonedx.model.bom import Bom > +from cyclonedx.model.component import Component > +from cyclonedx.output.json import JsonV1Dot4 > +import sys > +import re > + > +lFac = LicenseFactory() > +bom = Bom() > + > +for i in range(2, len(sys.argv)): > + pkg_report = sys.argv[1] + sys.argv[i] + ".yaml" > + with open(pkg_report, 'r') as file: > + content = file.read() > + name_ = re.search("name: \'(.+)\'", content).group(1) > + version_ = re.search("version: \'(.+)\'", content).group(1) > + > + # First see if we have a full CPE specified, then use that > + cpe_match = re.search("cpe: \'(.+)\'", content) > + cpe_ = None > + if cpe_match is not None: > + cpe_ = cpe_match.group(1) > + else: > + # See if we have the individual components > + cpe_vendor_match = re.search("cpe_vendor: \'(.+)\'", content) > + cpe_product_match = re.search("cpe_product: \'(.+)\'", content) > + cpe_version_match = re.search("cpe_version: \'(.+)\'", content) > + if cpe_vendor_match is not None and cpe_product_match is not None and cpe_version_match is not None: > + cpe_ = "cpe:2.3:a:{vendor}:{product}:{version}:*:*:*:*:*:*:*".format(vendor=cpe_vendor_match.group(1), > + product=cpe_product_match.group(1), > + version=cpe_version_match.group(1)) > + > + if cpe_ is not None: > + # We have a CPE, let's validate it. Regex from: https://csrc.nist.gov/schema/cpe/2.3/cpe-naming_2.3.xsd > + if not re.fullmatch("cpe:2\.3:[aho\*\-](:(((\?*|\*?)([a-zA-Z0-9\-\._]|(\\[\\\*\?!\"#$$%&'\(\)\+,/:;" > + "<=>@\[\]\^`\{\|}~]))+(\?*|\*?))|[\*\-])){5}(:(([a-zA-Z]{2,3}(-([a-zA-Z]{2}|[0-9]{3}))?)" > + "|[\*\-]))(:(((\?*|\*?)([a-zA-Z0-9\-\._]|(\\[\\\*\?!\"#$$%&'\(\)\+,/:;<=>@\[\]\^`\{\|}~]" > + "))+(\?*|\*?))|[\*\-])){4}", cpe_): > + raise ValueError("Constructed CPE is not valid: {cpe}".format(cpe=cpe_)) > + > + licenses_ = re.search("licenses: \'(.+)\'", content).group(1) > + comp = Component( > + name=name_, > + version=version_, > + cpe=cpe_, > + licenses=[lFac.make_from_string(licenses_)], > + bom_ref=name_ + "@" + version_ > + ) > + bom.components.add(comp) > + > +serializedJSON = JsonV1Dot4(bom).output_as_string() > +print(serializedJSON) This needs to be quite different. We have one yaml file with all the data and that should be used. As I said, I'm working on an SPDX SBOM. It's probably best if you wait for that to hit ptxdist master and build on top of that. That will probably happen this or next week, at least the initial version. Regards, Michael -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |