Re: [ptxdist] SBOM support

[Simon Falsig <sfalsig@verity.net>]

Hi Gavin, Michael,

> From: Gavin Schenk <g.schenk@eckelmann.de>
> Sent: Monday, September 11, 2023 15:11
> Hi,
> 
> > On Thu, Sep 07, 2023 at 03:03:47PM +0000, Simon Falsig wrote:
> >> I saw a post from 2021 to the mailing list on generating SBOMs from
> ptxdist.
> >> Has there been any further work on this?
> >
> > I've not worked on this and I'm not aware of any other efforts in this
> > direction.
> >
> >> We've been looking at implementing this internally - plan would be to
> >> generate the SBOM in CycloneDX format, and consume it with
> >> Dependency-Track (https://dependencytrack.org) for automatic
> >> vulnerability and license monitoring.
> >>
> >> Looks like we're quite close to having a working setup, but it'd make
> >> a lot more sense to have it upstreamed rather than as local patches,
> >> so would like to get a bit of input on the approach, and see if we
> >> can make that happen :)
> 
> we have a similar task. There is a rudimentary prototype running on our
> site. It uses licencse-complicance.yaml and a CPE-Dictionary from
> https://nvd.nist.gov/products/cpe to generate a bom.json.  This bom.json
> can than be feed into dependency tracker (We are using docker-compose for
> demonstration).
> 
> <snip bom>
> {
>     "bomFormat": "CycloneDX",
>     "specVersion": "1.4",
>     "serialNumber": "urn:uuid:f20af5d1-0480-477d-8bea-8cbcf6d9268c",
>     "components": [
>         {
>             "name": "attr",
>             "version": "2.4.47",
>             "cpe": "cpe:2.3:a:attr_project:attr:2.4.47:*:*:*:*:*:*:*",
>             "licenses": [
>                 {
>                     "license": {
>                         "id": "GPL-2.0-only"
>                     }
>                 },
>                 {
>                     "license": {
>                         "id": "LGPL-2.0-only"
>                     }
>                 }
>             ],
>             "hashes": [
>                 {
>                     "alg": "MD5",
>                     "content": "84f58dec00b60f2dc8fd1c9709291cc7"
>                 }
>             ]
>         },
>         ...
> </snip>
>

That sounds exactly like what we're working on :) I like how it also splits
up the multi-licensed specifications into separate statements. How does that
show up in dependency-track? (I just know that SPDX expressions with e.g. 
AND and OR don't really work for setting up policies.)

> 
> > I know very little about this stuff, but I'm open to add support for
> this.
> > So please sent patches once they are ready.
> >
> >> We've identified two main steps:
> >> 1. Generate the SBOM itself. A minimal version of this can be created
> from the
> >>    output of the existing fast-bsp-report in 40 lines of Python, using
> the
> >>    CycloneDX library.
> >>    I'd assume that such a script would just go into the scripts folder
> in ptxdist?
> >
> > Yes, just put it in the scripts/ folder.
> >
> 
> Our script has 110 lines python. It patches some package names that does
> not match with the db e.g.
> 
> expat -> libexpat
> gdbserver -> gdb
> kernel -> linux_kernel
> libmod -> kmod
> ...
> 
> I am unsure, if this is too error prone. Adding CPE vars to the recepies,
> where only the VERSION is variable, should be more robust.
> 
> >>    Is there a common way of tracking / documenting dependencies of such
> scripts?
> >
> > You mean Python packages used by the script? Not directly.
> 
> ptxdist-cyclonedx-bom master
>  ❯ cat requirements.txt
> pyyaml==6.0.0
> lxml==4.9.2

Yes, essentially, our current implementation needs Python3 and the
cyclonedx-bom Python library. From what I understand, this could probably
be set up using pipenv or something similar? 

> 
> > Is the SBOM something that should be created for an image or for the
> > BSP as a whole?
> 
> It should be per image imho. Because depending on configuration, layers,
> collectionconfig, content of images differs a lot. Starting from the
> image, should generate data that is 100% consistent. Not much of a
> difference to licence reports, I think.

I'm not 100% confident in the ptxdist terminology relating to image or bsp
here, but the SBOM should ideally be created based on what ends up on a
target - so I guess that's the image?
My current solution just takes this from the $PACKAGES variable, but not
sure how that works with more advanced setups than ours. Initial patchset
coming in a bit, so happy to hear about how it can be improved.

> 
> > For the license stuff I have plans to created documents for images.
> > Because those are delivered to customers, so license compliance really
> > needs to be done for each image.
> 
> And this is the same for SBOMi, as mentioned before.
> 
> > I'm not sure what makes sense. So you could either create a global
> > option and then generate the SBOM with each image. And then select
> > HOST_SYSTEM_PYTHON3_* in that option.
> > Or create an SBOM "image" for the whole BSP and select the options
> there.
> 
> One thing to keep in mind is that cyclonedx is one format, but not the
> only one. Today it is one of the most common used. But there are
> alternatives. So there might be other users who might need the sbom in
> different formats. I wonder if it is easier to adapt to other formats
> using the one, or other approach?
>

I guess this could be handled with a configuration selecting the format
to use, and just some different implementations based on that?

> 
> >> 2. To track vulnerabilities, it's necessary to track the Common
> Platform
> >>    Enumeration (CPE) name of each package (from
> >>    https://nvd.nist.gov/products/cpe).
> >>    This will allow matching packages to CVEs.
> >>    My suggestion would be to add a _CPE variable to each package (built
> >>    from whatever other variables make sense, typically _VERSION).
> >
> > Makes sense to me. From what I understand, the CPE is machine readable
> > and can be split into its components if needed, right?
> 
> I think it is. Here CPE number example for package expat:
> "cpe:2.3:a:attr_project:attr:2.4.47:*:*:*:*:*:*:*"
>                                 ^- PTXCONF_EXPAT_VERSION
> 
> >>    I managed to add this
> >>    for the fast report (extracting it to pkg_cpe in
> >>    rules/post/ptxd_make_world_common.make, and adding it to the report
> >>    in scripts/lib/ptxd_make_world_report.sh), but I wouldn't be
> >>    surprised if there are other places/report that need to track this
> >>    also for consistency?
> >
> > Just the fast and full reports are enough. Anything else in that
> > direction is legacy anyways and should be replaced with stuff based on
> > these reports anyways.
> >
> >>    Packages that specify _CPE would then have this included in their
> >>    report, and there'd be no change for the packages that don't
> >>    specify it.
> >>
> >>
> >> I'd be happy to get a bit of initial feedback on the approach. I'll
> >> have a look at putting up some initial patches in the coming days too.
> 
> I like it a lot. Need reviewers, testers, 1 mio Euro? We are willing to
> contribute.
> 
> Regards
> Gavin

Thanks! Let's see what we can put together then - I'll send in a few simple
patches in a bit, assuming that I can make this git email sending work
again...

- Simon