Re: [ptxdist] [PATCH] scripts: add update_check script

[Bruno Thomsen <bruno.thomsen@gmail.com>]

Den fre. 23. okt. 2026 kl. 12.37 skrev Sven Püschel <s.pueschel@pengutronix.de>:
>
> Hi Bruno,
>
> On 10/23/26 11:28, Bruno Thomsen wrote:
> > Den fre. 23. okt. 2026 kl. 11.01 skrev Sven Püschel <s.pueschel@pengutronix.de>:
> >> Add an update_check python script as an easy check which PTXdist
> >> rules can be bumped in their versions. It uses the report file generated
> >> from an existing PTXdist BSP as a basis to determine the package
> >> versions. Then it compares these package versions with the package
> >> versions against the newest release versions reported by
> >> release-monitoring.org .
> >>
> >> release-monitoring.org is a deployment of an open-source release
> >> monitoring project named Anitya [1]. It allows to define a Project
> >> which is some software you want to keep track of updates. A given
> >> software can have multiple Projects. An example is GTK+, where you want
> >> to have separate projects for GTK+ 3 and GTK+ 4. A Project can then
> >> define a mapping to a package name in a Distribution.
> >>
> >> I've created the PTXdist distribution [2] and created Project mappings
> >> for a good chunk of the PTXdist rules. I've excluded certain rules with
> >> user-defined versions (e.g. the linux kernel) and projects where it was
> >> unclear if PTXdist should track the latest major version or use the
> >> latest version of the current major version. Also I've ignored rules,
> >> where no Project existed yet. Furthermore I've skipped the `python-`
> >> rules, as they can probably be added in an automated fashion (given the
> >> pypy name can be used for matching purposes).
> > Nice that you have added PTXdist to release-monitoring.org, you can
> > also see it in Repology[1] where it has lists of outdated packages[2],
> > potentially vulnerable packages[3], and packages with problems[4] like
> > dead URLs or permanent redirects. Information is tracking the master branch.
>
> Yeah, Michael also mentioned your PTXDist patches towards repology and
> that he'd prefer some solution based on a report file.
>

Latest versions of the patches were not posted to the mailing list and we
continued development in a separate repo[1] which also contains regular
updates to the JSON output that repology consumes.

> Personally I didn't quite understand how Repology actually works. It
> looks like fuzzy matching of package names and getting the newest
> versions based on the comparison with other distro packages.
> Release-Monitoring looked dead-simple: Each Project has a dedicated
> definition where to look for new versions and a explicit mapping to a
> distro package name. Therefore I'd just went with it (with the side
> effect of also making PTXdist a bit present there). I'm not too
> invested, which Backend the script uses to actually get the necessary
> data (or it could also use multiple backends, etc.).

Yes, packages are fuzzy matched by download URL + name, and new versions
are detected when 2 or more distributions have a newer version.

I like Release-Monitoring as well, so I get a RHBZ ping to update a RPM spec :)

They are good for two slightly different things, but I hope you get
the patch series
complete.

/Bruno

[1] https://github.com/baxeno/ptxdist-repology

>
> Sincerely
>      Sven
>
> > /Bruno
> >
> > [1] https://repology.org/repository/ptxdist
> > [2] https://repology.org/projects/?inrepo=ptxdist&outdated=1&families_newest=2-
> > [3] https://repology.org/projects/?inrepo=ptxdist&vulnerable=1
> > [4] https://repology.org/repository/ptxdist/problems
> >
> >> This python script uses the unauthenticated API to get all current
> >> versions for the Projects with a mapped PTXdist name. It then caches
> >> these versions locally (updating only if the cache is older then 24h)
> >> and compares the versions against the versions in the report file.
> >> It displays all rules that can be updated and also counts the number of
> >> up-to-date rules and rules missing a mapping. With a command line flag
> >> the latter two can also be displayed.
> >>
> >> Note that this isn't perfect and just provides an approximation to the
> >> actual truth. There are a number of imperfections:
> >> - PTXdist using non-stable versions (e.g. iftop 1.0pre4 instead of 0.17)
> >> - Versions with different separators (dashes/underscores vs. dots)
> >> - Rules not intended to be tracked (e.g. cross-dummy-strip)
> >>
> >> Also there are some pitfalls in this design:
> >> - Version numbers are based on the BSP
> >>    - Updated rules might be already present in a newer PTXdist release
> >>    - Rules might be overwritten with an older version locally
> >> - BSP only rules (custom applications) will show up as untracked
> >> - Versions may lag behind a bit (Anitya cron job and rate-limits)
> >>
> >> [1] https://github.com/fedora-infra/anitya
> >> [2] https://release-monitoring.org/distro/PTXdist/
> >>
> >> Signed-off-by: Sven Püschel <s.pueschel@pengutronix.de>
> >> ---
> >>   scripts/update_check.py | 132 ++++++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 132 insertions(+)
> >>   create mode 100755 scripts/update_check.py
> >>
> >> diff --git a/scripts/update_check.py b/scripts/update_check.py
> >> new file mode 100755
> >> index 000000000..94ffe83bc
> >> --- /dev/null
> >> +++ b/scripts/update_check.py
> >> @@ -0,0 +1,132 @@
> >> +#!/usr/bin/env python3
> >> +#
> >> +# Copyright (C) 2026 by Sven Püschel <s.pueschel@pengutronix.de>
> >> +#
> >> +# For further information about the PTXdist project and license conditions
> >> +# see the README file.
> >> +#
> >> +
> >> +import argparse
> >> +import itertools
> >> +import json
> >> +import os
> >> +import time
> >> +from urllib import request
> >> +
> >> +import yaml
> >> +
> >> +API_ENDPOINT = 'https://release-monitoring.org/api/v2'
> >> +ITEMS_PER_PAGE = 25
> >> +
> >> +XDG_CACHE_DIR = os.getenv('XDG_CACHE_HOME')
> >> +if not XDG_CACHE_DIR:
> >> +    XDG_CACHE_DIR = os.path.join(os.getenv('HOME'), '.cache')
> >> +
> >> +VERSION_CACHE_FILENAME = 'upstream_pkg_versions.json'
> >> +VERSION_CACHE_EXPIRE_SEC = 24 * 60 * 60
> >> +VERSION_CACHE_DIR = os.path.join(XDG_CACHE_DIR, 'ptxdist')
> >> +VERSION_CACHE_FILE = os.path.join(VERSION_CACHE_DIR, VERSION_CACHE_FILENAME)
> >> +
> >> +def download_pkg_version_list():
> >> +    versions = []
> >> +
> >> +    for page in itertools.count(1):
> >> +        with request.urlopen(request.Request(
> >> +                f'{API_ENDPOINT}/packages/?distribution=PTXdist&page={page}&items_per_page={ITEMS_PER_PAGE}',
> >> +                headers={ 'User-Agent': 'PTXdist update_check script' }
> >> +            )) as url:
> >> +            data = json.load(url)
> >> +            assert data['page'] == page
> >> +            assert data['items_per_page'] == ITEMS_PER_PAGE
> >> +            total = data['total_items']
> >> +            versions.extend(data['items'])
> >> +            print(f'\rDownloading current package versions ({len(versions)}/{total})', end='')
> >> +            if len(versions) >= total:
> >> +                break
> >> +    print()
> >> +    return versions
> >> +
> >> +parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter)
> >> +parser.add_argument('--show-untracked', help='Show rules where no updated version is known', action='store_true')
> >> +parser.add_argument('--show-up-to-date', help='Show rules which are already at the newest versions', action='store_true')
> >> +parser.add_argument('report_file', help='report file generated by `ptxdist fast-bsp-report`')
> >> +
> >> +args = parser.parse_args()
> >> +
> >> +versions = None
> >> +try:
> >> +    if os.stat(VERSION_CACHE_FILE).st_mtime > time.time() - VERSION_CACHE_EXPIRE_SEC:
> >> +        print('Using cached version list')
> >> +        with open(VERSION_CACHE_FILE) as f:
> >> +            versions = json.load(f)
> >> +except FileNotFoundError:
> >> +    # No cache exists, just download it
> >> +    pass
> >> +except Exception as e:
> >> +    print(f'Unexpected error trying to read the version cache: {e}')
> >> +
> >> +if not versions:
> >> +    versions = download_pkg_version_list()
> >> +    try:
> >> +        os.makedirs(VERSION_CACHE_DIR, exist_ok=True)
> >> +        with open(VERSION_CACHE_FILE, 'w') as f:
> >> +            json.dump(versions, f, indent=1)
> >> +            f.write("\n")
> >> +    except Exception as e:
> >> +        print(f"Couldn't write package version cache: {e}")
> >> +print()
> >> +
> >> +stable_versions = {}
> >> +for package in versions:
> >> +    stable_versions[package['name']] = package['stable_version']
> >> +
> >> +with open(args.report_file) as f:
> >> +    report = yaml.safe_load(f)
> >> +
> >> +latest_version = []
> >> +needs_update = []
> >> +untracked = []
> >> +
> >> +max_pkgname_len = 0
> >> +max_version_len = 0
> >> +
> >> +for pkg in report['packages'].values():
> >> +    if not 'rulefile' in pkg or not 'version' in pkg:
> >> +        continue
> >> +
> >> +    max_pkgname_len = max(max_pkgname_len, len(pkg['name']))
> >> +    max_version_len = max(max_version_len, len(pkg['version']))
> >> +
> >> +    version = stable_versions.get(pkg['name'])
> >> +    if version is None:
> >> +        version = stable_versions.get(pkg['name'].removeprefix('host-'))
> >> +
> >> +    if version is None:
> >> +        untracked.append((pkg['name'], pkg['version']))
> >> +    else:
> >> +        if version != pkg['version']:
> >> +            needs_update.append((pkg['name'], pkg['version'], version))
> >> +        else:
> >> +            latest_version.append((pkg['name'], pkg['version']))
> >> +
> >> +if args.show_up_to_date:
> >> +    print('Up-to-date rules')
> >> +    print('================')
> >> +    for pkg in latest_version:
> >> +        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
> >> +    print()
> >> +
> >> +print('Updateable rules')
> >> +print('================')
> >> +for pkg in needs_update:
> >> +    print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)} -> {pkg[2]}")
> >> +print()
> >> +
> >> +if args.show_untracked:
> >> +    print('Untracked rules')
> >> +    print('===============')
> >> +    for pkg in untracked:
> >> +        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
> >> +    print()
> >> +
> >> +print(f"Updateable: {len(needs_update)} Up-to-date: {len(latest_version)} Untracked: {len(untracked)}")
> >> --
> >> 2.47.3
> >>
> >>