Re: [ptxdist] [APPLIED] scripts: add update_check script

[Michael Olbrich <m.olbrich@pengutronix.de>]

Thanks, applied as abd6d1ad5df147318c60039150492bb9e5d3d416.

Michael

[sent from post-receive hook]

On Fri, 13 Nov 2026 09:39:01 +0100, Sven Püschel <s.pueschel@pengutronix.de> wrote:
> Add an update_check python script as an easy check which PTXdist
> rules can be bumped in their versions. It uses the report file generated
> from an existing PTXdist BSP as a basis to determine the package
> versions. Then it compares these package versions with the package
> versions against the newest release versions reported by
> release-monitoring.org .
> 
> release-monitoring.org is a deployment of an open-source release
> monitoring project named Anitya [1]. It allows to define a Project
> which is some software you want to keep track of updates. A given
> software can have multiple Projects. An example is GTK+, where you want
> to have separate projects for GTK+ 3 and GTK+ 4. A Project can then
> define a mapping to a package name in a Distribution.
> 
> I've created the PTXdist distribution [2] and created Project mappings
> for a good chunk of the PTXdist rules. I've excluded certain rules with
> user-defined versions (e.g. the linux kernel) and projects where it was
> unclear if PTXdist should track the latest major version or use the
> latest version of the current major version. Also I've ignored rules,
> where no Project existed yet. Furthermore I've skipped the `python-`
> rules, as they can probably be added in an automated fashion (given the
> pypy name can be used for matching purposes).
> 
> This python script uses the unauthenticated API to get all current
> versions for the Projects with a mapped PTXdist name. It then caches
> these versions locally (updating only if the cache is older then 24h)
> and compares the versions against the versions in the report file.
> It displays all rules that can be updated and also counts the number of
> up-to-date rules and rules missing a mapping. With a command line flag
> the latter two can also be displayed.
> 
> Note that this isn't perfect and just provides an approximation to the
> actual truth. There are a number of imperfections:
> - PTXdist using non-stable versions (e.g. iftop 1.0pre4 instead of 0.17)
> - Versions with different separators (dashes/underscores vs. dots)
> - Rules not intended to be tracked (e.g. cross-dummy-strip)
> 
> Also there are some pitfalls in this design:
> - Version numbers are based on the BSP
>   - Updated rules might be already present in a newer PTXdist release
>   - Rules might be overwritten with an older version locally
> - BSP only rules (custom applications) will show up as untracked
> - Versions may lag behind a bit (Anitya cron job and rate-limits)
> 
> [1] https://github.com/fedora-infra/anitya
> [2] https://release-monitoring.org/distro/PTXdist/
> 
> Signed-off-by: Sven Püschel <s.pueschel@pengutronix.de>
> Message-Id: <20261023090104.3420495-1-s.pueschel@pengutronix.de>
> [mol: reformated with 'ruff format']
> Signed-off-by: Michael Olbrich <m.olbrich@pengutronix.de>
> 
> diff --git a/scripts/update_check.py b/scripts/update_check.py
> new file mode 100755
> index 000000000000..86570ae57901
> --- /dev/null
> +++ b/scripts/update_check.py
> @@ -0,0 +1,153 @@
> +#!/usr/bin/env python3
> +#
> +# Copyright (C) 2026 by Sven Püschel <s.pueschel@pengutronix.de>
> +#
> +# For further information about the PTXdist project and license conditions
> +# see the README file.
> +#
> +
> +import argparse
> +import itertools
> +import json
> +import os
> +import time
> +from urllib import request
> +
> +import yaml
> +
> +API_ENDPOINT = "https://release-monitoring.org/api/v2"
> +ITEMS_PER_PAGE = 25
> +
> +XDG_CACHE_DIR = os.getenv("XDG_CACHE_HOME")
> +if not XDG_CACHE_DIR:
> +    XDG_CACHE_DIR = os.path.join(os.getenv("HOME"), ".cache")
> +
> +VERSION_CACHE_FILENAME = "upstream_pkg_versions.json"
> +VERSION_CACHE_EXPIRE_SEC = 24 * 60 * 60
> +VERSION_CACHE_DIR = os.path.join(XDG_CACHE_DIR, "ptxdist")
> +VERSION_CACHE_FILE = os.path.join(VERSION_CACHE_DIR, VERSION_CACHE_FILENAME)
> +
> +
> +def download_pkg_version_list():
> +    versions = []
> +
> +    for page in itertools.count(1):
> +        with request.urlopen(
> +            request.Request(
> +                f"{API_ENDPOINT}/packages/?distribution=PTXdist&page={page}&items_per_page={ITEMS_PER_PAGE}",
> +                headers={"User-Agent": "PTXdist update_check script"},
> +            )
> +        ) as url:
> +            data = json.load(url)
> +            assert data["page"] == page
> +            assert data["items_per_page"] == ITEMS_PER_PAGE
> +            total = data["total_items"]
> +            versions.extend(data["items"])
> +            print(
> +                f"\rDownloading current package versions ({len(versions)}/{total})",
> +                end="",
> +            )
> +            if len(versions) >= total:
> +                break
> +    print()
> +    return versions
> +
> +
> +parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter)
> +parser.add_argument(
> +    "--show-untracked",
> +    help="Show rules where no updated version is known",
> +    action="store_true",
> +)
> +parser.add_argument(
> +    "--show-up-to-date",
> +    help="Show rules which are already at the newest versions",
> +    action="store_true",
> +)
> +parser.add_argument(
> +    "report_file", help="report file generated by `ptxdist fast-bsp-report`"
> +)
> +
> +args = parser.parse_args()
> +
> +versions = None
> +try:
> +    if os.stat(VERSION_CACHE_FILE).st_mtime > time.time() - VERSION_CACHE_EXPIRE_SEC:
> +        print("Using cached version list")
> +        with open(VERSION_CACHE_FILE) as f:
> +            versions = json.load(f)
> +except FileNotFoundError:
> +    # No cache exists, just download it
> +    pass
> +except Exception as e:
> +    print(f"Unexpected error trying to read the version cache: {e}")
> +
> +if not versions:
> +    versions = download_pkg_version_list()
> +    try:
> +        os.makedirs(VERSION_CACHE_DIR, exist_ok=True)
> +        with open(VERSION_CACHE_FILE, "w") as f:
> +            json.dump(versions, f, indent=1)
> +            f.write("\n")
> +    except Exception as e:
> +        print(f"Couldn't write package version cache: {e}")
> +print()
> +
> +stable_versions = {}
> +for package in versions:
> +    stable_versions[package["name"]] = package["stable_version"]
> +
> +with open(args.report_file) as f:
> +    report = yaml.safe_load(f)
> +
> +latest_version = []
> +needs_update = []
> +untracked = []
> +
> +max_pkgname_len = 0
> +max_version_len = 0
> +
> +for pkg in report["packages"].values():
> +    if not "rulefile" in pkg or not "version" in pkg:
> +        continue
> +
> +    max_pkgname_len = max(max_pkgname_len, len(pkg["name"]))
> +    max_version_len = max(max_version_len, len(pkg["version"]))
> +
> +    version = stable_versions.get(pkg["name"])
> +    if version is None:
> +        version = stable_versions.get(pkg["name"].removeprefix("host-"))
> +
> +    if version is None:
> +        untracked.append((pkg["name"], pkg["version"]))
> +    else:
> +        if version != pkg["version"]:
> +            needs_update.append((pkg["name"], pkg["version"], version))
> +        else:
> +            latest_version.append((pkg["name"], pkg["version"]))
> +
> +if args.show_up_to_date:
> +    print("Up-to-date rules")
> +    print("================")
> +    for pkg in latest_version:
> +        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
> +    print()
> +
> +print("Updateable rules")
> +print("================")
> +for pkg in needs_update:
> +    print(
> +        f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)} -> {pkg[2]}"
> +    )
> +print()
> +
> +if args.show_untracked:
> +    print("Untracked rules")
> +    print("===============")
> +    for pkg in untracked:
> +        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
> +    print()
> +
> +print(
> +    f"Updateable: {len(needs_update)} Up-to-date: {len(latest_version)} Untracked: {len(untracked)}"
> +)