From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Fri, 13 Nov 2026 09:40:06 +0100 Received: from metis.whiteo.stw.pengutronix.de ([2a0a:edc0:2:b01:1d::104]) by lore.white.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1xRmpJ-00AMHM-2w for lore@lore.pengutronix.de; Fri, 13 Nov 2026 09:40:06 +0100 Received: from [127.0.0.1] (helo=metis.whiteo.stw.pengutronix.de) by metis.whiteo.stw.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1xRmpG-0000B7-76; Fri, 13 Nov 2026 09:40:02 +0100 Received: from mx1.white.stw.pengutronix.de ([2a0a:edc0:0:b01:1d::107]) by metis.whiteo.stw.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1xRmoH-0004KK-SP for ptxdist@pengutronix.de; Fri, 13 Nov 2026 09:39:01 +0100 Received: from dude05.red.stw.pengutronix.de (dude05.red.stw.4.pengutronix.de [10.17.1.19]) (Authenticated sender: relay-from-dude05.red.stw.pengutronix.de) by mx1.white.stw.pengutronix.de (Postfix) with ESMTPSA id CCCD62058D5; Fri, 13 Nov 2026 09:39:01 +0100 (CET) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=pengutronix.de; s=20260414; t=1794559141; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YTo66UI6wygSmjq7CYzzJkQ2Y4nE5BvRvwT0sPXq/yY=; b=Jc+WwuoJy4jcLFfPGs9WYtKQeyeX9M0GUjLgDGbhYcKinjL0NuNW9J3snNQ2Wnfw7ehblb oM0BUxva3Zcx2+p/EBZOMWnmz34uXpXdUvPfjVuL4TvU/C/Asbk5alYlquzLIZF3uz10OU yvLMM4UY4t9K3hT5119GrxwINGcJNvvD+CSwVzqnQil5Mmeu13hX7GNKVk4bmc/sj2OpOQ J8thF1vPTIzfnFEjVrQPISus+mdGCwmFgHxUEClBZjkYOta32T+uf26Pbt1YEk9rNrvwBI J1/BTAhUlFSS9i5VQwDPILm9M1d9xG94urA0KEyOFFEIkGcQMOZ4Y6ZZH1UElA== ARC-Seal: i=1; s=20260414; d=pengutronix.de; t=1794559141; a=rsa-sha256; cv=none; b=OB2L03U/sMA1G9I0tv/2POfmyFzCZEo1GDPxzMb2nxSfvcIu9iVIC2qDKZsP9v0E+mSJLx XfkNHi9PkR3mfDBcgDOgrs8ynDPTzndWG9akDhkkfMg/kQ2BXR184Qki/dzO2j8VDHV1Bk YQOiO4JpyjFhXCbuK8rsFKdUEb1RLwZMiYTmg+ohioUk79WGHJarkAPR40jfOXSEWKoBYQ pJ2sousq69kDLEpa9L0Cao6alLo0W0tOsnsWjaOXNK3wPonm5m+4wYlMzKohIzCLsn6hEL psfLW35J86YUrS9m1l6vPF31ORrOeh6VZml30YSZV3CCBoX6ut6wHgbhn5+vJQ== ARC-Authentication-Results: i=1; ORIGINATING; auth=pass smtp.auth=relay-from-dude05.red.stw.pengutronix.de smtp.mailfrom=mol@pengutronix.de Received: by dude05.red.stw.pengutronix.de (Postfix, from userid 1014) id BDB7C743103; Fri, 13 Nov 2026 09:39:01 +0100 (CET) From: Michael Olbrich To: ptxdist@pengutronix.de Date: Fri, 13 Nov 2026 09:39:01 +0100 Message-ID: <20261113083901.1379293-1-m.olbrich@pengutronix.de> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261023090104.3420495-1-s.pueschel@pengutronix.de> References: <20261023090104.3420495-1-s.pueschel@pengutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: Re: [ptxdist] [APPLIED] scripts: add update_check script X-BeenThere: ptxdist@pengutronix.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: PTXdist Development Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ptxdist@pengutronix.de Cc: =?UTF-8?q?Sven=20P=C3=BCschel?= Sender: "ptxdist" X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: ptxdist-bounces@pengutronix.de X-SA-Exim-Scanned: No (on metis.whiteo.stw.pengutronix.de); SAEximRunCond expanded to false Thanks, applied as abd6d1ad5df147318c60039150492bb9e5d3d416. Michael [sent from post-receive hook] On Fri, 13 Nov 2026 09:39:01 +0100, Sven Püschel wrote: > Add an update_check python script as an easy check which PTXdist > rules can be bumped in their versions. It uses the report file generated > from an existing PTXdist BSP as a basis to determine the package > versions. Then it compares these package versions with the package > versions against the newest release versions reported by > release-monitoring.org . > > release-monitoring.org is a deployment of an open-source release > monitoring project named Anitya [1]. It allows to define a Project > which is some software you want to keep track of updates. A given > software can have multiple Projects. An example is GTK+, where you want > to have separate projects for GTK+ 3 and GTK+ 4. A Project can then > define a mapping to a package name in a Distribution. > > I've created the PTXdist distribution [2] and created Project mappings > for a good chunk of the PTXdist rules. I've excluded certain rules with > user-defined versions (e.g. the linux kernel) and projects where it was > unclear if PTXdist should track the latest major version or use the > latest version of the current major version. Also I've ignored rules, > where no Project existed yet. Furthermore I've skipped the `python-` > rules, as they can probably be added in an automated fashion (given the > pypy name can be used for matching purposes). > > This python script uses the unauthenticated API to get all current > versions for the Projects with a mapped PTXdist name. It then caches > these versions locally (updating only if the cache is older then 24h) > and compares the versions against the versions in the report file. > It displays all rules that can be updated and also counts the number of > up-to-date rules and rules missing a mapping. With a command line flag > the latter two can also be displayed. > > Note that this isn't perfect and just provides an approximation to the > actual truth. There are a number of imperfections: > - PTXdist using non-stable versions (e.g. iftop 1.0pre4 instead of 0.17) > - Versions with different separators (dashes/underscores vs. dots) > - Rules not intended to be tracked (e.g. cross-dummy-strip) > > Also there are some pitfalls in this design: > - Version numbers are based on the BSP > - Updated rules might be already present in a newer PTXdist release > - Rules might be overwritten with an older version locally > - BSP only rules (custom applications) will show up as untracked > - Versions may lag behind a bit (Anitya cron job and rate-limits) > > [1] https://github.com/fedora-infra/anitya > [2] https://release-monitoring.org/distro/PTXdist/ > > Signed-off-by: Sven Püschel > Message-Id: <20261023090104.3420495-1-s.pueschel@pengutronix.de> > [mol: reformated with 'ruff format'] > Signed-off-by: Michael Olbrich > > diff --git a/scripts/update_check.py b/scripts/update_check.py > new file mode 100755 > index 000000000000..86570ae57901 > --- /dev/null > +++ b/scripts/update_check.py > @@ -0,0 +1,153 @@ > +#!/usr/bin/env python3 > +# > +# Copyright (C) 2026 by Sven Püschel > +# > +# For further information about the PTXdist project and license conditions > +# see the README file. > +# > + > +import argparse > +import itertools > +import json > +import os > +import time > +from urllib import request > + > +import yaml > + > +API_ENDPOINT = "https://release-monitoring.org/api/v2" > +ITEMS_PER_PAGE = 25 > + > +XDG_CACHE_DIR = os.getenv("XDG_CACHE_HOME") > +if not XDG_CACHE_DIR: > + XDG_CACHE_DIR = os.path.join(os.getenv("HOME"), ".cache") > + > +VERSION_CACHE_FILENAME = "upstream_pkg_versions.json" > +VERSION_CACHE_EXPIRE_SEC = 24 * 60 * 60 > +VERSION_CACHE_DIR = os.path.join(XDG_CACHE_DIR, "ptxdist") > +VERSION_CACHE_FILE = os.path.join(VERSION_CACHE_DIR, VERSION_CACHE_FILENAME) > + > + > +def download_pkg_version_list(): > + versions = [] > + > + for page in itertools.count(1): > + with request.urlopen( > + request.Request( > + f"{API_ENDPOINT}/packages/?distribution=PTXdist&page={page}&items_per_page={ITEMS_PER_PAGE}", > + headers={"User-Agent": "PTXdist update_check script"}, > + ) > + ) as url: > + data = json.load(url) > + assert data["page"] == page > + assert data["items_per_page"] == ITEMS_PER_PAGE > + total = data["total_items"] > + versions.extend(data["items"]) > + print( > + f"\rDownloading current package versions ({len(versions)}/{total})", > + end="", > + ) > + if len(versions) >= total: > + break > + print() > + return versions > + > + > +parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter) > +parser.add_argument( > + "--show-untracked", > + help="Show rules where no updated version is known", > + action="store_true", > +) > +parser.add_argument( > + "--show-up-to-date", > + help="Show rules which are already at the newest versions", > + action="store_true", > +) > +parser.add_argument( > + "report_file", help="report file generated by `ptxdist fast-bsp-report`" > +) > + > +args = parser.parse_args() > + > +versions = None > +try: > + if os.stat(VERSION_CACHE_FILE).st_mtime > time.time() - VERSION_CACHE_EXPIRE_SEC: > + print("Using cached version list") > + with open(VERSION_CACHE_FILE) as f: > + versions = json.load(f) > +except FileNotFoundError: > + # No cache exists, just download it > + pass > +except Exception as e: > + print(f"Unexpected error trying to read the version cache: {e}") > + > +if not versions: > + versions = download_pkg_version_list() > + try: > + os.makedirs(VERSION_CACHE_DIR, exist_ok=True) > + with open(VERSION_CACHE_FILE, "w") as f: > + json.dump(versions, f, indent=1) > + f.write("\n") > + except Exception as e: > + print(f"Couldn't write package version cache: {e}") > +print() > + > +stable_versions = {} > +for package in versions: > + stable_versions[package["name"]] = package["stable_version"] > + > +with open(args.report_file) as f: > + report = yaml.safe_load(f) > + > +latest_version = [] > +needs_update = [] > +untracked = [] > + > +max_pkgname_len = 0 > +max_version_len = 0 > + > +for pkg in report["packages"].values(): > + if not "rulefile" in pkg or not "version" in pkg: > + continue > + > + max_pkgname_len = max(max_pkgname_len, len(pkg["name"])) > + max_version_len = max(max_version_len, len(pkg["version"])) > + > + version = stable_versions.get(pkg["name"]) > + if version is None: > + version = stable_versions.get(pkg["name"].removeprefix("host-")) > + > + if version is None: > + untracked.append((pkg["name"], pkg["version"])) > + else: > + if version != pkg["version"]: > + needs_update.append((pkg["name"], pkg["version"], version)) > + else: > + latest_version.append((pkg["name"], pkg["version"])) > + > +if args.show_up_to_date: > + print("Up-to-date rules") > + print("================") > + for pkg in latest_version: > + print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}") > + print() > + > +print("Updateable rules") > +print("================") > +for pkg in needs_update: > + print( > + f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)} -> {pkg[2]}" > + ) > +print() > + > +if args.show_untracked: > + print("Untracked rules") > + print("===============") > + for pkg in untracked: > + print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}") > + print() > + > +print( > + f"Updateable: {len(needs_update)} Up-to-date: {len(latest_version)} Untracked: {len(untracked)}" > +)