From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Fri, 23 Oct 2026 12:38:11 +0200 Received: from metis.whiteo.stw.pengutronix.de ([2a0a:edc0:2:b01:1d::104]) by lore.white.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1xKCf4-002DwD-38 for lore@lore.pengutronix.de; Fri, 23 Oct 2026 12:38:11 +0200 Received: from [127.0.0.1] (helo=metis.whiteo.stw.pengutronix.de) by metis.whiteo.stw.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1xKCf4-0003Iq-Fa; Fri, 23 Oct 2026 12:38:10 +0200 Received: from mx1.white.stw.pengutronix.de ([2a0a:edc0:0:b01:1d::107]) by metis.whiteo.stw.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1xKCeq-0003Ij-6N for ptxdist@pengutronix.de; Fri, 23 Oct 2026 12:37:56 +0200 Received: from [0.0.0.0] (ptz.office.stw.pengutronix.de [IPv6:2a0a:edc0:0:900:1d::77]) (Authenticated sender: spu@pengutronix.de) by mx1.white.stw.pengutronix.de (Postfix) with ESMTPSA id 21A7B200B9E; Fri, 23 Oct 2026 12:37:56 +0200 (CEST) ARC-Seal: i=1; s=20260414; d=pengutronix.de; t=1792751876; a=rsa-sha256; cv=none; b=Ea9SjDiP/jvIXlZuoqRWIBvW1qrIaXxHU+3AS5zdeOldR1jbggAtNHux7OKDJlDLYKGCMm YyYCZvdaMDWUbI4ibYrGuvbRUp1Wt0zmKmtxXOUmQbY+xrONqzvwN8026JP4OqnieHN1Xi /Vit5uvKFBcmb+MYoqOel1YlMaVPs4OtBfNGYL60qXf3Kuh81x84fHfsbwZwkBVLfu4Fmy paUhZAIRb733Y3KILgsTQVlXjsrIq6u1BJFhvk37LWA+or2azZRXMNN4vcKEmOBh/VFg7X cL0AqsIjTLKkcW0tHAAKxSk9EVoUUF92Y9CKghgcil7QH8jar5WDeZYDZ/xmJA== ARC-Authentication-Results: i=1; ORIGINATING; auth=pass smtp.auth=spu@pengutronix.de smtp.mailfrom=s.pueschel@pengutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=pengutronix.de; s=20260414; t=1792751876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ncyriMh6wwOoTuK+qWt7SfiJNCZU65Z9KXYpujsicM0=; b=I67v8oYgD56qiXloTbncNlWwXSuupNjNgBzN12h85b7jF5/nxoRNZlr/pMhYQgN9tqoPQG sA7G3t1YF0THSPdXaWlecFPlErW4tKvkMMJlvFQwpSpNbnKwvudzUot23bPdA3ybZfsCR0 j/2JgmAs+DxvIZJkmGA8/gbbeN11TKI2u8+GSBNE3XGhGuZh26LSIuow1x7W+65Ee3KPUw nEt5fbqOFtxPG8JiwF6qHIWd5yQdNKCJNDi8FbDBcw6T7GjOf4kvpLb7oPKV+Xwq/kMqaV x1OJoyfYX2HgGVh/MpcChwbsvrsgvnb0EWlwjN3yy856c3jE/L/zDxDlw+AJ3A== Message-ID: <33d2aeec-6f39-4b04-9734-780dd75c8701@pengutronix.de> Date: Fri, 23 Oct 2026 12:37:55 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Bruno Thomsen , ptxdist@pengutronix.de References: <20261023090104.3420495-1-s.pueschel@pengutronix.de> Content-Language: en-US From: =?UTF-8?Q?Sven_P=C3=BCschel?= In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [ptxdist] [PATCH] scripts: add update_check script X-BeenThere: ptxdist@pengutronix.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: PTXdist Development Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ptxdist@pengutronix.de Sender: "ptxdist" X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: ptxdist-bounces@pengutronix.de X-SA-Exim-Scanned: No (on metis.whiteo.stw.pengutronix.de); SAEximRunCond expanded to false Hi Bruno, On 10/23/26 11:28, Bruno Thomsen wrote: > Den fre. 23. okt. 2026 kl. 11.01 skrev Sven Püschel : >> Add an update_check python script as an easy check which PTXdist >> rules can be bumped in their versions. It uses the report file generated >> from an existing PTXdist BSP as a basis to determine the package >> versions. Then it compares these package versions with the package >> versions against the newest release versions reported by >> release-monitoring.org . >> >> release-monitoring.org is a deployment of an open-source release >> monitoring project named Anitya [1]. It allows to define a Project >> which is some software you want to keep track of updates. A given >> software can have multiple Projects. An example is GTK+, where you want >> to have separate projects for GTK+ 3 and GTK+ 4. A Project can then >> define a mapping to a package name in a Distribution. >> >> I've created the PTXdist distribution [2] and created Project mappings >> for a good chunk of the PTXdist rules. I've excluded certain rules with >> user-defined versions (e.g. the linux kernel) and projects where it was >> unclear if PTXdist should track the latest major version or use the >> latest version of the current major version. Also I've ignored rules, >> where no Project existed yet. Furthermore I've skipped the `python-` >> rules, as they can probably be added in an automated fashion (given the >> pypy name can be used for matching purposes). > Nice that you have added PTXdist to release-monitoring.org, you can > also see it in Repology[1] where it has lists of outdated packages[2], > potentially vulnerable packages[3], and packages with problems[4] like > dead URLs or permanent redirects. Information is tracking the master branch. Yeah, Michael also mentioned your PTXDist patches towards repology and that he'd prefer some solution based on a report file. Personally I didn't quite understand how Repology actually works. It looks like fuzzy matching of package names and getting the newest versions based on the comparison with other distro packages. Release-Monitoring looked dead-simple: Each Project has a dedicated definition where to look for new versions and a explicit mapping to a distro package name. Therefore I'd just went with it (with the side effect of also making PTXdist a bit present there). I'm not too invested, which Backend the script uses to actually get the necessary data (or it could also use multiple backends, etc.). Sincerely     Sven > /Bruno > > [1] https://repology.org/repository/ptxdist > [2] https://repology.org/projects/?inrepo=ptxdist&outdated=1&families_newest=2- > [3] https://repology.org/projects/?inrepo=ptxdist&vulnerable=1 > [4] https://repology.org/repository/ptxdist/problems > >> This python script uses the unauthenticated API to get all current >> versions for the Projects with a mapped PTXdist name. It then caches >> these versions locally (updating only if the cache is older then 24h) >> and compares the versions against the versions in the report file. >> It displays all rules that can be updated and also counts the number of >> up-to-date rules and rules missing a mapping. With a command line flag >> the latter two can also be displayed. >> >> Note that this isn't perfect and just provides an approximation to the >> actual truth. There are a number of imperfections: >> - PTXdist using non-stable versions (e.g. iftop 1.0pre4 instead of 0.17) >> - Versions with different separators (dashes/underscores vs. dots) >> - Rules not intended to be tracked (e.g. cross-dummy-strip) >> >> Also there are some pitfalls in this design: >> - Version numbers are based on the BSP >> - Updated rules might be already present in a newer PTXdist release >> - Rules might be overwritten with an older version locally >> - BSP only rules (custom applications) will show up as untracked >> - Versions may lag behind a bit (Anitya cron job and rate-limits) >> >> [1] https://github.com/fedora-infra/anitya >> [2] https://release-monitoring.org/distro/PTXdist/ >> >> Signed-off-by: Sven Püschel >> --- >> scripts/update_check.py | 132 ++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 132 insertions(+) >> create mode 100755 scripts/update_check.py >> >> diff --git a/scripts/update_check.py b/scripts/update_check.py >> new file mode 100755 >> index 000000000..94ffe83bc >> --- /dev/null >> +++ b/scripts/update_check.py >> @@ -0,0 +1,132 @@ >> +#!/usr/bin/env python3 >> +# >> +# Copyright (C) 2026 by Sven Püschel >> +# >> +# For further information about the PTXdist project and license conditions >> +# see the README file. >> +# >> + >> +import argparse >> +import itertools >> +import json >> +import os >> +import time >> +from urllib import request >> + >> +import yaml >> + >> +API_ENDPOINT = 'https://release-monitoring.org/api/v2' >> +ITEMS_PER_PAGE = 25 >> + >> +XDG_CACHE_DIR = os.getenv('XDG_CACHE_HOME') >> +if not XDG_CACHE_DIR: >> + XDG_CACHE_DIR = os.path.join(os.getenv('HOME'), '.cache') >> + >> +VERSION_CACHE_FILENAME = 'upstream_pkg_versions.json' >> +VERSION_CACHE_EXPIRE_SEC = 24 * 60 * 60 >> +VERSION_CACHE_DIR = os.path.join(XDG_CACHE_DIR, 'ptxdist') >> +VERSION_CACHE_FILE = os.path.join(VERSION_CACHE_DIR, VERSION_CACHE_FILENAME) >> + >> +def download_pkg_version_list(): >> + versions = [] >> + >> + for page in itertools.count(1): >> + with request.urlopen(request.Request( >> + f'{API_ENDPOINT}/packages/?distribution=PTXdist&page={page}&items_per_page={ITEMS_PER_PAGE}', >> + headers={ 'User-Agent': 'PTXdist update_check script' } >> + )) as url: >> + data = json.load(url) >> + assert data['page'] == page >> + assert data['items_per_page'] == ITEMS_PER_PAGE >> + total = data['total_items'] >> + versions.extend(data['items']) >> + print(f'\rDownloading current package versions ({len(versions)}/{total})', end='') >> + if len(versions) >= total: >> + break >> + print() >> + return versions >> + >> +parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter) >> +parser.add_argument('--show-untracked', help='Show rules where no updated version is known', action='store_true') >> +parser.add_argument('--show-up-to-date', help='Show rules which are already at the newest versions', action='store_true') >> +parser.add_argument('report_file', help='report file generated by `ptxdist fast-bsp-report`') >> + >> +args = parser.parse_args() >> + >> +versions = None >> +try: >> + if os.stat(VERSION_CACHE_FILE).st_mtime > time.time() - VERSION_CACHE_EXPIRE_SEC: >> + print('Using cached version list') >> + with open(VERSION_CACHE_FILE) as f: >> + versions = json.load(f) >> +except FileNotFoundError: >> + # No cache exists, just download it >> + pass >> +except Exception as e: >> + print(f'Unexpected error trying to read the version cache: {e}') >> + >> +if not versions: >> + versions = download_pkg_version_list() >> + try: >> + os.makedirs(VERSION_CACHE_DIR, exist_ok=True) >> + with open(VERSION_CACHE_FILE, 'w') as f: >> + json.dump(versions, f, indent=1) >> + f.write("\n") >> + except Exception as e: >> + print(f"Couldn't write package version cache: {e}") >> +print() >> + >> +stable_versions = {} >> +for package in versions: >> + stable_versions[package['name']] = package['stable_version'] >> + >> +with open(args.report_file) as f: >> + report = yaml.safe_load(f) >> + >> +latest_version = [] >> +needs_update = [] >> +untracked = [] >> + >> +max_pkgname_len = 0 >> +max_version_len = 0 >> + >> +for pkg in report['packages'].values(): >> + if not 'rulefile' in pkg or not 'version' in pkg: >> + continue >> + >> + max_pkgname_len = max(max_pkgname_len, len(pkg['name'])) >> + max_version_len = max(max_version_len, len(pkg['version'])) >> + >> + version = stable_versions.get(pkg['name']) >> + if version is None: >> + version = stable_versions.get(pkg['name'].removeprefix('host-')) >> + >> + if version is None: >> + untracked.append((pkg['name'], pkg['version'])) >> + else: >> + if version != pkg['version']: >> + needs_update.append((pkg['name'], pkg['version'], version)) >> + else: >> + latest_version.append((pkg['name'], pkg['version'])) >> + >> +if args.show_up_to_date: >> + print('Up-to-date rules') >> + print('================') >> + for pkg in latest_version: >> + print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}") >> + print() >> + >> +print('Updateable rules') >> +print('================') >> +for pkg in needs_update: >> + print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)} -> {pkg[2]}") >> +print() >> + >> +if args.show_untracked: >> + print('Untracked rules') >> + print('===============') >> + for pkg in untracked: >> + print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}") >> + print() >> + >> +print(f"Updateable: {len(needs_update)} Up-to-date: {len(latest_version)} Untracked: {len(untracked)}") >> -- >> 2.47.3 >> >>