From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Mon, 26 Oct 2026 16:42:31 +0100 Received: from metis.whiteo.stw.pengutronix.de ([2a0a:edc0:2:b01:1d::104]) by lore.white.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1xLMqE-003YpF-20 for lore@lore.pengutronix.de; Mon, 26 Oct 2026 16:42:31 +0100 Received: from [127.0.0.1] (helo=metis.whiteo.stw.pengutronix.de) by metis.whiteo.stw.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1xLMqE-00039O-29; Mon, 26 Oct 2026 16:42:30 +0100 Received: from mx1.white.stw.pengutronix.de ([2a0a:edc0:0:b01:1d::107]) by metis.whiteo.stw.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1xLMq4-00039E-4i for ptxdist@pengutronix.de; Mon, 26 Oct 2026 16:42:20 +0100 Received: from mail-ot1-x333.google.com (mail-ot1-x333.google.com [IPv6:2607:f8b0:4864:20::333]) by mx1.white.stw.pengutronix.de (Postfix) with ESMTPS id B847C200F75 for ; Mon, 26 Oct 2026 16:42:15 +0100 (CET) ARC-Seal: i=2; s=20260414; d=pengutronix.de; t=1793029335; a=rsa-sha256; cv=pass; b=B2vtf9G4qv+sxAmfFSrHyFS9AmZ9GeEpEfXUGgtYsfnoN1gQGcSCFrITsZEiHxu8yMJ6yM iSmZBX1BBFx4SxY/tnj02/1j+nsj9sUx3nuuVzx7V5XNb3VNNjVkqIpM33hO8SGragLLRZ 0FHazMTl/K0xKKC1M2WCePC3DiRyCTmakpdxs2Ggr/SfbQcZ/8N8JRCyLJTjjaW1CgRi4X Il4ozQjEgms4hXVO7HT+nUOpkjfsW2BoxqhtwAdDJ7I1t3t1Fz8Gqb5MWgOVj5et7kD1aG po3QYfIsqkNlHtonkqwYKGwJNQdTDEyzRQ+ODCzwsC3xAVvBsUp6+Cm+NaYEeg== ARC-Authentication-Results: i=2; mx1.white.stw.pengutronix.de; dkim=pass header.d=gmail.com header.s=20251104 header.b=JtutydEX; spf=pass (mx1.white.stw.pengutronix.de: domain of bruno.thomsen@gmail.com designates 2607:f8b0:4864:20::333 as permitted sender) smtp.mailfrom=bruno.thomsen@gmail.com; dmarc=pass (policy=none) header.from=gmail.com; arc=pass ("google.com:s=arc-20260327:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=pengutronix.de; s=20260414; t=1793029335; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TYyx4lu6Tx0a4RvtTwUmyeMyOnjzU+lfhSXR747e3dw=; b=M3rrNLhO1rVi9CPfbSnEyxfWkURg7hMfxfctsYHnt8rbtyfJDWkMYOKYdrlsYOvVVhfQbR doa+SoCohUfiVEMy5mkPIL+r4GwwqmBfl/ATbek6T8ZGXWxq8EyvqKA4QYZAMgjUXsPxa3 TU2VaB70e0UFU+9tHf9MoG5RhdlQLN0YE2Buxgqzii2MzBVbGDojIOxTiX+XSPHBKFY7p4 bUXWL68Oqc7FjzmLO3K2cq41weJyGvAHqZBQtl+HY+yS1ppebEVVLGuld34HnX0i8Vqyk/ ZJIu2HQgl9Qr4vKr7Wt3F943i80P57kjbitTadKITTYkASt49MkRg7ME0kDZog== Received: by mail-ot1-x333.google.com with SMTP id 46e09a7af769-84a248254f4so2066333a34.0 for ; Mon, 26 Oct 2026 08:42:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1793029334; cv=none; d=google.com; s=arc-20260327; b=rsBlsV96NOMCdbUkDl+yyKvYHMZDbdB8h4qxAwQIzz+JOVf0ZOcU1qKzvslscw0z2K gAJ1HSoJaThmRf7Oqv2irmbvLcxQqImhaINO2KqVpDWtX25HqsIZ2O4NT5TLXoMSBm4z yA/zLrgiDY2AQgg8jXX45yfI4CBuz7+Xgr8vLsGo9/NjSwO1EjE1glJH2O0NZYbYTnGr q28verqpDgWuinmaTO8dYKWKVS1qzuatHv34jt2Vl9Ww9P9TbAE08fTHSFEuUkheMV+q DK2wtM675Ov2WYydjfoAlSObHpkoulatxEkPoXgL+rCwM33m/BRDU9NBEqzn+xDCmlqE PBUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=TYyx4lu6Tx0a4RvtTwUmyeMyOnjzU+lfhSXR747e3dw=; fh=DHqXvQIVNw6lLVH+86wRWkovRDKdfMahlZNYDvPUa3k=; b=pHe0cQpXtFs+HXajW26omlkFlo5gnz6+550qTTyeEy9tfWBMqq+HWrtQ80ed/XNn/L ZstlN80IQQchausJNcRYFbFTvd5Xnm5s0XIAwTJIMFELT8IOLys920TGNMDKWs0Tnau5 jQufRu4UeqQKUPeqvcPr32xP8DrgL4//NXhMsqlqTHJ8XqLuQ9UDHqiICZLLy06vULmt mXEKpbtt/G50U/SfwgMy9x3MCeU/NhcvG7X+6G4j76qoGQ/QIlWeHR7Et3c5mTjqC/bX UrSqKdFvnOnoKESIx0C27IRHjuKzmNaQmcXw9HlTEV7QvNyIpw3k4oz0GTgP6ZvkI5/o 3MWQ==; darn=pengutronix.de ARC-Authentication-Results: i=1; mx.google.com; arc=none X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793029334; x=1793634134; h=content-transfer-encoding:content-type:cc:to:subject:message-id :date:from:in-reply-to:references:mime-version:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=TYyx4lu6Tx0a4RvtTwUmyeMyOnjzU+lfhSXR747e3dw=; b=Q7ghyWShwiJ/DfStmdFYTF3dOdi1aqMV5cqSwu5WoQ3RVX6C1w+8pi6gY0pPNkfUBn SAtHbhY0jyoG5TPmniP/GixitusbisiI5zRDqrL4ujj4XExSZ0CfHRJaxCsk3syUHsQK s4AqV0uON9s7j+26PoCKla+7hW0o5VLUivi9Y9Ncys/1zWkfpKV4x71eKXCSRd/8r2Ia 8Rmh6KUlXlRS4EFoj8So0JXGLuUS7MZmGtaYCx6ND2K/EFGdJEBOy4PoSjHkc5WSysvA bwXfPrXtifRZlVEGD58egOCj+pP41e9BshjcadNBMYxItAP9VQ8H9QEmEsJVnmQgACJ4 96tw== X-Gm-Message-State: AFq9FYImI5INQWJGyxVARa9NjLDpxfW4yN9QMIbQquwcjDyFYeIWynT4 C3DLf6uqjMKWWruwbXdp8rb9Ie+CfKBumTZ955fv6SJjGGluY6UtCpHnoiCtRDDJD2s7ogX6NqK yoceFX+7sDUF8Z87orT/6wS4lKnjjmlCbxivo X-Gm-Gg: AYBFou2q1EUh2mkpB0oPKdtW0w64tIKojqO/lsG2lB6i023bdxoHXZLPQib2UnVKjqF JRveOGegx6v6DtxyBueOUp+ioe2sIkkuhaIKxTqx0BjEtD1jpKxRGl8S3zzPDgfKL5mP19rUsMb wJ2kfg2WUs2mPB+ChDu927rCJyXQLR26si4vAi93zgKMIEXJgz6siU76kJ+/4/xQrZ3BVG9MQl/ qaW5nYsPK/h5UeinxVszRC1AmcPA8uQsCY79ilVRpEIeokXihp6j1aHZnMW7Xyirp4CqN0OfexE l6A9jc8NjNHzEaD3DYKZSoBm7xmUQjuuYIcbRYMh+KPcArqKWXQK5Da3CFtb8705k0b9NydKW39 dFs8R++QzlPECYLHYmwWL2QEtMNzKhRlzTtuydVXKbHvsYYm/d6OhdlGZ3w== X-Received: by 2002:a05:6808:c1b8:b0:4d3:6be7:c5e9 with SMTP id 5614622812f47-55c0026041fmr4691434b6e.24.1793029333630; Mon, 26 Oct 2026 08:42:13 -0700 (PDT) MIME-Version: 1.0 References: <20261023090104.3420495-1-s.pueschel@pengutronix.de> <33d2aeec-6f39-4b04-9734-780dd75c8701@pengutronix.de> In-Reply-To: <33d2aeec-6f39-4b04-9734-780dd75c8701@pengutronix.de> From: Bruno Thomsen Date: Mon, 26 Oct 2026 16:41:57 +0100 X-Gm-Features: AclHuK9IL2-GqjkafMXL_XNhNQSYfEuTTVmrQ-1Xr8atlFtwE3V3SoXNWeHk3nA Message-ID: To: =?UTF-8?Q?Sven_P=C3=BCschel?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: mx1 X-Rspamd-Queue-Id: B847C200F75 X-Spamd-Result: default: False [-5.00 / 15.00]; BAYES_HAM(-3.00)[99.99%]; ARC_ALLOW(-1.00)[google.com:s=arc-20260327:i=1]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20251104]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4864::/56]; MIME_GOOD(-0.10)[text/plain]; ARC_SIGNED(0.00)[pengutronix.de:s=20260414:i=2]; RCVD_TLS_LAST(0.00)[]; TAGGED_FROM(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MISSING_XM_UA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[ptxdist@pengutronix.de]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::333:from]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; NEURAL_HAM(-0.00)[-0.973]; RCVD_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-Rspamd-Action: no action X-Stat-Signature: qng1xfyenqnb5ktpw4rfpyhd3i14z7gg Subject: Re: [ptxdist] [PATCH] scripts: add update_check script X-BeenThere: ptxdist@pengutronix.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: PTXdist Development Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ptxdist@pengutronix.de Cc: ptxdist@pengutronix.de Sender: "ptxdist" X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: ptxdist-bounces@pengutronix.de X-SA-Exim-Scanned: No (on metis.whiteo.stw.pengutronix.de); SAEximRunCond expanded to false Den fre. 23. okt. 2026 kl. 12.37 skrev Sven P=C3=BCschel : > > Hi Bruno, > > On 10/23/26 11:28, Bruno Thomsen wrote: > > Den fre. 23. okt. 2026 kl. 11.01 skrev Sven P=C3=BCschel : > >> Add an update_check python script as an easy check which PTXdist > >> rules can be bumped in their versions. It uses the report file generat= ed > >> from an existing PTXdist BSP as a basis to determine the package > >> versions. Then it compares these package versions with the package > >> versions against the newest release versions reported by > >> release-monitoring.org . > >> > >> release-monitoring.org is a deployment of an open-source release > >> monitoring project named Anitya [1]. It allows to define a Project > >> which is some software you want to keep track of updates. A given > >> software can have multiple Projects. An example is GTK+, where you wan= t > >> to have separate projects for GTK+ 3 and GTK+ 4. A Project can then > >> define a mapping to a package name in a Distribution. > >> > >> I've created the PTXdist distribution [2] and created Project mappings > >> for a good chunk of the PTXdist rules. I've excluded certain rules wit= h > >> user-defined versions (e.g. the linux kernel) and projects where it wa= s > >> unclear if PTXdist should track the latest major version or use the > >> latest version of the current major version. Also I've ignored rules, > >> where no Project existed yet. Furthermore I've skipped the `python-` > >> rules, as they can probably be added in an automated fashion (given th= e > >> pypy name can be used for matching purposes). > > Nice that you have added PTXdist to release-monitoring.org, you can > > also see it in Repology[1] where it has lists of outdated packages[2], > > potentially vulnerable packages[3], and packages with problems[4] like > > dead URLs or permanent redirects. Information is tracking the master br= anch. > > Yeah, Michael also mentioned your PTXDist patches towards repology and > that he'd prefer some solution based on a report file. > Latest versions of the patches were not posted to the mailing list and we continued development in a separate repo[1] which also contains regular updates to the JSON output that repology consumes. > Personally I didn't quite understand how Repology actually works. It > looks like fuzzy matching of package names and getting the newest > versions based on the comparison with other distro packages. > Release-Monitoring looked dead-simple: Each Project has a dedicated > definition where to look for new versions and a explicit mapping to a > distro package name. Therefore I'd just went with it (with the side > effect of also making PTXdist a bit present there). I'm not too > invested, which Backend the script uses to actually get the necessary > data (or it could also use multiple backends, etc.). Yes, packages are fuzzy matched by download URL + name, and new versions are detected when 2 or more distributions have a newer version. I like Release-Monitoring as well, so I get a RHBZ ping to update a RPM spe= c :) They are good for two slightly different things, but I hope you get the patch series complete. /Bruno [1] https://github.com/baxeno/ptxdist-repology > > Sincerely > Sven > > > /Bruno > > > > [1] https://repology.org/repository/ptxdist > > [2] https://repology.org/projects/?inrepo=3Dptxdist&outdated=3D1&famili= es_newest=3D2- > > [3] https://repology.org/projects/?inrepo=3Dptxdist&vulnerable=3D1 > > [4] https://repology.org/repository/ptxdist/problems > > > >> This python script uses the unauthenticated API to get all current > >> versions for the Projects with a mapped PTXdist name. It then caches > >> these versions locally (updating only if the cache is older then 24h) > >> and compares the versions against the versions in the report file. > >> It displays all rules that can be updated and also counts the number o= f > >> up-to-date rules and rules missing a mapping. With a command line flag > >> the latter two can also be displayed. > >> > >> Note that this isn't perfect and just provides an approximation to the > >> actual truth. There are a number of imperfections: > >> - PTXdist using non-stable versions (e.g. iftop 1.0pre4 instead of 0.1= 7) > >> - Versions with different separators (dashes/underscores vs. dots) > >> - Rules not intended to be tracked (e.g. cross-dummy-strip) > >> > >> Also there are some pitfalls in this design: > >> - Version numbers are based on the BSP > >> - Updated rules might be already present in a newer PTXdist release > >> - Rules might be overwritten with an older version locally > >> - BSP only rules (custom applications) will show up as untracked > >> - Versions may lag behind a bit (Anitya cron job and rate-limits) > >> > >> [1] https://github.com/fedora-infra/anitya > >> [2] https://release-monitoring.org/distro/PTXdist/ > >> > >> Signed-off-by: Sven P=C3=BCschel > >> --- > >> scripts/update_check.py | 132 ++++++++++++++++++++++++++++++++++++++= ++ > >> 1 file changed, 132 insertions(+) > >> create mode 100755 scripts/update_check.py > >> > >> diff --git a/scripts/update_check.py b/scripts/update_check.py > >> new file mode 100755 > >> index 000000000..94ffe83bc > >> --- /dev/null > >> +++ b/scripts/update_check.py > >> @@ -0,0 +1,132 @@ > >> +#!/usr/bin/env python3 > >> +# > >> +# Copyright (C) 2026 by Sven P=C3=BCschel > >> +# > >> +# For further information about the PTXdist project and license condi= tions > >> +# see the README file. > >> +# > >> + > >> +import argparse > >> +import itertools > >> +import json > >> +import os > >> +import time > >> +from urllib import request > >> + > >> +import yaml > >> + > >> +API_ENDPOINT =3D 'https://release-monitoring.org/api/v2' > >> +ITEMS_PER_PAGE =3D 25 > >> + > >> +XDG_CACHE_DIR =3D os.getenv('XDG_CACHE_HOME') > >> +if not XDG_CACHE_DIR: > >> + XDG_CACHE_DIR =3D os.path.join(os.getenv('HOME'), '.cache') > >> + > >> +VERSION_CACHE_FILENAME =3D 'upstream_pkg_versions.json' > >> +VERSION_CACHE_EXPIRE_SEC =3D 24 * 60 * 60 > >> +VERSION_CACHE_DIR =3D os.path.join(XDG_CACHE_DIR, 'ptxdist') > >> +VERSION_CACHE_FILE =3D os.path.join(VERSION_CACHE_DIR, VERSION_CACHE_= FILENAME) > >> + > >> +def download_pkg_version_list(): > >> + versions =3D [] > >> + > >> + for page in itertools.count(1): > >> + with request.urlopen(request.Request( > >> + f'{API_ENDPOINT}/packages/?distribution=3DPTXdist&pag= e=3D{page}&items_per_page=3D{ITEMS_PER_PAGE}', > >> + headers=3D{ 'User-Agent': 'PTXdist update_check scrip= t' } > >> + )) as url: > >> + data =3D json.load(url) > >> + assert data['page'] =3D=3D page > >> + assert data['items_per_page'] =3D=3D ITEMS_PER_PAGE > >> + total =3D data['total_items'] > >> + versions.extend(data['items']) > >> + print(f'\rDownloading current package versions ({len(vers= ions)}/{total})', end=3D'') > >> + if len(versions) >=3D total: > >> + break > >> + print() > >> + return versions > >> + > >> +parser =3D argparse.ArgumentParser(formatter_class=3Dargparse.RawDesc= riptionHelpFormatter) > >> +parser.add_argument('--show-untracked', help=3D'Show rules where no u= pdated version is known', action=3D'store_true') > >> +parser.add_argument('--show-up-to-date', help=3D'Show rules which are= already at the newest versions', action=3D'store_true') > >> +parser.add_argument('report_file', help=3D'report file generated by `= ptxdist fast-bsp-report`') > >> + > >> +args =3D parser.parse_args() > >> + > >> +versions =3D None > >> +try: > >> + if os.stat(VERSION_CACHE_FILE).st_mtime > time.time() - VERSION_C= ACHE_EXPIRE_SEC: > >> + print('Using cached version list') > >> + with open(VERSION_CACHE_FILE) as f: > >> + versions =3D json.load(f) > >> +except FileNotFoundError: > >> + # No cache exists, just download it > >> + pass > >> +except Exception as e: > >> + print(f'Unexpected error trying to read the version cache: {e}') > >> + > >> +if not versions: > >> + versions =3D download_pkg_version_list() > >> + try: > >> + os.makedirs(VERSION_CACHE_DIR, exist_ok=3DTrue) > >> + with open(VERSION_CACHE_FILE, 'w') as f: > >> + json.dump(versions, f, indent=3D1) > >> + f.write("\n") > >> + except Exception as e: > >> + print(f"Couldn't write package version cache: {e}") > >> +print() > >> + > >> +stable_versions =3D {} > >> +for package in versions: > >> + stable_versions[package['name']] =3D package['stable_version'] > >> + > >> +with open(args.report_file) as f: > >> + report =3D yaml.safe_load(f) > >> + > >> +latest_version =3D [] > >> +needs_update =3D [] > >> +untracked =3D [] > >> + > >> +max_pkgname_len =3D 0 > >> +max_version_len =3D 0 > >> + > >> +for pkg in report['packages'].values(): > >> + if not 'rulefile' in pkg or not 'version' in pkg: > >> + continue > >> + > >> + max_pkgname_len =3D max(max_pkgname_len, len(pkg['name'])) > >> + max_version_len =3D max(max_version_len, len(pkg['version'])) > >> + > >> + version =3D stable_versions.get(pkg['name']) > >> + if version is None: > >> + version =3D stable_versions.get(pkg['name'].removeprefix('hos= t-')) > >> + > >> + if version is None: > >> + untracked.append((pkg['name'], pkg['version'])) > >> + else: > >> + if version !=3D pkg['version']: > >> + needs_update.append((pkg['name'], pkg['version'], version= )) > >> + else: > >> + latest_version.append((pkg['name'], pkg['version'])) > >> + > >> +if args.show_up_to_date: > >> + print('Up-to-date rules') > >> + print('=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D') > >> + for pkg in latest_version: > >> + print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_ver= sion_len)}") > >> + print() > >> + > >> +print('Updateable rules') > >> +print('=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D') > >> +for pkg in needs_update: > >> + print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version= _len)} -> {pkg[2]}") > >> +print() > >> + > >> +if args.show_untracked: > >> + print('Untracked rules') > >> + print('=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D') > >> + for pkg in untracked: > >> + print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_ver= sion_len)}") > >> + print() > >> + > >> +print(f"Updateable: {len(needs_update)} Up-to-date: {len(latest_versi= on)} Untracked: {len(untracked)}") > >> -- > >> 2.47.3 > >> > >>