[ptxdist] [PATCH] scripts: add update_check script

[ptxdist] [PATCH] scripts: add update_check script

[Sven Püschel]

Add an update_check python script as an easy check which PTXdist
rules can be bumped in their versions. It uses the report file generated
from an existing PTXdist BSP as a basis to determine the package
versions. Then it compares these package versions with the package
versions against the newest release versions reported by
release-monitoring.org .

release-monitoring.org is a deployment of an open-source release
monitoring project named Anitya [1]. It allows to define a Project
which is some software you want to keep track of updates. A given
software can have multiple Projects. An example is GTK+, where you want
to have separate projects for GTK+ 3 and GTK+ 4. A Project can then
define a mapping to a package name in a Distribution.

I've created the PTXdist distribution [2] and created Project mappings
for a good chunk of the PTXdist rules. I've excluded certain rules with
user-defined versions (e.g. the linux kernel) and projects where it was
unclear if PTXdist should track the latest major version or use the
latest version of the current major version. Also I've ignored rules,
where no Project existed yet. Furthermore I've skipped the `python-`
rules, as they can probably be added in an automated fashion (given the
pypy name can be used for matching purposes).

This python script uses the unauthenticated API to get all current
versions for the Projects with a mapped PTXdist name. It then caches
these versions locally (updating only if the cache is older then 24h)
and compares the versions against the versions in the report file.
It displays all rules that can be updated and also counts the number of
up-to-date rules and rules missing a mapping. With a command line flag
the latter two can also be displayed.

Note that this isn't perfect and just provides an approximation to the
actual truth. There are a number of imperfections:
- PTXdist using non-stable versions (e.g. iftop 1.0pre4 instead of 0.17)
- Versions with different separators (dashes/underscores vs. dots)
- Rules not intended to be tracked (e.g. cross-dummy-strip)

Also there are some pitfalls in this design:
- Version numbers are based on the BSP
  - Updated rules might be already present in a newer PTXdist release
  - Rules might be overwritten with an older version locally
- BSP only rules (custom applications) will show up as untracked
- Versions may lag behind a bit (Anitya cron job and rate-limits)

[1] https://github.com/fedora-infra/anitya
[2] https://release-monitoring.org/distro/PTXdist/

Signed-off-by: Sven Püschel <s.pueschel@pengutronix.de>
---
 scripts/update_check.py | 132 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 132 insertions(+)
 create mode 100755 scripts/update_check.py

diff --git a/scripts/update_check.py b/scripts/update_check.py
new file mode 100755
index 000000000..94ffe83bc
--- /dev/null
+++ b/scripts/update_check.py
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2026 by Sven Püschel <s.pueschel@pengutronix.de>
+#
+# For further information about the PTXdist project and license conditions
+# see the README file.
+#
+
+import argparse
+import itertools
+import json
+import os
+import time
+from urllib import request
+
+import yaml
+
+API_ENDPOINT = 'https://release-monitoring.org/api/v2'
+ITEMS_PER_PAGE = 25
+
+XDG_CACHE_DIR = os.getenv('XDG_CACHE_HOME')
+if not XDG_CACHE_DIR:
+    XDG_CACHE_DIR = os.path.join(os.getenv('HOME'), '.cache')
+
+VERSION_CACHE_FILENAME = 'upstream_pkg_versions.json'
+VERSION_CACHE_EXPIRE_SEC = 24 * 60 * 60
+VERSION_CACHE_DIR = os.path.join(XDG_CACHE_DIR, 'ptxdist')
+VERSION_CACHE_FILE = os.path.join(VERSION_CACHE_DIR, VERSION_CACHE_FILENAME)
+
+def download_pkg_version_list():
+    versions = []
+
+    for page in itertools.count(1):
+        with request.urlopen(request.Request(
+                f'{API_ENDPOINT}/packages/?distribution=PTXdist&page={page}&items_per_page={ITEMS_PER_PAGE}',
+                headers={ 'User-Agent': 'PTXdist update_check script' }
+            )) as url:
+            data = json.load(url)
+            assert data['page'] == page
+            assert data['items_per_page'] == ITEMS_PER_PAGE
+            total = data['total_items']
+            versions.extend(data['items'])
+            print(f'\rDownloading current package versions ({len(versions)}/{total})', end='')
+            if len(versions) >= total:
+                break
+    print()
+    return versions
+
+parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter)
+parser.add_argument('--show-untracked', help='Show rules where no updated version is known', action='store_true')
+parser.add_argument('--show-up-to-date', help='Show rules which are already at the newest versions', action='store_true')
+parser.add_argument('report_file', help='report file generated by `ptxdist fast-bsp-report`')
+
+args = parser.parse_args()
+
+versions = None
+try:
+    if os.stat(VERSION_CACHE_FILE).st_mtime > time.time() - VERSION_CACHE_EXPIRE_SEC:
+        print('Using cached version list')
+        with open(VERSION_CACHE_FILE) as f:
+            versions = json.load(f)
+except FileNotFoundError:
+    # No cache exists, just download it
+    pass
+except Exception as e:
+    print(f'Unexpected error trying to read the version cache: {e}')
+
+if not versions:
+    versions = download_pkg_version_list()
+    try:
+        os.makedirs(VERSION_CACHE_DIR, exist_ok=True)
+        with open(VERSION_CACHE_FILE, 'w') as f:
+            json.dump(versions, f, indent=1)
+            f.write("\n")
+    except Exception as e:
+        print(f"Couldn't write package version cache: {e}")
+print()
+
+stable_versions = {}
+for package in versions:
+    stable_versions[package['name']] = package['stable_version']
+
+with open(args.report_file) as f:
+    report = yaml.safe_load(f)
+
+latest_version = []
+needs_update = []
+untracked = []
+
+max_pkgname_len = 0
+max_version_len = 0
+
+for pkg in report['packages'].values():
+    if not 'rulefile' in pkg or not 'version' in pkg:
+        continue
+
+    max_pkgname_len = max(max_pkgname_len, len(pkg['name']))
+    max_version_len = max(max_version_len, len(pkg['version']))
+
+    version = stable_versions.get(pkg['name'])
+    if version is None:
+        version = stable_versions.get(pkg['name'].removeprefix('host-'))
+
+    if version is None:
+        untracked.append((pkg['name'], pkg['version']))
+    else:
+        if version != pkg['version']:
+            needs_update.append((pkg['name'], pkg['version'], version))
+        else:
+            latest_version.append((pkg['name'], pkg['version']))
+
+if args.show_up_to_date:
+    print('Up-to-date rules')
+    print('================')
+    for pkg in latest_version:
+        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
+    print()
+
+print('Updateable rules')
+print('================')
+for pkg in needs_update:
+    print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)} -> {pkg[2]}")
+print()
+
+if args.show_untracked:
+    print('Untracked rules')
+    print('===============')
+    for pkg in untracked:
+        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
+    print()
+
+print(f"Updateable: {len(needs_update)} Up-to-date: {len(latest_version)} Untracked: {len(untracked)}")
-- 
2.47.3

Re: [ptxdist] [PATCH] scripts: add update_check script

[Bruno Thomsen]

Den fre. 23. okt. 2026 kl. 11.01 skrev Sven Püschel <s.pueschel@pengutronix.de>:
>
> Add an update_check python script as an easy check which PTXdist
> rules can be bumped in their versions. It uses the report file generated
> from an existing PTXdist BSP as a basis to determine the package
> versions. Then it compares these package versions with the package
> versions against the newest release versions reported by
> release-monitoring.org .
>
> release-monitoring.org is a deployment of an open-source release
> monitoring project named Anitya [1]. It allows to define a Project
> which is some software you want to keep track of updates. A given
> software can have multiple Projects. An example is GTK+, where you want
> to have separate projects for GTK+ 3 and GTK+ 4. A Project can then
> define a mapping to a package name in a Distribution.
>
> I've created the PTXdist distribution [2] and created Project mappings
> for a good chunk of the PTXdist rules. I've excluded certain rules with
> user-defined versions (e.g. the linux kernel) and projects where it was
> unclear if PTXdist should track the latest major version or use the
> latest version of the current major version. Also I've ignored rules,
> where no Project existed yet. Furthermore I've skipped the `python-`
> rules, as they can probably be added in an automated fashion (given the
> pypy name can be used for matching purposes).

Nice that you have added PTXdist to release-monitoring.org, you can
also see it in Repology[1] where it has lists of outdated packages[2],
potentially vulnerable packages[3], and packages with problems[4] like
dead URLs or permanent redirects. Information is tracking the master branch.

/Bruno

[1] https://repology.org/repository/ptxdist
[2] https://repology.org/projects/?inrepo=ptxdist&outdated=1&families_newest=2-
[3] https://repology.org/projects/?inrepo=ptxdist&vulnerable=1
[4] https://repology.org/repository/ptxdist/problems

>
> This python script uses the unauthenticated API to get all current
> versions for the Projects with a mapped PTXdist name. It then caches
> these versions locally (updating only if the cache is older then 24h)
> and compares the versions against the versions in the report file.
> It displays all rules that can be updated and also counts the number of
> up-to-date rules and rules missing a mapping. With a command line flag
> the latter two can also be displayed.
>
> Note that this isn't perfect and just provides an approximation to the
> actual truth. There are a number of imperfections:
> - PTXdist using non-stable versions (e.g. iftop 1.0pre4 instead of 0.17)
> - Versions with different separators (dashes/underscores vs. dots)
> - Rules not intended to be tracked (e.g. cross-dummy-strip)
>
> Also there are some pitfalls in this design:
> - Version numbers are based on the BSP
>   - Updated rules might be already present in a newer PTXdist release
>   - Rules might be overwritten with an older version locally
> - BSP only rules (custom applications) will show up as untracked
> - Versions may lag behind a bit (Anitya cron job and rate-limits)
>
> [1] https://github.com/fedora-infra/anitya
> [2] https://release-monitoring.org/distro/PTXdist/
>
> Signed-off-by: Sven Püschel <s.pueschel@pengutronix.de>
> ---
>  scripts/update_check.py | 132 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 132 insertions(+)
>  create mode 100755 scripts/update_check.py
>
> diff --git a/scripts/update_check.py b/scripts/update_check.py
> new file mode 100755
> index 000000000..94ffe83bc
> --- /dev/null
> +++ b/scripts/update_check.py
> @@ -0,0 +1,132 @@
> +#!/usr/bin/env python3
> +#
> +# Copyright (C) 2026 by Sven Püschel <s.pueschel@pengutronix.de>
> +#
> +# For further information about the PTXdist project and license conditions
> +# see the README file.
> +#
> +
> +import argparse
> +import itertools
> +import json
> +import os
> +import time
> +from urllib import request
> +
> +import yaml
> +
> +API_ENDPOINT = 'https://release-monitoring.org/api/v2'
> +ITEMS_PER_PAGE = 25
> +
> +XDG_CACHE_DIR = os.getenv('XDG_CACHE_HOME')
> +if not XDG_CACHE_DIR:
> +    XDG_CACHE_DIR = os.path.join(os.getenv('HOME'), '.cache')
> +
> +VERSION_CACHE_FILENAME = 'upstream_pkg_versions.json'
> +VERSION_CACHE_EXPIRE_SEC = 24 * 60 * 60
> +VERSION_CACHE_DIR = os.path.join(XDG_CACHE_DIR, 'ptxdist')
> +VERSION_CACHE_FILE = os.path.join(VERSION_CACHE_DIR, VERSION_CACHE_FILENAME)
> +
> +def download_pkg_version_list():
> +    versions = []
> +
> +    for page in itertools.count(1):
> +        with request.urlopen(request.Request(
> +                f'{API_ENDPOINT}/packages/?distribution=PTXdist&page={page}&items_per_page={ITEMS_PER_PAGE}',
> +                headers={ 'User-Agent': 'PTXdist update_check script' }
> +            )) as url:
> +            data = json.load(url)
> +            assert data['page'] == page
> +            assert data['items_per_page'] == ITEMS_PER_PAGE
> +            total = data['total_items']
> +            versions.extend(data['items'])
> +            print(f'\rDownloading current package versions ({len(versions)}/{total})', end='')
> +            if len(versions) >= total:
> +                break
> +    print()
> +    return versions
> +
> +parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter)
> +parser.add_argument('--show-untracked', help='Show rules where no updated version is known', action='store_true')
> +parser.add_argument('--show-up-to-date', help='Show rules which are already at the newest versions', action='store_true')
> +parser.add_argument('report_file', help='report file generated by `ptxdist fast-bsp-report`')
> +
> +args = parser.parse_args()
> +
> +versions = None
> +try:
> +    if os.stat(VERSION_CACHE_FILE).st_mtime > time.time() - VERSION_CACHE_EXPIRE_SEC:
> +        print('Using cached version list')
> +        with open(VERSION_CACHE_FILE) as f:
> +            versions = json.load(f)
> +except FileNotFoundError:
> +    # No cache exists, just download it
> +    pass
> +except Exception as e:
> +    print(f'Unexpected error trying to read the version cache: {e}')
> +
> +if not versions:
> +    versions = download_pkg_version_list()
> +    try:
> +        os.makedirs(VERSION_CACHE_DIR, exist_ok=True)
> +        with open(VERSION_CACHE_FILE, 'w') as f:
> +            json.dump(versions, f, indent=1)
> +            f.write("\n")
> +    except Exception as e:
> +        print(f"Couldn't write package version cache: {e}")
> +print()
> +
> +stable_versions = {}
> +for package in versions:
> +    stable_versions[package['name']] = package['stable_version']
> +
> +with open(args.report_file) as f:
> +    report = yaml.safe_load(f)
> +
> +latest_version = []
> +needs_update = []
> +untracked = []
> +
> +max_pkgname_len = 0
> +max_version_len = 0
> +
> +for pkg in report['packages'].values():
> +    if not 'rulefile' in pkg or not 'version' in pkg:
> +        continue
> +
> +    max_pkgname_len = max(max_pkgname_len, len(pkg['name']))
> +    max_version_len = max(max_version_len, len(pkg['version']))
> +
> +    version = stable_versions.get(pkg['name'])
> +    if version is None:
> +        version = stable_versions.get(pkg['name'].removeprefix('host-'))
> +
> +    if version is None:
> +        untracked.append((pkg['name'], pkg['version']))
> +    else:
> +        if version != pkg['version']:
> +            needs_update.append((pkg['name'], pkg['version'], version))
> +        else:
> +            latest_version.append((pkg['name'], pkg['version']))
> +
> +if args.show_up_to_date:
> +    print('Up-to-date rules')
> +    print('================')
> +    for pkg in latest_version:
> +        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
> +    print()
> +
> +print('Updateable rules')
> +print('================')
> +for pkg in needs_update:
> +    print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)} -> {pkg[2]}")
> +print()
> +
> +if args.show_untracked:
> +    print('Untracked rules')
> +    print('===============')
> +    for pkg in untracked:
> +        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
> +    print()
> +
> +print(f"Updateable: {len(needs_update)} Up-to-date: {len(latest_version)} Untracked: {len(untracked)}")
> --
> 2.47.3
>
>

Re: [ptxdist] [PATCH] scripts: add update_check script

[Sven Püschel]

Hi Bruno,

On 10/23/26 11:28, Bruno Thomsen wrote:
> Den fre. 23. okt. 2026 kl. 11.01 skrev Sven Püschel <s.pueschel@pengutronix.de>:
>> Add an update_check python script as an easy check which PTXdist
>> rules can be bumped in their versions. It uses the report file generated
>> from an existing PTXdist BSP as a basis to determine the package
>> versions. Then it compares these package versions with the package
>> versions against the newest release versions reported by
>> release-monitoring.org .
>>
>> release-monitoring.org is a deployment of an open-source release
>> monitoring project named Anitya [1]. It allows to define a Project
>> which is some software you want to keep track of updates. A given
>> software can have multiple Projects. An example is GTK+, where you want
>> to have separate projects for GTK+ 3 and GTK+ 4. A Project can then
>> define a mapping to a package name in a Distribution.
>>
>> I've created the PTXdist distribution [2] and created Project mappings
>> for a good chunk of the PTXdist rules. I've excluded certain rules with
>> user-defined versions (e.g. the linux kernel) and projects where it was
>> unclear if PTXdist should track the latest major version or use the
>> latest version of the current major version. Also I've ignored rules,
>> where no Project existed yet. Furthermore I've skipped the `python-`
>> rules, as they can probably be added in an automated fashion (given the
>> pypy name can be used for matching purposes).
> Nice that you have added PTXdist to release-monitoring.org, you can
> also see it in Repology[1] where it has lists of outdated packages[2],
> potentially vulnerable packages[3], and packages with problems[4] like
> dead URLs or permanent redirects. Information is tracking the master branch.

Yeah, Michael also mentioned your PTXDist patches towards repology and 
that he'd prefer some solution based on a report file.

Personally I didn't quite understand how Repology actually works. It 
looks like fuzzy matching of package names and getting the newest 
versions based on the comparison with other distro packages. 
Release-Monitoring looked dead-simple: Each Project has a dedicated 
definition where to look for new versions and a explicit mapping to a 
distro package name. Therefore I'd just went with it (with the side 
effect of also making PTXdist a bit present there). I'm not too 
invested, which Backend the script uses to actually get the necessary 
data (or it could also use multiple backends, etc.).

Sincerely
     Sven

> /Bruno
>
> [1] https://repology.org/repository/ptxdist
> [2] https://repology.org/projects/?inrepo=ptxdist&outdated=1&families_newest=2-
> [3] https://repology.org/projects/?inrepo=ptxdist&vulnerable=1
> [4] https://repology.org/repository/ptxdist/problems
>
>> This python script uses the unauthenticated API to get all current
>> versions for the Projects with a mapped PTXdist name. It then caches
>> these versions locally (updating only if the cache is older then 24h)
>> and compares the versions against the versions in the report file.
>> It displays all rules that can be updated and also counts the number of
>> up-to-date rules and rules missing a mapping. With a command line flag
>> the latter two can also be displayed.
>>
>> Note that this isn't perfect and just provides an approximation to the
>> actual truth. There are a number of imperfections:
>> - PTXdist using non-stable versions (e.g. iftop 1.0pre4 instead of 0.17)
>> - Versions with different separators (dashes/underscores vs. dots)
>> - Rules not intended to be tracked (e.g. cross-dummy-strip)
>>
>> Also there are some pitfalls in this design:
>> - Version numbers are based on the BSP
>>    - Updated rules might be already present in a newer PTXdist release
>>    - Rules might be overwritten with an older version locally
>> - BSP only rules (custom applications) will show up as untracked
>> - Versions may lag behind a bit (Anitya cron job and rate-limits)
>>
>> [1] https://github.com/fedora-infra/anitya
>> [2] https://release-monitoring.org/distro/PTXdist/
>>
>> Signed-off-by: Sven Püschel <s.pueschel@pengutronix.de>
>> ---
>>   scripts/update_check.py | 132 ++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 132 insertions(+)
>>   create mode 100755 scripts/update_check.py
>>
>> diff --git a/scripts/update_check.py b/scripts/update_check.py
>> new file mode 100755
>> index 000000000..94ffe83bc
>> --- /dev/null
>> +++ b/scripts/update_check.py
>> @@ -0,0 +1,132 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Copyright (C) 2026 by Sven Püschel <s.pueschel@pengutronix.de>
>> +#
>> +# For further information about the PTXdist project and license conditions
>> +# see the README file.
>> +#
>> +
>> +import argparse
>> +import itertools
>> +import json
>> +import os
>> +import time
>> +from urllib import request
>> +
>> +import yaml
>> +
>> +API_ENDPOINT = 'https://release-monitoring.org/api/v2'
>> +ITEMS_PER_PAGE = 25
>> +
>> +XDG_CACHE_DIR = os.getenv('XDG_CACHE_HOME')
>> +if not XDG_CACHE_DIR:
>> +    XDG_CACHE_DIR = os.path.join(os.getenv('HOME'), '.cache')
>> +
>> +VERSION_CACHE_FILENAME = 'upstream_pkg_versions.json'
>> +VERSION_CACHE_EXPIRE_SEC = 24 * 60 * 60
>> +VERSION_CACHE_DIR = os.path.join(XDG_CACHE_DIR, 'ptxdist')
>> +VERSION_CACHE_FILE = os.path.join(VERSION_CACHE_DIR, VERSION_CACHE_FILENAME)
>> +
>> +def download_pkg_version_list():
>> +    versions = []
>> +
>> +    for page in itertools.count(1):
>> +        with request.urlopen(request.Request(
>> +                f'{API_ENDPOINT}/packages/?distribution=PTXdist&page={page}&items_per_page={ITEMS_PER_PAGE}',
>> +                headers={ 'User-Agent': 'PTXdist update_check script' }
>> +            )) as url:
>> +            data = json.load(url)
>> +            assert data['page'] == page
>> +            assert data['items_per_page'] == ITEMS_PER_PAGE
>> +            total = data['total_items']
>> +            versions.extend(data['items'])
>> +            print(f'\rDownloading current package versions ({len(versions)}/{total})', end='')
>> +            if len(versions) >= total:
>> +                break
>> +    print()
>> +    return versions
>> +
>> +parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter)
>> +parser.add_argument('--show-untracked', help='Show rules where no updated version is known', action='store_true')
>> +parser.add_argument('--show-up-to-date', help='Show rules which are already at the newest versions', action='store_true')
>> +parser.add_argument('report_file', help='report file generated by `ptxdist fast-bsp-report`')
>> +
>> +args = parser.parse_args()
>> +
>> +versions = None
>> +try:
>> +    if os.stat(VERSION_CACHE_FILE).st_mtime > time.time() - VERSION_CACHE_EXPIRE_SEC:
>> +        print('Using cached version list')
>> +        with open(VERSION_CACHE_FILE) as f:
>> +            versions = json.load(f)
>> +except FileNotFoundError:
>> +    # No cache exists, just download it
>> +    pass
>> +except Exception as e:
>> +    print(f'Unexpected error trying to read the version cache: {e}')
>> +
>> +if not versions:
>> +    versions = download_pkg_version_list()
>> +    try:
>> +        os.makedirs(VERSION_CACHE_DIR, exist_ok=True)
>> +        with open(VERSION_CACHE_FILE, 'w') as f:
>> +            json.dump(versions, f, indent=1)
>> +            f.write("\n")
>> +    except Exception as e:
>> +        print(f"Couldn't write package version cache: {e}")
>> +print()
>> +
>> +stable_versions = {}
>> +for package in versions:
>> +    stable_versions[package['name']] = package['stable_version']
>> +
>> +with open(args.report_file) as f:
>> +    report = yaml.safe_load(f)
>> +
>> +latest_version = []
>> +needs_update = []
>> +untracked = []
>> +
>> +max_pkgname_len = 0
>> +max_version_len = 0
>> +
>> +for pkg in report['packages'].values():
>> +    if not 'rulefile' in pkg or not 'version' in pkg:
>> +        continue
>> +
>> +    max_pkgname_len = max(max_pkgname_len, len(pkg['name']))
>> +    max_version_len = max(max_version_len, len(pkg['version']))
>> +
>> +    version = stable_versions.get(pkg['name'])
>> +    if version is None:
>> +        version = stable_versions.get(pkg['name'].removeprefix('host-'))
>> +
>> +    if version is None:
>> +        untracked.append((pkg['name'], pkg['version']))
>> +    else:
>> +        if version != pkg['version']:
>> +            needs_update.append((pkg['name'], pkg['version'], version))
>> +        else:
>> +            latest_version.append((pkg['name'], pkg['version']))
>> +
>> +if args.show_up_to_date:
>> +    print('Up-to-date rules')
>> +    print('================')
>> +    for pkg in latest_version:
>> +        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
>> +    print()
>> +
>> +print('Updateable rules')
>> +print('================')
>> +for pkg in needs_update:
>> +    print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)} -> {pkg[2]}")
>> +print()
>> +
>> +if args.show_untracked:
>> +    print('Untracked rules')
>> +    print('===============')
>> +    for pkg in untracked:
>> +        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
>> +    print()
>> +
>> +print(f"Updateable: {len(needs_update)} Up-to-date: {len(latest_version)} Untracked: {len(untracked)}")
>> --
>> 2.47.3
>>
>>

Re: [ptxdist] [PATCH] scripts: add update_check script

[Bruno Thomsen]

Den fre. 23. okt. 2026 kl. 12.37 skrev Sven Püschel <s.pueschel@pengutronix.de>:
>
> Hi Bruno,
>
> On 10/23/26 11:28, Bruno Thomsen wrote:
> > Den fre. 23. okt. 2026 kl. 11.01 skrev Sven Püschel <s.pueschel@pengutronix.de>:
> >> Add an update_check python script as an easy check which PTXdist
> >> rules can be bumped in their versions. It uses the report file generated
> >> from an existing PTXdist BSP as a basis to determine the package
> >> versions. Then it compares these package versions with the package
> >> versions against the newest release versions reported by
> >> release-monitoring.org .
> >>
> >> release-monitoring.org is a deployment of an open-source release
> >> monitoring project named Anitya [1]. It allows to define a Project
> >> which is some software you want to keep track of updates. A given
> >> software can have multiple Projects. An example is GTK+, where you want
> >> to have separate projects for GTK+ 3 and GTK+ 4. A Project can then
> >> define a mapping to a package name in a Distribution.
> >>
> >> I've created the PTXdist distribution [2] and created Project mappings
> >> for a good chunk of the PTXdist rules. I've excluded certain rules with
> >> user-defined versions (e.g. the linux kernel) and projects where it was
> >> unclear if PTXdist should track the latest major version or use the
> >> latest version of the current major version. Also I've ignored rules,
> >> where no Project existed yet. Furthermore I've skipped the `python-`
> >> rules, as they can probably be added in an automated fashion (given the
> >> pypy name can be used for matching purposes).
> > Nice that you have added PTXdist to release-monitoring.org, you can
> > also see it in Repology[1] where it has lists of outdated packages[2],
> > potentially vulnerable packages[3], and packages with problems[4] like
> > dead URLs or permanent redirects. Information is tracking the master branch.
>
> Yeah, Michael also mentioned your PTXDist patches towards repology and
> that he'd prefer some solution based on a report file.
>

Latest versions of the patches were not posted to the mailing list and we
continued development in a separate repo[1] which also contains regular
updates to the JSON output that repology consumes.

> Personally I didn't quite understand how Repology actually works. It
> looks like fuzzy matching of package names and getting the newest
> versions based on the comparison with other distro packages.
> Release-Monitoring looked dead-simple: Each Project has a dedicated
> definition where to look for new versions and a explicit mapping to a
> distro package name. Therefore I'd just went with it (with the side
> effect of also making PTXdist a bit present there). I'm not too
> invested, which Backend the script uses to actually get the necessary
> data (or it could also use multiple backends, etc.).

Yes, packages are fuzzy matched by download URL + name, and new versions
are detected when 2 or more distributions have a newer version.

I like Release-Monitoring as well, so I get a RHBZ ping to update a RPM spec :)

They are good for two slightly different things, but I hope you get
the patch series
complete.

/Bruno

[1] https://github.com/baxeno/ptxdist-repology

>
> Sincerely
>      Sven
>
> > /Bruno
> >
> > [1] https://repology.org/repository/ptxdist
> > [2] https://repology.org/projects/?inrepo=ptxdist&outdated=1&families_newest=2-
> > [3] https://repology.org/projects/?inrepo=ptxdist&vulnerable=1
> > [4] https://repology.org/repository/ptxdist/problems
> >
> >> This python script uses the unauthenticated API to get all current
> >> versions for the Projects with a mapped PTXdist name. It then caches
> >> these versions locally (updating only if the cache is older then 24h)
> >> and compares the versions against the versions in the report file.
> >> It displays all rules that can be updated and also counts the number of
> >> up-to-date rules and rules missing a mapping. With a command line flag
> >> the latter two can also be displayed.
> >>
> >> Note that this isn't perfect and just provides an approximation to the
> >> actual truth. There are a number of imperfections:
> >> - PTXdist using non-stable versions (e.g. iftop 1.0pre4 instead of 0.17)
> >> - Versions with different separators (dashes/underscores vs. dots)
> >> - Rules not intended to be tracked (e.g. cross-dummy-strip)
> >>
> >> Also there are some pitfalls in this design:
> >> - Version numbers are based on the BSP
> >>    - Updated rules might be already present in a newer PTXdist release
> >>    - Rules might be overwritten with an older version locally
> >> - BSP only rules (custom applications) will show up as untracked
> >> - Versions may lag behind a bit (Anitya cron job and rate-limits)
> >>
> >> [1] https://github.com/fedora-infra/anitya
> >> [2] https://release-monitoring.org/distro/PTXdist/
> >>
> >> Signed-off-by: Sven Püschel <s.pueschel@pengutronix.de>
> >> ---
> >>   scripts/update_check.py | 132 ++++++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 132 insertions(+)
> >>   create mode 100755 scripts/update_check.py
> >>
> >> diff --git a/scripts/update_check.py b/scripts/update_check.py
> >> new file mode 100755
> >> index 000000000..94ffe83bc
> >> --- /dev/null
> >> +++ b/scripts/update_check.py
> >> @@ -0,0 +1,132 @@
> >> +#!/usr/bin/env python3
> >> +#
> >> +# Copyright (C) 2026 by Sven Püschel <s.pueschel@pengutronix.de>
> >> +#
> >> +# For further information about the PTXdist project and license conditions
> >> +# see the README file.
> >> +#
> >> +
> >> +import argparse
> >> +import itertools
> >> +import json
> >> +import os
> >> +import time
> >> +from urllib import request
> >> +
> >> +import yaml
> >> +
> >> +API_ENDPOINT = 'https://release-monitoring.org/api/v2'
> >> +ITEMS_PER_PAGE = 25
> >> +
> >> +XDG_CACHE_DIR = os.getenv('XDG_CACHE_HOME')
> >> +if not XDG_CACHE_DIR:
> >> +    XDG_CACHE_DIR = os.path.join(os.getenv('HOME'), '.cache')
> >> +
> >> +VERSION_CACHE_FILENAME = 'upstream_pkg_versions.json'
> >> +VERSION_CACHE_EXPIRE_SEC = 24 * 60 * 60
> >> +VERSION_CACHE_DIR = os.path.join(XDG_CACHE_DIR, 'ptxdist')
> >> +VERSION_CACHE_FILE = os.path.join(VERSION_CACHE_DIR, VERSION_CACHE_FILENAME)
> >> +
> >> +def download_pkg_version_list():
> >> +    versions = []
> >> +
> >> +    for page in itertools.count(1):
> >> +        with request.urlopen(request.Request(
> >> +                f'{API_ENDPOINT}/packages/?distribution=PTXdist&page={page}&items_per_page={ITEMS_PER_PAGE}',
> >> +                headers={ 'User-Agent': 'PTXdist update_check script' }
> >> +            )) as url:
> >> +            data = json.load(url)
> >> +            assert data['page'] == page
> >> +            assert data['items_per_page'] == ITEMS_PER_PAGE
> >> +            total = data['total_items']
> >> +            versions.extend(data['items'])
> >> +            print(f'\rDownloading current package versions ({len(versions)}/{total})', end='')
> >> +            if len(versions) >= total:
> >> +                break
> >> +    print()
> >> +    return versions
> >> +
> >> +parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter)
> >> +parser.add_argument('--show-untracked', help='Show rules where no updated version is known', action='store_true')
> >> +parser.add_argument('--show-up-to-date', help='Show rules which are already at the newest versions', action='store_true')
> >> +parser.add_argument('report_file', help='report file generated by `ptxdist fast-bsp-report`')
> >> +
> >> +args = parser.parse_args()
> >> +
> >> +versions = None
> >> +try:
> >> +    if os.stat(VERSION_CACHE_FILE).st_mtime > time.time() - VERSION_CACHE_EXPIRE_SEC:
> >> +        print('Using cached version list')
> >> +        with open(VERSION_CACHE_FILE) as f:
> >> +            versions = json.load(f)
> >> +except FileNotFoundError:
> >> +    # No cache exists, just download it
> >> +    pass
> >> +except Exception as e:
> >> +    print(f'Unexpected error trying to read the version cache: {e}')
> >> +
> >> +if not versions:
> >> +    versions = download_pkg_version_list()
> >> +    try:
> >> +        os.makedirs(VERSION_CACHE_DIR, exist_ok=True)
> >> +        with open(VERSION_CACHE_FILE, 'w') as f:
> >> +            json.dump(versions, f, indent=1)
> >> +            f.write("\n")
> >> +    except Exception as e:
> >> +        print(f"Couldn't write package version cache: {e}")
> >> +print()
> >> +
> >> +stable_versions = {}
> >> +for package in versions:
> >> +    stable_versions[package['name']] = package['stable_version']
> >> +
> >> +with open(args.report_file) as f:
> >> +    report = yaml.safe_load(f)
> >> +
> >> +latest_version = []
> >> +needs_update = []
> >> +untracked = []
> >> +
> >> +max_pkgname_len = 0
> >> +max_version_len = 0
> >> +
> >> +for pkg in report['packages'].values():
> >> +    if not 'rulefile' in pkg or not 'version' in pkg:
> >> +        continue
> >> +
> >> +    max_pkgname_len = max(max_pkgname_len, len(pkg['name']))
> >> +    max_version_len = max(max_version_len, len(pkg['version']))
> >> +
> >> +    version = stable_versions.get(pkg['name'])
> >> +    if version is None:
> >> +        version = stable_versions.get(pkg['name'].removeprefix('host-'))
> >> +
> >> +    if version is None:
> >> +        untracked.append((pkg['name'], pkg['version']))
> >> +    else:
> >> +        if version != pkg['version']:
> >> +            needs_update.append((pkg['name'], pkg['version'], version))
> >> +        else:
> >> +            latest_version.append((pkg['name'], pkg['version']))
> >> +
> >> +if args.show_up_to_date:
> >> +    print('Up-to-date rules')
> >> +    print('================')
> >> +    for pkg in latest_version:
> >> +        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
> >> +    print()
> >> +
> >> +print('Updateable rules')
> >> +print('================')
> >> +for pkg in needs_update:
> >> +    print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)} -> {pkg[2]}")
> >> +print()
> >> +
> >> +if args.show_untracked:
> >> +    print('Untracked rules')
> >> +    print('===============')
> >> +    for pkg in untracked:
> >> +        print(f"{pkg[0].ljust(max_pkgname_len)} {pkg[1].rjust(max_version_len)}")
> >> +    print()
> >> +
> >> +print(f"Updateable: {len(needs_update)} Up-to-date: {len(latest_version)} Untracked: {len(untracked)}")
> >> --
> >> 2.47.3
> >>
> >>