[ptxdist] [RFC PATCH] ptxdist: add repology sub command

[ptxdist] [RFC PATCH] ptxdist: add repology sub command

[Bruno Thomsen]

Improve ptxdist project visibility by adding it to repology.

Generate JSON output about all packages in ptxdist for repology.

ptxdist repology | tail -n +7 | jq

This is just a POC for integrating ptxdist with repology. 

Add PTXdist support #1487:
https://github.com/repology/repology-updater/issues/1487

I have run this patch against ptxdist-2024.12.0 and got this output:
https://github.com/baxeno/ptxdist-repology/blob/main/repology.json

Signed-off-by: Bruno Thomsen <bruno.thomsen@gmail.com>
---
Any feedback is welcome as this is a very rough implementation.

 bin/ptxdist                             |  6 +++
 rules/post/ptxd_make_repology.make      | 20 +++++++++
 scripts/bash_completion                 |  2 +-
 scripts/lib/ptxd_make_world_repology.sh | 54 +++++++++++++++++++++++++
 4 files changed, 81 insertions(+), 1 deletion(-)
 create mode 100644 rules/post/ptxd_make_repology.make
 create mode 100644 scripts/lib/ptxd_make_world_repology.sh

diff --git a/bin/ptxdist b/bin/ptxdist
index 1f561fdf6..24a8e2ed9 100755
--- a/bin/ptxdist
+++ b/bin/ptxdist
@@ -799,6 +799,7 @@ Misc:
 				variable and print the contents of this variable
   licensecheck			check md5sums of license files for all packages
   lint				run some basic checks for the bsp and PTXdist
+  repology			generate repology JSON for PTXdist
   list-packages			print a list of all selected packages
   local-src <pkg> [<directory>]	overwrite a package source with a locally provided
 				directory containing the sourcecode.
@@ -1916,6 +1917,11 @@ EOF
 				ptxd_bailout "lint checks failed"
 			fi
 			;;
+		repology)
+			check_premake_compiler &&
+			ptxd_make_log ptxdist-repology
+			exit
+			;;
 		list-packages)
 			check_config &&
 			check_deps &&
diff --git a/rules/post/ptxd_make_repology.make b/rules/post/ptxd_make_repology.make
new file mode 100644
index 000000000..b243ed03e
--- /dev/null
+++ b/rules/post/ptxd_make_repology.make
@@ -0,0 +1,20 @@
+# -*-makefile-*-
+#
+# Copyright (C) 2025 by Bruno Thomsen <bruno.thomsen@gmail.com>
+#
+# For further information about the PTXdist project and license conditions
+# see the README file.
+#
+
+PHONY += ptxdist-repology
+
+ptx/repology = \
+	$(call ptx/env) \
+	ptx_dgen_rulesfiles_make="$(PTX_DGEN_RULESFILES_MAKE)" \
+	ptxd_make_world_repology
+
+ptxdist-repology:
+	@$(call targetinfo)
+	@$(call ptx/repology)
+
+# vim: syntax=make
diff --git a/scripts/bash_completion b/scripts/bash_completion
index 4bff2d4b6..2bb21a848 100644
--- a/scripts/bash_completion
+++ b/scripts/bash_completion
@@ -123,7 +123,7 @@ _ptxdist_completion()
 	clean)
 		COMPREPLY=( $( compgen -W root -- $cur ) )
 		;&
-	get|extract|prepare|compile|install|targetinstall|tags|urlcheck|licensecheck|package-info|cargosync)
+	get|extract|prepare|compile|install|targetinstall|tags|urlcheck|licensecheck|repology|package-info|cargosync)
 		COMPREPLY+=( $( compgen -W "$(__ptxdist_completion_packages)" -- $cur ) )
 		;;
 	drop)
diff --git a/scripts/lib/ptxd_make_world_repology.sh b/scripts/lib/ptxd_make_world_repology.sh
new file mode 100644
index 000000000..2b3a0a098
--- /dev/null
+++ b/scripts/lib/ptxd_make_world_repology.sh
@@ -0,0 +1,54 @@
+#!/bin/bash
+#
+# Copyright (C) 2025 by Bruno Thomsen <bruno.thomsen@gmail.com>
+#
+# For further information about the PTXdist project and license conditions
+# see the README file.
+#
+
+ptxd_make_world_repology() {
+	local filefd file
+	local first
+	ptxd_make_world_init
+
+	echo "Locating all target packages ..."
+
+	first=1
+	exec {filefd}< <(ptxd_make_world_lint_makefiles)
+	while read file <&${filefd}; do
+	filename="${file##*/}"
+
+	case "${filename}" in
+		host-system-*|host-*|cross-*|image-*)
+		continue
+		;;
+		*)
+		;;
+	esac
+	case "${file}" in
+		*/rules/post/*|*/rules/pre/*)
+			continue
+			;;
+		*)
+			;;
+	esac
+	grep -q '^[^ 	]*_VERSION[ 	:]*=' "${file}" || continue
+
+	pkg_name=$(grep '^PACKAGES-$(PTXCONF_' "${file}" | cut -d '=' -f 2 | xargs)
+	pkg_version=$(grep '^[^ 	]*_VERSION[ 	:]*=' "${file}" | cut -d '=' -f 2 | xargs)
+	pkg_license=$(grep '^[^ 	]*_LICENSE[ 	:]*=' "${file}" | cut -d '=' -f 2 | xargs)
+
+	[[ "$pkg_license" == *"call remove_quotes"* ]] && continue
+	[[ "$pkg_version" == *"call ptx/"* ]] && continue
+
+	if [ $first -eq 1 ]; then
+		first=0
+		echo "["
+	else
+		echo ","
+	fi
+	echo "{\"name\": \"${pkg_name}\", \"version\": \"${pkg_version}\", \"license\": \"${pkg_license}\"}"
+	done
+	echo "]"
+}
+export -f ptxd_make_world_repology

base-commit: f61905c23240642dec3e5390ee15e83cf1e016e9
-- 
2.49.0

Re: [ptxdist] [RFC PATCH] ptxdist: add repology sub command

[Ladislav Michl]

Hello Bruno,

On Fri, Apr 18, 2025 at 01:33:57PM +0200, Bruno Thomsen wrote:
> Improve ptxdist project visibility by adding it to repology.

This is just cool :) A little remark bellow...

> Generate JSON output about all packages in ptxdist for repology.
> 
> ptxdist repology | tail -n +7 | jq
> 
> This is just a POC for integrating ptxdist with repology. 
> 
> Add PTXdist support #1487:
> https://github.com/repology/repology-updater/issues/1487
> 
> I have run this patch against ptxdist-2024.12.0 and got this output:
> https://github.com/baxeno/ptxdist-repology/blob/main/repology.json
> 
> Signed-off-by: Bruno Thomsen <bruno.thomsen@gmail.com>
> ---
> Any feedback is welcome as this is a very rough implementation.
> 
>  bin/ptxdist                             |  6 +++
>  rules/post/ptxd_make_repology.make      | 20 +++++++++
>  scripts/bash_completion                 |  2 +-
>  scripts/lib/ptxd_make_world_repology.sh | 54 +++++++++++++++++++++++++
>  4 files changed, 81 insertions(+), 1 deletion(-)
>  create mode 100644 rules/post/ptxd_make_repology.make
>  create mode 100644 scripts/lib/ptxd_make_world_repology.sh
> 
> diff --git a/bin/ptxdist b/bin/ptxdist
> index 1f561fdf6..24a8e2ed9 100755
> --- a/bin/ptxdist
> +++ b/bin/ptxdist
> @@ -799,6 +799,7 @@ Misc:
>  				variable and print the contents of this variable
>    licensecheck			check md5sums of license files for all packages
>    lint				run some basic checks for the bsp and PTXdist
> +  repology			generate repology JSON for PTXdist
>    list-packages			print a list of all selected packages
>    local-src <pkg> [<directory>]	overwrite a package source with a locally provided
>  				directory containing the sourcecode.
> @@ -1916,6 +1917,11 @@ EOF
>  				ptxd_bailout "lint checks failed"
>  			fi
>  			;;
> +		repology)
> +			check_premake_compiler &&
> +			ptxd_make_log ptxdist-repology
> +			exit
> +			;;
>  		list-packages)
>  			check_config &&
>  			check_deps &&
> diff --git a/rules/post/ptxd_make_repology.make b/rules/post/ptxd_make_repology.make
> new file mode 100644
> index 000000000..b243ed03e
> --- /dev/null
> +++ b/rules/post/ptxd_make_repology.make
> @@ -0,0 +1,20 @@
> +# -*-makefile-*-
> +#
> +# Copyright (C) 2025 by Bruno Thomsen <bruno.thomsen@gmail.com>
> +#
> +# For further information about the PTXdist project and license conditions
> +# see the README file.
> +#
> +
> +PHONY += ptxdist-repology
> +
> +ptx/repology = \
> +	$(call ptx/env) \
> +	ptx_dgen_rulesfiles_make="$(PTX_DGEN_RULESFILES_MAKE)" \
> +	ptxd_make_world_repology

Here would be probably better to pass $(PTX_PACKAGES_INSTALL) and possibly sort it.

> +ptxdist-repology:
> +	@$(call targetinfo)
> +	@$(call ptx/repology)
> +
> +# vim: syntax=make
> diff --git a/scripts/bash_completion b/scripts/bash_completion
> index 4bff2d4b6..2bb21a848 100644
> --- a/scripts/bash_completion
> +++ b/scripts/bash_completion
> @@ -123,7 +123,7 @@ _ptxdist_completion()
>  	clean)
>  		COMPREPLY=( $( compgen -W root -- $cur ) )
>  		;&
> -	get|extract|prepare|compile|install|targetinstall|tags|urlcheck|licensecheck|package-info|cargosync)
> +	get|extract|prepare|compile|install|targetinstall|tags|urlcheck|licensecheck|repology|package-info|cargosync)
>  		COMPREPLY+=( $( compgen -W "$(__ptxdist_completion_packages)" -- $cur ) )
>  		;;
>  	drop)
> diff --git a/scripts/lib/ptxd_make_world_repology.sh b/scripts/lib/ptxd_make_world_repology.sh
> new file mode 100644
> index 000000000..2b3a0a098
> --- /dev/null
> +++ b/scripts/lib/ptxd_make_world_repology.sh
> @@ -0,0 +1,54 @@
> +#!/bin/bash
> +#
> +# Copyright (C) 2025 by Bruno Thomsen <bruno.thomsen@gmail.com>
> +#
> +# For further information about the PTXdist project and license conditions
> +# see the README file.
> +#
> +
> +ptxd_make_world_repology() {
> +	local filefd file
> +	local first
> +	ptxd_make_world_init
> +
> +	echo "Locating all target packages ..."
> +
> +	first=1
> +	exec {filefd}< <(ptxd_make_world_lint_makefiles)
> +	while read file <&${filefd}; do
> +	filename="${file##*/}"
> +
> +	case "${filename}" in
> +		host-system-*|host-*|cross-*|image-*)
> +		continue
> +		;;
> +		*)
> +		;;
> +	esac
> +	case "${file}" in
> +		*/rules/post/*|*/rules/pre/*)
> +			continue
> +			;;
> +		*)
> +			;;
> +	esac
> +	grep -q '^[^ 	]*_VERSION[ 	:]*=' "${file}" || continue
> +
> +	pkg_name=$(grep '^PACKAGES-$(PTXCONF_' "${file}" | cut -d '=' -f 2 | xargs)
> +	pkg_version=$(grep '^[^ 	]*_VERSION[ 	:]*=' "${file}" | cut -d '=' -f 2 | xargs)
> +	pkg_license=$(grep '^[^ 	]*_LICENSE[ 	:]*=' "${file}" | cut -d '=' -f 2 | xargs)

...and here use that list as this loop fails for valgring, which lists:
PACKAGES-$(PTXCONF_ARCH_X86)-$(PTXCONF_VALGRIND) += valgrind
PACKAGES-$(PTXCONF_ARCH_PPC)-$(PTXCONF_VALGRIND) += valgrind
PACKAGES-$(PTXCONF_ARCH_ARM)-$(PTXCONF_VALGRIND) += valgrind
PACKAGES-$(PTXCONF_ARCH_ARM64)-$(PTXCONF_VALGRIND) += valgrind

> +	[[ "$pkg_license" == *"call remove_quotes"* ]] && continue
> +	[[ "$pkg_version" == *"call ptx/"* ]] && continue
> +
> +	if [ $first -eq 1 ]; then
> +		first=0
> +		echo "["
> +	else
> +		echo ","
> +	fi
> +	echo "{\"name\": \"${pkg_name}\", \"version\": \"${pkg_version}\", \"license\": \"${pkg_license}\"}"
> +	done
> +	echo "]"
> +}
> +export -f ptxd_make_world_repology
> 
> base-commit: f61905c23240642dec3e5390ee15e83cf1e016e9
> -- 
> 2.49.0
> 

Re: [ptxdist] [RFC PATCH] ptxdist: add repology sub command

[Ladislav Michl]

On Sat, Apr 19, 2025 at 11:10:47PM +0200, Ladislav Michl wrote:
> On Fri, Apr 18, 2025 at 01:33:57PM +0200, Bruno Thomsen wrote:
> > +ptx/repology = \
> > +	$(call ptx/env) \
> > +	ptx_dgen_rulesfiles_make="$(PTX_DGEN_RULESFILES_MAKE)" \
> > +	ptxd_make_world_repology
> 
> Here would be probably better to pass $(PTX_PACKAGES_INSTALL) and possibly sort it.

Actually tried that with $(PTX_PACKAGES_ALL) and it does not seem to be viable
solution. Too much has to be set up just to generate JSON report. Now trying to
model it similar way ptxdist-lint target is made.

	l.

Re: [ptxdist] [RFC PATCH] ptxdist: add repology sub command

[Bruno Thomsen]

Den søn. 20. apr. 2025 kl. 10.41 skrev Ladislav Michl <oss-lists@triops.cz>:
>
> On Sat, Apr 19, 2025 at 11:10:47PM +0200, Ladislav Michl wrote:
> > On Fri, Apr 18, 2025 at 01:33:57PM +0200, Bruno Thomsen wrote:
> > > +ptx/repology = \
> > > +   $(call ptx/env) \
> > > +   ptx_dgen_rulesfiles_make="$(PTX_DGEN_RULESFILES_MAKE)" \
> > > +   ptxd_make_world_repology
> >
> > Here would be probably better to pass $(PTX_PACKAGES_INSTALL) and possibly sort it.
>
> Actually tried that with $(PTX_PACKAGES_ALL) and it does not seem to be viable
> solution. Too much has to be set up just to generate JSON report. Now trying to
> model it similar way ptxdist-lint target is made.

Hi

I also started with ptxdist-lint but might have removed a little too
much of the makefile
and relied too much on the bash script part :)
It would be nice if we could run more makefile to get dynamic versions
and licenses resolved
before handoff to bash script. Or maybe we should do it more like spdx
sbom in Python.

The sorting and indentation is currently done externally by jq, but
that could be moved inside
ptxd_make_world_repology script function.

Also I am a bit unsure if this cross rule variable access is actually
more of a bug then a feature.

$ rg XORG_LIB_X11_LICENSE
rules/libx11-locale.make
24:LIBX11_LOCALE_LICENSE    = $(XORG_LIB_X11_LICENSE)

rules/xorg-lib-X11.make
24:XORG_LIB_X11_LICENSE    := MIT


Ladislav you can find latest development version of this repology support here:
https://github.com/baxeno/ptxdist/tree/repology_support
(you can send PR directly if you wish)

Output from RFC PATCH have at least 19 issues with "$(":
https://github.com/baxeno/ptxdist-repology/blob/main/repology.json

/Bruno

>
>         l.

Re: [ptxdist] [RFC PATCH] ptxdist: add repology sub command

[Ladislav Michl]

On Sun, Apr 20, 2025 at 06:18:32PM +0200, Bruno Thomsen wrote:
> Hi
> 
> I also started with ptxdist-lint but might have removed a little too
> much of the makefile
> and relied too much on the bash script part :)
> It would be nice if we could run more makefile to get dynamic versions
> and licenses resolved
> before handoff to bash script. Or maybe we should do it more like spdx
> sbom in Python.
> 
> The sorting and indentation is currently done externally by jq, but
> that could be moved inside
> ptxd_make_world_repology script function.
> 
> Also I am a bit unsure if this cross rule variable access is actually
> more of a bug then a feature.
> 
> $ rg XORG_LIB_X11_LICENSE
> rules/libx11-locale.make
> 24:LIBX11_LOCALE_LICENSE    = $(XORG_LIB_X11_LICENSE)
> 
> rules/xorg-lib-X11.make
> 24:XORG_LIB_X11_LICENSE    := MIT

I guess we are overcomplicating it. What about simply (working concept):
$ cat scripts/lib/ptxd_make_world_repology.sh 

PTX_PACKAGES_TARGET := \
	$(sort $(filter-out host-system-%,$(filter-out host-%,$(filter-out image-%,$(PTX_PACKAGES_ALL)))))

PHONY += ptxdist-repology

ptx/repology = \
	@$(foreach pkg,$(PTX_PACKAGES_TARGET), \
		echo "{\"name\": \"${pkg}\", \"version\": \"$($(PTX_MAP_TO_PACKAGE_$(pkg))_VERSION)\", \"license\": \"$($(PTX_MAP_TO_PACKAGE_$(pkg))_LICENSE)\"}"$(ptx/nl))

ptxdist-repology:
	@$(call targetinfo)
	@$(call ptx/repology)

# vim: syntax=make
EOF

Doing all inside makefile quite simplifies it and performance is
probably not critical here.

> Ladislav you can find latest development version of this repology support here:
> https://github.com/baxeno/ptxdist/tree/repology_support
> (you can send PR directly if you wish)

I'll try to cook up something but not sooner than next week.

	l.

Re: [ptxdist] [RFC PATCH] ptxdist: add repology sub command

[Ladislav Michl]

On Sun, Apr 20, 2025 at 08:59:53PM +0200, Ladislav Michl wrote:
> Doing all inside makefile quite simplifies it and performance is
> probably not critical here.

echo in a subshell is the bottleneck. Here's complete version as a patch
on the top of original one (there are more packages to ignore, which is
left as an excercise for the reader):

diff --git a/rules/post/ptxd_make_repology.make b/rules/post/ptxd_make_repology.make
index b243ed03e..0083ded34 100644
--- a/rules/post/ptxd_make_repology.make
+++ b/rules/post/ptxd_make_repology.make
@@ -6,15 +6,40 @@
 # see the README file.
 #
 
+PTX_PACKAGES_IGNORE	:= \
+	base \
+	cryptodev-api
+
 PHONY += ptxdist-repology
 
+ptx_packages_reported:=$(sort $(filter-out host-system-%,$(filter-out host-%,$(filter-out image-%,$(PTX_PACKAGES_ALL)))))
+ptx_packages_reported:=$(filter-out $(PTX_PACKAGES_IGNORE),$(ptx_packages_reported))
+ptx_package_last:=$(lastword $(ptx_packages_reported))
+ptx_packages_reported:=$(filter-out $(ptx_package_last),$(ptx_packages_reported))
+
+ptx/repology-print = \
+	$(info $(space)   "name": "$(1)"$(comma)) \
+	$(info $(space)   "version": "$($(PTX_MAP_TO_PACKAGE_$(1))_VERSION)"$(comma)) \
+	$(info $(space)   "license": "$($(PTX_MAP_TO_PACKAGE_$(1))_LICENSE)")
+
+ptx/repology-item = \
+	$(info $(space) {) \
+	$(call ptx/repology-print,$(1)) \
+	$(info $(space) }$(comma))
+
+ptx/repology-lastitem = \
+	$(info $(space) {) \
+	$(call ptx/repology-print,$(1)) \
+	$(info $(space) })
+
 ptx/repology = \
-	$(call ptx/env) \
-	ptx_dgen_rulesfiles_make="$(PTX_DGEN_RULESFILES_MAKE)" \
-	ptxd_make_world_repology
+	$(info [) \
+	$(foreach pkg,$(ptx_packages_reported), \
+		$(call ptx/repology-item,$(pkg))$(ptx/nl)) \
+	$(call ptx/repology-lastitem,$(ptx_package_last)) \
+	$(info ])
 
 ptxdist-repology:
-	@$(call targetinfo)
 	@$(call ptx/repology)
 
 # vim: syntax=make

Re: [ptxdist] [RFC PATCH] ptxdist: add repology sub command

[Michael Olbrich]

On Fri, Apr 18, 2025 at 01:33:57PM +0200, Bruno Thomsen wrote:
> Improve ptxdist project visibility by adding it to repology.
> 
> Generate JSON output about all packages in ptxdist for repology.
> 
> ptxdist repology | tail -n +7 | jq
> 
> This is just a POC for integrating ptxdist with repology. 
> 
> Add PTXdist support #1487:
> https://github.com/repology/repology-updater/issues/1487
> 
> I have run this patch against ptxdist-2024.12.0 and got this output:
> https://github.com/baxeno/ptxdist-repology/blob/main/repology.json
> 
> Signed-off-by: Bruno Thomsen <bruno.thomsen@gmail.com>
> ---
> Any feedback is welcome as this is a very rough implementation.

Hmm, I think it makes more sense to add this to the report stuff in
scripts/report/. That should be a pretty simple python script.
You can still add a command and make target to call this but there is no
need for manual parsing.

Michael

> 
>  bin/ptxdist                             |  6 +++
>  rules/post/ptxd_make_repology.make      | 20 +++++++++
>  scripts/bash_completion                 |  2 +-
>  scripts/lib/ptxd_make_world_repology.sh | 54 +++++++++++++++++++++++++
>  4 files changed, 81 insertions(+), 1 deletion(-)
>  create mode 100644 rules/post/ptxd_make_repology.make
>  create mode 100644 scripts/lib/ptxd_make_world_repology.sh
> 
> diff --git a/bin/ptxdist b/bin/ptxdist
> index 1f561fdf6..24a8e2ed9 100755
> --- a/bin/ptxdist
> +++ b/bin/ptxdist
> @@ -799,6 +799,7 @@ Misc:
>  				variable and print the contents of this variable
>    licensecheck			check md5sums of license files for all packages
>    lint				run some basic checks for the bsp and PTXdist
> +  repology			generate repology JSON for PTXdist
>    list-packages			print a list of all selected packages
>    local-src <pkg> [<directory>]	overwrite a package source with a locally provided
>  				directory containing the sourcecode.
> @@ -1916,6 +1917,11 @@ EOF
>  				ptxd_bailout "lint checks failed"
>  			fi
>  			;;
> +		repology)
> +			check_premake_compiler &&
> +			ptxd_make_log ptxdist-repology
> +			exit
> +			;;
>  		list-packages)
>  			check_config &&
>  			check_deps &&
> diff --git a/rules/post/ptxd_make_repology.make b/rules/post/ptxd_make_repology.make
> new file mode 100644
> index 000000000..b243ed03e
> --- /dev/null
> +++ b/rules/post/ptxd_make_repology.make
> @@ -0,0 +1,20 @@
> +# -*-makefile-*-
> +#
> +# Copyright (C) 2025 by Bruno Thomsen <bruno.thomsen@gmail.com>
> +#
> +# For further information about the PTXdist project and license conditions
> +# see the README file.
> +#
> +
> +PHONY += ptxdist-repology
> +
> +ptx/repology = \
> +	$(call ptx/env) \
> +	ptx_dgen_rulesfiles_make="$(PTX_DGEN_RULESFILES_MAKE)" \
> +	ptxd_make_world_repology
> +
> +ptxdist-repology:
> +	@$(call targetinfo)
> +	@$(call ptx/repology)
> +
> +# vim: syntax=make
> diff --git a/scripts/bash_completion b/scripts/bash_completion
> index 4bff2d4b6..2bb21a848 100644
> --- a/scripts/bash_completion
> +++ b/scripts/bash_completion
> @@ -123,7 +123,7 @@ _ptxdist_completion()
>  	clean)
>  		COMPREPLY=( $( compgen -W root -- $cur ) )
>  		;&
> -	get|extract|prepare|compile|install|targetinstall|tags|urlcheck|licensecheck|package-info|cargosync)
> +	get|extract|prepare|compile|install|targetinstall|tags|urlcheck|licensecheck|repology|package-info|cargosync)
>  		COMPREPLY+=( $( compgen -W "$(__ptxdist_completion_packages)" -- $cur ) )
>  		;;
>  	drop)
> diff --git a/scripts/lib/ptxd_make_world_repology.sh b/scripts/lib/ptxd_make_world_repology.sh
> new file mode 100644
> index 000000000..2b3a0a098
> --- /dev/null
> +++ b/scripts/lib/ptxd_make_world_repology.sh
> @@ -0,0 +1,54 @@
> +#!/bin/bash
> +#
> +# Copyright (C) 2025 by Bruno Thomsen <bruno.thomsen@gmail.com>
> +#
> +# For further information about the PTXdist project and license conditions
> +# see the README file.
> +#
> +
> +ptxd_make_world_repology() {
> +	local filefd file
> +	local first
> +	ptxd_make_world_init
> +
> +	echo "Locating all target packages ..."
> +
> +	first=1
> +	exec {filefd}< <(ptxd_make_world_lint_makefiles)
> +	while read file <&${filefd}; do
> +	filename="${file##*/}"
> +
> +	case "${filename}" in
> +		host-system-*|host-*|cross-*|image-*)
> +		continue
> +		;;
> +		*)
> +		;;
> +	esac
> +	case "${file}" in
> +		*/rules/post/*|*/rules/pre/*)
> +			continue
> +			;;
> +		*)
> +			;;
> +	esac
> +	grep -q '^[^ 	]*_VERSION[ 	:]*=' "${file}" || continue
> +
> +	pkg_name=$(grep '^PACKAGES-$(PTXCONF_' "${file}" | cut -d '=' -f 2 | xargs)
> +	pkg_version=$(grep '^[^ 	]*_VERSION[ 	:]*=' "${file}" | cut -d '=' -f 2 | xargs)
> +	pkg_license=$(grep '^[^ 	]*_LICENSE[ 	:]*=' "${file}" | cut -d '=' -f 2 | xargs)
> +
> +	[[ "$pkg_license" == *"call remove_quotes"* ]] && continue
> +	[[ "$pkg_version" == *"call ptx/"* ]] && continue
> +
> +	if [ $first -eq 1 ]; then
> +		first=0
> +		echo "["
> +	else
> +		echo ","
> +	fi
> +	echo "{\"name\": \"${pkg_name}\", \"version\": \"${pkg_version}\", \"license\": \"${pkg_license}\"}"
> +	done
> +	echo "]"
> +}
> +export -f ptxd_make_world_repology
> 
> base-commit: f61905c23240642dec3e5390ee15e83cf1e016e9
> -- 
> 2.49.0
> 
> 
> 

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [ptxdist] [RFC PATCH] ptxdist: add repology sub command

[Bruno Thomsen]

Den lør. 26. apr. 2025 kl. 12.13 skrev Michael Olbrich
<m.olbrich@pengutronix.de>:
>
> On Fri, Apr 18, 2025 at 01:33:57PM +0200, Bruno Thomsen wrote:
> > Improve ptxdist project visibility by adding it to repology.
> >
> > Generate JSON output about all packages in ptxdist for repology.
> >
> > ptxdist repology | tail -n +7 | jq
> >
> > This is just a POC for integrating ptxdist with repology.
> >
> > Add PTXdist support #1487:
> > https://github.com/repology/repology-updater/issues/1487
> >
> > I have run this patch against ptxdist-2024.12.0 and got this output:
> > https://github.com/baxeno/ptxdist-repology/blob/main/repology.json
> >
> > Signed-off-by: Bruno Thomsen <bruno.thomsen@gmail.com>
> > ---
> > Any feedback is welcome as this is a very rough implementation.
>
> Hmm, I think it makes more sense to add this to the report stuff in
> scripts/report/. That should be a pretty simple python script.
> You can still add a command and make target to call this but there is no
> need for manual parsing.

Welcome back :)
There is a newer patch version that removes the bash script and does it all
in makefile. Maybe it should be converted to a python script. This "report"
is a bit different since it includes all ptxdist packages and not just
the packages
used in a project.

/Bruno

>
> Michael
>
> >
> >  bin/ptxdist                             |  6 +++
> >  rules/post/ptxd_make_repology.make      | 20 +++++++++
> >  scripts/bash_completion                 |  2 +-
> >  scripts/lib/ptxd_make_world_repology.sh | 54 +++++++++++++++++++++++++
> >  4 files changed, 81 insertions(+), 1 deletion(-)
> >  create mode 100644 rules/post/ptxd_make_repology.make
> >  create mode 100644 scripts/lib/ptxd_make_world_repology.sh
> >
> > diff --git a/bin/ptxdist b/bin/ptxdist
> > index 1f561fdf6..24a8e2ed9 100755
> > --- a/bin/ptxdist
> > +++ b/bin/ptxdist
> > @@ -799,6 +799,7 @@ Misc:
> >                               variable and print the contents of this variable
> >    licensecheck                       check md5sums of license files for all packages
> >    lint                               run some basic checks for the bsp and PTXdist
> > +  repology                   generate repology JSON for PTXdist
> >    list-packages                      print a list of all selected packages
> >    local-src <pkg> [<directory>]      overwrite a package source with a locally provided
> >                               directory containing the sourcecode.
> > @@ -1916,6 +1917,11 @@ EOF
> >                               ptxd_bailout "lint checks failed"
> >                       fi
> >                       ;;
> > +             repology)
> > +                     check_premake_compiler &&
> > +                     ptxd_make_log ptxdist-repology
> > +                     exit
> > +                     ;;
> >               list-packages)
> >                       check_config &&
> >                       check_deps &&
> > diff --git a/rules/post/ptxd_make_repology.make b/rules/post/ptxd_make_repology.make
> > new file mode 100644
> > index 000000000..b243ed03e
> > --- /dev/null
> > +++ b/rules/post/ptxd_make_repology.make
> > @@ -0,0 +1,20 @@
> > +# -*-makefile-*-
> > +#
> > +# Copyright (C) 2025 by Bruno Thomsen <bruno.thomsen@gmail.com>
> > +#
> > +# For further information about the PTXdist project and license conditions
> > +# see the README file.
> > +#
> > +
> > +PHONY += ptxdist-repology
> > +
> > +ptx/repology = \
> > +     $(call ptx/env) \
> > +     ptx_dgen_rulesfiles_make="$(PTX_DGEN_RULESFILES_MAKE)" \
> > +     ptxd_make_world_repology
> > +
> > +ptxdist-repology:
> > +     @$(call targetinfo)
> > +     @$(call ptx/repology)
> > +
> > +# vim: syntax=make
> > diff --git a/scripts/bash_completion b/scripts/bash_completion
> > index 4bff2d4b6..2bb21a848 100644
> > --- a/scripts/bash_completion
> > +++ b/scripts/bash_completion
> > @@ -123,7 +123,7 @@ _ptxdist_completion()
> >       clean)
> >               COMPREPLY=( $( compgen -W root -- $cur ) )
> >               ;&
> > -     get|extract|prepare|compile|install|targetinstall|tags|urlcheck|licensecheck|package-info|cargosync)
> > +     get|extract|prepare|compile|install|targetinstall|tags|urlcheck|licensecheck|repology|package-info|cargosync)
> >               COMPREPLY+=( $( compgen -W "$(__ptxdist_completion_packages)" -- $cur ) )
> >               ;;
> >       drop)
> > diff --git a/scripts/lib/ptxd_make_world_repology.sh b/scripts/lib/ptxd_make_world_repology.sh
> > new file mode 100644
> > index 000000000..2b3a0a098
> > --- /dev/null
> > +++ b/scripts/lib/ptxd_make_world_repology.sh
> > @@ -0,0 +1,54 @@
> > +#!/bin/bash
> > +#
> > +# Copyright (C) 2025 by Bruno Thomsen <bruno.thomsen@gmail.com>
> > +#
> > +# For further information about the PTXdist project and license conditions
> > +# see the README file.
> > +#
> > +
> > +ptxd_make_world_repology() {
> > +     local filefd file
> > +     local first
> > +     ptxd_make_world_init
> > +
> > +     echo "Locating all target packages ..."
> > +
> > +     first=1
> > +     exec {filefd}< <(ptxd_make_world_lint_makefiles)
> > +     while read file <&${filefd}; do
> > +     filename="${file##*/}"
> > +
> > +     case "${filename}" in
> > +             host-system-*|host-*|cross-*|image-*)
> > +             continue
> > +             ;;
> > +             *)
> > +             ;;
> > +     esac
> > +     case "${file}" in
> > +             */rules/post/*|*/rules/pre/*)
> > +                     continue
> > +                     ;;
> > +             *)
> > +                     ;;
> > +     esac
> > +     grep -q '^[^    ]*_VERSION[     :]*=' "${file}" || continue
> > +
> > +     pkg_name=$(grep '^PACKAGES-$(PTXCONF_' "${file}" | cut -d '=' -f 2 | xargs)
> > +     pkg_version=$(grep '^[^         ]*_VERSION[     :]*=' "${file}" | cut -d '=' -f 2 | xargs)
> > +     pkg_license=$(grep '^[^         ]*_LICENSE[     :]*=' "${file}" | cut -d '=' -f 2 | xargs)
> > +
> > +     [[ "$pkg_license" == *"call remove_quotes"* ]] && continue
> > +     [[ "$pkg_version" == *"call ptx/"* ]] && continue
> > +
> > +     if [ $first -eq 1 ]; then
> > +             first=0
> > +             echo "["
> > +     else
> > +             echo ","
> > +     fi
> > +     echo "{\"name\": \"${pkg_name}\", \"version\": \"${pkg_version}\", \"license\": \"${pkg_license}\"}"
> > +     done
> > +     echo "]"
> > +}
> > +export -f ptxd_make_world_repology
> >
> > base-commit: f61905c23240642dec3e5390ee15e83cf1e016e9
> > --
> > 2.49.0
> >
> >
> >
>
> --
> Pengutronix e.K.                           |                             |
> Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
> 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [ptxdist] [RFC PATCH] ptxdist: add repology sub command

[Michael Olbrich]

On Sat, Apr 26, 2025 at 06:23:35PM +0200, Bruno Thomsen wrote:
> Den lør. 26. apr. 2025 kl. 12.13 skrev Michael Olbrich
> <m.olbrich@pengutronix.de>:
> >
> > On Fri, Apr 18, 2025 at 01:33:57PM +0200, Bruno Thomsen wrote:
> > > Improve ptxdist project visibility by adding it to repology.
> > >
> > > Generate JSON output about all packages in ptxdist for repology.
> > >
> > > ptxdist repology | tail -n +7 | jq
> > >
> > > This is just a POC for integrating ptxdist with repology.
> > >
> > > Add PTXdist support #1487:
> > > https://github.com/repology/repology-updater/issues/1487
> > >
> > > I have run this patch against ptxdist-2024.12.0 and got this output:
> > > https://github.com/baxeno/ptxdist-repology/blob/main/repology.json
> > >
> > > Signed-off-by: Bruno Thomsen <bruno.thomsen@gmail.com>
> > > ---
> > > Any feedback is welcome as this is a very rough implementation.
> >
> > Hmm, I think it makes more sense to add this to the report stuff in
> > scripts/report/. That should be a pretty simple python script.
> > You can still add a command and make target to call this but there is no
> > need for manual parsing.
> 
> Welcome back :)

Thanks, I've been rather busy the last few weeks...

> There is a newer patch version that removes the bash script and does it all
> in makefile. Maybe it should be converted to a python script. This "report"
> is a bit different since it includes all ptxdist packages and not just
> the packages
> used in a project.

No, the report includes all enabled packages. And it includes all
dependencies and for images like the rootfs, all packages that are used to
build the image. With all that information available in one place, it
rather easy to limit what to use.

I've seen the new version and really prefer the Python scripts for that
kind of thing. Ist a lot more flexible. For example I'm pretty sure that
most of the ignore list can by automatically generated by looking at the
package (Does it have a URL?). Doing stuff like that in the Makefile is
painful.

Michael

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [ptxdist] [RFC PATCH] ptxdist: add repology sub command

[Ladislav Michl]

Hi Michael!

On Sun, Apr 27, 2025 at 04:58:26PM +0200, Michael Olbrich wrote:
> No, the report includes all enabled packages. And it includes all
> dependencies and for images like the rootfs, all packages that are used to
> build the image. With all that information available in one place, it
> rather easy to limit what to use.

you do want to show all available packages to repology, not only enabled
ones. In the ideal world, ptxdist should be able to produce that output
even without BSP.

> I've seen the new version and really prefer the Python scripts for that
> kind of thing. Ist a lot more flexible. For example I'm pretty sure that
> most of the ignore list can by automatically generated by looking at the
> package (Does it have a URL?). Doing stuff like that in the Makefile is
> painful.

Hmm, I found it pretty easy to do. Even that no-url-trick seem easy enough.
Now not so sure using $(info ...) was that good idea after all, but that
is only minor issue here.

	ladis