[ptxdist] [PATCH] ptxdist getdev: Download dev packages from ${PTXCONF_SETUP_DEVMIRROR}

[ptxdist] [PATCH] ptxdist getdev: Download dev packages from ${PTXCONF_SETUP_DEVMIRROR}

[jon]

From: Jon Ringle <jringle@gridpoint.com>

Version 2

This patch allows you to setup a dev package mirror to hold *-dev.tar.gz
packages.

Upload your *-dev.tar.gz packages to your mirror, then you can avoid
rebuilding all your packages by doing:

ptxdist getdev


Signed-off-by: Jon Ringle <jringle@gridpoint.com>
---
 bin/ptxdist                         |    5 +++++
 config/setup/Kconfig                |   10 ++++++++++
 rules/post/ptxd_make_world_get.make |    7 +++++++
 scripts/lib/ptxd_lib_dgen.awk       |    7 +++++++
 scripts/lib/ptxd_make_get.sh        |   15 ++++++++-------
 5 files changed, 37 insertions(+), 7 deletions(-)

diff --git a/bin/ptxdist b/bin/ptxdist
index f9cbdec..663c41e 100755
--- a/bin/ptxdist
+++ b/bin/ptxdist
@@ -1789,6 +1789,11 @@ EOF
 			fi
 			exit
 			;;
+		getdev)
+			check_premake &&
+			ptxd_make_log getdev
+			exit
+			;;
 		go|images)
 			do_${cmd}
 			exit
diff --git a/config/setup/Kconfig b/config/setup/Kconfig
index 790276b..e90949b 100644
--- a/config/setup/Kconfig
+++ b/config/setup/Kconfig
@@ -93,6 +93,16 @@ config SETUP_PROJECTPATH
 
 endmenu
 
+menu "Dev Download      "
+
+config SETUP_DEVMIRROR
+	string "Dev package Mirror"
+	help
+	  Enter a URL where *-dev.tar.gz packages can be downloaded
+	  Use: ptxdist getdev
+
+endmenu
+
 menu "Source Directories"
 
 config SETUP_SRCDIR
diff --git a/rules/post/ptxd_make_world_get.make b/rules/post/ptxd_make_world_get.make
index 152674c..ae11db9 100644
--- a/rules/post/ptxd_make_world_get.make
+++ b/rules/post/ptxd_make_world_get.make
@@ -26,6 +26,9 @@ $(SRCDIR)/%:
 	@$(call targetinfo)
 	@$(call get, $($@))
 
+$(call remove_quotes, $(PTXCONF_PROJECT_DEVPKGDIR))/%:
+	@$(call targetinfo)
+	@$(call getdev, $@)
 
 $(STATEDIR)/%.get:
 	@$(call targetinfo)
@@ -59,4 +62,8 @@ get = \
 check_src = \
 	ptxd_make_check_src "$($(strip $(1))_SOURCE)" "$($(strip $(1))_MD5)"
 
+getdev = \
+	ptxd_make_get_nofail=y \
+	ptxd_make_get "$(strip $(1))" "$(call remove_quotes, $(PTXCONF_SETUP_DEVMIRROR))/$(notdir $(1))"
+
 # vim: syntax=make
diff --git a/scripts/lib/ptxd_lib_dgen.awk b/scripts/lib/ptxd_lib_dgen.awk
index 7f790c9..0cd2ac1 100644
--- a/scripts/lib/ptxd_lib_dgen.awk
+++ b/scripts/lib/ptxd_lib_dgen.awk
@@ -241,6 +241,13 @@ function write_deps_pkg_all(this_PKG, this_pkg) {
 	# .get rule
 	#
 	print "$(STATEDIR)/" this_pkg ".get: $(" this_PKG "_SOURCES)"	> DGEN_DEPS_POST;
+	print "ifneq ($(call remove_quotes, $(PTXCONF_SETUP_DEVMIRROR)),)" > DGEN_DEPS_POST;
+	print "ifneq ($($(" this_PKG "_DEVPKG)),NO)" > DGEN_DEPS_POST;
+	print "ifneq ($(" this_PKG "_CFGHASH),)" > DGEN_DEPS_POST;
+	print "getdev: $(call remove_quotes, $(PTXCONF_PROJECT_DEVPKGDIR))/$(" this_PKG "_DEVPKG)" > DGEN_DEPS_POST;
+	print "endif" > DGEN_DEPS_POST;
+	print "endif" > DGEN_DEPS_POST;
+	print "endif" > DGEN_DEPS_POST;
 }
 
 function write_deps_pkg_active(this_PKG, this_pkg, prefix) {
diff --git a/scripts/lib/ptxd_make_get.sh b/scripts/lib/ptxd_make_get.sh
index f4d7922..185dc77 100644
--- a/scripts/lib/ptxd_make_get.sh
+++ b/scripts/lib/ptxd_make_get.sh
@@ -280,7 +280,7 @@ ptxd_make_get() {
 		shift
 
 		case "${url}" in
-		${PTXCONF_SETUP_PTXMIRROR}/*/*)
+		${PTXCONF_SETUP_PTXMIRROR}/*/*|${PTXCONF_SETUP_DEVMIRROR}/*)
 			# keep original URL, for stuff like glibc
 			argv[${#argv[@]}]="${url}"
 			mrd=true
@@ -397,11 +397,12 @@ ptxd_make_get() {
 		esac
 	done
 
-	echo
-	echo "Could not download package"
-	echo "URL: ${orig_argv[@]}"
-	echo
-	exit 1
+	if [ "${ptxd_make_get_nofail}" != "y" ]; then
+		echo
+		echo "Could not download package"
+		echo "URL: ${orig_argv[@]}"
+		echo
+		exit 1
+	fi
 }
-
 export -f ptxd_make_get
-- 
1.7.10.4


-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH] ptxdist getdev: Download dev packages from ${PTXCONF_SETUP_DEVMIRROR}

[Michael Olbrich]

On Mon, Jan 20, 2014 at 09:11:54PM -0500, jon@ringle.org wrote:
> From: Jon Ringle <jringle@gridpoint.com>
> 
> Version 2
> 
> This patch allows you to setup a dev package mirror to hold *-dev.tar.gz
> packages.
> 
> Upload your *-dev.tar.gz packages to your mirror, then you can avoid
> rebuilding all your packages by doing:
> 
> ptxdist getdev
> 
> 
> Signed-off-by: Jon Ringle <jringle@gridpoint.com>
> ---
>  bin/ptxdist                         |    5 +++++
>  config/setup/Kconfig                |   10 ++++++++++
>  rules/post/ptxd_make_world_get.make |    7 +++++++
>  scripts/lib/ptxd_lib_dgen.awk       |    7 +++++++
>  scripts/lib/ptxd_make_get.sh        |   15 ++++++++-------
>  5 files changed, 37 insertions(+), 7 deletions(-)
> 
> diff --git a/bin/ptxdist b/bin/ptxdist
> index f9cbdec..663c41e 100755
> --- a/bin/ptxdist
> +++ b/bin/ptxdist
> @@ -1789,6 +1789,11 @@ EOF
>  			fi
>  			exit
>  			;;
> +		getdev)
> +			check_premake &&
> +			ptxd_make_log getdev
> +			exit
> +			;;
>  		go|images)
>  			do_${cmd}
>  			exit
> diff --git a/config/setup/Kconfig b/config/setup/Kconfig
> index 790276b..e90949b 100644
> --- a/config/setup/Kconfig
> +++ b/config/setup/Kconfig
> @@ -93,6 +93,16 @@ config SETUP_PROJECTPATH
>  
>  endmenu
>  
> +menu "Dev Download      "
> +
> +config SETUP_DEVMIRROR

we use:
	string
	prompt "..."

> +	string "Dev package Mirror"

Can you change this to "Mirror for pre-build archives" or something like
that. Take a look at the latest commits. Juergen pushed a patch to change
the corresponding descriptions in rules/project-name.in

jbe: any better idea for the prompt?

> +	help
> +	  Enter a URL where *-dev.tar.gz packages can be downloaded
> +	  Use: ptxdist getdev
> +
> +endmenu
> +
>  menu "Source Directories"
>  
>  config SETUP_SRCDIR
> diff --git a/rules/post/ptxd_make_world_get.make b/rules/post/ptxd_make_world_get.make
> index 152674c..ae11db9 100644
> --- a/rules/post/ptxd_make_world_get.make
> +++ b/rules/post/ptxd_make_world_get.make
> @@ -26,6 +26,9 @@ $(SRCDIR)/%:
>  	@$(call targetinfo)
>  	@$(call get, $($@))
>  

ifneq ($(call remove_quotes, $(PTXCONF_PROJECT_DEVPKGDIR)),)
$(call remove_quotes, $(PTXCONF_PROJECT_DEVPKGDIR))/%-dev.tar.gz:
[...]

Otherwise this may match other stuff. I was rather confused when ptxdist
tied to download a prepare stage :-)

Michael

> +$(call remove_quotes, $(PTXCONF_PROJECT_DEVPKGDIR))/%:
> +	@$(call targetinfo)
> +	@$(call getdev, $@)
>  
>  $(STATEDIR)/%.get:
>  	@$(call targetinfo)
> @@ -59,4 +62,8 @@ get = \
>  check_src = \
>  	ptxd_make_check_src "$($(strip $(1))_SOURCE)" "$($(strip $(1))_MD5)"
>  
> +getdev = \
> +	ptxd_make_get_nofail=y \
> +	ptxd_make_get "$(strip $(1))" "$(call remove_quotes, $(PTXCONF_SETUP_DEVMIRROR))/$(notdir $(1))"
> +
>  # vim: syntax=make
> diff --git a/scripts/lib/ptxd_lib_dgen.awk b/scripts/lib/ptxd_lib_dgen.awk
> index 7f790c9..0cd2ac1 100644
> --- a/scripts/lib/ptxd_lib_dgen.awk
> +++ b/scripts/lib/ptxd_lib_dgen.awk
> @@ -241,6 +241,13 @@ function write_deps_pkg_all(this_PKG, this_pkg) {
>  	# .get rule
>  	#
>  	print "$(STATEDIR)/" this_pkg ".get: $(" this_PKG "_SOURCES)"	> DGEN_DEPS_POST;
> +	print "ifneq ($(call remove_quotes, $(PTXCONF_SETUP_DEVMIRROR)),)" > DGEN_DEPS_POST;
> +	print "ifneq ($($(" this_PKG "_DEVPKG)),NO)" > DGEN_DEPS_POST;
> +	print "ifneq ($(" this_PKG "_CFGHASH),)" > DGEN_DEPS_POST;
> +	print "getdev: $(call remove_quotes, $(PTXCONF_PROJECT_DEVPKGDIR))/$(" this_PKG "_DEVPKG)" > DGEN_DEPS_POST;
> +	print "endif" > DGEN_DEPS_POST;
> +	print "endif" > DGEN_DEPS_POST;
> +	print "endif" > DGEN_DEPS_POST;
>  }
>  
>  function write_deps_pkg_active(this_PKG, this_pkg, prefix) {
> diff --git a/scripts/lib/ptxd_make_get.sh b/scripts/lib/ptxd_make_get.sh
> index f4d7922..185dc77 100644
> --- a/scripts/lib/ptxd_make_get.sh
> +++ b/scripts/lib/ptxd_make_get.sh
> @@ -280,7 +280,7 @@ ptxd_make_get() {
>  		shift
>  
>  		case "${url}" in
> -		${PTXCONF_SETUP_PTXMIRROR}/*/*)
> +		${PTXCONF_SETUP_PTXMIRROR}/*/*|${PTXCONF_SETUP_DEVMIRROR}/*)
>  			# keep original URL, for stuff like glibc
>  			argv[${#argv[@]}]="${url}"
>  			mrd=true
> @@ -397,11 +397,12 @@ ptxd_make_get() {
>  		esac
>  	done
>  
> -	echo
> -	echo "Could not download package"
> -	echo "URL: ${orig_argv[@]}"
> -	echo
> -	exit 1
> +	if [ "${ptxd_make_get_nofail}" != "y" ]; then
> +		echo
> +		echo "Could not download package"
> +		echo "URL: ${orig_argv[@]}"
> +		echo
> +		exit 1
> +	fi
>  }
> -
>  export -f ptxd_make_get
> -- 
> 1.7.10.4
> 
> 
> -- 
> ptxdist mailing list
> ptxdist@pengutronix.de
> 

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH] ptxdist getdev: Download dev packages from ${PTXCONF_SETUP_DEVMIRROR}

[Michael Olbrich]

On Mon, Jan 20, 2014 at 09:11:54PM -0500, jon@ringle.org wrote:
> From: Jon Ringle <jringle@gridpoint.com>
> 
> Version 2
> 
> This patch allows you to setup a dev package mirror to hold *-dev.tar.gz
> packages.
> 
> Upload your *-dev.tar.gz packages to your mirror, then you can avoid
> rebuilding all your packages by doing:
> 
> ptxdist getdev
> 
> 
> Signed-off-by: Jon Ringle <jringle@gridpoint.com>
> ---
>  bin/ptxdist                         |    5 +++++
>  config/setup/Kconfig                |   10 ++++++++++
>  rules/post/ptxd_make_world_get.make |    7 +++++++
>  scripts/lib/ptxd_lib_dgen.awk       |    7 +++++++
>  scripts/lib/ptxd_make_get.sh        |   15 ++++++++-------
>  5 files changed, 37 insertions(+), 7 deletions(-)
> 
> diff --git a/bin/ptxdist b/bin/ptxdist
> index f9cbdec..663c41e 100755
> --- a/bin/ptxdist
> +++ b/bin/ptxdist
> @@ -1789,6 +1789,11 @@ EOF
>  			fi
>  			exit
>  			;;
> +		getdev)
> +			check_premake &&
> +			ptxd_make_log getdev
> +			exit
> +			;;
>  		go|images)
>  			do_${cmd}
>  			exit
> diff --git a/config/setup/Kconfig b/config/setup/Kconfig
> index 790276b..e90949b 100644
> --- a/config/setup/Kconfig
> +++ b/config/setup/Kconfig
> @@ -93,6 +93,16 @@ config SETUP_PROJECTPATH
>  
>  endmenu
>  
> +menu "Dev Download      "
> +
> +config SETUP_DEVMIRROR
> +	string "Dev package Mirror"
> +	help
> +	  Enter a URL where *-dev.tar.gz packages can be downloaded
> +	  Use: ptxdist getdev
> +
> +endmenu
> +
>  menu "Source Directories"
>  
>  config SETUP_SRCDIR
> diff --git a/rules/post/ptxd_make_world_get.make b/rules/post/ptxd_make_world_get.make
> index 152674c..ae11db9 100644
> --- a/rules/post/ptxd_make_world_get.make
> +++ b/rules/post/ptxd_make_world_get.make
> @@ -26,6 +26,9 @@ $(SRCDIR)/%:
>  	@$(call targetinfo)
>  	@$(call get, $($@))
>  
> +$(call remove_quotes, $(PTXCONF_PROJECT_DEVPKGDIR))/%:
> +	@$(call targetinfo)
> +	@$(call getdev, $@)
>  
>  $(STATEDIR)/%.get:
>  	@$(call targetinfo)
> @@ -59,4 +62,8 @@ get = \
>  check_src = \
>  	ptxd_make_check_src "$($(strip $(1))_SOURCE)" "$($(strip $(1))_MD5)"
>  
> +getdev = \
> +	ptxd_make_get_nofail=y \
> +	ptxd_make_get "$(strip $(1))" "$(call remove_quotes, $(PTXCONF_SETUP_DEVMIRROR))/$(notdir $(1))"
> +
>  # vim: syntax=make
> diff --git a/scripts/lib/ptxd_lib_dgen.awk b/scripts/lib/ptxd_lib_dgen.awk
> index 7f790c9..0cd2ac1 100644
> --- a/scripts/lib/ptxd_lib_dgen.awk
> +++ b/scripts/lib/ptxd_lib_dgen.awk
> @@ -241,6 +241,13 @@ function write_deps_pkg_all(this_PKG, this_pkg) {
>  	# .get rule
>  	#
>  	print "$(STATEDIR)/" this_pkg ".get: $(" this_PKG "_SOURCES)"	> DGEN_DEPS_POST;
> +	print "ifneq ($(call remove_quotes, $(PTXCONF_SETUP_DEVMIRROR)),)" > DGEN_DEPS_POST;
> +	print "ifneq ($($(" this_PKG "_DEVPKG)),NO)" > DGEN_DEPS_POST;
> +	print "ifneq ($(" this_PKG "_CFGHASH),)" > DGEN_DEPS_POST;
> +	print "getdev: $(call remove_quotes, $(PTXCONF_PROJECT_DEVPKGDIR))/$(" this_PKG "_DEVPKG)" > DGEN_DEPS_POST;
> +	print "endif" > DGEN_DEPS_POST;
> +	print "endif" > DGEN_DEPS_POST;
> +	print "endif" > DGEN_DEPS_POST;
>  }
>  
>  function write_deps_pkg_active(this_PKG, this_pkg, prefix) {
> diff --git a/scripts/lib/ptxd_make_get.sh b/scripts/lib/ptxd_make_get.sh
> index f4d7922..185dc77 100644
> --- a/scripts/lib/ptxd_make_get.sh
> +++ b/scripts/lib/ptxd_make_get.sh
> @@ -280,7 +280,7 @@ ptxd_make_get() {
>  		shift
>  
>  		case "${url}" in
> -		${PTXCONF_SETUP_PTXMIRROR}/*/*)
> +		${PTXCONF_SETUP_PTXMIRROR}/*/*|${PTXCONF_SETUP_DEVMIRROR}/*)

add this to the next block ("${PTXCONF_SETUP_PTXMIRROR}/*)"). This on
should be removed anyways. I think this was only used for some realy old
toolchains.

Michael

>  			# keep original URL, for stuff like glibc
>  			argv[${#argv[@]}]="${url}"
>  			mrd=true
> @@ -397,11 +397,12 @@ ptxd_make_get() {
>  		esac
>  	done
>  
> -	echo
> -	echo "Could not download package"
> -	echo "URL: ${orig_argv[@]}"
> -	echo
> -	exit 1
> +	if [ "${ptxd_make_get_nofail}" != "y" ]; then
> +		echo
> +		echo "Could not download package"
> +		echo "URL: ${orig_argv[@]}"
> +		echo
> +		exit 1
> +	fi
>  }
> -
>  export -f ptxd_make_get
> -- 
> 1.7.10.4
> 
> 
> -- 
> ptxdist mailing list
> ptxdist@pengutronix.de
> 

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH] ptxdist getdev: Download dev packages from ${PTXCONF_SETUP_DEVMIRROR}

[Juergen Beisert]

Hi Jon, Michael,

On Thursday 23 January 2014 17:08:21 Michael Olbrich wrote:
> [...]
>
> > diff --git a/config/setup/Kconfig b/config/setup/Kconfig
> > index 790276b..e90949b 100644
> > --- a/config/setup/Kconfig
> > +++ b/config/setup/Kconfig
> > @@ -93,6 +93,16 @@ config SETUP_PROJECTPATH
> >
> >  endmenu
> >
> > +menu "Dev Download      "
> > +
> > +config SETUP_DEVMIRROR
>
> we use:
> 	string
> 	prompt "..."
>
> > +	string "Dev package Mirror"
>
> Can you change this to "Mirror for pre-build archives" or something like
> that. Take a look at the latest commits. Juergen pushed a patch to change
> the corresponding descriptions in rules/project-name.in
>
> jbe: any better idea for the prompt?

To be clear I use the term "pre-build archives" everywhere for this feature. 
Currently I'm also extending the "How to become a PTXdist Guru" manual with a 
description of this feature and using exactly this term. So as long this term 
is part of the menu entry (or at least of the help text) I'm happy.

IMHO: due to currently existing restrictions using this feature (fixed to a 
specific PTXdist release, patch stacks, toolchain a.s.o.), should this setting 
be a *project local* setting instead.

jbe

-- 
Pengutronix e.K.                              | Juergen Beisert             |
Linux Solutions for Science and Industry      | Phone: +49-5121-206917-5128 |
Peiner Str. 6-8, 31137 Hildesheim, Germany    | Fax:   +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686              | http://www.pengutronix.de/  |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH] ptxdist getdev: Download dev packages from ${PTXCONF_SETUP_DEVMIRROR}

[Michael Olbrich]

On Fri, Jan 24, 2014 at 09:43:29AM +0100, Juergen Beisert wrote:
> Hi Jon, Michael,
> 
> On Thursday 23 January 2014 17:08:21 Michael Olbrich wrote:
> > [...]
> >
> > > diff --git a/config/setup/Kconfig b/config/setup/Kconfig
> > > index 790276b..e90949b 100644
> > > --- a/config/setup/Kconfig
> > > +++ b/config/setup/Kconfig
> > > @@ -93,6 +93,16 @@ config SETUP_PROJECTPATH
> > >
> > >  endmenu
> > >
> > > +menu "Dev Download      "
> > > +
> > > +config SETUP_DEVMIRROR
> >
> > we use:
> > 	string
> > 	prompt "..."
> >
> > > +	string "Dev package Mirror"
> >
> > Can you change this to "Mirror for pre-build archives" or something like
> > that. Take a look at the latest commits. Juergen pushed a patch to change
> > the corresponding descriptions in rules/project-name.in
> >
> > jbe: any better idea for the prompt?
> 
> To be clear I use the term "pre-build archives" everywhere for this feature. 
> Currently I'm also extending the "How to become a PTXdist Guru" manual with a 
> description of this feature and using exactly this term. So as long this term 
> is part of the menu entry (or at least of the help text) I'm happy.
> 
> IMHO: due to currently existing restrictions using this feature (fixed to a 
> specific PTXdist release, patch stacks, toolchain a.s.o.), should this setting 
> be a *project local* setting instead.

Now that I think about this, you are correct.
Jon: can you move this right next to the directory option?

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de