Re: [ptxdist] [PATCH v4 3/3] nodejs_packages: new package

[Michael Olbrich <m.olbrich@pengutronix.de>]

On Wed, Jan 29, 2020 at 12:06:19PM +0100, Ulrich Ölmann wrote:
> From: Björn Esser <b.esser@pengutronix.de>
> 
> This is a dummy package to specify and install required Node.js packages for the
> target system.
> 
> The management of those packages and their dependencies is handled with
> Yarn (run through host-nodejs) to maintain a reproducible offline cache that is
> stored in local_src/nodejs_packages.
> 
> Signed-off-by: Björn Esser <b.esser@pengutronix.de>
> [uol: adjusted commit message and Kconfig menu file's help texts & fixed
> NODEJS_PACKAGES_URL]
> Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
> ---
> Range-diff of v4 against v3:
> 3:  2a6984f0297b ! 3:  ccd2b44a8d92 nodejs_packages: New package.
>     @@ Metadata
>      Author: Björn Esser <b.esser@pengutronix.de>
> 
>       ## Commit message ##
>     -    nodejs_packages: New package.
>     +    nodejs_packages: new package
> 
>     -    This is a dummy package to specifiy and install the required
>     -    Node.js packages for the target system.
>     +    This is a dummy package to specify and install required Node.js packages for the
>     +    target system.
> 
>     -    The management of the Node.js packages and their dependencies
>     -    are handled with Yarn (run through host-nodejs) to maintain
>     -    a reprocible offline cache.
>     -
>     -    The cache is stored in local_src/nodejs_packages, which must
>     -    be an existing path inside of the BSP.
>     +    The management of those packages and their dependencies is handled with
>     +    Yarn (run through host-nodejs) to maintain a reproducible offline cache that is
>     +    stored in local_src/nodejs_packages.
> 
>          Signed-off-by: Björn Esser <b.esser@pengutronix.de>
>     +    [uol: adjusted commit message and Kconfig menu file's help texts & fixed
>     +    NODEJS_PACKAGES_URL]
>     +    Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
> 
>       ## rules/nodejs_packages.in (new) ##
>      @@
>     @@ rules/nodejs_packages.in (new)
>      +	  Reproducibly install Node.js packages that will be available
>      +	  system-wide on the target.
>      +
>     -+	  You MUST specify at least one Node.js package in the shown
>     -+	  submenu after enabling this option.
>     ++	  You MUST specify at least one Node.js package in the submenu that is
>     ++	  shown after enabling this option.
>      +
>      +
>      +if NODEJS_PACKAGES
>     @@ rules/nodejs_packages.in (new)
>      +	string
>      +	prompt "Node.js packages to install"
>      +	help
>     -+	  A white-space separated list of Node.js packages including
>     -+	  its version (e.g. express@4.17.x). npm semvers are fully
>     -+	  supported.
>     ++	  A white-space separated list of Node.js packages including their
>     ++	  versions (e.g. express@4.17.x or socket.io@'>=2.1.0 <=2.3.0'). NPM
>     ++	  semvers are fully supported, see [1].
>      +
>      +	  The packages are managed with Yarn and are guaranteed to be
>     -+	  reproducible and immutable as long as the download location
>     -+	  stays accessible or the download-cache does not get altered.
>     ++	  reproducible and immutable as long as the download location stays
>     ++	  accessible or the offline cache is not altered.
>     ++
>     ++	  If the list of Node.js packages to be installed has been populated for
>     ++	  the first time or if it has been modified later on one needs to
>     ++	  (re-)build this package to download the desired Node.js packages and
>     ++	  update the offline cache. Afterwards it is highly recommended to
>     ++	  commit the path '${PTXDIST_WORKSPACE}/local_src/nodejs_packages/' into
>     ++	  the version control system and to lock the package cache.
>      +
>     -+	  After downloading the packages and creating the package cache,
>     -+	  by building this package the first time after altering the list
>     -+	  of installed packages, it is highly recommended to commit the
>     -+	  path: './local_src/nodejs_packages/' into the git repository
>     -+	  and to lock the package cache.
>     ++	  As a prerequisite the path '${PTXDIST_WORKSPACE}/local_src/nodejs_
>     ++	  packages/' must exist in the BSP *before* building this package.
>      +
>     -+	  Also the path: './local_src/nodejs_packages/' must exist in the
>     -+	  BSP *before* before building this package.
>     ++	  [1] https://legacy.yarnpkg.com/en/docs/dependency-versions/
>      +
>      +config NODEJS_PACKAGES_OFFLINE
>      +	bool
>     -+	prompt "lock package cache for offline / production use"
>     ++	prompt "lock offline cache for offline/production use"
>      +	help
>     -+	  This option locks the cache for installing the nodejs
>     -+	  packages and performs their installation during the
>     -+	  build of the BSP without any need for internet connectivity.
>     ++	  This option locks the offline cache for installing the nodejs packages
>     ++	  and allows their installation during the build of the BSP without any
>     ++	  need for internet connectivity.
>      +
>     -+	  Checking this option is highly recommended for production use.
>     ++	  Enabling this option is highly recommended for production use.
>      +
>     -+	  Do NOT forget to commit the path: './local_src/nodejs_packages/'
>     -+	  into the BSP's git repository!
>     ++	  Do NOT forget to commit the path '${PTXDIST_WORKSPACE}/local_src/
>     ++	  nodejs_packages/' into the BSP's version control system!
>      +
>      +endif
> 
>     @@ rules/nodejs_packages.make (new)
>      +NODEJS_PACKAGES_VERSION	:= 0.0.1
>      +NODEJS_PACKAGES		:= nodejs_packages-$(NODEJS_PACKAGES_VERSION)
>      +NODEJS_PACKAGES_LOCAL	:= local_src/nodejs_packages
>     -+NODEJS_PACKAGES_URL	:= lndir://$(NODEJS_PACKAGES_LOCAL)
>     ++NODEJS_PACKAGES_URL	:= lndir://$(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)
>      +NODEJS_PACKAGES_DIR	:= $(BUILDDIR)/$(NODEJS_PACKAGES)
>      +NODEJS_PACKAGES_CACHE	:= $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn_cache
>      +NODEJS_PACKAGES_LICENSE	:= unknown
> 
> 
>  rules/nodejs_packages.in   | 55 +++++++++++++++++++++++
>  rules/nodejs_packages.make | 90 ++++++++++++++++++++++++++++++++++++++
>  2 files changed, 145 insertions(+)
>  create mode 100644 rules/nodejs_packages.in
>  create mode 100644 rules/nodejs_packages.make
> 
> diff --git a/rules/nodejs_packages.in b/rules/nodejs_packages.in
> new file mode 100644
> index 000000000000..e6d33d30aaa9
> --- /dev/null
> +++ b/rules/nodejs_packages.in
> @@ -0,0 +1,55 @@
> +## SECTION=bytecode_engines
> +
> +menuconfig NODEJS_PACKAGES
> +	tristate
> +	select HOST_YARN
> +	select NODEJS
> +	prompt "Node.js packages (target)     "
> +	help
> +	  Reproducibly install Node.js packages that will be available
> +	  system-wide on the target.
> +
> +	  You MUST specify at least one Node.js package in the submenu that is
> +	  shown after enabling this option.
> +
> +
> +if NODEJS_PACKAGES
> +
> +config NODEJS_PACKAGES_LIST
> +	string
> +	prompt "Node.js packages to install"
> +	help
> +	  A white-space separated list of Node.js packages including their
> +	  versions (e.g. express@4.17.x or socket.io@'>=2.1.0 <=2.3.0'). NPM
> +	  semvers are fully supported, see [1].
> +
> +	  The packages are managed with Yarn and are guaranteed to be
> +	  reproducible and immutable as long as the download location stays
> +	  accessible or the offline cache is not altered.
> +
> +	  If the list of Node.js packages to be installed has been populated for
> +	  the first time or if it has been modified later on one needs to
> +	  (re-)build this package to download the desired Node.js packages and
> +	  update the offline cache. Afterwards it is highly recommended to
> +	  commit the path '${PTXDIST_WORKSPACE}/local_src/nodejs_packages/' into
> +	  the version control system and to lock the package cache.
> +
> +	  As a prerequisite the path '${PTXDIST_WORKSPACE}/local_src/nodejs_
> +	  packages/' must exist in the BSP *before* building this package.
> +
> +	  [1] https://legacy.yarnpkg.com/en/docs/dependency-versions/
> +
> +config NODEJS_PACKAGES_OFFLINE
> +	bool
> +	prompt "lock offline cache for offline/production use"
> +	help
> +	  This option locks the offline cache for installing the nodejs packages
> +	  and allows their installation during the build of the BSP without any
> +	  need for internet connectivity.
> +
> +	  Enabling this option is highly recommended for production use.
> +
> +	  Do NOT forget to commit the path '${PTXDIST_WORKSPACE}/local_src/
> +	  nodejs_packages/' into the BSP's version control system!
> +
> +endif
> diff --git a/rules/nodejs_packages.make b/rules/nodejs_packages.make
> new file mode 100644
> index 000000000000..514db3252279
> --- /dev/null
> +++ b/rules/nodejs_packages.make
> @@ -0,0 +1,90 @@
> +# -*-makefile-*-
> +#
> +# Copyright (C) 2020 by Bjoern Esser <bes@pengutronix.de>
> +#
> +# For further information about the PTXdist project and license conditions
> +# see the README file.
> +#
> +
> +#
> +# We provide this package
> +#
> +PACKAGES-$(PTXCONF_NODEJS_PACKAGES) += nodejs_packages
> +
> +#
> +# Paths and names
> +#
> +NODEJS_PACKAGES_VERSION	:= 0.0.1
> +NODEJS_PACKAGES		:= nodejs_packages-$(NODEJS_PACKAGES_VERSION)
> +NODEJS_PACKAGES_LOCAL	:= local_src/nodejs_packages
> +NODEJS_PACKAGES_URL	:= lndir://$(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)
> +NODEJS_PACKAGES_DIR	:= $(BUILDDIR)/$(NODEJS_PACKAGES)
> +NODEJS_PACKAGES_CACHE	:= $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn_cache
> +NODEJS_PACKAGES_LICENSE	:= unknown

Kconfig option for the license?

> +
> +NODEJS_PACKAGES_LIST	:= $(call remove_quotes, $(PTXCONF_NODEJS_PACKAGES_LIST))
> +
> +YARN_LOCK	:= $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn.lock
> +YARN_OPTS	:= \
> +	--cwd "$(NODEJS_PACKAGES_DIR)" \
> +	--cache-folder "$(NODEJS_PACKAGES_CACHE)" \
> +	--link-duplicates \
> +	--production=true
> +
> +ifdef PTXCONF_NODEJS_PACKAGES_OFFLINE
> +YARN_OPTS	+= \
> +	--frozen-lockfile \
> +	--offline
> +endif

Hmmm, I'm not sure if I like this option, but I don't have a good idea
either.

> +
> +# ----------------------------------------------------------------------------
> +# Compile
> +# ----------------------------------------------------------------------------
> +
> +$(STATEDIR)/nodejs_packages.compile:
> +	@$(call targetinfo)
> +	mkdir -p $(NODEJS_PACKAGES_CACHE)
> +	printf "{ \
> +		\"name\": \"nodejs_packages\", \
> +		\"version\": \"$(NODEJS_PACKAGES_VERSION)\", \
> +		\"license\": \"UNLICENSED\", \
> +		\"private\": true \
> +	}" > $(NODEJS_PACKAGES_DIR)/package.json
> +	if [[ ! -f $(YARN_LOCK) ]]; then \

single [ is sufficient.

> +		touch $(YARN_LOCK); \
> +		ln -fs $(YARN_LOCK) $(NODEJS_PACKAGES_DIR)/yarn.lock; \
> +	fi
> +	yarn $(YARN_OPTS) add $(NODEJS_PACKAGES_LIST)
> +	find $(NODEJS_PACKAGES_CACHE) -type f -name '.yarn-tarball.tgz' -delete
> +	@$(call touch)
> +
> +# ----------------------------------------------------------------------------
> +# Install
> +# ----------------------------------------------------------------------------
> +
> +$(STATEDIR)/nodejs_packages.install:
> +	@$(call targetinfo)
> +	install -dm 0755 $(NODEJS_PACKAGES_PKGDIR)/usr/lib
> +	cp -pr $(NODEJS_PACKAGES_DIR)/node_modules $(NODEJS_PACKAGES_PKGDIR)/usr/lib

You need to ensure that NODEJS_PACKAGES_PKGDIR is empty. Use world/execute
for the first and just execute for the second.

Michael

> +	@$(call touch)
> +
> +# ----------------------------------------------------------------------------
> +# Target-Install
> +# ----------------------------------------------------------------------------
> +
> +$(STATEDIR)/nodejs_packages.targetinstall:
> +	@$(call targetinfo)
> +
> +	@$(call install_init, nodejs_packages)
> +	@$(call install_fixup, nodejs_packages,PRIORITY,optional)
> +	@$(call install_fixup, nodejs_packages,SECTION,base)
> +	@$(call install_fixup, nodejs_packages,AUTHOR,"Bjoern Esser <bes@pengutronix.de>")
> +	@$(call install_fixup, nodejs_packages,DESCRIPTION,missing)
> +
> +	$(call install_tree, nodejs_packages, 0, 0, -, /usr/lib/node_modules/)
> +
> +	@$(call install_finish, nodejs_packages)
> +
> +	@$(call touch)
> +
> +# vim: syntax=make
> -- 
> 2.25.0
> 
> 
> _______________________________________________
> ptxdist mailing list
> ptxdist@pengutronix.de

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de