Re: [ptxdist] [PATCH v5 3/3] nodejs_packages: new package

[Michael Olbrich <m.olbrich@pengutronix.de>]

On Tue, Feb 04, 2020 at 08:53:59AM +0100, Ulrich Ölmann wrote:
> From: Björn Esser <b.esser@pengutronix.de>
> 
> This is a dummy package to specify and install required Node.js packages for the
> target system.
> 
> The management of those packages and their dependencies is handled with
> Yarn (run through host-nodejs) to maintain a reproducible offline cache that is
> stored in local_src/nodejs_packages.
> 
> Signed-off-by: Björn Esser <b.esser@pengutronix.de>
> [uol: adjusted commit message and Kconfig menu file's help texts, integrated
> Kconfig symbol NODEJS_PACKAGES_LICENSE, fixed NODEJS_PACKAGES_URL, removed one
> square bracket in a shell test and wrapped install/cp into call of execute and
> world/execute]
> Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
> ---
> v3 --> v4: adjusted commit message and Kconfig menu file's help texts & fixed
>            NODEJS_PACKAGES_URL
> Range-diff of v5 against v4:
> 3:  ccd2b44a8d92 ! 3:  1a73ebbd6e7e nodejs_packages: new package
>     @@ Commit message
>          stored in local_src/nodejs_packages.
> 
>          Signed-off-by: Björn Esser <b.esser@pengutronix.de>
>     -    [uol: adjusted commit message and Kconfig menu file's help texts & fixed
>     -    NODEJS_PACKAGES_URL]
>     +    [uol: adjusted commit message and Kconfig menu file's help texts, integrated
>     +    Kconfig symbol NODEJS_PACKAGES_LICENSE, fixed NODEJS_PACKAGES_URL, removed one
>     +    square bracket in a shell test and wrapped install/cp into call of execute and
>     +    world/execute]
>          Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
> 
>       ## rules/nodejs_packages.in (new) ##
>     @@ rules/nodejs_packages.in (new)
>      +
>      +	  [1] https://legacy.yarnpkg.com/en/docs/dependency-versions/
>      +
>     ++config NODEJS_PACKAGES_LICENSE
>     ++	string
>     ++	prompt "Licenses of to be installed Node.js packages"
>     ++	default "unknown"
>     ++	help
>     ++	  Manually compiled list of licenses that the to be installed Node.js
>     ++	  packages are distributed under, see description of the variable
>     ++	  *_LICENSE in [1].
>     ++
>     ++	  ATTENTION: be aware of the fact that Node.js packages usually depend
>     ++	  on additional Node.js packages that Yarn installs as well and that
>     ++	  this propagates recursively resulting in an unknown number of Node.js
>     ++	  packages on the target with an unknown set of licenses that they may
>     ++	  be utilized under. Hence you carefully need to examine all of them
>     ++	  individually to create a reliable list of licenses.
>     ++
>     ++	  [1] https://www.ptxdist.org/doc/dev_manual.html#some-notes-about-licenses
>     ++
>      +config NODEJS_PACKAGES_OFFLINE
>      +	bool
>     -+	prompt "lock offline cache for offline/production use"
>     ++	prompt "Lock offline cache for offline/production use"
>      +	help
>     -+	  This option locks the offline cache for installing the nodejs packages
>     -+	  and allows their installation during the build of the BSP without any
>     -+	  need for internet connectivity.
>     ++	  This option locks the offline cache for installing the Node.js
>     ++	  packages and allows their installation during the build of the BSP
>     ++	  without any need for internet connectivity.
>      +
>      +	  Enabling this option is highly recommended for production use.
>      +
>     @@ rules/nodejs_packages.make (new)
>      +NODEJS_PACKAGES_URL	:= lndir://$(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)
>      +NODEJS_PACKAGES_DIR	:= $(BUILDDIR)/$(NODEJS_PACKAGES)
>      +NODEJS_PACKAGES_CACHE	:= $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn_cache
>     -+NODEJS_PACKAGES_LICENSE	:= unknown
>     ++NODEJS_PACKAGES_LICENSE	:= $(call remove_quotes, $(PTXCONF_NODEJS_PACKAGES_LICENSE))
>      +
>      +NODEJS_PACKAGES_LIST	:= $(call remove_quotes, $(PTXCONF_NODEJS_PACKAGES_LIST))
>      +
>     @@ rules/nodejs_packages.make (new)
>      +		\"license\": \"UNLICENSED\", \
>      +		\"private\": true \
>      +	}" > $(NODEJS_PACKAGES_DIR)/package.json
>     -+	if [[ ! -f $(YARN_LOCK) ]]; then \
>     ++	if [ ! -f $(YARN_LOCK) ]; then \
>      +		touch $(YARN_LOCK); \
>      +		ln -fs $(YARN_LOCK) $(NODEJS_PACKAGES_DIR)/yarn.lock; \
>      +	fi
>     @@ rules/nodejs_packages.make (new)
>      +
>      +$(STATEDIR)/nodejs_packages.install:
>      +	@$(call targetinfo)
>     -+	install -dm 0755 $(NODEJS_PACKAGES_PKGDIR)/usr/lib
>     -+	cp -pr $(NODEJS_PACKAGES_DIR)/node_modules $(NODEJS_PACKAGES_PKGDIR)/usr/lib
>     ++	@$(call world/execute, HOST_UA_NODESET, \
>     ++		install -dm 0755 $(NODEJS_PACKAGES_PKGDIR)/usr/lib)
>     ++	@$(call execute, HOST_UA_NODESET, \
>     ++		cp -pr $(NODEJS_PACKAGES_DIR)/node_modules \
>     ++			$(NODEJS_PACKAGES_PKGDIR)/usr/lib)
>      +	@$(call touch)
>      +
>      +# ----------------------------------------------------------------------------
> 
>  rules/nodejs_packages.in   | 73 ++++++++++++++++++++++++++++++
>  rules/nodejs_packages.make | 93 ++++++++++++++++++++++++++++++++++++++
>  2 files changed, 166 insertions(+)
>  create mode 100644 rules/nodejs_packages.in
>  create mode 100644 rules/nodejs_packages.make
> 
> diff --git a/rules/nodejs_packages.in b/rules/nodejs_packages.in
> new file mode 100644
> index 000000000000..aea44205da41
> --- /dev/null
> +++ b/rules/nodejs_packages.in
> @@ -0,0 +1,73 @@
> +## SECTION=bytecode_engines
> +
> +menuconfig NODEJS_PACKAGES
> +	tristate
> +	select HOST_YARN
> +	select NODEJS
> +	prompt "Node.js packages (target)     "
> +	help
> +	  Reproducibly install Node.js packages that will be available
> +	  system-wide on the target.
> +
> +	  You MUST specify at least one Node.js package in the submenu that is
> +	  shown after enabling this option.
> +
> +
> +if NODEJS_PACKAGES
> +
> +config NODEJS_PACKAGES_LIST
> +	string
> +	prompt "Node.js packages to install"
> +	help
> +	  A white-space separated list of Node.js packages including their
> +	  versions (e.g. express@4.17.x or socket.io@'>=2.1.0 <=2.3.0'). NPM
> +	  semvers are fully supported, see [1].
> +
> +	  The packages are managed with Yarn and are guaranteed to be
> +	  reproducible and immutable as long as the download location stays
> +	  accessible or the offline cache is not altered.
> +
> +	  If the list of Node.js packages to be installed has been populated for
> +	  the first time or if it has been modified later on one needs to
> +	  (re-)build this package to download the desired Node.js packages and
> +	  update the offline cache. Afterwards it is highly recommended to
> +	  commit the path '${PTXDIST_WORKSPACE}/local_src/nodejs_packages/' into
> +	  the version control system and to lock the package cache.
> +
> +	  As a prerequisite the path '${PTXDIST_WORKSPACE}/local_src/nodejs_
> +	  packages/' must exist in the BSP *before* building this package.
> +
> +	  [1] https://legacy.yarnpkg.com/en/docs/dependency-versions/
> +
> +config NODEJS_PACKAGES_LICENSE
> +	string
> +	prompt "Licenses of to be installed Node.js packages"
> +	default "unknown"
> +	help
> +	  Manually compiled list of licenses that the to be installed Node.js
> +	  packages are distributed under, see description of the variable
> +	  *_LICENSE in [1].
> +
> +	  ATTENTION: be aware of the fact that Node.js packages usually depend
> +	  on additional Node.js packages that Yarn installs as well and that
> +	  this propagates recursively resulting in an unknown number of Node.js
> +	  packages on the target with an unknown set of licenses that they may
> +	  be utilized under. Hence you carefully need to examine all of them
> +	  individually to create a reliable list of licenses.
> +
> +	  [1] https://www.ptxdist.org/doc/dev_manual.html#some-notes-about-licenses
> +
> +config NODEJS_PACKAGES_OFFLINE
> +	bool
> +	prompt "Lock offline cache for offline/production use"
> +	help
> +	  This option locks the offline cache for installing the Node.js
> +	  packages and allows their installation during the build of the BSP
> +	  without any need for internet connectivity.
> +
> +	  Enabling this option is highly recommended for production use.
> +
> +	  Do NOT forget to commit the path '${PTXDIST_WORKSPACE}/local_src/
> +	  nodejs_packages/' into the BSP's version control system!
> +
> +endif
> diff --git a/rules/nodejs_packages.make b/rules/nodejs_packages.make
> new file mode 100644
> index 000000000000..cdde30f38b0c
> --- /dev/null
> +++ b/rules/nodejs_packages.make
> @@ -0,0 +1,93 @@
> +# -*-makefile-*-
> +#
> +# Copyright (C) 2020 by Bjoern Esser <bes@pengutronix.de>
> +#
> +# For further information about the PTXdist project and license conditions
> +# see the README file.
> +#
> +
> +#
> +# We provide this package
> +#
> +PACKAGES-$(PTXCONF_NODEJS_PACKAGES) += nodejs_packages
> +
> +#
> +# Paths and names
> +#
> +NODEJS_PACKAGES_VERSION	:= 0.0.1
> +NODEJS_PACKAGES		:= nodejs_packages-$(NODEJS_PACKAGES_VERSION)
> +NODEJS_PACKAGES_LOCAL	:= local_src/nodejs_packages
> +NODEJS_PACKAGES_URL	:= lndir://$(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)
> +NODEJS_PACKAGES_DIR	:= $(BUILDDIR)/$(NODEJS_PACKAGES)
> +NODEJS_PACKAGES_CACHE	:= $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn_cache
> +NODEJS_PACKAGES_LICENSE	:= $(call remove_quotes, $(PTXCONF_NODEJS_PACKAGES_LICENSE))
> +
> +NODEJS_PACKAGES_LIST	:= $(call remove_quotes, $(PTXCONF_NODEJS_PACKAGES_LIST))
> +
> +YARN_LOCK	:= $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn.lock
> +YARN_OPTS	:= \
> +	--cwd "$(NODEJS_PACKAGES_DIR)" \
> +	--cache-folder "$(NODEJS_PACKAGES_CACHE)" \
> +	--link-duplicates \
> +	--production=true
> +
> +ifdef PTXCONF_NODEJS_PACKAGES_OFFLINE
> +YARN_OPTS	+= \
> +	--frozen-lockfile \
> +	--offline
> +endif
> +
> +# ----------------------------------------------------------------------------
> +# Compile
> +# ----------------------------------------------------------------------------
> +
> +$(STATEDIR)/nodejs_packages.compile:
> +	@$(call targetinfo)
> +	mkdir -p $(NODEJS_PACKAGES_CACHE)
> +	printf "{ \
> +		\"name\": \"nodejs_packages\", \
> +		\"version\": \"$(NODEJS_PACKAGES_VERSION)\", \
> +		\"license\": \"UNLICENSED\", \
> +		\"private\": true \
> +	}" > $(NODEJS_PACKAGES_DIR)/package.json
> +	if [ ! -f $(YARN_LOCK) ]; then \
> +		touch $(YARN_LOCK); \
> +		ln -fs $(YARN_LOCK) $(NODEJS_PACKAGES_DIR)/yarn.lock; \
> +	fi

I think we should fail here if NODEJS_PACKAGES_OFFLINE is set and the lock
file is missing. Or will yarn produce a sensible error message in this
case?

> +	yarn $(YARN_OPTS) add $(NODEJS_PACKAGES_LIST)
> +	find $(NODEJS_PACKAGES_CACHE) -type f -name '.yarn-tarball.tgz' -delete
> +	@$(call touch)
> +
> +# ----------------------------------------------------------------------------
> +# Install
> +# ----------------------------------------------------------------------------
> +
> +$(STATEDIR)/nodejs_packages.install:
> +	@$(call targetinfo)
> +	@$(call world/execute, HOST_UA_NODESET, \
> +		install -dm 0755 $(NODEJS_PACKAGES_PKGDIR)/usr/lib)

add -v to make it visible at runtime.

> +	@$(call execute, HOST_UA_NODESET, \
> +		cp -pr $(NODEJS_PACKAGES_DIR)/node_modules \
> +			$(NODEJS_PACKAGES_PKGDIR)/usr/lib)

Here as well.

Michael

> +	@$(call touch)
> +
> +# ----------------------------------------------------------------------------
> +# Target-Install
> +# ----------------------------------------------------------------------------
> +
> +$(STATEDIR)/nodejs_packages.targetinstall:
> +	@$(call targetinfo)
> +
> +	@$(call install_init, nodejs_packages)
> +	@$(call install_fixup, nodejs_packages,PRIORITY,optional)
> +	@$(call install_fixup, nodejs_packages,SECTION,base)
> +	@$(call install_fixup, nodejs_packages,AUTHOR,"Bjoern Esser <bes@pengutronix.de>")
> +	@$(call install_fixup, nodejs_packages,DESCRIPTION,missing)
> +
> +	$(call install_tree, nodejs_packages, 0, 0, -, /usr/lib/node_modules/)
> +
> +	@$(call install_finish, nodejs_packages)
> +
> +	@$(call touch)
> +
> +# vim: syntax=make
> -- 
> 2.25.0
> 
> 
> _______________________________________________
> ptxdist mailing list
> ptxdist@pengutronix.de

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de