Re: [ptxdist] [PATCH v4 3/3] nodejs_packages: new package

["Ulrich Ölmann" <u.oelmann@pengutronix.de>]

On Thu, Jan 30 2020 at 15:22 +0100, Michael Olbrich <m.olbrich@pengutronix.de> wrote:
> On Wed, Jan 29, 2020 at 12:06:19PM +0100, Ulrich Ölmann wrote:
>> From: Björn Esser <b.esser@pengutronix.de>
>>
>> This is a dummy package to specify and install required Node.js packages for the
>> target system.
>>
>> The management of those packages and their dependencies is handled with
>> Yarn (run through host-nodejs) to maintain a reproducible offline cache that is
>> stored in local_src/nodejs_packages.
>>
>> Signed-off-by: Björn Esser <b.esser@pengutronix.de>
>> [uol: adjusted commit message and Kconfig menu file's help texts & fixed
>> NODEJS_PACKAGES_URL]
>> Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
>> ---
>> Range-diff of v4 against v3:
>> 3:  2a6984f0297b ! 3:  ccd2b44a8d92 nodejs_packages: New package.
>>     @@ Metadata
>>      Author: Björn Esser <b.esser@pengutronix.de>
>>
>>       ## Commit message ##
>>     -    nodejs_packages: New package.
>>     +    nodejs_packages: new package
>>
>>     -    This is a dummy package to specifiy and install the required
>>     -    Node.js packages for the target system.
>>     +    This is a dummy package to specify and install required Node.js packages for the
>>     +    target system.
>>
>>     -    The management of the Node.js packages and their dependencies
>>     -    are handled with Yarn (run through host-nodejs) to maintain
>>     -    a reprocible offline cache.
>>     -
>>     -    The cache is stored in local_src/nodejs_packages, which must
>>     -    be an existing path inside of the BSP.
>>     +    The management of those packages and their dependencies is handled with
>>     +    Yarn (run through host-nodejs) to maintain a reproducible offline cache that is
>>     +    stored in local_src/nodejs_packages.
>>
>>          Signed-off-by: Björn Esser <b.esser@pengutronix.de>
>>     +    [uol: adjusted commit message and Kconfig menu file's help texts & fixed
>>     +    NODEJS_PACKAGES_URL]
>>     +    Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
>>
>>       ## rules/nodejs_packages.in (new) ##
>>      @@
>>     @@ rules/nodejs_packages.in (new)
>>      +	  Reproducibly install Node.js packages that will be available
>>      +	  system-wide on the target.
>>      +
>>     -+	  You MUST specify at least one Node.js package in the shown
>>     -+	  submenu after enabling this option.
>>     ++	  You MUST specify at least one Node.js package in the submenu that is
>>     ++	  shown after enabling this option.
>>      +
>>      +
>>      +if NODEJS_PACKAGES
>>     @@ rules/nodejs_packages.in (new)
>>      +	string
>>      +	prompt "Node.js packages to install"
>>      +	help
>>     -+	  A white-space separated list of Node.js packages including
>>     -+	  its version (e.g. express@4.17.x). npm semvers are fully
>>     -+	  supported.
>>     ++	  A white-space separated list of Node.js packages including their
>>     ++	  versions (e.g. express@4.17.x or socket.io@'>=2.1.0 <=2.3.0'). NPM
>>     ++	  semvers are fully supported, see [1].
>>      +
>>      +	  The packages are managed with Yarn and are guaranteed to be
>>     -+	  reproducible and immutable as long as the download location
>>     -+	  stays accessible or the download-cache does not get altered.
>>     ++	  reproducible and immutable as long as the download location stays
>>     ++	  accessible or the offline cache is not altered.
>>     ++
>>     ++	  If the list of Node.js packages to be installed has been populated for
>>     ++	  the first time or if it has been modified later on one needs to
>>     ++	  (re-)build this package to download the desired Node.js packages and
>>     ++	  update the offline cache. Afterwards it is highly recommended to
>>     ++	  commit the path '${PTXDIST_WORKSPACE}/local_src/nodejs_packages/' into
>>     ++	  the version control system and to lock the package cache.
>>      +
>>     -+	  After downloading the packages and creating the package cache,
>>     -+	  by building this package the first time after altering the list
>>     -+	  of installed packages, it is highly recommended to commit the
>>     -+	  path: './local_src/nodejs_packages/' into the git repository
>>     -+	  and to lock the package cache.
>>     ++	  As a prerequisite the path '${PTXDIST_WORKSPACE}/local_src/nodejs_
>>     ++	  packages/' must exist in the BSP *before* building this package.
>>      +
>>     -+	  Also the path: './local_src/nodejs_packages/' must exist in the
>>     -+	  BSP *before* before building this package.
>>     ++	  [1] https://legacy.yarnpkg.com/en/docs/dependency-versions/
>>      +
>>      +config NODEJS_PACKAGES_OFFLINE
>>      +	bool
>>     -+	prompt "lock package cache for offline / production use"
>>     ++	prompt "lock offline cache for offline/production use"
>>      +	help
>>     -+	  This option locks the cache for installing the nodejs
>>     -+	  packages and performs their installation during the
>>     -+	  build of the BSP without any need for internet connectivity.
>>     ++	  This option locks the offline cache for installing the nodejs packages
>>     ++	  and allows their installation during the build of the BSP without any
>>     ++	  need for internet connectivity.
>>      +
>>     -+	  Checking this option is highly recommended for production use.
>>     ++	  Enabling this option is highly recommended for production use.
>>      +
>>     -+	  Do NOT forget to commit the path: './local_src/nodejs_packages/'
>>     -+	  into the BSP's git repository!
>>     ++	  Do NOT forget to commit the path '${PTXDIST_WORKSPACE}/local_src/
>>     ++	  nodejs_packages/' into the BSP's version control system!
>>      +
>>      +endif
>>
>>     @@ rules/nodejs_packages.make (new)
>>      +NODEJS_PACKAGES_VERSION	:= 0.0.1
>>      +NODEJS_PACKAGES		:= nodejs_packages-$(NODEJS_PACKAGES_VERSION)
>>      +NODEJS_PACKAGES_LOCAL	:= local_src/nodejs_packages
>>     -+NODEJS_PACKAGES_URL	:= lndir://$(NODEJS_PACKAGES_LOCAL)
>>     ++NODEJS_PACKAGES_URL	:= lndir://$(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)
>>      +NODEJS_PACKAGES_DIR	:= $(BUILDDIR)/$(NODEJS_PACKAGES)
>>      +NODEJS_PACKAGES_CACHE	:= $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn_cache
>>      +NODEJS_PACKAGES_LICENSE	:= unknown
>>
>>
>>  rules/nodejs_packages.in   | 55 +++++++++++++++++++++++
>>  rules/nodejs_packages.make | 90 ++++++++++++++++++++++++++++++++++++++
>>  2 files changed, 145 insertions(+)
>>  create mode 100644 rules/nodejs_packages.in
>>  create mode 100644 rules/nodejs_packages.make
>>
>> diff --git a/rules/nodejs_packages.in b/rules/nodejs_packages.in
>> new file mode 100644
>> index 000000000000..e6d33d30aaa9
>> --- /dev/null
>> +++ b/rules/nodejs_packages.in
>> @@ -0,0 +1,55 @@
>> +## SECTION=bytecode_engines
>> +
>> +menuconfig NODEJS_PACKAGES
>> +	tristate
>> +	select HOST_YARN
>> +	select NODEJS
>> +	prompt "Node.js packages (target)     "
>> +	help
>> +	  Reproducibly install Node.js packages that will be available
>> +	  system-wide on the target.
>> +
>> +	  You MUST specify at least one Node.js package in the submenu that is
>> +	  shown after enabling this option.
>> +
>> +
>> +if NODEJS_PACKAGES
>> +
>> +config NODEJS_PACKAGES_LIST
>> +	string
>> +	prompt "Node.js packages to install"
>> +	help
>> +	  A white-space separated list of Node.js packages including their
>> +	  versions (e.g. express@4.17.x or socket.io@'>=2.1.0 <=2.3.0'). NPM
>> +	  semvers are fully supported, see [1].
>> +
>> +	  The packages are managed with Yarn and are guaranteed to be
>> +	  reproducible and immutable as long as the download location stays
>> +	  accessible or the offline cache is not altered.
>> +
>> +	  If the list of Node.js packages to be installed has been populated for
>> +	  the first time or if it has been modified later on one needs to
>> +	  (re-)build this package to download the desired Node.js packages and
>> +	  update the offline cache. Afterwards it is highly recommended to
>> +	  commit the path '${PTXDIST_WORKSPACE}/local_src/nodejs_packages/' into
>> +	  the version control system and to lock the package cache.
>> +
>> +	  As a prerequisite the path '${PTXDIST_WORKSPACE}/local_src/nodejs_
>> +	  packages/' must exist in the BSP *before* building this package.
>> +
>> +	  [1] https://legacy.yarnpkg.com/en/docs/dependency-versions/
>> +
>> +config NODEJS_PACKAGES_OFFLINE
>> +	bool
>> +	prompt "lock offline cache for offline/production use"
>> +	help
>> +	  This option locks the offline cache for installing the nodejs packages
>> +	  and allows their installation during the build of the BSP without any
>> +	  need for internet connectivity.
>> +
>> +	  Enabling this option is highly recommended for production use.
>> +
>> +	  Do NOT forget to commit the path '${PTXDIST_WORKSPACE}/local_src/
>> +	  nodejs_packages/' into the BSP's version control system!
>> +
>> +endif
>> diff --git a/rules/nodejs_packages.make b/rules/nodejs_packages.make
>> new file mode 100644
>> index 000000000000..514db3252279
>> --- /dev/null
>> +++ b/rules/nodejs_packages.make
>> @@ -0,0 +1,90 @@
>> +# -*-makefile-*-
>> +#
>> +# Copyright (C) 2020 by Bjoern Esser <bes@pengutronix.de>
>> +#
>> +# For further information about the PTXdist project and license conditions
>> +# see the README file.
>> +#
>> +
>> +#
>> +# We provide this package
>> +#
>> +PACKAGES-$(PTXCONF_NODEJS_PACKAGES) += nodejs_packages
>> +
>> +#
>> +# Paths and names
>> +#
>> +NODEJS_PACKAGES_VERSION	:= 0.0.1
>> +NODEJS_PACKAGES		:= nodejs_packages-$(NODEJS_PACKAGES_VERSION)
>> +NODEJS_PACKAGES_LOCAL	:= local_src/nodejs_packages
>> +NODEJS_PACKAGES_URL	:= lndir://$(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)
>> +NODEJS_PACKAGES_DIR	:= $(BUILDDIR)/$(NODEJS_PACKAGES)
>> +NODEJS_PACKAGES_CACHE	:= $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn_cache
>> +NODEJS_PACKAGES_LICENSE	:= unknown
>
> Kconfig option for the license?

That's a good idea as the set of to be installed Node.js packages is
parameterized as well. Nevertheless the new Kconfig option should get a
default of "unknown" so that our users are hopefully going to stumble
across it if they did not change it manually and study the automatically
generated license report later on.

Ulrich


>> +
>> +NODEJS_PACKAGES_LIST	:= $(call remove_quotes, $(PTXCONF_NODEJS_PACKAGES_LIST))
>> +
>> +YARN_LOCK	:= $(PTXDIST_WORKSPACE)/$(NODEJS_PACKAGES_LOCAL)/yarn.lock
>> +YARN_OPTS	:= \
>> +	--cwd "$(NODEJS_PACKAGES_DIR)" \
>> +	--cache-folder "$(NODEJS_PACKAGES_CACHE)" \
>> +	--link-duplicates \
>> +	--production=true
>> +
>> +ifdef PTXCONF_NODEJS_PACKAGES_OFFLINE
>> +YARN_OPTS	+= \
>> +	--frozen-lockfile \
>> +	--offline
>> +endif
>
> Hmmm, I'm not sure if I like this option, but I don't have a good idea
> either.
>
>> +
>> +# ----------------------------------------------------------------------------
>> +# Compile
>> +# ----------------------------------------------------------------------------
>> +
>> +$(STATEDIR)/nodejs_packages.compile:
>> +	@$(call targetinfo)
>> +	mkdir -p $(NODEJS_PACKAGES_CACHE)
>> +	printf "{ \
>> +		\"name\": \"nodejs_packages\", \
>> +		\"version\": \"$(NODEJS_PACKAGES_VERSION)\", \
>> +		\"license\": \"UNLICENSED\", \
>> +		\"private\": true \
>> +	}" > $(NODEJS_PACKAGES_DIR)/package.json
>> +	if [[ ! -f $(YARN_LOCK) ]]; then \
>
> single [ is sufficient.
>
>> +		touch $(YARN_LOCK); \
>> +		ln -fs $(YARN_LOCK) $(NODEJS_PACKAGES_DIR)/yarn.lock; \
>> +	fi
>> +	yarn $(YARN_OPTS) add $(NODEJS_PACKAGES_LIST)
>> +	find $(NODEJS_PACKAGES_CACHE) -type f -name '.yarn-tarball.tgz' -delete
>> +	@$(call touch)
>> +
>> +# ----------------------------------------------------------------------------
>> +# Install
>> +# ----------------------------------------------------------------------------
>> +
>> +$(STATEDIR)/nodejs_packages.install:
>> +	@$(call targetinfo)
>> +	install -dm 0755 $(NODEJS_PACKAGES_PKGDIR)/usr/lib
>> +	cp -pr $(NODEJS_PACKAGES_DIR)/node_modules $(NODEJS_PACKAGES_PKGDIR)/usr/lib
>
> You need to ensure that NODEJS_PACKAGES_PKGDIR is empty. Use world/execute
> for the first and just execute for the second.
>
> Michael
>
>> +	@$(call touch)
>> +
>> +# ----------------------------------------------------------------------------
>> +# Target-Install
>> +# ----------------------------------------------------------------------------
>> +
>> +$(STATEDIR)/nodejs_packages.targetinstall:
>> +	@$(call targetinfo)
>> +
>> +	@$(call install_init, nodejs_packages)
>> +	@$(call install_fixup, nodejs_packages,PRIORITY,optional)
>> +	@$(call install_fixup, nodejs_packages,SECTION,base)
>> +	@$(call install_fixup, nodejs_packages,AUTHOR,"Bjoern Esser <bes@pengutronix.de>")
>> +	@$(call install_fixup, nodejs_packages,DESCRIPTION,missing)
>> +
>> +	$(call install_tree, nodejs_packages, 0, 0, -, /usr/lib/node_modules/)
>> +
>> +	@$(call install_finish, nodejs_packages)
>> +
>> +	@$(call touch)
>> +
>> +# vim: syntax=make
>> --
>> 2.25.0
>>
>>
>> _______________________________________________
>> ptxdist mailing list
>> ptxdist@pengutronix.de
-- 
Pengutronix e.K.                           | Ulrich Ölmann               |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de