[ptxdist] [PATCH 0/6] New BSP from template

[ptxdist] [PATCH 0/6] New BSP from template

[Ladislav Michl]

Running freshly deployed PTXdist currently gives you:

ladis@lenoch:/tmp/test_bsp$ ~/src/ptxdist/bin/ptxdist menuconfig

ptxdist: error: PTXdist in /home/ladis/src/ptxdist is not built.

Simply put:
"Why on _earth_ would you need to install source code before you're
allowed to configure and compile it?  The only thing it actually
built when I did the first "make" was the kconfig binaries.  The
kernel (where kconfig comes from) does not require you to make and
install kconfig before using menuconfig.  Neither do long-term users
of it like busybox or uClibc..." -- Rob Landley
(https://sourceware.org/legacy-ml/crossgcc/2009-04/msg00038.html)

I'd suggest read whole Design discussion archived at:
https://sourceware.org/legacy-ml/crossgcc/2009-04/msg00015.html
before reading the rest or even looking at the patchset.

I had this in mind for a quite long time and was carrying patches
to achieve that, but still was not quite happy with them until last
rainy weekend (figuring out where I read above rant was a pretty hard
task itself...)

Back then I was making excuses PTXdist still being in development,
which was, with respect to PTXdist age at that time, a bit silly, so
now the following patchset brings you this:

ladis@lenoch:/tmp/test_bsp$ ~/src/ptxdist/bin/ptxdist menuconfig
ptxdist: preparing host tools in '/home/ladis/.cache/ptxdist/git' ...

ptxdist: there is no PTXdist project in '/tmp/test_bsp'
ptxdist: create a new one? [y/N] y

ptxdist: project name.................: test_bsp
ptxdist: select option by number:
ptxdist: [1] ALPHA
ptxdist: [2] AVR32
ptxdist: [3] AVR
ptxdist: [4] ARM
ptxdist: [5] ARM64
ptxdist: [6] BLACKFIN
ptxdist: [7] X86
ptxdist: [8] MINGW
ptxdist: [9] PPC
ptxdist: [10] M68K
ptxdist: [11] SPARC
ptxdist: [12] MICROBLAZE
ptxdist: [13] MIPS
ptxdist: [14] CRIS
ptxdist: [15] PARISC
ptxdist: [16] SH
ptxdist: [17] RISCV
ptxdist: [18] LOONGARCH
ptxdist: architecture.................: 13
ptxdist: platform name................: mips

ptxdist: creating project 'test_bsp'
ptxdist: creating platform 'mips' (mips)

[here menuconfig is presented]

ptxdist: done. There is no toolchain yet, select one with
ptxdist: 'ptxdist toolchain <path>'. 'ptxdist platformconfig' refines
ptxdist: the platform, 'ptxdist newpackage <type>' adds own packages.

*** End of the configuration.
*** Execute 'ptxdist go' to build the project or try 'ptxdist help'.

And that's it. Previous configure && make && make install still works
the same as before in case you are installing system-wide or into
a docker image.

The following patchset it trying to address all Rob's remarks about
PTXdist, except those the time invalidated: flex and bison are now
required even by kernel, so we still do the same.

Comments and suggestions welcome, as always.

Ladislav Michl (6):
  scripts: add ptxdist_bootstrap.sh with out-of-tree build support
  ptxdist: introduce PTXDIST_HOSTTOOLS_DIR
  ptxdist: prepare host tools on demand
  ptxdist: don't depend on bash being /bin/bash
  configure: use ptxdist_bootstrap.sh to check the host
  ptxdist: let new project raise from template

 .gitignore                             |   1 +
 Makefile.in                            |  65 ++---
 README                                 |  16 +-
 bin/ptxdist                            | 259 ++++++++++++++++++--
 bin/ptxdist-auto-version               |   4 +-
 configure.ac                           | 314 +++---------------------
 doc/ref_parameter.rst                  |   4 +
 rules/host-qemu.make                   |   2 +-
 rules/other/Toplevel.make              |   6 +-
 rules/templates/project/gitignore      |   9 +
 rules/templates/project/platformconfig |  10 +
 rules/templates/project/ptxconfig      |   6 +
 scripts/lib/ptxd_lib_kconfig.sh        |   6 +-
 scripts/lib/ptxd_make_nested.sh        |   2 +-
 scripts/lib/ptxd_make_report.sh        |   2 +-
 scripts/ptxdist_bootstrap.sh           | 315 +++++++++++++++++++++++++
 16 files changed, 661 insertions(+), 360 deletions(-)
 create mode 100644 rules/templates/project/gitignore
 create mode 100644 rules/templates/project/platformconfig
 create mode 100644 rules/templates/project/ptxconfig
 create mode 100755 scripts/ptxdist_bootstrap.sh

-- 
2.47.3

[ptxdist] [PATCH 1/6] scripts: add ptxdist_bootstrap.sh with out-of-tree build support

[Ladislav Michl]

From: Ladislav Michl <ladis@triops.cz>

The bootstrap script makes ready everything PTXdist needs from the host:
- links to the GNU tools in <builddir>/bin
- kconfig in <builddir>/scripts/kconfig
All the missing prerequisites are reported at once.

Add support for out-of-tree builds to the Makefile; all generated files
ends in the build directory, 'install' and 'dist' targets are only
supported in-tree.

Signed-off-by: Ladislav Michl <ladis@triops.cz>
---
 .gitignore                   |   1 +
 Makefile.in                  |  65 +++-------
 configure.ac                 |   2 +-
 scripts/ptxdist_bootstrap.sh | 230 +++++++++++++++++++++++++++++++++++
 4 files changed, 250 insertions(+), 48 deletions(-)
 create mode 100755 scripts/ptxdist_bootstrap.sh

diff --git a/.gitignore b/.gitignore
index 1e2cd1c31..79657ecab 100644
--- a/.gitignore
+++ b/.gitignore
@@ -24,3 +24,4 @@ man/ptxdist.1.gz
 /env/
 
 /scripts/report/__pycache__/
+/build.log
diff --git a/Makefile.in b/Makefile.in
index da8cb9d7c..5eb7b407b 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -1,9 +1,10 @@
-version		:= $(shell . ./scripts/ptxdist_version.sh && echo $${PTXDIST_VERSION_FULL})
+version		:= $(shell PTXDIST_TOPDIR=@abs_srcdir@ && . @abs_srcdir@/scripts/ptxdist_version.sh && echo $${PTXDIST_VERSION_FULL})
 project		:= ptxdist-$(version)
 
 prefix		:= @prefix@
 exec_prefix	:= @exec_prefix@
 abs_srcdir 	:= @abs_srcdir@
+abs_builddir	:= @abs_builddir@
 libdir	 	:= @libdir@
 bindir	 	:= @bindir@
 instdir		:= $(libdir)/$(project)
@@ -13,49 +14,13 @@ RST2MAN		:= @RST2MAN@
 
 export SHELL
 
-all: kconfig environment
-	@touch .done
-
-kconfig:
-	@echo "building conf and mconf ..."
-	$(MAKE) -C "$(abs_srcdir)/scripts/kconfig"
+all:
+	@echo "preparing PTXdist host tools ..."
+	@CC="@CC@" CFLAGS="@CFLAGS@" CPPFLAGS="@CPPFLAGS@" LDFLAGS="@LDFLAGS@" \
+		LEX="@LEX@" YACC="@YACC@" \
+		"@BASH@" "$(abs_srcdir)/scripts/ptxdist_bootstrap.sh" "$(abs_builddir)"
 	@echo "done."
 
-environment:
-	@echo -n "preparing PTXdist environment ..."
-	@ln -sf @AWK@ "$(abs_srcdir)/bin/awk"
-	@ln -sf @GNU_chmod@ "$(abs_srcdir)/bin/chmod"
-	@ln -sf @GNU_chown@ "$(abs_srcdir)/bin/chown"
-	@ln -sf @GNU_mv@ "$(abs_srcdir)/bin/mv"
-	@ln -sf @GNU_cp@ "$(abs_srcdir)/bin/cp"
-	@ln -sf @GNU_rm@ "$(abs_srcdir)/bin/rm"
-	@ln -sf @GNU_ln@ "$(abs_srcdir)/bin/ln"
-	@ln -sf @GNU_rmdir@ "$(abs_srcdir)/bin/rmdir"
-	@ln -sf @GNU_md5sum@ "$(abs_srcdir)/bin/md5sum"
-	@ln -sf @GNU_mkdir@ "$(abs_srcdir)/bin/mkdir"
-	@ln -sf @GNU_mktemp@ "$(abs_srcdir)/bin/mktemp"
-	@ln -sf @GNU_install@ "$(abs_srcdir)/bin/install"
-	@ln -sf @GNU_stat@ "$(abs_srcdir)/bin/stat"
-	@ln -sf @GNU_mknod@ "$(abs_srcdir)/bin/mknod"
-	@ln -sf @GNU_tty@ "$(abs_srcdir)/bin/tty"
-	@ln -sf @GNU_tar@ "$(abs_srcdir)/bin/tar"
-	@ln -sf @GNU_find@ "$(abs_srcdir)/bin/find"
-	@ln -sf @PTXDIST_FILE@ "$(abs_srcdir)/bin/file"
-	@ln -sf @GNU_cat@ "$(abs_srcdir)/bin/cat"
-	@ln -sf @GNU_dirname@ "$(abs_srcdir)/bin/dirname"
-	@ln -sf @GNU_readlink@ "$(abs_srcdir)/bin/readlink"
-	@ln -sf @GNU_realpath@ "$(abs_srcdir)/bin/realpath"
-	@ln -sf @GNU_sort@ "$(abs_srcdir)/bin/sort"
-	@ln -sf @SED@ "$(abs_srcdir)/bin/sed"
-	@ln -sf @GNU_xargs@ "$(abs_srcdir)/bin/xargs"
-	@ln -sf @BASH@  "$(abs_srcdir)/bin/bash"
-	@ln -sf @SYSCTL@ "$(abs_srcdir)/bin/sysctl"
-	@ln -sf @MAKE@ "$(abs_srcdir)/bin/make"
-	@if [ @SPHINX@ != "sphinx-build" ]; then \
-		ln -sf @SPHINX@ "$(abs_srcdir)/bin/sphinx-build"; \
-	fi
-	@echo " done"
-
 man: man/ptxdist.1.gz
 man/ptxdist.1: doc/ptxdist.man doc/ref_parameter.rst
 	@mkdir -pv man
@@ -66,16 +31,22 @@ man/ptxdist.1.gz: man/ptxdist.1
 	gzip -f $<
 
 clean:
-	@rm -f .done
-	@find "$(abs_srcdir)/bin" -type l -print0 | xargs -0 rm -f
-	@$(MAKE) -C "$(abs_srcdir)/scripts/kconfig" clean
+	@rm -f "$(abs_builddir)/.done"
+	@find "$(abs_builddir)/bin" -type l -print0 | xargs -0 rm -f
+	@$(MAKE) -C "$(abs_builddir)/scripts/kconfig" \
+		-f "$(abs_srcdir)/scripts/kconfig/Makefile" clean
+
+intree-check:
+	@if [ "$(abs_srcdir)" != "$(abs_builddir)" ]; then \
+		echo "install and dist are only supported for in-tree builds!" ; exit 1 ; \
+	fi
 
-dirty-check:
+dirty-check: intree-check
 	@case "$(version)" in \
 		*-dirty) echo "refusing to install or package a dirty git tree!" ; exit 1 ;; \
 	esac
 
-install: all dirty-check
+install: intree-check all dirty-check
 	@echo "installing PTXdist to $(DESTDIR)$(prefix)..."
 	@rm -fr   "$(DESTDIR)$(instdir)"
 	@mkdir -p "$(DESTDIR)$(instdir)"
diff --git a/configure.ac b/configure.ac
index c368e46b1..2590ebbed 100644
--- a/configure.ac
+++ b/configure.ac
@@ -15,7 +15,7 @@ dnl
 dnl Check if the user has extracted the patches archive
 dnl
 AC_MSG_CHECKING(for ptxdist patches)
-if test -d `dirname $0`/patches
+if test -d "$srcdir/patches"
 then
 	AC_MSG_RESULT(yes)
 else
diff --git a/scripts/ptxdist_bootstrap.sh b/scripts/ptxdist_bootstrap.sh
new file mode 100755
index 000000000..b9f4c7a5c
--- /dev/null
+++ b/scripts/ptxdist_bootstrap.sh
@@ -0,0 +1,230 @@
+#!/bin/sh
+#
+# Prepare the host tools PTXdist needs:
+# - links to sane (GNU) tools found on the host in <builddir>/bin
+# - kconfig (conf, mconf and, if possible, nconf) in <builddir>/scripts/kconfig
+#
+# All missing prerequisites are reported at once. Only tools that PTXdist
+# itself needs are checked, everything required by some packages only is
+# left to the package build.
+#
+# usage: ptxdist_bootstrap.sh <builddir>
+#
+# CC, CFLAGS, CPPFLAGS, LDFLAGS, LEX, YACC and PKG_CONFIG from the
+# environment are used if set.
+#
+
+PTXDIST_LOG_PROMPT="ptxdist: "
+
+#
+# Everything up to the bash version check must be plain POSIX sh: the
+# script may be started by any shell and re-executes itself with bash.
+#
+if [ -z "${BASH_VERSION}" ]; then
+	bash="$(command -v bash)" || {
+		echo "${PTXDIST_LOG_PROMPT}error: bash >= 4.2 is required, please install it" >&2
+		exit 1
+	}
+	exec "${bash}" "${0}" "${@}"
+fi
+
+case "${BASH_VERSION}" in
+[0-3].*|4.[01].*)
+	echo "${PTXDIST_LOG_PROMPT}error: bash >= 4.2 is required, found ${BASH_VERSION} as '${BASH}'" >&2
+	exit 1
+	;;
+esac
+
+# e.g. when started as 'sh' with /bin/sh -> bash
+set +o posix
+
+topdir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" || exit
+builddir="${1}"
+if [ -z "${builddir}" ]; then
+	echo "usage: ${0} <builddir>" >&2
+	exit 1
+fi
+
+declare -A links
+declare -a missing
+
+#
+# $1: version found, $2: version required
+#
+version_ge() {
+	local -a have want
+	local i
+
+	IFS=. read -r -a have <<< "${1}"
+	IFS=. read -r -a want <<< "${2}"
+	for (( i = 0; i < ${#want[@]}; i++ )); do
+		if [ "${have[i]:-0}" -gt "${want[i]}" ]; then
+			return 0
+		elif [ "${have[i]:-0}" -lt "${want[i]}" ]; then
+			return 1
+		fi
+	done
+}
+
+#
+# find a GNU tool and register a link for it
+#
+# $1: link name
+# $2: expected '--version' output, e.g. 'GNU coreutils'
+# $3: minimum version, may be empty
+# $4: package that provides it (to print the hint)
+# $5...: names to look for in PATH
+#
+find_gnu() {
+	local link="${1}" id="${2}" min="${3}" pkg="${4}"
+	local name path version
+	shift 4
+
+	for name in "${@}"; do
+		path="$(type -P "${name}")" || continue
+		version="$("${path}" --version 2>/dev/null | head -n 1)"
+		[[ "${version}" =~ ${id} ]] || continue
+		if [ -n "${min}" ]; then
+			[[ "${version}" =~ ([0-9]+(\.[0-9]+)+) ]] &&
+			version_ge "${BASH_REMATCH[1]}" "${min}" || continue
+		fi
+		links[${link}]="${path}"
+		return
+	done
+	missing+=( "${id}${min:+ >= ${min}} as '${link}' (package: ${pkg})" )
+}
+
+for tool in cat chmod chown cp dirname install ln md5sum mkdir mknod \
+		mktemp mv readlink realpath rm rmdir sort stat tty; do
+	find_gnu "${tool}" "GNU coreutils" "" coreutils "g${tool}" "${tool}"
+done
+find_gnu tar	"GNU tar"	""	tar		gtar tar
+find_gnu find	"GNU findutils"	4	findutils	gfind find
+find_gnu xargs	"GNU findutils"	4	findutils	gxargs xargs
+find_gnu sed	"GNU sed"	4	sed		gsed sed
+find_gnu awk	"GNU Awk"	""	gawk		gawk awk
+find_gnu make	"GNU Make"	3.81	make		gmake gnumake make
+
+links[bash]="${BASH}"
+
+if path="$(type -P file)"; then
+	links[file]="${path}"
+else
+	missing+=( "'file' (package: file)" )
+fi
+
+# optional
+links[sysctl]="$(type -P sysctl || type -P false)"
+if path="$(type -P sphinx-build)"; then
+	links[sphinx-build]="${path}"
+fi
+
+#
+# kconfig
+#
+if [ -z "${CC}" ]; then
+	CC="$(type -P gcc || type -P cc)" || CC=cc
+fi
+if ! type -P "${CC%% *}" > /dev/null; then
+	missing+=( "C compiler '${CC}' (package: gcc)" )
+fi
+: ${LEX:=flex} ${YACC:=bison} ${PKG_CONFIG:=pkg-config}
+if ! type -P "${LEX%% *}" > /dev/null; then
+	missing+=( "'${LEX}' (package: flex)" )
+fi
+if ! type -P "${YACC%% *}" > /dev/null; then
+	missing+=( "'${YACC}' (package: bison)" )
+fi
+
+#
+# $1: test program; possibly followed by compiler and linker flags
+#
+try_link() {
+	local prog="${1}"
+	shift
+	echo "${prog}" |
+		${CC} ${CFLAGS} ${CPPFLAGS} -x c - -o /dev/null ${LDFLAGS} "${@}" > /dev/null 2>&1
+}
+
+#
+# $1: pkg-config modules (space separated, all required)
+# $2: fallback libs
+# $3: test program
+# out: libs_cflags, libs_libs
+#
+find_libs() {
+	local modules="${1}" fallback="${2}" prog="${3}"
+
+	if ${PKG_CONFIG} --exists ${modules} 2>/dev/null; then
+		libs_cflags="$(${PKG_CONFIG} --cflags ${modules})"
+		libs_libs="$(${PKG_CONFIG} --libs ${modules})"
+	else
+		libs_cflags=""
+		libs_libs="${fallback}"
+	fi
+	try_link "${prog}" ${libs_cflags} ${libs_libs}
+}
+
+ncurses_prog='#include <ncurses.h>
+int main(void) { initscr(); endwin(); return 0; }'
+nconf_prog='#include <ncurses.h>
+#include <menu.h>
+#include <panel.h>
+int main(void) { initscr(); new_panel(stdscr); free_menu(NULL); return 0; }'
+
+if type -P "${CC%% *}" > /dev/null; then
+	if find_libs "ncursesw" "-lncursesw" "${ncurses_prog}" ||
+		find_libs "ncurses" "-lncurses" "${ncurses_prog}"; then
+		mconf_cflags="${libs_cflags}"
+		mconf_libs="${libs_libs}"
+	else
+		missing+=( "ncurses development files (package: libncurses-dev or ncurses-devel)" )
+	fi
+	# nconf is optional
+	if find_libs "ncursesw menuw panelw" "-lmenuw -lpanelw -lncursesw" "${nconf_prog}" ||
+		find_libs "ncurses menu panel" "-lmenu -lpanel -lncurses" "${nconf_prog}"; then
+		nconf=nconf
+		nconf_libs="${libs_libs}"
+	fi
+fi
+
+if [ ${#missing[@]} -gt 0 ]; then
+	{
+		echo
+		echo "${PTXDIST_LOG_PROMPT}error: missing host tools, please install:"
+		echo
+		printf "  - %s\n" "${missing[@]}"
+		echo
+	} >&2
+	exit 1
+fi
+
+#
+# all prerequisites are there, set up the build directory
+#
+mkdir -p "${builddir}/bin" "${builddir}/scripts/kconfig" &&
+for link in "${!links[@]}"; do
+	ln -sfn "${links[${link}]}" "${builddir}/bin/${link}" || exit
+done
+
+if ! "${links[make]}" -C "${builddir}/scripts/kconfig" \
+	-f "${topdir}/scripts/kconfig/Makefile" \
+	CC="${CC}" \
+	${CFLAGS:+CFLAGS="${CFLAGS}"} \
+	${CPPFLAGS:+CPPFLAGS="${CPPFLAGS}"} \
+	${LDFLAGS:+LDFLAGS="${LDFLAGS}"} \
+	LEX="${LEX}" \
+	YACC="${YACC}" \
+	MCONF_CFLAGS="${mconf_cflags}" \
+	MCONF_LIBS="${mconf_libs}" \
+	NCONF_LIBS="${nconf_libs}" \
+	NCONF="${nconf}" \
+	> "${builddir}/build.log" 2>&1; then
+	tail -n 20 "${builddir}/build.log" >&2
+	echo >&2
+	echo "${PTXDIST_LOG_PROMPT}error: building kconfig failed, see '${builddir}/build.log'" >&2
+	echo >&2
+	exit 1
+fi
+
+echo "${topdir}" > "${builddir}/.done"
-- 
2.47.3

[ptxdist] [PATCH 2/6] ptxdist: introduce PTXDIST_HOSTTOOLS_DIR

[Ladislav Michl]

From: Ladislav Michl <ladis@triops.cz>

Store the links to the host tools and kconfig in a separate variable
for now the same as PTXDIST_TOPDIR.

Signed-off-by: Ladislav Michl <ladis@triops.cz>
---
 bin/ptxdist                     | 7 +++++--
 rules/host-qemu.make            | 2 +-
 rules/other/Toplevel.make       | 6 +++---
 scripts/lib/ptxd_lib_kconfig.sh | 6 +++---
 scripts/lib/ptxd_make_nested.sh | 2 +-
 scripts/lib/ptxd_make_report.sh | 2 +-
 6 files changed, 14 insertions(+), 11 deletions(-)

diff --git a/bin/ptxdist b/bin/ptxdist
index 2557cb9a2..94c41412f 100755
--- a/bin/ptxdist
+++ b/bin/ptxdist
@@ -2118,6 +2118,7 @@ _realpath()
 #
 # out: PTXDIST
 #      PTXDIST_TOPDIR
+#      PTXDIST_HOSTTOOLS_DIR
 #      PTXDIST_WORKSPACE
 #      PTXDIST_PTXCONFIG_DEFAULT
 #      PTXDIST_PLATFORMCONFIG_DEFAULT
@@ -2142,6 +2143,7 @@ setup_topdir() {
 
 	PTXDIST="${ptxdist}"
 	PTXDIST_TOPDIR="${topdir}"
+	PTXDIST_HOSTTOOLS_DIR="${topdir}"
 	PTXDIST_WORKSPACE="$(pwd)"
 
 	eval PTXDIST_PTXCONFIG_DEFAULT="${PTXDIST_PTXCONFIG_DEFAULT}"
@@ -2149,8 +2151,8 @@ setup_topdir() {
 	eval PTXDIST_COLLECTIONCONFIG_DEFAULT="${PTXDIST_COLLECTIONCONFIG_DEFAULT}"
 	eval PTXDIST_TOOLCHAIN_DEFAULT="${PTXDIST_TOOLCHAIN_DEFAULT}"
 
-	# ${PTXDIST_TOPDIR}/bin contains links to sane tools found by configure
-	export PATH="${PTXDIST_TOPDIR}/bin:${PATH}"
+	# ${PTXDIST_HOSTTOOLS_DIR}/bin contains links to sane tools found by configure
+	export PATH="${PTXDIST_HOSTTOOLS_DIR}/bin:${PATH}"
 }
 
 
@@ -2727,6 +2729,7 @@ setup_export() {
 		\
 		PTXDIST \
 		PTXDIST_TOPDIR \
+		PTXDIST_HOSTTOOLS_DIR \
 		PTXDIST_SRCDIR \
 		PTXDIST_TEMPDIR \
 		PTXDIST_LIB_DIR \
diff --git a/rules/host-qemu.make b/rules/host-qemu.make
index eef2dc133..77cd8dcea 100644
--- a/rules/host-qemu.make
+++ b/rules/host-qemu.make
@@ -249,7 +249,7 @@ ifdef PTXCONF_HOST_QEMU_USR
 		-e 's|RTLDLIST=.*|RTLDLIST="$(PTXDIST_SYSROOT_TOOLCHAIN)$(QEMU_CROSS_DL)"|' \
 		-e 's|eval $$add_env|eval $(PTXDIST_SYSROOT_CROSS)/usr/bin/qemu-cross -E "$${add_env// /,}"|' \
 		-e 's|verify_out=`|verify_out=`$(PTXDIST_SYSROOT_CROSS)/usr/bin/qemu-cross |' \
-		-e 's|#! */.*$$|#!$(shell readlink $(PTXDIST_TOPDIR)/bin/bash)|' \
+		-e 's|#! */.*$$|#!$(shell readlink $(PTXDIST_HOSTTOOLS_DIR)/bin/bash)|' \
 		$(PTXDIST_SYSROOT_TOOLCHAIN)/usr/bin/ldd > $(PTXDIST_SYSROOT_CROSS)/usr/bin/qemu/ldd
 	@chmod +x $(PTXDIST_SYSROOT_CROSS)/usr/bin/qemu/ldd
 endif
diff --git a/rules/other/Toplevel.make b/rules/other/Toplevel.make
index 44a030825..5dec61bb3 100644
--- a/rules/other/Toplevel.make
+++ b/rules/other/Toplevel.make
@@ -8,7 +8,7 @@ ifneq ($(findstring n,$(filter-out --%,$(MAKEFLAGS))),)
 MAKE=true
 SHELL=true
 define ptx/force-shell
-$(eval SHELL=$(realpath $(PTXDIST_TOPDIR)/bin/bash))$(shell $(1))$(eval SHELL=true)
+$(eval SHELL=$(realpath $(PTXDIST_HOSTTOOLS_DIR)/bin/bash))$(shell $(1))$(eval SHELL=true)
 endef
 define ptx/force-sh
 $(eval SHELL=/bin/sh)$(shell $(1))$(eval SHELL=true)
@@ -17,12 +17,12 @@ define ptx/sh
 endef
 else
 # make sure bash is used to execute commands from makefiles
-SHELL=$(realpath $(PTXDIST_TOPDIR)/bin/bash)
+SHELL=$(realpath $(PTXDIST_HOSTTOOLS_DIR)/bin/bash)
 define ptx/force-shell
 $(shell $(1))
 endef
 define ptx/force-sh
-$(eval SHELL=/bin/sh)$(shell $(1))$(eval SHELL=$(realpath $(PTXDIST_TOPDIR)/bin/bash))
+$(eval SHELL=/bin/sh)$(shell $(1))$(eval SHELL=$(realpath $(PTXDIST_HOSTTOOLS_DIR)/bin/bash))
 endef
 define ptx/sh
 $(call ptx/force-sh,$(1))
diff --git a/scripts/lib/ptxd_lib_kconfig.sh b/scripts/lib/ptxd_lib_kconfig.sh
index 0cec1edac..625a32c8a 100644
--- a/scripts/lib/ptxd_lib_kconfig.sh
+++ b/scripts/lib/ptxd_lib_kconfig.sh
@@ -781,9 +781,9 @@ ptxd_kconfig() {
 	;;
     esac
 
-    local conf="${PTXDIST_TOPDIR}/scripts/kconfig/conf"
-    local mconf="${PTXDIST_TOPDIR}/scripts/kconfig/mconf"
-    local nconf="${PTXDIST_TOPDIR}/scripts/kconfig/nconf"
+    local conf="${PTXDIST_HOSTTOOLS_DIR}/scripts/kconfig/conf"
+    local mconf="${PTXDIST_HOSTTOOLS_DIR}/scripts/kconfig/mconf"
+    local nconf="${PTXDIST_HOSTTOOLS_DIR}/scripts/kconfig/nconf"
 
     export \
 	KCONFIG_NOTIMESTAMP="1" \
diff --git a/scripts/lib/ptxd_make_nested.sh b/scripts/lib/ptxd_make_nested.sh
index 54535377f..35087034c 100644
--- a/scripts/lib/ptxd_make_nested.sh
+++ b/scripts/lib/ptxd_make_nested.sh
@@ -47,7 +47,7 @@ ptxd_make_nested_ptxdist() {
     args=( "cd" "${pkg_workspace}" "&&" "${PTXDIST_TOPDIR}/bin/ptxdist" )
 
     PTXDIST_TOOLCHAIN="$(readlink "${PTXDIST_PLATFORMDIR}/selected_toolchain")"
-    PATH=$(sed -e 's;[^:]*/sysroot-\(host\|cross\)/[^:]*:;;g' -e "s;${PTXDIST_TOPDIR}/bin:;;" -e "s;${PTXDIST_TOOLCHAIN}:;;" <<< "${PATH}")
+    PATH=$(sed -e 's;[^:]*/sysroot-\(host\|cross\)/[^:]*:;;g' -e "s;${PTXDIST_HOSTTOOLS_DIR}/bin:;;" -e "s;${PTXDIST_TOOLCHAIN}:;;" <<< "${PATH}")
 
     if [ "${PTXDIST_WORKSPACE}" != "${pkg_workspace}" ]; then
 	unset PTXDIST_PLATFORMDIR
diff --git a/scripts/lib/ptxd_make_report.sh b/scripts/lib/ptxd_make_report.sh
index 7eea60900..7f8ad07ad 100644
--- a/scripts/lib/ptxd_make_report.sh
+++ b/scripts/lib/ptxd_make_report.sh
@@ -24,7 +24,7 @@ ptxd_make_bsp_report_header() {
     do_path() {
 	local orig_IFS="${IFS}"
 	# only the paths added by PTXdist
-	local path="${2%:${PTXDIST_TOPDIR}/bin:*}:${PTXDIST_TOPDIR}/bin"
+	local path="${2%:${PTXDIST_HOSTTOOLS_DIR}/bin:*}:${PTXDIST_HOSTTOOLS_DIR}/bin"
 	echo "${indent}${1}"
 	awk "BEGIN { RS=\":\" } { if (\$1) print \"${indent}- '\" \$1 \"'\" }" <<<"${path}"
     }
-- 
2.47.3

[ptxdist] [PATCH 3/6] ptxdist: prepare host tools on demand

[Ladislav Michl]

From: Ladislav Michl <ladis@triops.cz>

Run ptxdist_bootstrap.sh on first use if PTXdist is neither installed
nor configured in-tree. The PTXdist tree itself is never written to.

Concurrent PTXdist runs are serialized using lock file, host tools are
prepared again if they were built for another PTXdist tree or a linked
host tool vanished.

Signed-off-by: Ladislav Michl <ladis@triops.cz>
---
 README      | 13 ++++++--
 bin/ptxdist | 86 +++++++++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 84 insertions(+), 15 deletions(-)

diff --git a/README b/README
index debf098d8..5774e046a 100644
--- a/README
+++ b/README
@@ -20,9 +20,16 @@ to install it. When you start using PTXdist, make sure your $PATH
 environment variable points to <installpath>/bin, because that's where
 the PTXdist frontend program is being installed to.
 
-For developers who want to work with git versions of PTXdist it is only
-necessary to run "./autogen.sh && ./configure && make" and set the PATH
-variable to the bin/ directory inside the PTXdist tree.
+Installing is optional. PTXdist can be used directly from an unpacked
+tarball or a git checkout, without any configure or build step:
+
+	cd <your BSP>
+	/path/to/ptxdist/bin/ptxdist menuconfig
+
+On first use, PTXdist checks the host for everything it needs, reports
+all missing tools at once, and builds kconfig in .ptxdist/<version>/
+(.ptxdist/git/ for git checkouts) inside the BSP. The PTXdist tree
+itself is never written to.
 
 Idea
 ----
diff --git a/bin/ptxdist b/bin/ptxdist
index 94c41412f..7b20026bb 100755
--- a/bin/ptxdist
+++ b/bin/ptxdist
@@ -2131,19 +2131,8 @@ setup_topdir() {
 	ptxdist="$(_realpath "${0}")" &&
 	topdir="$(cd "$(dirname "${ptxdist}")"/.. && pwd)" || return
 
-	#
-	# sanity check: is PTXdist already configured?
-	#
-	if [ ! -e "${topdir}/.done" ]; then
-		echo
-		echo "${PTXDIST_LOG_PROMPT}error: PTXdist in ${topdir} is not built."
-		echo
-		exit 1
-	fi
-
 	PTXDIST="${ptxdist}"
 	PTXDIST_TOPDIR="${topdir}"
-	PTXDIST_HOSTTOOLS_DIR="${topdir}"
 	PTXDIST_WORKSPACE="$(pwd)"
 
 	eval PTXDIST_PTXCONFIG_DEFAULT="${PTXDIST_PTXCONFIG_DEFAULT}"
@@ -2151,11 +2140,84 @@ setup_topdir() {
 	eval PTXDIST_COLLECTIONCONFIG_DEFAULT="${PTXDIST_COLLECTIONCONFIG_DEFAULT}"
 	eval PTXDIST_TOOLCHAIN_DEFAULT="${PTXDIST_TOOLCHAIN_DEFAULT}"
 
-	# ${PTXDIST_HOSTTOOLS_DIR}/bin contains links to sane tools found by configure
+	setup_hosttools || return
+
+	# ${PTXDIST_HOSTTOOLS_DIR}/bin contains links to sane tools found on the host
 	export PATH="${PTXDIST_HOSTTOOLS_DIR}/bin:${PATH}"
 }
 
 
+#
+# check that the host tools in ${1} are complete and built for this
+# PTXdist, that no linked host tool vanished meanwhile and that the
+# sources did not change since, e.g. after 'git pull'
+#
+_check_hosttools() {
+	local link
+
+	[ "$(cat "${1}/.done" 2>/dev/null)" = "${PTXDIST_TOPDIR}" ] || return
+	for link in "${1}/bin/"*; do
+		[ -e "${link}" ] || return
+	done
+	[ -z "$(find "${PTXDIST_TOPDIR}/scripts/kconfig" \
+		"${PTXDIST_TOPDIR}/scripts/ptxdist_bootstrap.sh" \
+		-newer "${1}/.done" -print -quit)" ]
+}
+
+#
+# prepare the host tools (kconfig and links to sane tools) out-of-tree
+# unless PTXdist is installed or configured in-tree.
+#
+# in:  PTXDIST_TOPDIR
+#      PTXDIST_WORKSPACE
+# out: PTXDIST_HOSTTOOLS_DIR
+#
+setup_hosttools() {
+	local hosttools version lock
+
+	if [ -e "${PTXDIST_TOPDIR}/.done" ]; then
+		# installed or configured in-tree
+		PTXDIST_HOSTTOOLS_DIR="${PTXDIST_TOPDIR}"
+		return
+	fi
+
+	source "${PTXDIST_TOPDIR}/scripts/ptxdist_version.sh" || return
+
+	# like ptxdistrc: one for all git versions, they change with each commit
+	version="${PTXDIST_VERSION_FULL}"
+	if [ -n "${PTXDIST_VERSION_SCM}" ]; then
+		version=git
+	fi
+
+	if [ -e "${PTXDIST_WORKSPACE}/selected_ptxconfig" -o \
+		-e "${PTXDIST_WORKSPACE}/configs/ptxconfig" ]; then
+		hosttools="${PTXDIST_WORKSPACE}/.ptxdist/${version}"
+	else
+		# not in a BSP, e.g. 'ptxdist setup'
+		hosttools="${XDG_CACHE_HOME:-${HOME}/.cache}/ptxdist/${version}"
+	fi
+	PTXDIST_HOSTTOOLS_DIR="${hosttools}"
+
+	if _check_hosttools "${hosttools}"; then
+		return
+	fi
+
+	mkdir -p "$(dirname "${hosttools}")" &&
+	exec {lock}>"${hosttools}.lock" &&
+	flock "${lock}" || return
+
+	# a concurrent PTXdist may have finished first
+	if ! _check_hosttools "${hosttools}"; then
+		echo "${PTXDIST_LOG_PROMPT}preparing host tools in '${hosttools}' ..." >&2
+		rm -rf "${hosttools}" &&
+		mkdir "${hosttools}" || return
+		"${BASH}" "${PTXDIST_TOPDIR}/scripts/ptxdist_bootstrap.sh" "${hosttools}" || exit 1
+	fi
+
+	exec {lock}>&-
+}
+
+
 ensure_ptxrc() {
 	local ptxrc tmpptxrc
 	local rc_default="${PTXDIST_TOPDIR}/config/setup/ptxdistrc.default"
-- 
2.47.3

[ptxdist] [PATCH 4/6] ptxdist: don't depend on bash being /bin/bash

[Ladislav Michl]

From: Ladislav Michl <ladis@triops.cz>

Provide a helpful message instead of 'bad interpreter' and check
bash version.

Signed-off-by: Ladislav Michl <ladis@triops.cz>
---
 bin/ptxdist              | 24 +++++++++++++++++++++++-
 bin/ptxdist-auto-version |  4 ++--
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/bin/ptxdist b/bin/ptxdist
index 7b20026bb..3c86aa225 100755
--- a/bin/ptxdist
+++ b/bin/ptxdist
@@ -1,4 +1,26 @@
-#!/bin/bash
+#!/bin/sh
+
+#
+# Everything up to the bash version check must be plain POSIX sh: PTXdist
+# may be started by any shell and re-executes itself with bash.
+#
+if [ -z "${BASH_VERSION}" ]; then
+	bash="$(command -v bash)" || {
+		echo "ptxdist: error: bash >= 4.2 is required, please install it" >&2
+		exit 1
+	}
+	exec "${bash}" "${0}" "${@}"
+fi
+
+case "${BASH_VERSION}" in
+[0-3].*|4.[01].*)
+	echo "ptxdist: error: bash >= 4.2 is required, found ${BASH_VERSION} as '${BASH}'" >&2
+	exit 1
+	;;
+esac
+
+# e.g. when started as 'sh' with /bin/sh -> bash
+set +o posix
 
 LANG=C
 LC_ALL=POSIX
diff --git a/bin/ptxdist-auto-version b/bin/ptxdist-auto-version
index 2f6414ae0..92c50fc3d 100755
--- a/bin/ptxdist-auto-version
+++ b/bin/ptxdist-auto-version
@@ -1,7 +1,7 @@
-#!/bin/bash
+#!/bin/sh
 
 #
-# bash implementation of realpath / readlink -f
+# shell implementation of realpath / readlink -f
 #   arg1 - filename
 realpath()
 {
-- 
2.47.3

[ptxdist] [PATCH 5/6] configure: use ptxdist_bootstrap.sh to check the host

[Ladislav Michl]

From: Ladislav Michl <ladis@triops.cz>

Drop checks for things nothing uses: C++ compiler, headers, types and
functions (kconfig has no config.h), the #! test, the bison++ test,
and AC_CANONICAL_*.

Move the remaining runtime checks to the bootstrap, so both ways check
the same things. While there try to unify things a bit; every host
tool used ends linked, so that all host tools PTXdist depends on are
in one place, show up in the BSP report and trigger a rebuild of the
host tools when any of them vanishes.

patch keeps its special handling: GNU patch >= 2.1 is preferred
(gpatch, then patch). Anything else is used with a warning, as before;
Sun's patch is known to be buggy and patch 2.0 cannot create directories.

CC, CFLAGS, CPPFLAGS, LDFLAGS, LEX, YACC, PATCH and PKG_CONFIG can
still be passed to configure and end up in the bootstrap when running
make.

Signed-off-by: Ladislav Michl <ladis@triops.cz>
---
 Makefile.in                  |   2 +-
 configure.ac                 | 312 ++++-------------------------------
 scripts/ptxdist_bootstrap.sh | 111 +++++++++++--
 3 files changed, 130 insertions(+), 295 deletions(-)

diff --git a/Makefile.in b/Makefile.in
index 5eb7b407b..f16a3744f 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -17,7 +17,7 @@ export SHELL
 all:
 	@echo "preparing PTXdist host tools ..."
 	@CC="@CC@" CFLAGS="@CFLAGS@" CPPFLAGS="@CPPFLAGS@" LDFLAGS="@LDFLAGS@" \
-		LEX="@LEX@" YACC="@YACC@" \
+		LEX="@LEX@" YACC="@YACC@" PATCH="@PATCH@" PKG_CONFIG="@PKG_CONFIG@" \
 		"@BASH@" "$(abs_srcdir)/scripts/ptxdist_bootstrap.sh" "$(abs_builddir)"
 	@echo "done."
 
diff --git a/configure.ac b/configure.ac
index 2590ebbed..17edbd769 100644
--- a/configure.ac
+++ b/configure.ac
@@ -8,8 +8,6 @@ AC_INIT([ptxdist],
        [m4_esyscmd([scripts/kernel/setlocalversion .tarball-version])],
        [ptxdist@pengutronix.de])
 AC_CONFIG_AUX_DIR(scripts/autoconf)
-AC_CANONICAL_BUILD
-AC_CANONICAL_HOST
 
 dnl
 dnl Check if the user has extracted the patches archive
@@ -24,62 +22,8 @@ else
 fi
 
 dnl
-dnl Check for C and C++ compilers
+dnl Check for Bourne-Again Shell, needed to run anything else
 dnl
-AC_PROG_CC
-AC_PROG_CXX
-
-dnl
-dnl Check for pkg-config
-dnl
-PKG_PROG_PKG_CONFIG
-AS_IF([test -z "$PKG_CONFIG"], AC_MSG_ERROR([pkg-config not found.]))
-
-AC_CHECK_HEADERS([fcntl.h inttypes.h libintl.h limits.h locale.h malloc.h stddef.h stdlib.h string.h sys/time.h unistd.h])
-
-AC_CHECK_HEADER([regex.h], [], AC_MSG_ERROR([Cannot find regex.h.]))
-
-PKG_CHECK_MODULES([MCONF], [ncursesw],, [
-	PKG_CHECK_MODULES([MCONF], [ncurses],, [
-		AC_MSG_ERROR([Cannot find ncurses library.])
-	])
-])
-CFLAGS="${CFLAGS} ${MCONF_CFLAGS}"
-
-dnl
-dnl Checks for typedefs, structures, and compiler characteristics.
-dnl
-AC_CHECK_HEADER_STDBOOL
-AC_C_INLINE
-AC_TYPE_INT16_T
-AC_TYPE_INT32_T
-AC_TYPE_INT8_T
-AC_TYPE_OFF_T
-AC_TYPE_SIZE_T
-AC_TYPE_SSIZE_T
-AC_TYPE_UINT16_T
-AC_TYPE_UINT32_T
-AC_TYPE_UINT8_T
-
-dnl
-dnl Checks for library functions.
-dnl
-AC_FUNC_ALLOCA
-AC_FUNC_MALLOC
-AC_FUNC_REALLOC
-AC_CHECK_FUNCS([bzero gettimeofday memmove memset mkdir regcomp setlocale strcasecmp strchr strcspn strdup strncasecmp strpbrk strrchr strspn strtol strtoull uname])
-
-
-AC_SYS_INTERPRETER
-if test "$interpval" != yes ; then
-	AC_MSG_WARN([no
-
-bash scripts may not be invoked correctly due to problems with your
-systems implementation of #! being either broken or non-existant.
-])
-fi
-
-dnl Check for Bourne-Again Shell
 unset BASH  # bash sets this itself!
 AC_ARG_WITH(bash, AS_HELP_STRING([--with-bash],[name of the bash executable to use]),
     [
@@ -93,230 +37,42 @@ if test -z "$BASH" ; then
     AC_MSG_ERROR([Please specify the location of bash with the option '--with-bash'])
 fi
 
-# It would be nice not to have to use backticks, but too many retarded sh
-# implementations still don't support $( )
-# BEWARE:  There is a distinct possibility that we are currently running under
-# bash in this configure script (/bin/sh being a symlink to /bin/bash).  Even
-# though the result /could/ be available to us directly as $BASH_VERSION we
-# don't want to use, or trust it, incase the user is specifying a different
-# bash executable.
-if `$BASH -c '[[ "$BASH_VERSION" \< "4.2" ]]'` ; then
-    AC_MSG_ERROR([
-$PACKAGE_NAME requires at least version 4.2 of bash, you can download a current
-version of bash from ftp.gnu.org
-])
-fi
-
-
-dnl Check for sed >= 4.0
-AC_PATH_PROGS(SED, gsed sed,, $PATH)
-if test -z "$SED"; then
-	AC_MSG_ERROR([sed could not be found, please install])
-fi
-
-AC_MSG_CHECKING([sed version])
-SED_VERSION=`$SED --version 2>/dev/null | $SED -ne "1 s/.*GNU.*\s\([[0-9\+\.]]\+\).*/\1/p"`
-case "$SED_VERSION" in
-3.*) AC_MSG_ERROR([we need at least GNU sed 4.x but found $SED_VERSION]) ;;
-4.*) ;;
-*)   AC_MSG_ERROR([we need at least GNU sed 4.x]) ;;
-esac
-AC_MSG_RESULT([$SED_VERSION])
-
-dnl check for gnu tools
-dnl   $1: tool to search for
-dnl   $2: package where the tool is (e.g. 'coreutils')
-AC_DEFUN([GNU_TOOL],
-	[AC_CACHE_CHECK([for GNU $1], [ac_cv_path_gnu_$1],
-		[AC_PATH_PROGS_FEATURE_CHECK([gnu_$1], [g$1 $1.gnu $1-gnu $1],
-			[[out=`$ac_path_gnu_$1 --version 2>/dev/null | $SED -ne "s/.*\(GNU\) $2.*/\1/p"`
-				test "x$out" = xGNU && ac_cv_path_gnu_$1=$ac_path_gnu_$1 ac_path_gnu_$1_found=:]],
-			[AC_MSG_ERROR([could not find GNU $1])])])
-	AC_SUBST([GNU_$1], [$ac_cv_path_gnu_$1])])
-
-GNU_TOOL(cat, coreutils)
-GNU_TOOL(chmod, coreutils)
-GNU_TOOL(chown, coreutils)
-GNU_TOOL(cp, coreutils)
-GNU_TOOL(dirname, coreutils)
-GNU_TOOL(install, coreutils)
-GNU_TOOL(ln, coreutils)
-GNU_TOOL(md5sum, coreutils)
-GNU_TOOL(mkdir, coreutils)
-GNU_TOOL(mknod, coreutils)
-GNU_TOOL(mktemp, coreutils)
-GNU_TOOL(mv, coreutils)
-GNU_TOOL(readlink, coreutils)
-GNU_TOOL(realpath, coreutils)
-GNU_TOOL(rm, coreutils)
-GNU_TOOL(rmdir, coreutils)
-GNU_TOOL(sha256sum, coreutils)
-GNU_TOOL(sort, coreutils)
-GNU_TOOL(stat, coreutils)
-GNU_TOOL(touch, coreutils)
-GNU_TOOL(tty, coreutils)
-GNU_TOOL(tar, tar)
-GNU_TOOL(find, findutils)
-GNU_TOOL(xargs, findutils)
-
-dnl Check for awk
-AC_PROG_AWK
-AC_PATH_PROGS(AWK, ${AWK},)
-AC_MSG_CHECKING([awk version])
-AWK_VERSION=`$AWK --version 2>/dev/null | $SED -ne "s/GNU [[Aa]]wk \([[0-9\.]]*\)/\1/p"`
-case "$AWK_VERSION" in
-"") AC_MSG_ERROR([we need GNU awk]) ;;
-*)  ;;
-esac
-AC_MSG_RESULT([$AWK_VERSION])
-
-# fall back to 'false' for systems which do not provide a 'sysctl' tool
-AC_PATH_PROGS(SYSCTL, sysctl false,, $PATH)
-
-dnl Check for lex
-AC_PROG_LEX(noyywrap)
-AC_PATH_PROGS(LEX, ${LEX},, $PATH)
-if test -z "${LEX}"; then
-   AC_MSG_ERROR([(f)lex could not be found, please install])
-fi
-
-dnl Check for yacc
-AC_PROG_YACC
-AC_PATH_PROGS(YACC, ${YACC},, $PATH)
-if test -z "${YACC}"; then
-   AC_MSG_ERROR([yacc/bison could not be found, please install])
-fi
-
-
-AC_MSG_CHECKING([whether yacc is bison++])
-if $YACC --version 2>/dev/null | grep bison++ -q; then
-   AC_MSG_ERROR([your $YACC turns out to be bison++, please install plain bison])
-else
-   AC_MSG_RESULT([no, this is good])
-fi
-
-AC_MSG_CHECKING([yacc/bison version])
-YACC_VERSION=`$YACC --version 2>/dev/null | $SED -ne "s/.*GNU Bison.* \([[0-9.]]*\).*/\1/p"`
-case "${YACC_VERSION}" in
-     1.*)
-     AC_MSG_ERROR([$YACC_VERSION, your bison version is too old, install 2.3 or sth. like that]) ;;
-     "")
-     AC_MSG_WARN([no GNU bison found, continuing anyway....]) ;;
-     *)
-     AC_MSG_RESULT([$YACC_VERSION]) ;;
-esac
-
-dnl Check for bunzip2
-AC_PATH_PROGS(BUNZIP2, bunzip2,, $PATH)
-if test -z "$BUNZIP2"; then
-	AC_MSG_ERROR([bunzip2 could not be found, please install])
-fi
-
-dnl Check for gunzip
-AC_PATH_PROGS(GUNZIP, gunzip,, $PATH)
-if test -z "$GUNZIP"; then
-	AC_MSG_ERROR([gunzip could not be found, please install])
-fi
-
-dnl Check for unzip
-AC_PATH_PROGS(UNZIP, unzip,, $PATH)
-if test -z "$UNZIP"; then
-	AC_MSG_ERROR([unzip could not be found, please install])
-fi
-
-dnl Check for xz
-AC_PATH_PROGS(XZ, xz,, $PATH)
-if test -z "$XZ"; then
-	AC_MSG_ERROR([xz could not be found, please install])
-fi
-
-dnl Check for wget
-AC_PATH_PROGS(WGET, wget,, $PATH)
-if test -z "$WGET"; then
-	AC_MSG_ERROR([wget could not be found, please install])
-fi
-
-AC_MSG_CHECKING([find version])
-FIND_VERSION=`$GNU_find --version 2>/dev/null | $SED -ne "s/.*\s\+\([[0-9]]\+\.[[0-9]]\+\.[[0-9]]\+\).*$/\1/p"`
-case "$FIND_VERSION" in
-4.*) ;;
-*)   AC_MSG_ERROR([we need at least GNU find 4.x]) ;;
-esac
-AC_MSG_RESULT([$FIND_VERSION])
-
-dnl Check for make
-AC_PATH_PROGS(MAKE, gmake gnumake make,, $PATH)
-if test -z "$MAKE"; then
-	AC_MSG_ERROR([make could not be found, please install])
-fi
-MAKE_VERSION=`$MAKE --version 2>/dev/null | $SED -ne "s/^GNU Make \([[0-9]]\+\.[[0-9.]]\+\).*$/\1/p"`
-MAJOR_MAKE_VERSION="${MAKE_VERSION%%.*}"
-MINOR_MAKE_VERSION="${MAKE_VERSION#*.}"
-MINOR_MAKE_VERSION="${MINOR_MAKE_VERSION%%.*}"
-if test -z "${MAJOR_MAKE_VERSION}"  -o -z "${MINOR_MAKE_VERSION}" ; then
-	AC_MSG_ERROR([could not parse make version. GNU make >= 3.81 required])
-fi
-if test $MAJOR_MAKE_VERSION -eq 3 -a $MINOR_MAKE_VERSION -lt 81 -o $MAJOR_MAKE_VERSION -lt 3 ; then
-	AC_MSG_ERROR([GNU make >= 3.81 required ($MAKE_VERSION found)])
-fi
-
-dnl Check for file
-AC_PATH_PROGS(FILE, file,, $PATH)
-if test -z "$FILE"; then
-	AC_MSG_ERROR([file could not be found, please install])
-fi
-AC_SUBST([PTXDIST_FILE], [$FILE])
-
-AC_PATH_PROG(SPHINX, sphinx-build,, $PATH)
-if test -z "$SPHINX"; then
-	AC_MSG_NOTICE([sphinx not found, required to build the documentation])
-	SPHINX=sphinx-build
+dnl
+dnl Everything else is checked by the bootstrap script, which reports all
+dnl missing tools at once. It is run again by 'make' to set things up.
+dnl
+AC_ARG_VAR([CC], [C compiler command for kconfig])
+AC_ARG_VAR([CFLAGS], [C compiler flags for kconfig])
+AC_ARG_VAR([CPPFLAGS], [C preprocessor flags for kconfig])
+AC_ARG_VAR([LDFLAGS], [linker flags for kconfig])
+AC_ARG_VAR([LEX], [lexer generator, default: flex])
+AC_ARG_VAR([YACC], [parser generator, default: bison])
+AC_ARG_VAR([PKG_CONFIG], [path to pkg-config])
+AC_ARG_VAR([PATCH], [patch executable to use, default: GNU patch from PATH])
+AC_ARG_WITH(patch, AS_HELP_STRING([--with-patch],[name of the patch executable to use, same as PATCH=]),
+	[PATCH="$withval"])
+
+AC_MSG_CHECKING([host tools])
+if CC="$CC" CFLAGS="$CFLAGS" CPPFLAGS="$CPPFLAGS" LDFLAGS="$LDFLAGS" \
+	LEX="$LEX" YACC="$YACC" PATCH="$PATCH" PKG_CONFIG="$PKG_CONFIG" \
+	"$BASH" "$srcdir/scripts/ptxdist_bootstrap.sh" --check >&AS_MESSAGE_LOG_FD 2>config.bootstrap
+then
+	AC_MSG_RESULT([ok])
+	# warnings
+	cat config.bootstrap >&2
+	cat config.bootstrap >&AS_MESSAGE_LOG_FD
+	rm -f config.bootstrap
 else
-	AC_MSG_NOTICE([sphinx found, run 'ptxdist docs-html' to get documentation])
+	AC_MSG_RESULT([failed])
+	cat config.bootstrap >&2
+	cat config.bootstrap >&AS_MESSAGE_LOG_FD
+	rm -f config.bootstrap
+	AC_MSG_ERROR([host check failed])
 fi
-AC_SUBST(SPHINX)
 
 AC_PATH_PROGS(RST2MAN, rst2man,, $PATH)
 AC_SUBST(RST2MAN)
 
-dnl Check for patch
-AC_ARG_WITH(patch, AS_HELP_STRING([--with-patch],[name of the patch executable to use]),
-    [
-	PATCH="$withval"
-	AC_SUBST(PATCH)
-	AC_MSG_NOTICE([Using patch executable $PATCH])
-    ],[
-	AC_PATH_PROG(PATCH, patch)
-    ])
-if test -z "$PATCH"; then
-    AC_MSG_ERROR([Please specify the location of the patch executable with the option '--with-patch'])
-fi
-
-# Sun's patch is a mess, issue a warning.  But we are going to continue with
-# the build because you might just be lucky.
-AC_MSG_CHECKING([whether $PATCH will work])
-if $PATCH -v 2>&1 | grep -q "Sun" >/dev/null 2>&1; then
-	AC_MSG_RESULT(no)
-	AC_MSG_WARN([
-Sorry, you have a Sun version of patch which is notoriously buggy.  $PACKAGE_NAME
-may function correctly, or minor errors may occur due to Sun's patch tool.
-Please consider upgrading to GNU patch, if you already have GNU patch then you
-can supply its path with the '--with-patch=' option.
-])
-elif $PATCH --version 2>&1 | grep -q "patch 2.0" >/dev/null 2>&1; then
-	AC_MSG_RESULT(no)
-	AC_MSG_WARN([
-Sorry, the version of patch you are using can cause severe problems when a patch
-creates a directory.  $PACKAGE_NAME may well function correctly with this version
-of patch or small problems could creep in.
-Please consider upgrading your patch to a more recent version, if you already
-have a more recent version of patch then you can supply its path with the
-'--with-patch=' option.
-])
-else
-	AC_MSG_RESULT(yes)
-fi
-
 dnl check if bash completion is desired (adapted from 'source-highlight' by Lorenzo Bettini)
 dnl don't use sysconfdir! as bash_completion does not check multiple directories, we use the system default
 
@@ -347,12 +103,6 @@ dnl Print results
 AC_MSG_RESULT([$PACKAGE_NAME version $PACKAGE_VERSION configured.])
 AC_MSG_RESULT([Using '$prefix' for installation prefix.])
 
-# we don't need to see this just for the backup-files command
-# but we may as well spec it for the future
-#AC_MSG_RESULT([Using '$CC' for C compiler.])
-#AC_MSG_RESULT([Building with '$CFLAGS' for C compiler flags.])
-#AC_MSG_RESULT([Building with '$LIBS' for linker flags.])
-
 AC_MSG_RESULT([])
 AC_MSG_RESULT([Report bugs to $PACKAGE_BUGREPORT])
 AC_MSG_RESULT([])
diff --git a/scripts/ptxdist_bootstrap.sh b/scripts/ptxdist_bootstrap.sh
index b9f4c7a5c..081bace1a 100755
--- a/scripts/ptxdist_bootstrap.sh
+++ b/scripts/ptxdist_bootstrap.sh
@@ -9,9 +9,12 @@
 # left to the package build.
 #
 # usage: ptxdist_bootstrap.sh <builddir>
+#        ptxdist_bootstrap.sh --check
 #
-# CC, CFLAGS, CPPFLAGS, LDFLAGS, LEX, YACC and PKG_CONFIG from the
-# environment are used if set.
+# With --check, only the prerequisites are checked and nothing is built.
+#
+# CC, CFLAGS, CPPFLAGS, LDFLAGS, LEX, YACC, PATCH and PKG_CONFIG from
+# the environment are used if set and not empty.
 #
 
 PTXDIST_LOG_PROMPT="ptxdist: "
@@ -39,14 +42,26 @@ esac
 set +o posix
 
 topdir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" || exit
-builddir="${1}"
-if [ -z "${builddir}" ]; then
-	echo "usage: ${0} <builddir>" >&2
+case "${1}" in
+"")
+	echo "usage: ${0} <builddir>|--check" >&2
 	exit 1
-fi
+	;;
+--check)
+	check_only=true
+	;;
+*)
+	builddir="${1}"
+	;;
+esac
+
+# empty means unset, e.g. when passed from configure
+for var in CC CFLAGS CPPFLAGS LDFLAGS LEX YACC PATCH PKG_CONFIG; do
+	[ -n "${!var}" ] || unset "${var}"
+done
 
 declare -A links
-declare -a missing
+declare -a missing warnings
 
 #
 # $1: version found, $2: version required
@@ -95,7 +110,7 @@ find_gnu() {
 }
 
 for tool in cat chmod chown cp dirname install ln md5sum mkdir mknod \
-		mktemp mv readlink realpath rm rmdir sort stat tty; do
+		mktemp mv readlink realpath rm rmdir sha256sum sort stat touch tty; do
 	find_gnu "${tool}" "GNU coreutils" "" coreutils "g${tool}" "${tool}"
 done
 find_gnu tar	"GNU tar"	""	tar		gtar tar
@@ -107,11 +122,69 @@ find_gnu make	"GNU Make"	3.81	make		gmake gnumake make
 
 links[bash]="${BASH}"
 
-if path="$(type -P file)"; then
-	links[file]="${path}"
-else
-	missing+=( "'file' (package: file)" )
-fi
+#
+# find a tool, any implementation will do, and register a link for it
+#
+# $1: tool, $2: package that provides it
+#
+find_tool() {
+	local path
+
+	if path="$(type -P "${1}")"; then
+		links[${1}]="${path}"
+	else
+		missing+=( "'${1}' (package: ${2})" )
+	fi
+}
+
+find_tool file	file
+find_tool bzip2	bzip2
+find_tool gzip	gzip
+find_tool xz	xz
+find_tool unzip	unzip
+find_tool wget	wget
+
+#
+# patch: any implementation works in principle, but some are known to
+# be broken, e.g. Sun's patch or patch 2.0, which cannot create
+# directories. Prefer GNU patch and warn about anything else.
+#
+find_patch() {
+	local -a names=( gpatch patch )
+	local name path version
+
+	if [ -n "${PATCH}" ]; then
+		names=( "${PATCH}" )
+	fi
+
+	for name in "${names[@]}"; do
+		path="$(type -P "${name}")" || continue
+		version="$("${path}" --version 2>/dev/null | head -n 1)"
+		if [[ "${version}" =~ ^(GNU )?patch\ ([0-9]+(\.[0-9]+)+) ]] &&
+			version_ge "${BASH_REMATCH[2]}" 2.1; then
+			links[patch]="${path}"
+			return
+		fi
+	done
+
+	# no GNU patch, take what is there
+	for name in "${names[@]}"; do
+		path="$(type -P "${name}")" && break
+	done
+	if [ -z "${path}" ]; then
+		missing+=( "'${names[-1]}' (package: patch)" )
+		return
+	fi
+	links[patch]="${path}"
+	if "${path}" -v 2>&1 | grep -q Sun; then
+		warnings+=( "'${path}' is Sun's patch, which is notoriously buggy." )
+	else
+		warnings+=( "'${path}' is not GNU patch >= 2.1, it may not work correctly." )
+	fi
+	warnings+=( "Please consider installing GNU patch (package: patch) or set PATCH." )
+}
+
+find_patch
 
 # optional
 links[sysctl]="$(type -P sysctl || type -P false)"
@@ -188,6 +261,14 @@ if type -P "${CC%% *}" > /dev/null; then
 	fi
 fi
 
+if [ ${#warnings[@]} -gt 0 ]; then
+	{
+		echo
+		printf "${PTXDIST_LOG_PROMPT}warning: %s\n" "${warnings[@]}"
+		echo
+	} >&2
+fi
+
 if [ ${#missing[@]} -gt 0 ]; then
 	{
 		echo
@@ -199,6 +280,10 @@ if [ ${#missing[@]} -gt 0 ]; then
 	exit 1
 fi
 
+if [ -n "${check_only}" ]; then
+	exit 0
+fi
+
 #
 # all prerequisites are there, set up the build directory
 #
-- 
2.47.3

[ptxdist] [PATCH 6/6] ptxdist: let new project raise from template

[Ladislav Michl]

From: Ladislav Michl <ladis@triops.cz>

Let PTXdist offer to create the project from template when BSP
directory it is running from appears unconfigured.

Nothing is written before all questions are answered, so declining or
Ctrl-C leaves the directory untouched.

Without a terminal or for configs given with --ptxconfig/--platformconfig,
the error is shown as before. The toolchain check is disabled in the new
platformconfig, as there is no toolchain yet.

The host tools prepared in ~/.cache before the project existed are
copied into it, so they are not built twice.

Signed-off-by: Ladislav Michl <ladis@triops.cz>
---
 README                                 |   3 +
 bin/ptxdist                            | 146 ++++++++++++++++++++++++-
 doc/ref_parameter.rst                  |   4 +
 rules/templates/project/gitignore      |   9 ++
 rules/templates/project/platformconfig |  10 ++
 rules/templates/project/ptxconfig      |   6 +
 6 files changed, 174 insertions(+), 4 deletions(-)
 create mode 100644 rules/templates/project/gitignore
 create mode 100644 rules/templates/project/platformconfig
 create mode 100644 rules/templates/project/ptxconfig

diff --git a/README b/README
index 5774e046a..5ac1c6468 100644
--- a/README
+++ b/README
@@ -31,6 +31,9 @@ all missing tools at once, and builds kconfig in .ptxdist/<version>/
 (.ptxdist/git/ for git checkouts) inside the BSP. The PTXdist tree
 itself is never written to.
 
+In case the BSP directory appears uninitialized, PTXdist offers to
+create the project from template.
+
 Idea
 ----
 
diff --git a/bin/ptxdist b/bin/ptxdist
index 3c86aa225..65bc70ce8 100755
--- a/bin/ptxdist
+++ b/bin/ptxdist
@@ -264,11 +264,18 @@ check_version() {
 
 
 
+_init_hint() {
+	if [ -z "${ptxd_init_asked}" -a ! -t 0 ]; then
+		echo "	or run ptxdist in a terminal to create a new ${1}"
+	fi
+}
+
 _get_config_ptx() {
-	if [ ! -e "${PTXDIST_PTXCONFIG}" ]; then
+	if [ ! -e "${PTXDIST_PTXCONFIG}" ] && ! do_init ptx; then
 		ptxd_dialog_msgbox \
 			"error:	'${PTXDIST_PTXCONFIG#${PTXDIST_WORKSPACE}/}' file is missing\n" \
-			"	try 'ptxdist select <ptxconfig>'\n"
+			"	try 'ptxdist select <ptxconfig>'\n" \
+			"$(_init_hint project)\n"
 		return 1
 	fi
 
@@ -321,10 +328,11 @@ _check_config_platform() {
 	#
 	ptxd_get_ptxconf PTXCONF_NO_PLATFORM >/dev/null && return || true
 
-	if [ ! -e "${PTXDIST_PLATFORMCONFIG}" ]; then
+	if [ ! -e "${PTXDIST_PLATFORMCONFIG}" ] && ! do_init platform; then
 		ptxd_dialog_msgbox \
 			"error:	'${PTXDIST_PLATFORMCONFIG#${PTXDIST_WORKSPACE}/}' is missing\n" \
-			"	try 'ptxdist platform <platformconfig>'"
+			"	try 'ptxdist platform <platformconfig>'\n" \
+			"$(_init_hint platform)"
 		return 1
 	fi
 
@@ -1507,6 +1515,136 @@ do_select()
 }
 
 
+#
+# create what is missing for a new project on demand:
+# the ptxconfig and, if not selected either, the platformconfig,
+# or just the platformconfig; returns 1 if nothing was created.
+#
+# $1: what is missing: 'ptx' or 'platform'
+#
+do_init()
+{
+	local part="${1}"
+	local project platform arch answer
+	local -a archs
+	local templates="${PTXDIST_TOPDIR}/rules/templates/project"
+	local ptxconfig="configs/ptxconfig"
+	local platformconfig
+
+	# only interactive and only for the default locations
+	if [ ! -t 0 -o ! -t 1 -o -n "${PTX_MENU}" ]; then
+		return 1
+	fi
+	if [ "${part}" = ptx -a "${PTX_ptxconfig_SET}" = true ]; then
+		return 1
+	fi
+	if [ "${PTX_platformconfig_SET}" = true ]; then
+		[ "${part}" = platform ] && return 1
+	elif [ ! -e "${PTXDIST_PLATFORMCONFIG}" ]; then
+		platformconfig=new
+	fi
+
+	if [ "${part}" = ptx -a -e "${ptxconfig}" ]; then
+		return 1
+	fi
+
+	ptxd_init_asked=true
+	echo
+	if [ "${part}" = ptx ]; then
+		echo "${PTXDIST_LOG_PROMPT}there is no PTXdist project in '${PTXDIST_WORKSPACE}'"
+		read -e -p "${PTXDIST_LOG_PROMPT}create a new one? [y/N] " answer || return 1
+	else
+		echo "${PTXDIST_LOG_PROMPT}no platform is selected"
+		read -e -p "${PTXDIST_LOG_PROMPT}create a new one? [y/N] " answer || return 1
+	fi
+	case "${answer}" in
+	y|Y|yes) ;;
+	*) return 1 ;;
+	esac
+	echo
+
+	if [ "${part}" = ptx ]; then
+		project="$(basename "${PTXDIST_WORKSPACE}")"
+		ptxd_template_read "project name" project "${project}" || return 1
+		if [ -z "${project}" ]; then
+			ptxd_bailout "the project name must not be empty"
+		fi
+	fi
+	if [ -n "${platformconfig}" ]; then
+		# the architectures from the main choice in architecture.in
+		archs=( $(awk '/^choice/ { c++ } c == 1 && /^\tconfig ARCH_/ { print $2 } /^endchoice/ && c == 1 { exit }' \
+			"${PTXDIST_TOPDIR}/platforms/architecture.in") ) &&
+		ptxd_template_read_options "architecture" arch "${archs[@]#ARCH_}" || return 1
+		platform="${arch,,}"
+		while :; do
+			ptxd_template_read "platform name" platform "${platform}" || return 1
+			platformconfig="configs/platform-${platform}/platformconfig"
+			if [ -z "${platform}" ]; then
+				echo "${PTXDIST_LOG_PROMPT}the platform name must not be empty"
+			elif [ -e "${platformconfig}" ]; then
+				echo "${PTXDIST_LOG_PROMPT}'${platformconfig}' already exists"
+			else
+				break
+			fi
+		done
+	fi
+
+	# create everything after all questions answered
+	echo
+	if [ -n "${project}" ]; then
+		echo "${PTXDIST_LOG_PROMPT}creating project '${project}'"
+		mkdir -p configs rules projectroot &&
+		if [ ! -e .gitignore ]; then
+			ptxd_replace_magic "${templates}/gitignore" > .gitignore
+		fi &&
+		PROJECT="${project}" \
+			ptxd_replace_magic "${templates}/ptxconfig" > "${ptxconfig}" &&
+		ln -sf "${ptxconfig}" "${PTXDIST_PTXCONFIG_DEFAULT}" || return
+		PTXDIST_PTXCONFIG="${PTXDIST_PTXCONFIG_DEFAULT}"
+	fi
+	if [ -n "${platform}" ]; then
+		echo "${PTXDIST_LOG_PROMPT}creating platform '${platform}' (${arch,,})"
+		mkdir -p "$(dirname "${platformconfig}")" &&
+		PLATFORM="${platform}" ARCH="ARCH_${arch}" \
+			ptxd_replace_magic "${templates}/platformconfig" > "${platformconfig}" &&
+		ln -sf "${platformconfig}" "${PTXDIST_PLATFORMCONFIG_DEFAULT}" || return
+		PTXDIST_PLATFORMCONFIG="${PTXDIST_PLATFORMCONFIG_DEFAULT}"
+	fi
+
+	# this is a project now, keep the host tools with it
+	case "${PTXDIST_HOSTTOOLS_DIR}" in
+	"${PTXDIST_TOPDIR}"|"${PTXDIST_WORKSPACE}/.ptxdist/"*)
+		;;
+	*)
+		if [ ! -e ".ptxdist/${PTXDIST_HOSTTOOLS_DIR##*/}" ]; then
+			mkdir -p .ptxdist &&
+			cp -a "${PTXDIST_HOSTTOOLS_DIR}" ".ptxdist/" || return
+		fi
+		;;
+	esac
+
+	unset PTXDIST_PLATFORMDIR
+	setup_platform &&
+	setup_path &&
+	setup_export || return
+
+	# fill in the defaults for everything else
+	if [ -n "${project}" ]; then
+		KCONFIG_ALLCONFIG="${PTXDIST_WORKSPACE}/${ptxconfig}" \
+			do_config alldefconfig ptx > /dev/null || return
+	fi
+	if [ -n "${platform}" ]; then
+		KCONFIG_ALLCONFIG="${PTXDIST_WORKSPACE}/${platformconfig}" \
+			do_config alldefconfig platform > /dev/null || return
+	fi
+
+	echo
+	echo "${PTXDIST_LOG_PROMPT}done. There is no toolchain yet, select one with"
+	echo "${PTXDIST_LOG_PROMPT}'ptxdist toolchain <path>'. 'ptxdist platformconfig' refines"
+	echo "${PTXDIST_LOG_PROMPT}the platform, 'ptxdist newpackage <type>' adds own packages."
+	echo
+}
+
 
 #
 # guess and find the toolchain
diff --git a/doc/ref_parameter.rst b/doc/ref_parameter.rst
index e167f28fe..52d470304 100644
--- a/doc/ref_parameter.rst
+++ b/doc/ref_parameter.rst
@@ -28,6 +28,8 @@ Setup and Project Actions
   the same project. The default location for the configuration file is
   ``configs/ptxconfig``. PTXdist will use this if no other configuration is
   selected.
+  If there is no configuration at all and PTXdist runs in a terminal, it
+  offers to create a new project instead, see ``rules/templates/project/``.
 
 ``platform <config>``
   this action will select a platform
@@ -38,6 +40,8 @@ Setup and Project Actions
   The default location for the configuration file is
   ``configs/*/platformconfig``. PTXdist will use that if the pattern matches
   exactly one file and no other configuration is selected.
+  If no platform is selected and PTXdist runs in a terminal, it offers to
+  create a new one instead.
 
 ``toolchain [<path>]``
   this action will select the toolchain to use. If no path is specified
diff --git a/rules/templates/project/gitignore b/rules/templates/project/gitignore
new file mode 100644
index 000000000..20ede9441
--- /dev/null
+++ b/rules/templates/project/gitignore
@@ -0,0 +1,9 @@
+/.ptxdist/
+/configs/*.old
+/configs/platform-*/*.old
+/platform-*
+/selected_collectionconfig
+/selected_platformconfig
+/selected_ptxconfig
+/selected_toolchain*
+/src
diff --git a/rules/templates/project/platformconfig b/rules/templates/project/platformconfig
new file mode 100644
index 000000000..165143199
--- /dev/null
+++ b/rules/templates/project/platformconfig
@@ -0,0 +1,10 @@
+#
+# template for a new project, everything not set here gets its default
+#
+PTXCONF__platformconfig_MAGIC__=y
+PTXCONF_PLATFORMCONFIG_VERSION="@PTXDIST_VERSION_FULL@"
+PTXCONF_PLATFORM="@PLATFORM@"
+PTXCONF_@ARCH@=y
+# no toolchain yet, select one with 'ptxdist toolchain <path>'
+PTXCONF_CROSSCHAIN_VENDOR=""
+PTXCONF_CROSSCHAIN_CHECK=""
diff --git a/rules/templates/project/ptxconfig b/rules/templates/project/ptxconfig
new file mode 100644
index 000000000..2b4b71024
--- /dev/null
+++ b/rules/templates/project/ptxconfig
@@ -0,0 +1,6 @@
+#
+# template for a new project, everything not set here gets its default
+#
+PTXCONF__ptxconfig_MAGIC__=y
+PTXCONF_CONFIGFILE_VERSION="@PTXDIST_VERSION_FULL@"
+PTXCONF_PROJECT="@PROJECT@"
-- 
2.47.3