[ptxdist] [RFC] scripts: add template patch backporter

[ptxdist] [RFC] scripts: add template patch backporter

[Roland Hieber]

Backporting PTXdist patches that changed rules/templates/* to a recipe
that was created with a template via 'ptxdist newpackage' can be quite
tedious and error-prone, as there are many @VARIABLES@ in the templates
that have been expanded in the recipe. (And it's especially tedious if
there are several such recipes in the BSP.) Try to automate that process
with a bit of sed magic, and give the resulting patches a nice commit
message too.

Signed-off-by: Roland Hieber <rhi@pengutronix.de>
---
 scripts/port-template-commits | 103 ++++++++++++++++++++++++++++++++++
 1 file changed, 103 insertions(+)
 create mode 100755 scripts/port-template-commits

diff --git a/scripts/port-template-commits b/scripts/port-template-commits
new file mode 100755
index 000000000000..0907c833b4a9
--- /dev/null
+++ b/scripts/port-template-commits
@@ -0,0 +1,103 @@
+#!/bin/bash
+shopt -s extglob
+self=$(basename "$0")
+
+usage() {
+	echo "Usage: ${self} <type> <packagename> <commit-ranges...>"
+	echo
+	echo "Given a range of PTXdist commits, rewrite commits which changed"
+	echo "the given template into patches that can be applied to a package"
+	echo "that was created with that template."
+	echo
+	echo "<type> is the parameter that was supplied to 'ptxdist newpackage'"
+	echo "<packagename> is the package name entered during 'ptxdist newpackage'"
+	echo "    (e.g. 'foobar' for rules/barebox-foobar.rules)"
+	echo "<commit-ranges> is one or more Git commit ranges in the PTXdist repository"
+	echo
+	echo "Example:"
+	echo "  ${self} barebox foobar 6882ddc29fda^..6882ddc29fda ptxdist-2019.09.0.."
+	echo "    Port all commits that touched rules/templates/template-barebox-{in,make}"
+	echo "    from commit 6882ddc29fda and since PTXdist 2019.09.0 so that the"
+	echo "    resulting patches can be applied to 'barebox-foobar'."
+	echo "    The patches will be written to ./barebox-foobar-template-patches/."
+}
+
+if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
+	usage
+	exit 1
+fi
+if [ -n "$(grep -- ' --help\|-h ' <<< " $@ ")" ]; then
+	usage
+	exit
+fi
+
+if [ -z "$(git ls-files | grep '^bin/ptxdist$')" ]; then
+	echo "Not inside a PTXdist Git repository." >&2
+	exit 1
+fi
+
+template_type="$1"
+package_name="$2"
+shift 2
+commit_ranges="$@"
+
+template_files="rules/templates/template-${template_type}-@(config|in|make)"
+package="$(tr "[A-Z]" "[a-z]" <<< "${package_name}")"
+packagedash="$(tr "[_]" "[\-]" <<< "${package}")"
+PACKAGE="$(tr "[a-z-]" "[A-Z_]" <<< "${package_name}")"
+
+outdir="./${template_type}-${package_name}-template-patches"
+startnumber=0
+patches=""
+for range in $commit_ranges; do
+	series=$(git format-patch --stat=90,80 --numbered --start-number ${startnumber} \
+		--output-directory=${outdir} "${range}" -- ${template_files} || exit 3)
+	patches="$patches $series"
+	startnumber=$((startnumber+100))
+done
+
+tmpfile="$(mktemp ./.commitmsg-XXXXXXXX)"
+trap "rm -f '$tmpfile'" EXIT
+
+for patch in ${patches}; do
+	# format commit message
+	commit=$(head -n 1 ${patch} | cut -d' ' -f2)
+	commit_short=$(cut -b1-20 <<< "${commit}")
+	if [ -z "$commit_short" ]; then
+		echo "Could not detect commit ID in '$patch'!" >&2
+		continue
+	fi
+
+	# remove everything after first empty line until before the diff
+	# TODO: would be nicer to keep the diffstat… need to find the correct regex
+	sed -i -e '
+		/^$/,/^diff / {
+			/^diff / { b; }
+			d;
+		}
+		' ${patch}
+	# and insert the new commit message before the diff
+	{
+		echo
+		echo "This ports PTXdist commit ${commit_short}:"
+		echo
+		git log -1 ${commit} | sed -e 's;^;    | ;' -e 's;\s\+$;;g'
+		echo
+		echo "Link: https://git.pengutronix.de/cgit/ptxdist/commit/?id=${commit_short}"
+		echo "---"
+	} > ${tmpfile}
+	sed -i -e '
+		/^diff / {
+			r '${tmpfile}'
+			N
+		}' ${patch}
+
+	# replace filenames and template variables
+	sed -i '
+		s,rules/templates/template-.*-\(config\|in\|make\),'"${template_type}-${package_name}"'.\1,g;
+		s,@PACKAGE@,'"${PACKAGE}"',g;
+		s,@package@,'"${package_name}"',g;
+	' ${patch}
+
+	printf "%s\n" ${patch}
+done
-- 
2.29.2


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de

Re: [ptxdist] [RFC] scripts: add template patch backporter

[Michael Olbrich]

On Mon, May 31, 2021 at 02:40:07PM +0200, Roland Hieber wrote:
> Backporting PTXdist patches that changed rules/templates/* to a recipe
> that was created with a template via 'ptxdist newpackage' can be quite
> tedious and error-prone, as there are many @VARIABLES@ in the templates
> that have been expanded in the recipe. (And it's especially tedious if
> there are several such recipes in the BSP.) Try to automate that process
> with a bit of sed magic, and give the resulting patches a nice commit
> message too.

I've not tested this yet but I like the idea.

> Signed-off-by: Roland Hieber <rhi@pengutronix.de>
> ---
>  scripts/port-template-commits | 103 ++++++++++++++++++++++++++++++++++
>  1 file changed, 103 insertions(+)
>  create mode 100755 scripts/port-template-commits
> 
> diff --git a/scripts/port-template-commits b/scripts/port-template-commits
> new file mode 100755
> index 000000000000..0907c833b4a9
> --- /dev/null
> +++ b/scripts/port-template-commits
> @@ -0,0 +1,103 @@
> +#!/bin/bash
> +shopt -s extglob
> +self=$(basename "$0")
> +
> +usage() {
> +	echo "Usage: ${self} <type> <packagename> <commit-ranges...>"
> +	echo
> +	echo "Given a range of PTXdist commits, rewrite commits which changed"
> +	echo "the given template into patches that can be applied to a package"
> +	echo "that was created with that template."
> +	echo
> +	echo "<type> is the parameter that was supplied to 'ptxdist newpackage'"
> +	echo "<packagename> is the package name entered during 'ptxdist newpackage'"
> +	echo "    (e.g. 'foobar' for rules/barebox-foobar.rules)"
> +	echo "<commit-ranges> is one or more Git commit ranges in the PTXdist repository"
> +	echo
> +	echo "Example:"
> +	echo "  ${self} barebox foobar 6882ddc29fda^..6882ddc29fda ptxdist-2019.09.0.."
> +	echo "    Port all commits that touched rules/templates/template-barebox-{in,make}"
> +	echo "    from commit 6882ddc29fda and since PTXdist 2019.09.0 so that the"
> +	echo "    resulting patches can be applied to 'barebox-foobar'."
> +	echo "    The patches will be written to ./barebox-foobar-template-patches/."
> +}
> +
> +if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
> +	usage
> +	exit 1
> +fi
> +if [ -n "$(grep -- ' --help\|-h ' <<< " $@ ")" ]; then
> +	usage
> +	exit
> +fi
> +
> +if [ -z "$(git ls-files | grep '^bin/ptxdist$')" ]; then

if !  git ls-files 2>/dev/null | grep -q '^bin/ptxdist$'; then

> +	echo "Not inside a PTXdist Git repository." >&2
> +	exit 1
> +fi
> +
> +template_type="$1"
> +package_name="$2"
> +shift 2
> +commit_ranges="$@"
> +
> +template_files="rules/templates/template-${template_type}-@(config|in|make)"
> +package="$(tr "[A-Z]" "[a-z]" <<< "${package_name}")"
> +packagedash="$(tr "[_]" "[\-]" <<< "${package}")"
> +PACKAGE="$(tr "[a-z-]" "[A-Z_]" <<< "${package_name}")"
> +
> +outdir="./${template_type}-${package_name}-template-patches"
> +startnumber=0
> +patches=""
> +for range in $commit_ranges; do
> +	series=$(git format-patch --stat=90,80 --numbered --start-number ${startnumber} \
> +		--output-directory=${outdir} "${range}" -- ${template_files} || exit 3)
> +	patches="$patches $series"
> +	startnumber=$((startnumber+100))
> +done
> +
> +tmpfile="$(mktemp ./.commitmsg-XXXXXXXX)"
> +trap "rm -f '$tmpfile'" EXIT
> +
> +for patch in ${patches}; do
> +	# format commit message
> +	commit=$(head -n 1 ${patch} | cut -d' ' -f2)
> +	commit_short=$(cut -b1-20 <<< "${commit}")
> +	if [ -z "$commit_short" ]; then
> +		echo "Could not detect commit ID in '$patch'!" >&2
> +		continue
> +	fi
> +
> +	# remove everything after first empty line until before the diff
> +	# TODO: would be nicer to keep the diffstat… need to find the correct regex

I think the '---' line is the cut of point, or am I missing something?

	sed -i '0,/^---$/d' ${patch}

> +	sed -i -e '
> +		/^$/,/^diff / {
> +			/^diff / { b; }
> +			d;
> +		}
> +		' ${patch}
> +	# and insert the new commit message before the diff
> +	{
> +		echo
> +		echo "This ports PTXdist commit ${commit_short}:"
> +		echo
> +		git log -1 ${commit} | sed -e 's;^;    | ;' -e 's;\s\+$;;g'
> +		echo
> +		echo "Link: https://git.pengutronix.de/cgit/ptxdist/commit/?id=${commit_short}"
> +		echo "---"
> +	} > ${tmpfile}

Add a comment, what the 'sed' is doing here.

Michael

> +	sed -i -e '
> +		/^diff / {
> +			r '${tmpfile}'
> +			N
> +		}' ${patch}
> +
> +	# replace filenames and template variables
> +	sed -i '
> +		s,rules/templates/template-.*-\(config\|in\|make\),'"${template_type}-${package_name}"'.\1,g;
> +		s,@PACKAGE@,'"${PACKAGE}"',g;
> +		s,@package@,'"${package_name}"',g;
> +	' ${patch}
> +
> +	printf "%s\n" ${patch}
> +done
> -- 
> 2.29.2
> 
> 
> _______________________________________________
> ptxdist mailing list
> ptxdist@pengutronix.de
> To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de