[ptxdist] [PATCH 0/2] license-report: also create CSV lists

[ptxdist] [PATCH 0/2] license-report: also create CSV lists

[Andreas Pretzsch]

When calling "ptxdist make license-report", license reports for
both the target- and the host-tools-packages are created.

Based upon the license info files created during the regular build
process, LaTeX files are generated, which are then converted into
matching PDFs, containing package informations, license tags as well
as the licenses themselves.
These are called "report/license-report.pdf" respectively
"report/license-report-tools.pdf".

Extend this process to also create a list of these packages with the
most basic fields as CSV:
    "name"|"version"|"licenses"|"flags"
    "somepackage"|"1.2.3"|"Apache-2.0"|"attribution"
    [...]

These are called "report/license-report.csv" respectively
"report/license-report-tools.csv".

This minimal report is much more suited for a comparison of different
builds resp. releases, and can be easily imported into some spreadsheet.
Which in turn is useful both for developers and legal department.
Keeping it minimal (e.g. no URL) most likely avoids CSV escaping hell.

Tested with PTXdist 2016.01.0, but given the minimal differences
against PTXdist 2018.03.0, I assume this is still fine. Albeit I did
not test it.

It is split into two commits (to ease local maintenance), but of course
could be squashed into one.
Also the commit subjects could be adapted, removing "scripts/lib", or
if squashed even simply to "license-report: also create CSV lists".

While all of this is working (and sufficient for my use case), it is
also a bit of a RFC. Just don't expect I'll do some _substantial_ rework
or improvement in a timely fashion...
So, feel free to squash and/or fixup at your own discretion.

Last, sorry for having this one delayed a year and a half before sending
mainline...

Andreas Pretzsch (2):
  scripts/lib: make_world_license: add suffix "_tex" to
    ptxd_make_world_license_write
  scripts/lib: also create CSV lists on "ptxdist make license-report"

 scripts/lib/ptxd_make_license_report.sh | 21 +++++++++++
 scripts/lib/ptxd_make_world_license.sh  | 65 +++++++++++++++++++++++++++++++--
 2 files changed, 83 insertions(+), 3 deletions(-)

-- 
2.9.3


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

[ptxdist] [PATCH 1/2] scripts/lib: make_world_license: add suffix "_tex" to ptxd_make_world_license_write

[Andreas Pretzsch]

Rename "function ptxd_make_world_license_write" to
"ptxd_make_world_license_write_tex", to be clear about
what it creates (LaTeX file, then a PDF from it).

This change is purely cosmetic.
But we will add other generation functions later,
similiar to this. Therefore rename it.

Signed-off-by: Andreas Pretzsch <apr@cn-eng.de>
---
Tested with PTXdist 2016.01.0, but given the minimal differences
against PTXdist 2018.03.0, I assume this is still fine.

 scripts/lib/ptxd_make_world_license.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/ptxd_make_world_license.sh b/scripts/lib/ptxd_make_world_license.sh
index 1a79387..bba8ca3 100644
--- a/scripts/lib/ptxd_make_world_license.sh
+++ b/scripts/lib/ptxd_make_world_license.sh
@@ -125,7 +125,7 @@ export -f ptxd_make_latex_escape
 # $pkg_license = the license string
 # $pkg_license_texts/$pkg_license_texts_guessed = license text files
 #
-ptxd_make_world_license_write() {
+ptxd_make_world_license_write_tex() {
     local guess
     local brl='{' brr='}'
     local pkg_chapter="$(ptxd_make_latex_escape ${pkg_label})"
@@ -207,7 +207,7 @@ ptxd_make_world_license_write() {
 	EOF
     done
 }
-export -f ptxd_make_world_license_write
+export -f ptxd_make_world_license_write_tex
 
 # Copy all patches according to the series file
 # $1 full path to the series file
@@ -461,7 +461,7 @@ changed: ${md5} -> $(md5sum "${lic}" | sed 's/ .*//')
 	ptxd_bailout "Failed to copy '$(ptxd_print_path "${file}")'"
     done &&
 
-    ptxd_make_world_license_write | \
+    ptxd_make_world_license_write_tex | \
         sed -e 's/%/\\%/g' > "${pkg_license_dir}/license-report.tex" &&
     check_pipe_status &&
 
-- 
2.9.3


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

[ptxdist] [PATCH 2/2] scripts/lib: also create CSV lists on "ptxdist make license-report"

[Andreas Pretzsch]

When calling "ptxdist make license-report", license reports for
both the target- and the host-tools-packages are created.

Based upon the license info files created during the regular build
process, LaTeX files are generated, which are then converted into
matching PDFs, containing package informations, license tags as well
as the licenses themselves.
These are called "report/license-report.pdf" respectively
"report/license-report-tools.pdf".

Extend this process to also create a list of these packages with the
most basic fields as CSV:
    "name"|"version"|"licenses"|"flags"
    "somepackage"|"1.2.3"|"Apache-2.0"|"attribution"
    [...]

These are called "report/license-report.csv" respectively
"report/license-report-tools.csv".

The compliance report and distribution creation, trigger via
"ptxdist make license-compliance-distribution", is not modified,
as it essentially contains just a subset of the above packages,
namely all the Free and Open Source Software.
This could be extended by a CSV file, too, but it will not aid
in any verification of the overall distribution, as this has to
be checked in total, not only with respect to the FOSS parts.

Signed-off-by: Andreas Pretzsch <apr@cn-eng.de>
---
Tested with PTXdist 2016.01.0, but given the minimal differences
against PTXdist 2018.03.0, I assume this is still fine.

 scripts/lib/ptxd_make_license_report.sh | 21 ++++++++++++
 scripts/lib/ptxd_make_world_license.sh  | 59 +++++++++++++++++++++++++++++++++
 2 files changed, 80 insertions(+)

diff --git a/scripts/lib/ptxd_make_license_report.sh b/scripts/lib/ptxd_make_license_report.sh
index a6410d1..d6570ca 100644
--- a/scripts/lib/ptxd_make_license_report.sh
+++ b/scripts/lib/ptxd_make_license_report.sh
@@ -194,6 +194,27 @@ ptxd_make_license_report() {
 	ptxd_make_license_report_footer
     ) > "${ptx_license_target_tex}"
 
+    ptx_license_target_csv="${ptx_report_dir}/${pkg_section}/$(basename "${ptx_license_target%.pdf}.csv")"
+    (
+	local pkg_csv_file
+	ptxd_make_world_license_write_csv_header
+	for pkg in ${@}; do
+		pkg_lic="${ptxd_package_license_association[${pkg}]}"
+		if [ -z ${pkg_lic} ]; then
+			continue
+		fi
+		pkg_lic="${pkg_lic}/${pkg}"
+
+		pkg_csv_file="${ptx_report_dir}/${pkg_lic}/license-report.csv"
+		if [ -r "${pkg_csv_file}" ]; then
+			cat "${pkg_csv_file}"
+		else
+			ptxd_bailout "no $(ptxd_print_path "${pkg_csv_file}")"
+		fi
+	done
+	ptxd_make_world_license_write_csv_footer
+    ) > "${ptx_license_target_csv}"
+
     ptxd_make_license_report_build
 #    mv "${ptx_license_target_tex%.tex}.pdf" "${ptx_license_target}"
 }
diff --git a/scripts/lib/ptxd_make_world_license.sh b/scripts/lib/ptxd_make_world_license.sh
index bba8ca3..39c696c 100644
--- a/scripts/lib/ptxd_make_world_license.sh
+++ b/scripts/lib/ptxd_make_world_license.sh
@@ -209,6 +209,63 @@ ptxd_make_world_license_write_tex() {
 }
 export -f ptxd_make_world_license_write_tex
 
+#
+# echo header for generated CSV:
+# "name"|"version"|"licenses"|"flags"
+#
+ptxd_make_world_license_write_csv_header() {
+	local -r STRING_QUOTE='"'
+	local -r CSV_SEPARATOR='|'
+	local l
+	l="${STRING_QUOTE}name${STRING_QUOTE}"
+	l="${l}${CSV_SEPARATOR}"
+	l="${l}${STRING_QUOTE}version${STRING_QUOTE}"
+	l="${l}${CSV_SEPARATOR}"
+	l="${l}${STRING_QUOTE}licenses${STRING_QUOTE}"
+	l="${l}${CSV_SEPARATOR}"
+	l="${l}${STRING_QUOTE}flags${STRING_QUOTE}"
+	l="${l}\n"
+	echo -ne "${l}"
+}
+export -f ptxd_make_world_license_write_csv_header
+#
+# generate a CSV for the package:
+# "somepackage"|"1.2.3"|"Apache-2.0"|"attribution"
+#
+ptxd_make_world_license_write_csv() {
+	local -r STRING_QUOTE='"'
+	local -r CSV_SEPARATOR='|'
+	local -a flags=( "${!pkg_license_flags[@]}" )
+	local l
+
+	# not required, seems to be filtered before:
+	#case "${pkg_license}" in
+	#*ignore*)
+	#	# ignore this package, e.g. do not list it in the report
+	#	return 0
+	#	;;
+	#esac
+
+	l="${STRING_QUOTE}${pkg_label}${STRING_QUOTE}"
+	l="${l}${CSV_SEPARATOR}"
+	[ -n "${pkg_version}" ]	&& l="${l}${STRING_QUOTE}${pkg_version}${STRING_QUOTE}"
+	l="${l}${CSV_SEPARATOR}"
+	[ -n "${pkg_license}" ]	&& l="${l}${STRING_QUOTE}${pkg_license}${STRING_QUOTE}"
+	l="${l}${CSV_SEPARATOR}"
+	[ -n "${flags[*]}" ]	&& l="${l}${STRING_QUOTE}${flags[*]}${STRING_QUOTE}"
+	l="${l}\n"
+	echo -ne "${l}"
+}
+export -f ptxd_make_world_license_write_csv
+#
+# echo footer for generated CSV
+#
+ptxd_make_world_license_write_csv_footer() {
+	# nothing to to
+	/bin/true
+}
+export -f ptxd_make_world_license_write_csv_footer
+
 # Copy all patches according to the series file
 # $1 full path to the series file
 # $2 source directory
@@ -465,6 +522,8 @@ changed: ${md5} -> $(md5sum "${lic}" | sed 's/ .*//')
         sed -e 's/%/\\%/g' > "${pkg_license_dir}/license-report.tex" &&
     check_pipe_status &&
 
+    ptxd_make_world_license_write_csv > "${pkg_license_dir}/license-report.csv" &&
+
     echo "${pkg_license}" > "${pkg_license_dir}/license-name" &&
     if [ "${#pkg_license_flags[@]}" -gt 0 ]; then
 	echo "${!pkg_license_flags[@]}" > "${pkg_license_dir}/license-flags"
-- 
2.9.3


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH 2/2] scripts/lib: also create CSV lists on "ptxdist make license-report"

[Michael Olbrich]

On Wed, Mar 28, 2018 at 10:23:07PM +0200, Andreas Pretzsch wrote:
> When calling "ptxdist make license-report", license reports for
> both the target- and the host-tools-packages are created.
> 
> Based upon the license info files created during the regular build
> process, LaTeX files are generated, which are then converted into
> matching PDFs, containing package informations, license tags as well
> as the licenses themselves.
> These are called "report/license-report.pdf" respectively
> "report/license-report-tools.pdf".
> 
> Extend this process to also create a list of these packages with the
> most basic fields as CSV:
>     "name"|"version"|"licenses"|"flags"
>     "somepackage"|"1.2.3"|"Apache-2.0"|"attribution"
>     [...]
> 
> These are called "report/license-report.csv" respectively
> "report/license-report-tools.csv".
> 
> The compliance report and distribution creation, trigger via
> "ptxdist make license-compliance-distribution", is not modified,
> as it essentially contains just a subset of the above packages,
> namely all the Free and Open Source Software.
> This could be extended by a CSV file, too, but it will not aid
> in any verification of the overall distribution, as this has to
> be checked in total, not only with respect to the FOSS parts.
> 
> Signed-off-by: Andreas Pretzsch <apr@cn-eng.de>
> ---
> Tested with PTXdist 2016.01.0, but given the minimal differences
> against PTXdist 2018.03.0, I assume this is still fine.
> 
>  scripts/lib/ptxd_make_license_report.sh | 21 ++++++++++++
>  scripts/lib/ptxd_make_world_license.sh  | 59 +++++++++++++++++++++++++++++++++
>  2 files changed, 80 insertions(+)
> 
> diff --git a/scripts/lib/ptxd_make_license_report.sh b/scripts/lib/ptxd_make_license_report.sh
> index a6410d1..d6570ca 100644
> --- a/scripts/lib/ptxd_make_license_report.sh
> +++ b/scripts/lib/ptxd_make_license_report.sh
> @@ -194,6 +194,27 @@ ptxd_make_license_report() {
>  	ptxd_make_license_report_footer
>      ) > "${ptx_license_target_tex}"
>  
> +    ptx_license_target_csv="${ptx_report_dir}/${pkg_section}/$(basename "${ptx_license_target%.pdf}.csv")"
> +    (
> +	local pkg_csv_file
> +	ptxd_make_world_license_write_csv_header
> +	for pkg in ${@}; do
> +		pkg_lic="${ptxd_package_license_association[${pkg}]}"
> +		if [ -z ${pkg_lic} ]; then
> +			continue
> +		fi
> +		pkg_lic="${pkg_lic}/${pkg}"
> +
> +		pkg_csv_file="${ptx_report_dir}/${pkg_lic}/license-report.csv"
> +		if [ -r "${pkg_csv_file}" ]; then
> +			cat "${pkg_csv_file}"
> +		else
> +			ptxd_bailout "no $(ptxd_print_path "${pkg_csv_file}")"
> +		fi
> +	done
> +	ptxd_make_world_license_write_csv_footer
> +    ) > "${ptx_license_target_csv}"
> +

Please create separate ptxd_make_license_report_tex /
ptxd_make_license_report_csv functions that are called by
ptxd_make_license_report.

>      ptxd_make_license_report_build
>  #    mv "${ptx_license_target_tex%.tex}.pdf" "${ptx_license_target}"
>  }
> diff --git a/scripts/lib/ptxd_make_world_license.sh b/scripts/lib/ptxd_make_world_license.sh
> index bba8ca3..39c696c 100644
> --- a/scripts/lib/ptxd_make_world_license.sh
> +++ b/scripts/lib/ptxd_make_world_license.sh
> @@ -209,6 +209,63 @@ ptxd_make_world_license_write_tex() {
>  }
>  export -f ptxd_make_world_license_write_tex
>  
> +#
> +# echo header for generated CSV:
> +# "name"|"version"|"licenses"|"flags"
> +#
> +ptxd_make_world_license_write_csv_header() {
> +	local -r STRING_QUOTE='"'
> +	local -r CSV_SEPARATOR='|'

Lower case for local variables. Also, I think a helper to write one line
makes sense. Untested:

helper() {
	string_quote='"'
	csv_separator='|'
	while [ ${#} -gt 0 ]; do
		if [ -n "${1}" ]; then
			echo -n "${string_quote}${1}${string_quote}"
		fi
		shift
		if [ [ ${#} -gt 0 ]; do
			echo -n ${csv_separator}"
		fi
	done
	echo
}

use it here...

> +	local l
> +	l="${STRING_QUOTE}name${STRING_QUOTE}"
> +	l="${l}${CSV_SEPARATOR}"
> +	l="${l}${STRING_QUOTE}version${STRING_QUOTE}"
> +	l="${l}${CSV_SEPARATOR}"
> +	l="${l}${STRING_QUOTE}licenses${STRING_QUOTE}"
> +	l="${l}${CSV_SEPARATOR}"
> +	l="${l}${STRING_QUOTE}flags${STRING_QUOTE}"
> +	l="${l}\n"
> +	echo -ne "${l}"
> +}
> +export -f ptxd_make_world_license_write_csv_header
> +#
> +# generate a CSV for the package:
> +# "somepackage"|"1.2.3"|"Apache-2.0"|"attribution"
> +#
> +ptxd_make_world_license_write_csv() {
> +	local -r STRING_QUOTE='"'
> +	local -r CSV_SEPARATOR='|'
> +	local -a flags=( "${!pkg_license_flags[@]}" )
> +	local l
> +
> +	# not required, seems to be filtered before:
> +	#case "${pkg_license}" in
> +	#*ignore*)
> +	#	# ignore this package, e.g. do not list it in the report
> +	#	return 0
> +	#	;;
> +	#esac
> +
> +	l="${STRING_QUOTE}${pkg_label}${STRING_QUOTE}"
> +	l="${l}${CSV_SEPARATOR}"
> +	[ -n "${pkg_version}" ]	&& l="${l}${STRING_QUOTE}${pkg_version}${STRING_QUOTE}"
> +	l="${l}${CSV_SEPARATOR}"
> +	[ -n "${pkg_license}" ]	&& l="${l}${STRING_QUOTE}${pkg_license}${STRING_QUOTE}"
> +	l="${l}${CSV_SEPARATOR}"
> +	[ -n "${flags[*]}" ]	&& l="${l}${STRING_QUOTE}${flags[*]}${STRING_QUOTE}"
> +	l="${l}\n"
> +	echo -ne "${l}"

... and here.


Michael

> +}
> +export -f ptxd_make_world_license_write_csv
> +#
> +# echo footer for generated CSV
> +#
> +ptxd_make_world_license_write_csv_footer() {
> +	# nothing to to
> +	/bin/true
> +}
> +export -f ptxd_make_world_license_write_csv_footer
> +
>  # Copy all patches according to the series file
>  # $1 full path to the series file
>  # $2 source directory
> @@ -465,6 +522,8 @@ changed: ${md5} -> $(md5sum "${lic}" | sed 's/ .*//')
>          sed -e 's/%/\\%/g' > "${pkg_license_dir}/license-report.tex" &&
>      check_pipe_status &&
>  
> +    ptxd_make_world_license_write_csv > "${pkg_license_dir}/license-report.csv" &&
> +
>      echo "${pkg_license}" > "${pkg_license_dir}/license-name" &&
>      if [ "${#pkg_license_flags[@]}" -gt 0 ]; then
>  	echo "${!pkg_license_flags[@]}" > "${pkg_license_dir}/license-flags"
> -- 
> 2.9.3
> 
> 
> _______________________________________________
> ptxdist mailing list
> ptxdist@pengutronix.de

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH 2/2] scripts/lib: also create CSV lists on "ptxdist make license-report"

[Andreas Pretzsch]

On Do, 2018-03-29 at 11:46 +0200, Michael Olbrich wrote:
> Please create separate ptxd_make_license_report_tex /
> ptxd_make_license_report_csv functions that are called by
> ptxd_make_license_report.

Ok.

> > +ptxd_make_world_license_write_csv_header() {
> > +	local -r STRING_QUOTE='"'
> > +	local -r CSV_SEPARATOR='|'
> 
> Lower case for local variables.

Ok.

>  Also, I think a helper to write one line
> makes sense. Untested:
> 
> helper() {
> 	string_quote='"'
> 	csv_separator='|'
> 	while [ ${#} -gt 0 ]; do
> 		if [ -n "${1}" ]; then
> 			echo -n "${string_quote}${1}${string_quote}"
> 		fi
> 		shift
> 		if [ [ ${#} -gt 0 ]; do
> 			echo -n ${csv_separator}"
> 		fi
> 	done
> 	echo
> }

Good point. Will do.


> > +ptxd_make_world_license_write_csv() {
> > +	local -r STRING_QUOTE='"'
> > +	local -r CSV_SEPARATOR='|'
> > +	local -a flags=( "${!pkg_license_flags[@]}" )
> > +	local l
> > +
> > +	# not required, seems to be filtered before:
> > +	#case "${pkg_license}" in
> > +	#*ignore*)
> > +	#	# ignore this package, e.g. do not list it in the report
> > +	#	return 0
> > +	#	;;
> > +	#esac

Any comment on this commented section ?
Back then, I copied the logic from somewhere, but filtering was not
required in a real run. Still, kept it as reference.


I'll rework the patch as above over Easter time, plus
  - include section tag (see Denis' patch)
  - squash it into one with "add suffix _tex"


Best regards,
  Andreas

-- 

carpe noctem engineering
Ingenieurbuero fuer Hard- & Software-Entwicklung Andreas Pretzsch
Dipl.-Ing. (FH) Andreas Pretzsch        Tel. +49-(0)7307-936088-1
Lange Strasse 28a                       Fax: +49-(0)7307-936088-9
89250 Senden, Germany                   email: apr@cn-eng.de


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH 2/2] scripts/lib: also create CSV lists on "ptxdist make license-report"

[Michael Olbrich]

On Thu, Mar 29, 2018 at 02:08:51PM +0200, Andreas Pretzsch wrote:
> On Do, 2018-03-29 at 11:46 +0200, Michael Olbrich wrote:
> > > +ptxd_make_world_license_write_csv() {
> > > +	local -r STRING_QUOTE='"'
> > > +	local -r CSV_SEPARATOR='|'
> > > +	local -a flags=( "${!pkg_license_flags[@]}" )
> > > +	local l
> > > +
> > > +	# not required, seems to be filtered before:
> > > +	#case "${pkg_license}" in
> > > +	#*ignore*)
> > > +	#	# ignore this package, e.g. do not list it in the report
> > > +	#	return 0
> > > +	#	;;
> > > +	#esac
> 
> Any comment on this commented section ?
> Back then, I copied the logic from somewhere, but filtering was not
> required in a real run. Still, kept it as reference.

Just drop this. The license is in the csv, so the processing tool can
filter if necessary.

> I'll rework the patch as above over Easter time, plus
>   - include section tag (see Denis' patch)
>   - squash it into one with "add suffix _tex"

Sounds good, thanks.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de