[ptxdist] [PATCH] license-csv: Add rules to generate CSV license report.

[ptxdist] [PATCH] license-csv: Add rules to generate CSV license report.

[Denis OSTERLAND]

Generates CSV license report along with PDF license report.

Signed-off-by: Denis Osterland <Denis.Osterland@diehl.com>
---
 rules/post/ptxd_make_license_csv.make | 14 ++++++++++++++
 scripts/lib/ptxd_make_license_csv.sh  | 16 ++++++++++++++++
 2 files changed, 30 insertions(+)
 create mode 100644 rules/post/ptxd_make_license_csv.make
 create mode 100644 scripts/lib/ptxd_make_license_csv.sh

diff --git a/rules/post/ptxd_make_license_csv.make b/rules/post/ptxd_make_license_csv.make
new file mode 100644
index 0000000..d74fed8
--- /dev/null
+++ b/rules/post/ptxd_make_license_csv.make
@@ -0,0 +1,14 @@
+
+license-report: $(REPORTDIR)/license-report.csv
+
+$(foreach p,$(PTX_PACKAGES_INSTALL),$(eval $(addprefix $(STATEDIR)/,$(addsuffix .report-csv,$p)): $(addprefix $(STATEDIR)/,$(addsuffix .report,$p))))
+
+$(REPORTDIR)/license-report.csv: $(addprefix $(STATEDIR)/,$(addsuffix .report-csv,$(PTX_PACKAGES_INSTALL)))
+	@$(call targetinfo)
+	cat $(wildcard $(addprefix $(REPORTDIR)/*/,$(addsuffix /report.csv,$(sort $(PTX_PACKAGES_INSTALL))))) > $@
+
+$(STATEDIR)/%.report-csv:
+	@$(call targetinfo)
+	@$(call world/env, $(PTX_MAP_TO_PACKAGE_$(*))) ptxd_make_license_csv
+	@$(call touch)
+
diff --git a/scripts/lib/ptxd_make_license_csv.sh b/scripts/lib/ptxd_make_license_csv.sh
new file mode 100644
index 0000000..f6149f5
--- /dev/null
+++ b/scripts/lib/ptxd_make_license_csv.sh
@@ -0,0 +1,16 @@
+#! /bin/bash
+#
+# Copyright (C) 2017 by Denis Osterland <Denis.Osterland@diehl.com>
+#
+
+ptxd_make_license_csv() {
+	declare -A pkg_license_flags
+	ptxd_make_world_license_init || return
+
+	if [ "${pkg_section}" != "ignore" ]
+		then
+		echo "${pkg_label};${pkg_version};${pkg_section};${pkg_license};${pkg_url}" > ${pkg_license_dir}/report.csv
+		fi
+}
+export -f ptxd_make_license_csv 
+
-- 
2.7.4


Diehl AKO Stiftung & Co. KG, Pfannerstraße 75-83, 88239 Wangen im Allgäu
Bereichsvorstand: Dr.-Ing. Michael Siedentop (Sprecher), Josef Fellner (Mitglied)
Sitz der Gesellschaft: Wangen i.A. – Registergericht: Amtsgericht Ulm HRA 620609 – Persönlich haftende Gesellschafterin: Diehl Verwaltungs-Stiftung – Sitz: Nürnberg – Registergericht: Amtsgericht Nürnberg HRA 11756 –
Vorstand: Dr.-Ing. E.h. Thomas Diehl (†) (Vorsitzender), Herr Dipl.-Wirtsch.-Ing. Wolfgang Weggen (stellvertretender Vorsitzender), Dipl.-Kfm. Claus Günther, Dipl.-Kfm. Frank Gutzeit, Dr.-Ing. Heinrich Schunk, Dr.-Ing. Michael Siedentop , Dipl.-Kfm. Dr.-Ing. Martin Sommer, Dipl.-Ing. (FH) Rainer von Borstel, Vorsitzender des Aufsichtsrates: Dr. Klaus Maier
___________________________________________________________________________________________________
Der Inhalt der vorstehenden E-Mail ist nicht rechtlich bindend. Diese E-Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen.
Informieren Sie uns bitte, wenn Sie diese E-Mail faelschlicherweise erhalten haben. Bitte loeschen Sie in diesem Fall die Nachricht. Jede unerlaubte Form der Reproduktion, Bekanntgabe, Aenderung, Verteilung und/oder Publikation dieser E-Mail ist strengstens untersagt.
The contents of the above mentioned e-mail is not legally binding. This e-mail contains confidential and/or legally protected information. Please inform us if you have received this e-mail by mistake and delete it in such a case. Each unauthorized reproduction, disclosure, alteration, distribution and/or publication of this e-mail is strictly prohibited.

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH] license-csv: Add rules to generate CSV license report.

[Michael Olbrich]

On Thu, Mar 29, 2018 at 07:29:56AM +0000, Denis OSTERLAND wrote:
> Generates CSV license report along with PDF license report.

Something like this seems to be interesting for multiple users... :-)

I prefer the other implementation. I'm sure we can extend the field list to
match your use-case too.

Michael

> Signed-off-by: Denis Osterland <Denis.Osterland@diehl.com>
> ---
>  rules/post/ptxd_make_license_csv.make | 14 ++++++++++++++
>  scripts/lib/ptxd_make_license_csv.sh  | 16 ++++++++++++++++
>  2 files changed, 30 insertions(+)
>  create mode 100644 rules/post/ptxd_make_license_csv.make
>  create mode 100644 scripts/lib/ptxd_make_license_csv.sh
> 
> diff --git a/rules/post/ptxd_make_license_csv.make b/rules/post/ptxd_make_license_csv.make
> new file mode 100644
> index 0000000..d74fed8
> --- /dev/null
> +++ b/rules/post/ptxd_make_license_csv.make
> @@ -0,0 +1,14 @@
> +
> +license-report: $(REPORTDIR)/license-report.csv
> +
> +$(foreach p,$(PTX_PACKAGES_INSTALL),$(eval $(addprefix $(STATEDIR)/,$(addsuffix .report-csv,$p)): $(addprefix $(STATEDIR)/,$(addsuffix .report,$p))))
> +
> +$(REPORTDIR)/license-report.csv: $(addprefix $(STATEDIR)/,$(addsuffix .report-csv,$(PTX_PACKAGES_INSTALL)))
> +	@$(call targetinfo)
> +	cat $(wildcard $(addprefix $(REPORTDIR)/*/,$(addsuffix /report.csv,$(sort $(PTX_PACKAGES_INSTALL))))) > $@
> +
> +$(STATEDIR)/%.report-csv:
> +	@$(call targetinfo)
> +	@$(call world/env, $(PTX_MAP_TO_PACKAGE_$(*))) ptxd_make_license_csv
> +	@$(call touch)
> +
> diff --git a/scripts/lib/ptxd_make_license_csv.sh b/scripts/lib/ptxd_make_license_csv.sh
> new file mode 100644
> index 0000000..f6149f5
> --- /dev/null
> +++ b/scripts/lib/ptxd_make_license_csv.sh
> @@ -0,0 +1,16 @@
> +#! /bin/bash
> +#
> +# Copyright (C) 2017 by Denis Osterland <Denis.Osterland@diehl.com>
> +#
> +
> +ptxd_make_license_csv() {
> +	declare -A pkg_license_flags
> +	ptxd_make_world_license_init || return
> +
> +	if [ "${pkg_section}" != "ignore" ]
> +		then
> +		echo "${pkg_label};${pkg_version};${pkg_section};${pkg_license};${pkg_url}" > ${pkg_license_dir}/report.csv
> +		fi
> +}
> +export -f ptxd_make_license_csv 
> +
> -- 
> 2.7.4
> 
> 
> Diehl AKO Stiftung & Co. KG, Pfannerstraße 75-83, 88239 Wangen im Allgäu
> Bereichsvorstand: Dr.-Ing. Michael Siedentop (Sprecher), Josef Fellner (Mitglied)
> Sitz der Gesellschaft: Wangen i.A. – Registergericht: Amtsgericht Ulm HRA 620609 – Persönlich haftende Gesellschafterin: Diehl Verwaltungs-Stiftung – Sitz: Nürnberg – Registergericht: Amtsgericht Nürnberg HRA 11756 –
> Vorstand: Dr.-Ing. E.h. Thomas Diehl (†) (Vorsitzender), Herr Dipl.-Wirtsch.-Ing. Wolfgang Weggen (stellvertretender Vorsitzender), Dipl.-Kfm. Claus Günther, Dipl.-Kfm. Frank Gutzeit, Dr.-Ing. Heinrich Schunk, Dr.-Ing. Michael Siedentop , Dipl.-Kfm. Dr.-Ing. Martin Sommer, Dipl.-Ing. (FH) Rainer von Borstel, Vorsitzender des Aufsichtsrates: Dr. Klaus Maier
> ___________________________________________________________________________________________________
> Der Inhalt der vorstehenden E-Mail ist nicht rechtlich bindend. Diese E-Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen.
> Informieren Sie uns bitte, wenn Sie diese E-Mail faelschlicherweise erhalten haben. Bitte loeschen Sie in diesem Fall die Nachricht. Jede unerlaubte Form der Reproduktion, Bekanntgabe, Aenderung, Verteilung und/oder Publikation dieser E-Mail ist strengstens untersagt.
> The contents of the above mentioned e-mail is not legally binding. This e-mail contains confidential and/or legally protected information. Please inform us if you have received this e-mail by mistake and delete it in such a case. Each unauthorized reproduction, disclosure, alteration, distribution and/or publication of this e-mail is strictly prohibited.
> 
> _______________________________________________
> ptxdist mailing list
> ptxdist@pengutronix.de

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH] license-csv: Add rules to generate CSV license report.

[Andreas Pretzsch]

On Do, 2018-03-29 at 07:29 +0000, Denis OSTERLAND wrote:
> +++ b/scripts/lib/ptxd_make_license_csv.sh
> @@ -0,0 +1,16 @@
> +#! /bin/bash
> +#
> +# Copyright (C) 2017 by Denis Osterland <Denis.Osterland@diehl.com>
> +#
> +
> +ptxd_make_license_csv() {
> +       declare -A pkg_license_flags
> +       ptxd_make_world_license_init || return
> +
> +       if [ "${pkg_section}" != "ignore" ]
> +               then
> +               echo "${pkg_label};${pkg_version};${pkg_section};
> ${pkg_license};${pkg_url}" > ${pkg_license_dir}/report.csv
> +               fi
> +}
> +export -f ptxd_make_license_csv 

Extending the field list by section is a good point, thanks. Will do.

But I would refrain from including the URL (for now), simply as it will
include all sorts of characters, which then will require proper
escaping. And in 20+ years, I've seen way too many broken parsers (and
escapers)...
Beside, the key point of the CSV list (to me / customers) is to have a
diff'able list of included packages (which ones, version, license),
beside the full fledged report. To see if one needs to look into
something. So URL is nice, but not mandatory.

Last, about the CSV separator, I chose | out of experience, avoiding
most collisions. Both , and ; show up often in text and even version
tags. Yes, they are marked as string with "", but back to "broken
parsers"...

So I still tend to a CSV output like:

"name"|"version"|"section"|"licenses"|"flags"
"somepackage"|"1.2.3"|"osi-conform"|"Apache-2.0"|"attribution"
[...]


Denis, what do you think ?
Michael, any experience/opinion on this ?

Best regards,
  Andreas

-- 

carpe noctem engineering
Ingenieurbuero fuer Hard- & Software-Entwicklung Andreas Pretzsch
Dipl.-Ing. (FH) Andreas Pretzsch        Tel. +49-(0)7307-936088-1
Lange Strasse 28a                       Fax: +49-(0)7307-936088-9
89250 Senden, Germany                   email: apr@cn-eng.de


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH] license-csv: Add rules to generate CSV license report.

[Denis OSTERLAND]

Am Donnerstag, den 29.03.2018, 14:08 +0200 schrieb Andreas Pretzsch:
> On Do, 2018-03-29 at 07:29 +0000, Denis OSTERLAND wrote:
> > 
> > +++ b/scripts/lib/ptxd_make_license_csv.sh
> > @@ -0,0 +1,16 @@
> > +#! /bin/bash
> > +#
> > +# Copyright (C) 2017 by Denis Osterland <Denis.Osterland@diehl.com>
> > +#
> > +
> > +ptxd_make_license_csv() {
> > +       declare -A pkg_license_flags
> > +       ptxd_make_world_license_init || return
> > +
> > +       if [ "${pkg_section}" != "ignore" ]
> > +               then
> > +               echo "${pkg_label};${pkg_version};${pkg_section};
> > ${pkg_license};${pkg_url}" > ${pkg_license_dir}/report.csv
> > +               fi
> > +}
> > +export -f ptxd_make_license_csv 
> Extending the field list by section is a good point, thanks. Will do.
> 
> But I would refrain from including the URL (for now), simply as it will
> include all sorts of characters, which then will require proper
> escaping. And in 20+ years, I've seen way too many broken parsers (and
> escapers)...
> Beside, the key point of the CSV list (to me / customers) is to have a
> diff'able list of included packages (which ones, version, license),
> beside the full fledged report. To see if one needs to look into
> something. So URL is nice, but not mandatory.
> 
> Last, about the CSV separator, I chose | out of experience, avoiding
> most collisions. Both , and ; show up often in text and even version
> tags. Yes, they are marked as string with "", but back to "broken
> parsers"...
> 
> So I still tend to a CSV output like:
> 
> "name"|"version"|"section"|"licenses"|"flags"
> "somepackage"|"1.2.3"|"osi-conform"|"Apache-2.0"|"attribution"
> [...]
> 
> 
> Denis, what do you think ?
You are right, URL tend to be brittle.
We use it as input for an overview page on embedded web server.
This will need some refactoring.
By the way, is it possible depend on your csv report from make?
I guess I will need to add some wrapper target to call sh function.

Regards Denis
> Michael, any experience/opinion on this ?
> 
> Best regards,
>   Andreas
> 
Diehl AKO Stiftung & Co. KG, Pfannerstraße 75-83, 88239 Wangen im Allgäu
Bereichsvorstand: Dr.-Ing. Michael Siedentop (Sprecher), Josef Fellner (Mitglied)
Sitz der Gesellschaft: Wangen i.A. – Registergericht: Amtsgericht Ulm HRA 620609 – Persönlich haftende Gesellschafterin: Diehl Verwaltungs-Stiftung – Sitz: Nürnberg – Registergericht: Amtsgericht Nürnberg HRA 11756 –
Vorstand: Dr.-Ing. E.h. Thomas Diehl (†) (Vorsitzender), Herr Dipl.-Wirtsch.-Ing. Wolfgang Weggen (stellvertretender Vorsitzender), Dipl.-Kfm. Claus Günther, Dipl.-Kfm. Frank Gutzeit, Dr.-Ing. Heinrich Schunk, Dr.-Ing. Michael Siedentop , Dipl.-Kfm. Dr.-Ing. Martin Sommer, Dipl.-Ing. (FH) Rainer von Borstel, Vorsitzender des Aufsichtsrates: Dr. Klaus Maier
___________________________________________________________________________________________________
Der Inhalt der vorstehenden E-Mail ist nicht rechtlich bindend. Diese E-Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen.
Informieren Sie uns bitte, wenn Sie diese E-Mail faelschlicherweise erhalten haben. Bitte loeschen Sie in diesem Fall die Nachricht. Jede unerlaubte Form der Reproduktion, Bekanntgabe, Aenderung, Verteilung und/oder Publikation dieser E-Mail ist strengstens untersagt.
The contents of the above mentioned e-mail is not legally binding. This e-mail contains confidential and/or legally protected information. Please inform us if you have received this e-mail by mistake and delete it in such a case. Each unauthorized reproduction, disclosure, alteration, distribution and/or publication of this e-mail is strictly prohibited.
_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH] license-csv: Add rules to generate CSV license report.

[Michael Olbrich]

On Thu, Mar 29, 2018 at 12:46:16PM +0000, Denis OSTERLAND wrote:
> Am Donnerstag, den 29.03.2018, 14:08 +0200 schrieb Andreas Pretzsch:
> > On Do, 2018-03-29 at 07:29 +0000, Denis OSTERLAND wrote:
> > Extending the field list by section is a good point, thanks. Will do.
> > 
> > But I would refrain from including the URL (for now), simply as it will
> > include all sorts of characters, which then will require proper
> > escaping. And in 20+ years, I've seen way too many broken parsers (and
> > escapers)...
> > Beside, the key point of the CSV list (to me / customers) is to have a
> > diff'able list of included packages (which ones, version, license),
> > beside the full fledged report. To see if one needs to look into
> > something. So URL is nice, but not mandatory.
> > 
> > Last, about the CSV separator, I chose | out of experience, avoiding
> > most collisions. Both , and ; show up often in text and even version
> > tags. Yes, they are marked as string with "", but back to "broken
> > parsers"...
> > 
> > So I still tend to a CSV output like:
> > 
> > "name"|"version"|"section"|"licenses"|"flags"
> > "somepackage"|"1.2.3"|"osi-conform"|"Apache-2.0"|"attribution"
> > [...]
> > 
> > 
> > Denis, what do you think ?
> You are right, URL tend to be brittle.
> We use it as input for an overview page on embedded web server.
> This will need some refactoring.

You could overwrite the function that generates the per package file in
your BSP.

> By the way, is it possible depend on your csv report from make?
> I guess I will need to add some wrapper target to call sh function.

There is a make target for the license report. If necessary we can
restructure this as needed.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [PATCH] license-csv: Add rules to generate CSV license report.

[Andreas Pretzsch]

On Do, 2018-03-29 at 12:46 +0000, Denis OSTERLAND wrote:
> > But I would refrain from including the URL (for now), simply as it
> will
> > include all sorts of characters, which then will require proper
> > escaping.
> > [...] 
> > 
> > Denis, what do you think ?
> You are right, URL tend to be brittle.
> We use it as input for an overview page on embedded web server.
> This will need some refactoring.

Hmmmm... Based on research and quite some debates with lawyers, I'd say
to have "only" the URL in a package list is critical at least.
For a solid conformance with the various FOSS licenses, you should
include most (if not all) of the content of the ptxdist license report.
The one from "license-compliance-distribution" would be enough, but I'd
suggest just to include the complete one.

My suggestion here would be to omit the URL in the overview page, and
also link to the license-blah.pdf (on the embedded webserver). Given you
have the storage space available.
So you provide everything in the device, and need no special hacks.

Of course, you could always add a (modified) version of 
  scripts/lib/ptxd_make_license_report.sh
  scripts/lib/ptxd_make_world_license.sh
to your BSP folder under "scripts/lib/". Those will override the ones
from "official" PTXdist.


> By the way, is it possible depend on your csv report from make?
> I guess I will need to add some wrapper target to call sh function.

About adding a wrapper called from make, see
"rules/post/ptxd_make_license_report.make".
Maybe as an additional step to "license-compliance-distribution".

Having something special for CSV, well, I just included it in the normal
flow, beside the TeX/PDF generation. Would say, that's a good place.

Of course, some hook to generate such a page would be nice, too.
But this is something Michael has to decide, mainly from a conceptual
point of view.


Best regards,
  Andreas

-- 

carpe noctem engineering
Ingenieurbuero fuer Hard- & Software-Entwicklung Andreas Pretzsch
Dipl.-Ing. (FH) Andreas Pretzsch        Tel. +49-(0)7307-936088-1
Lange Strasse 28a                       Fax: +49-(0)7307-936088-9
89250 Senden, Germany                   email: apr@cn-eng.de


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de