[ptxdist] [RFC 0/2] add a way to extract license text from packages

[ptxdist] [RFC 0/2] add a way to extract license text from packages

[Markus Niebel]

From: Markus Niebel <Markus.Niebel@tq-group.com>

just as an RFC: it would be good to have above said features to ease
deployment and auditing.

First patch implements this feature as an addon of the targetinstall 
stage. Second patch shows the feature for a package that uses it's own
license

TODO: 
maybe targetinstall is not the right place, just in case the license
is altered during patching the package.

Markus Niebel (2):
  ptxdist: add license text extraction
  icu: add LICENSE_TEXT

 rules/icu.make                         |  5 +++--
 rules/post/ptxd_make_world_common.make |  1 +
 scripts/lib/ptxd_make_xpkg_common.sh   |  3 +++
 scripts/lib/ptxd_make_xpkg_finish.sh   | 11 +++++++++++
 4 files changed, 18 insertions(+), 2 deletions(-)

-- 
2.2.1


-- 
ptxdist mailing list
ptxdist@pengutronix.de

[ptxdist] [RFC 1/2] ptxdist: add license text extraction

[Markus Niebel]

From: Markus Niebel <Markus.Niebel@tq-group.com>

It would be good to have the possibility to extract license texts for
target packages. This would add a benefit to ptxdist since it makes
auditing a BSP easier.

This patch adds

- a new (optional) variable to package rule file to encode the
  license file(s) for this package
- set a pkg var with the value new var from the package rule
  or use default list of filenames
- copy the license text files in the targetinstall.post stage to
  a directory under $PTXDIST_PLATFORMDIR/<pkg>-<pkg_version>

Signed-off-by: Markus Niebel <Markus.Niebel@tq-group.com>
---
 rules/post/ptxd_make_world_common.make |  1 +
 scripts/lib/ptxd_make_xpkg_common.sh   |  3 +++
 scripts/lib/ptxd_make_xpkg_finish.sh   | 11 +++++++++++
 3 files changed, 15 insertions(+)

diff --git a/rules/post/ptxd_make_world_common.make b/rules/post/ptxd_make_world_common.make
index 4820d8c..01d2c88 100644
--- a/rules/post/ptxd_make_world_common.make
+++ b/rules/post/ptxd_make_world_common.make
@@ -55,6 +55,7 @@ world/env/impl = \
 	pkg_pkg_dir="$(call ptx/escape,$($(1)_PKGDIR))"				\
 	pkg_pkg_dev="$(call ptx/escape,$($(1)_DEVPKG))"				\
 	pkg_license="$(call ptx/escape,$($(1)_LICENSE))"			\
+	pkg_license_text="$(call ptx/escape,$($(1)_LICENSE_TEXT))"	\
 	pkg_build_deps="$(call ptx/escape,$(PTX_MAP_B_dep_$(1)))"		\
 	pkg_run_deps="$(call ptx/escape,$(PTX_MAP_R_dep_$(1)))"			\
 										\
diff --git a/scripts/lib/ptxd_make_xpkg_common.sh b/scripts/lib/ptxd_make_xpkg_common.sh
index 279348f..90501eb 100644
--- a/scripts/lib/ptxd_make_xpkg_common.sh
+++ b/scripts/lib/ptxd_make_xpkg_common.sh
@@ -87,6 +87,9 @@ ptxd_make_xpkg_init() {
     pkg_license="${pkg_license:-unknown}"
     pkg_xpkg_license="${pkg_xpkg_license:-${pkg_license}}"
     pkg_xpkg_license_file="${ptx_state_dir}/${pkg_xpkg}.license"
+    pkg_license_text="${pkg_license_text:-COPYING COPYING3 COPYING.LIB COPYING3.LIB}"
+    pkg_xpkg_license_text="${pkg_xpkg_license_text:-${pkg_license_text}}"
+    pkg_xpkg_license_dir="${PTXDIST_PLATFORMDIR}/licenses/${pkg_xpkg}-${pkg_xpkg_version}"
 
     # packaging stuff
     pkg_xpkg_install_deps="${ptx_state_dir}/${pkg_xpkg}.deps"
diff --git a/scripts/lib/ptxd_make_xpkg_finish.sh b/scripts/lib/ptxd_make_xpkg_finish.sh
index e56be89..ec28710 100644
--- a/scripts/lib/ptxd_make_xpkg_finish.sh
+++ b/scripts/lib/ptxd_make_xpkg_finish.sh
@@ -63,6 +63,17 @@ EOF
     #
     echo -n "xpkg_finish:	collecting license (${pkg_xpkg_license}) ... "
     echo "${pkg_xpkg_license}" > "${pkg_xpkg_license_file}"
+    #
+    # create a package specific dir to store the license information
+    #
+    mkdir -p "${pkg_xpkg_license_dir}"
+    cp "${pkg_xpkg_license_file}" "${pkg_xpkg_license_dir}/license"
+    for f in ${pkg_xpkg_license_text}
+    do
+        if [ -f "${pkg_dir}/${f}" ]; then
+            cp "${pkg_dir}/${f}" "${pkg_xpkg_license_dir}"
+        fi
+    done
     echo "done."
 
     #
-- 
2.2.1


-- 
ptxdist mailing list
ptxdist@pengutronix.de

[ptxdist] [RFC 2/2] icu: add LICENSE_TEXT

[Markus Niebel]

From: Markus Niebel <Markus.Niebel@tq-group.com>

this patch enables to extract the license text
when at it, name the license for the package

Signed-off-by: Markus Niebel <Markus.Niebel@tq-group.com>
---
 rules/icu.make | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/rules/icu.make b/rules/icu.make
index 312a7b9..ca6d3be 100644
--- a/rules/icu.make
+++ b/rules/icu.make
@@ -24,8 +24,9 @@ ICU_URL		:= http://download.icu-project.org/files/icu4c/$(ICU_VERSION)/$(ICU).$(
 ICU_SOURCE	:= $(SRCDIR)/$(ICU).$(ICU_SUFFIX)
 ICU_DIR		:= $(BUILDDIR)/$(ICU)
 ICU_SUBDIR	:= source
-ICU_LICENSE	:= unknown
-
+# license currently not in SPDX database, maybe short name will change
+ICU_LICENSE	:= ICU
+ICU_LICENSE_TEXT	:= license.html
 # ----------------------------------------------------------------------------
 # Prepare
 # ----------------------------------------------------------------------------
-- 
2.2.1


-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [RFC 0/2] add a way to extract license text from packages

[Markus Niebel]

Am 21.01.2015 um 09:12 schrieb Markus Niebel:
> From: Markus Niebel <Markus.Niebel@tq-group.com>
> 
> just as an RFC: it would be good to have above said features to ease
> deployment and auditing.
> 
> First patch implements this feature as an addon of the targetinstall 
> stage. Second patch shows the feature for a package that uses it's own
> license
> 
> TODO: 
> maybe targetinstall is not the right place, just in case the license
> is altered during patching the package.
> 
> Markus Niebel (2):
>   ptxdist: add license text extraction
>   icu: add LICENSE_TEXT
> 
>  rules/icu.make                         |  5 +++--
>  rules/post/ptxd_make_world_common.make |  1 +
>  scripts/lib/ptxd_make_xpkg_common.sh   |  3 +++
>  scripts/lib/ptxd_make_xpkg_finish.sh   | 11 +++++++++++
>  4 files changed, 18 insertions(+), 2 deletions(-)
> 
ping ...

Any interest in this topic?

Regards

Markus

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [RFC 0/2] add a way to extract license text from packages

[Guillermo Rodriguez Garcia]

Hi,

If I understand correctly the auditing procedure before this patch would be:

1. Collect all existing files in state/<pkg>.license
2. Each .license file contains the license name (e.g. "GPL") -- use
this to find the actual license text
3. Audit each license

What would be the difference after this patch?

Thanks,

Guillermo


2015-01-29 8:45 GMT+01:00 Markus Niebel <list-09@tqsc.de>:
> Am 21.01.2015 um 09:12 schrieb Markus Niebel:
>> From: Markus Niebel <Markus.Niebel@tq-group.com>
>>
>> just as an RFC: it would be good to have above said features to ease
>> deployment and auditing.
>>
>> First patch implements this feature as an addon of the targetinstall
>> stage. Second patch shows the feature for a package that uses it's own
>> license
>>
>> TODO:
>> maybe targetinstall is not the right place, just in case the license
>> is altered during patching the package.
>>
>> Markus Niebel (2):
>>   ptxdist: add license text extraction
>>   icu: add LICENSE_TEXT
>>
>>  rules/icu.make                         |  5 +++--
>>  rules/post/ptxd_make_world_common.make |  1 +
>>  scripts/lib/ptxd_make_xpkg_common.sh   |  3 +++
>>  scripts/lib/ptxd_make_xpkg_finish.sh   | 11 +++++++++++
>>  4 files changed, 18 insertions(+), 2 deletions(-)
>>
> ping ...
>
> Any interest in this topic?
>
> Regards
>
> Markus
>
> --
> ptxdist mailing list
> ptxdist@pengutronix.de



-- 
Guillermo Rodriguez Garcia
guille.rodriguez@gmail.com

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [RFC 0/2] add a way to extract license text from packages

[Markus Niebel]

Hello,
Am 29.01.2015 um 21:23 schrieb Guillermo Rodriguez Garcia:
> Hi,
> 
> If I understand correctly the auditing procedure before this patch would be:
> 
> 1. Collect all existing files in state/<pkg>.license
> 2. Each .license file contains the license name (e.g. "GPL") -- use
> this to find the actual license text
> 3. Audit each license
> 
> What would be the difference after this patch?
> 

My goal is to have a single place where the licenses of the target packages
are collected (think of it like export_src for license information - license
names and license text). 

Motivation:
- Some packages have additional clauses to the license text (Qt for instances). 
  So it would be a benefit to have the text from the package.
- If you deliver binaries of a BSP (preinstalled or for download) some licenses
  force you to supply the license text with the binary
- In some cases non developers need to look over the licenses
- AFAIK other build tools (yocto, buildroot) has similar features

Hope this helps

Markus

> Thanks,
> 
> Guillermo
> 
> 
> 2015-01-29 8:45 GMT+01:00 Markus Niebel <list-09@tqsc.de>:
>> Am 21.01.2015 um 09:12 schrieb Markus Niebel:
>>> From: Markus Niebel <Markus.Niebel@tq-group.com>
>>>
>>> just as an RFC: it would be good to have above said features to ease
>>> deployment and auditing.
>>>
>>> First patch implements this feature as an addon of the targetinstall
>>> stage. Second patch shows the feature for a package that uses it's own
>>> license
>>>
>>> TODO:
>>> maybe targetinstall is not the right place, just in case the license
>>> is altered during patching the package.
>>>
>>> Markus Niebel (2):
>>>   ptxdist: add license text extraction
>>>   icu: add LICENSE_TEXT
>>>
>>>  rules/icu.make                         |  5 +++--
>>>  rules/post/ptxd_make_world_common.make |  1 +
>>>  scripts/lib/ptxd_make_xpkg_common.sh   |  3 +++
>>>  scripts/lib/ptxd_make_xpkg_finish.sh   | 11 +++++++++++
>>>  4 files changed, 18 insertions(+), 2 deletions(-)
>>>
>> ping ...
>>
>> Any interest in this topic?
>>
>> Regards
>>
>> Markus
>>
>> --
>> ptxdist mailing list
>> ptxdist@pengutronix.de
> 
> 
> 


-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [RFC 0/2] add a way to extract license text from packages

[Guillermo Rodriguez Garcia]

Hello Markus,

I'm no expert at this topic but here is how I would handle this:

1. Collect all existing files in state/<pkg>.license into a license
list. Example:

#!/bin/sh

if [ $# -eq 0 -o ! -e $1/state ]; then
    echo "usage: $0 <ptxdist_platform_dir>"
    exit 1
fi
for file in $1/state/*.license ; do
    pkg=${file##*/}
    pkg=${pkg%%.*}
    echo "$pkg: `cat $file`"
done

2. Build a tool that would go through this list and download/collect
the actual license texts.

i.e. very similar to what you already did, but outside of ptxdist.

Then if this can be easily done "outside" of ptxdist (as a separate
script) I guess that the real question is whether this is a desirable
feature to have in ptxdist itself. But this is something for the
ptxdist experts to decide :-)

Guillermo

2015-01-30 8:48 GMT+01:00 Markus Niebel <list-09_ptxdist@tqsc.de>:
> Hello,
> Am 29.01.2015 um 21:23 schrieb Guillermo Rodriguez Garcia:
>> Hi,
>>
>> If I understand correctly the auditing procedure before this patch would be:
>>
>> 1. Collect all existing files in state/<pkg>.license
>> 2. Each .license file contains the license name (e.g. "GPL") -- use
>> this to find the actual license text
>> 3. Audit each license
>>
>> What would be the difference after this patch?
>>
>
> My goal is to have a single place where the licenses of the target packages
> are collected (think of it like export_src for license information - license
> names and license text).
>
> Motivation:
> - Some packages have additional clauses to the license text (Qt for instances).
>   So it would be a benefit to have the text from the package.
> - If you deliver binaries of a BSP (preinstalled or for download) some licenses
>   force you to supply the license text with the binary
> - In some cases non developers need to look over the licenses
> - AFAIK other build tools (yocto, buildroot) has similar features
>
> Hope this helps
>
> Markus
>
>> Thanks,
>>
>> Guillermo
>>
>>
>> 2015-01-29 8:45 GMT+01:00 Markus Niebel <list-09@tqsc.de>:
>>> Am 21.01.2015 um 09:12 schrieb Markus Niebel:
>>>> From: Markus Niebel <Markus.Niebel@tq-group.com>
>>>>
>>>> just as an RFC: it would be good to have above said features to ease
>>>> deployment and auditing.
>>>>
>>>> First patch implements this feature as an addon of the targetinstall
>>>> stage. Second patch shows the feature for a package that uses it's own
>>>> license
>>>>
>>>> TODO:
>>>> maybe targetinstall is not the right place, just in case the license
>>>> is altered during patching the package.
>>>>
>>>> Markus Niebel (2):
>>>>   ptxdist: add license text extraction
>>>>   icu: add LICENSE_TEXT
>>>>
>>>>  rules/icu.make                         |  5 +++--
>>>>  rules/post/ptxd_make_world_common.make |  1 +
>>>>  scripts/lib/ptxd_make_xpkg_common.sh   |  3 +++
>>>>  scripts/lib/ptxd_make_xpkg_finish.sh   | 11 +++++++++++
>>>>  4 files changed, 18 insertions(+), 2 deletions(-)
>>>>
>>> ping ...
>>>
>>> Any interest in this topic?
>>>
>>> Regards
>>>
>>> Markus
>>>
>>> --
>>> ptxdist mailing list
>>> ptxdist@pengutronix.de
>>
>>
>>
>
>
> --
> ptxdist mailing list
> ptxdist@pengutronix.de



-- 
Guillermo Rodriguez Garcia
guille.rodriguez@gmail.com

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [RFC 0/2] add a way to extract license text from packages

[Hubert Feurstein]

Hello,

IMHO the proposal by Markus makes sense. I have had the same issues
with collecting license texts in the past. So a solution integrated in
PTXdist would be great. Because then it would be possible to collect
the license texts depending on the config options of a package
(sometimes plugins have a different license).

Hubert

2015-01-30 8:48 GMT+01:00 Markus Niebel <list-09_ptxdist@tqsc.de>:
> Hello,
> Am 29.01.2015 um 21:23 schrieb Guillermo Rodriguez Garcia:
>> Hi,
>>
>> If I understand correctly the auditing procedure before this patch would be:
>>
>> 1. Collect all existing files in state/<pkg>.license
>> 2. Each .license file contains the license name (e.g. "GPL") -- use
>> this to find the actual license text
>> 3. Audit each license
>>
>> What would be the difference after this patch?
>>
>
> My goal is to have a single place where the licenses of the target packages
> are collected (think of it like export_src for license information - license
> names and license text).
>
> Motivation:
> - Some packages have additional clauses to the license text (Qt for instances).
>   So it would be a benefit to have the text from the package.
> - If you deliver binaries of a BSP (preinstalled or for download) some licenses
>   force you to supply the license text with the binary
> - In some cases non developers need to look over the licenses
> - AFAIK other build tools (yocto, buildroot) has similar features
>
> Hope this helps
>
> Markus
>
>> Thanks,
>>
>> Guillermo
>>
>>
>> 2015-01-29 8:45 GMT+01:00 Markus Niebel <list-09@tqsc.de>:
>>> Am 21.01.2015 um 09:12 schrieb Markus Niebel:
>>>> From: Markus Niebel <Markus.Niebel@tq-group.com>
>>>>
>>>> just as an RFC: it would be good to have above said features to ease
>>>> deployment and auditing.
>>>>
>>>> First patch implements this feature as an addon of the targetinstall
>>>> stage. Second patch shows the feature for a package that uses it's own
>>>> license
>>>>
>>>> TODO:
>>>> maybe targetinstall is not the right place, just in case the license
>>>> is altered during patching the package.
>>>>
>>>> Markus Niebel (2):
>>>>   ptxdist: add license text extraction
>>>>   icu: add LICENSE_TEXT
>>>>
>>>>  rules/icu.make                         |  5 +++--
>>>>  rules/post/ptxd_make_world_common.make |  1 +
>>>>  scripts/lib/ptxd_make_xpkg_common.sh   |  3 +++
>>>>  scripts/lib/ptxd_make_xpkg_finish.sh   | 11 +++++++++++
>>>>  4 files changed, 18 insertions(+), 2 deletions(-)
>>>>
>>> ping ...
>>>
>>> Any interest in this topic?
>>>
>>> Regards
>>>
>>> Markus
>>>
>>> --
>>> ptxdist mailing list
>>> ptxdist@pengutronix.de
>>
>>
>>
>
>
> --
> ptxdist mailing list
> ptxdist@pengutronix.de

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [RFC 0/2] add a way to extract license text from packages

[Juergen Borleis]

Hi,

On Wednesday 11 February 2015 11:50:16 Hubert Feurstein wrote:
> IMHO the proposal by Markus makes sense. I have had the same issues
> with collecting license texts in the past. So a solution integrated in
> PTXdist would be great. Because then it would be possible to collect
> the license texts depending on the config options of a package
> (sometimes plugins have a different license).

Please checkout current PTXdist master and run "ptxdist make license-report" 
inside a PTXdist project. You need "LATEX" and "dot" to make it work.

Regards,
Juergen

-- 
Pengutronix e.K.                              | Juergen Borleis             |
Industrial Linux Solutions                    | http://www.pengutronix.de/  |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [RFC 0/2] add a way to extract license text from packages

[Michael Olbrich]

Hi,

On Wed, Feb 11, 2015 at 12:19:34PM +0100, Juergen Borleis wrote:
> On Wednesday 11 February 2015 11:50:16 Hubert Feurstein wrote:
> > IMHO the proposal by Markus makes sense. I have had the same issues
> > with collecting license texts in the past. So a solution integrated in
> > PTXdist would be great. Because then it would be possible to collect
> > the license texts depending on the config options of a package
> > (sometimes plugins have a different license).
> 
> Please checkout current PTXdist master and run "ptxdist make license-report" 
> inside a PTXdist project. You need "LATEX" and "dot" to make it work.

xelatex (from texlive-xetex in Debian) and dot2tex are needed as well.

The raw license data for each package is collected in
platform-<something>/state/report/<pkg>/ if you have use for that.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [RFC 0/2] add a way to extract license text from packages

[Markus Niebel]

Hello,
Am 11.02.2015 um 14:02 schrieb Michael Olbrich:
> Hi,
> 
> On Wed, Feb 11, 2015 at 12:19:34PM +0100, Juergen Borleis wrote:
>> On Wednesday 11 February 2015 11:50:16 Hubert Feurstein wrote:
>>> IMHO the proposal by Markus makes sense. I have had the same issues
>>> with collecting license texts in the past. So a solution integrated in
>>> PTXdist would be great. Because then it would be possible to collect
>>> the license texts depending on the config options of a package
>>> (sometimes plugins have a different license).
>>
>> Please checkout current PTXdist master and run "ptxdist make license-report" 
>> inside a PTXdist project. You need "LATEX" and "dot" to make it work.
> 
> xelatex (from texlive-xetex in Debian) and dot2tex are needed as well.
> 

Thank you, test looks promising and generated data are very helpfull. 
Just a hint for documentation - recent versions of the LATEX packages are needed, 
because of style packages used.

Markus

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [RFC 0/2] add a way to extract license text from packages

[Michael Olbrich]

On Tue, Feb 17, 2015 at 09:10:09AM +0100, Markus Niebel wrote:
> Am 11.02.2015 um 14:02 schrieb Michael Olbrich:
> > On Wed, Feb 11, 2015 at 12:19:34PM +0100, Juergen Borleis wrote:
> >> On Wednesday 11 February 2015 11:50:16 Hubert Feurstein wrote:
> >>> IMHO the proposal by Markus makes sense. I have had the same issues
> >>> with collecting license texts in the past. So a solution integrated in
> >>> PTXdist would be great. Because then it would be possible to collect
> >>> the license texts depending on the config options of a package
> >>> (sometimes plugins have a different license).
> >>
> >> Please checkout current PTXdist master and run "ptxdist make license-report" 
> >> inside a PTXdist project. You need "LATEX" and "dot" to make it work.
> > 
> > xelatex (from texlive-xetex in Debian) and dot2tex are needed as well.
> > 
> 
> Thank you, test looks promising and generated data are very helpfull. 
> Just a hint for documentation - recent versions of the LATEX packages are needed, 
> because of style packages used.

It worked on my Debian unstable :-). I'd be happy to take patches that make
it work with older versions if there is a need for that.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [RFC 0/2] add a way to extract license text from packages

[Markus Niebel]

Hello Juergen,

Am 11.02.2015 um 12:19 schrieb Juergen Borleis:
> Hi,
> 
> On Wednesday 11 February 2015 11:50:16 Hubert Feurstein wrote:
>> IMHO the proposal by Markus makes sense. I have had the same issues
>> with collecting license texts in the past. So a solution integrated in
>> PTXdist would be great. Because then it would be possible to collect
>> the license texts depending on the config options of a package
>> (sometimes plugins have a different license).
> 
> Please checkout current PTXdist master and run "ptxdist make license-report" 
> inside a PTXdist project. You need "LATEX" and "dot" to make it work.
> 

Sounds great, will try it. Thanks

Regards,
Markus

> Regards,
> Juergen
> 


-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] [RFC 0/2] add a way to extract license text from packages

[Robert Schwebel]

Hi,

On Thu, Feb 12, 2015 at 08:33:32AM +0100, Markus Niebel wrote:
> > On Wednesday 11 February 2015 11:50:16 Hubert Feurstein wrote:
> >> IMHO the proposal by Markus makes sense. I have had the same issues
> >> with collecting license texts in the past. So a solution integrated in
> >> PTXdist would be great. Because then it would be possible to collect
> >> the license texts depending on the config options of a package
> >> (sometimes plugins have a different license).
> > 
> > Please checkout current PTXdist master and run "ptxdist make license-report" 
> > inside a PTXdist project. You need "LATEX" and "dot" to make it work.
> 
> Sounds great, will try it. Thanks

Please do so - it's a good opportunity to share forces here. The whole
license compliance topic is gaining speed recently.

rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

[ptxdist] license-report: extend to generate textual list of package + licenses

[Andreas Pretzsch]

On Mi, 2015-02-11 at 12:19 +0100, Juergen Borleis wrote:
> Hi,
> 
> On Wednesday 11 February 2015 11:50:16 Hubert Feurstein wrote:
> > IMHO the proposal by Markus makes sense. I have had the same issues
> > with collecting license texts in the past. So a solution integrated in
> > PTXdist would be great. Because then it would be possible to collect
> > the license texts depending on the config options of a package
> > (sometimes plugins have a different license).
> 
> Please checkout current PTXdist master and run "ptxdist make license-report" 
> inside a PTXdist project. You need "LATEX" and "dot" to make it work.

First, this one is wonderful, and works like a charm. Thanks a lot.

What I am missing is an overview.
Mainly as an external list, but maybe also in the reports.

I'm talking about e.g. a simple CSV, listing all the included packages,
like in "pkgname | pkgversion | licenses". So something like
[...]
busybox|1.24.2|GPL-2.0
[...]
Sensible escaping would be something to look at, but anyway.

Point is, this list would be easily diff'able between releases.
And lawyers tend to be more expensive than developers...

Also, one can sort it and make matrices of any kind, e.g. sort by GPLv3,
and so on.

Above would be the minimum information, for anything else, one can look
into the big report. Maybe the flags like "attribution" and "choice"
would be interessting, too.

Now, this can be done with some shell scripting, outside of the build
process.
But as there is already the license-report generation, this would be the
perfect place for it.

Has anyone already (partly) implemented this ?
Would be happy to pick up the pieces and also mainline it. Just asking,
before I start hacking things twice...


-- 

carpe noctem engineering
Ingenieurbuero fuer Hard- & Software-Entwicklung Andreas Pretzsch
Dipl.-Ing. (FH) Andreas Pretzsch        Tel. +49-(0)7307-936088-1
Lange Strasse 28a                       Fax: +49-(0)7307-936088-9
89250 Senden, Germany                   email: apr@cn-eng.de


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] license-report: extend to generate textual list of package + licenses

[Michael Olbrich]

Hi,

On Tue, Oct 04, 2016 at 12:23:00AM +0200, Andreas Pretzsch wrote:
> On Mi, 2015-02-11 at 12:19 +0100, Juergen Borleis wrote:
> > On Wednesday 11 February 2015 11:50:16 Hubert Feurstein wrote:
> > > IMHO the proposal by Markus makes sense. I have had the same issues
> > > with collecting license texts in the past. So a solution integrated in
> > > PTXdist would be great. Because then it would be possible to collect
> > > the license texts depending on the config options of a package
> > > (sometimes plugins have a different license).
> > 
> > Please checkout current PTXdist master and run "ptxdist make license-report" 
> > inside a PTXdist project. You need "LATEX" and "dot" to make it work.
> 
> First, this one is wonderful, and works like a charm. Thanks a lot.
> 
> What I am missing is an overview.
> Mainly as an external list, but maybe also in the reports.
> 
> I'm talking about e.g. a simple CSV, listing all the included packages,
> like in "pkgname | pkgversion | licenses". So something like
> [...]
> busybox|1.24.2|GPL-2.0
> [...]
> Sensible escaping would be something to look at, but anyway.
> 
> Point is, this list would be easily diff'able between releases.
> And lawyers tend to be more expensive than developers...
> 
> Also, one can sort it and make matrices of any kind, e.g. sort by GPLv3,
> and so on.
> 
> Above would be the minimum information, for anything else, one can look
> into the big report. Maybe the flags like "attribution" and "choice"
> would be interessting, too.
> 
> Now, this can be done with some shell scripting, outside of the build
> process.
> But as there is already the license-report generation, this would be the
> perfect place for it.
> 
> Has anyone already (partly) implemented this ?
> Would be happy to pick up the pieces and also mainline it. Just asking,
> before I start hacking things twice...

I don't think there are any implementations like this.

If we do something like this, then I'd like to take this a step further. I
think there are more use-cases for data like this. And there a probably
conflicting requirements for the different files and output formats.

The main issue with PTXdist is that we cannot just iterate over all
packages is one script. So what I'd like to see is something like the
current report stage to aggregate all possibly interesting data about the
packages and later convert an filter that into the required output.
What would be a good intermediate format for this? I'd like to avoid
escaping problems if possible.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] license-report: extend to generate textual list of package + licenses

[Andreas Pretzsch]

On Di, 2016-10-04 at 15:02 +0200, Michael Olbrich wrote:
> > > [ptxdist make license-report]
> > 
> > First, this one is wonderful, and works like a charm. Thanks a lot.
> > 
> > What I am missing is an overview.
> > Mainly as an external list, but maybe also in the reports.
> > 
> > I'm talking about e.g. a simple CSV, listing all the included packages,
> > like in "pkgname | pkgversion | licenses". So something like
> > [...]
> > busybox|1.24.2|GPL-2.0
> > [...]
> > Sensible escaping would be something to look at, but anyway.
> > [...]
> > 
> > Has anyone already (partly) implemented this ?
> > Would be happy to pick up the pieces and also mainline it. Just asking,
> > before I start hacking things twice...
> 
> I don't think there are any implementations like this.
> 
> If we do something like this, then I'd like to take this a step further. I
> think there are more use-cases for data like this. And there a probably
> conflicting requirements for the different files and output formats.

For sure. Both.
Just, I'd like to start with a simple one, to have something. Or to be
more precise, I have to, due to customer requirement. So in any case,
I'll hack something the next few days. In case I'll hook it into the
ptxdist scripts, I'll CC the list, as a RFC.
Of course, I'm happy about any pointers where to put my fingers on in
the scripts ;-)

Out of my experience with these topics (license compliance, due
diligence by laywer), the todays report is wonderful !
And sufficient here. Maybe it could be condensed in some points (like
referencing fixed texts as the GPL instead of copying each time), but
even this is not necessary. And/or converted to some other format (text,
html), to be included in a product delivery itself (menu item, web
interface, whatever).

But to ease everyones task, and esp. have both developers and lawyers
easily check what changed, we need a simple list. In whatever format,
and included in the report or not.


> The main issue with PTXdist is that we cannot just iterate over all
> packages is one script. So what I'd like to see is something like the
> current report stage to aggregate all possibly interesting data about the
> packages and later convert an filter that into the required output.
> What would be a good intermediate format for this?

Sounds like the best way to do it. About an intermediate format, hmmm,
good question. Not sure if there is one per se. Or from an pragmatic
point of view, we already have one, the information in the report
generation scripts.


>  I'd like to avoid escaping problems if possible.

Me too :-)
CSV is always fun...

Looking at the minimal information (package name, version, license tags,
maybe flags), I think we will not run into encoding and escaping issues
here. As long as we leave out the URL. And stick with western package
names.
Just take everything as a string, and use | as separator. Should be good
for now. At least, this would be my first try.
And even if the format changes some day, this could be converted
externally with minimal effort.


-- 

carpe noctem engineering
Ingenieurbuero fuer Hard- & Software-Entwicklung Andreas Pretzsch
Dipl.-Ing. (FH) Andreas Pretzsch        Tel. +49-(0)7307-936088-1
Lange Strasse 28a                       Fax: +49-(0)7307-936088-9
89250 Senden, Germany                   email: apr@cn-eng.de


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] license-report: extend to generate textual list of package + licenses

[Michael Olbrich]

On Tue, Oct 04, 2016 at 07:15:07PM +0200, Andreas Pretzsch wrote:
> On Di, 2016-10-04 at 15:02 +0200, Michael Olbrich wrote:
> > > > [ptxdist make license-report]
> > > 
> > > First, this one is wonderful, and works like a charm. Thanks a lot.
> > > 
> > > What I am missing is an overview.
> > > Mainly as an external list, but maybe also in the reports.
> > > 
> > > I'm talking about e.g. a simple CSV, listing all the included packages,
> > > like in "pkgname | pkgversion | licenses". So something like
> > > [...]
> > > busybox|1.24.2|GPL-2.0
> > > [...]
> > > Sensible escaping would be something to look at, but anyway.
> > > [...]
> > > 
> > > Has anyone already (partly) implemented this ?
> > > Would be happy to pick up the pieces and also mainline it. Just asking,
> > > before I start hacking things twice...
> > 
> > I don't think there are any implementations like this.
> > 
> > If we do something like this, then I'd like to take this a step further. I
> > think there are more use-cases for data like this. And there a probably
> > conflicting requirements for the different files and output formats.
> 
> For sure. Both.
> Just, I'd like to start with a simple one, to have something. Or to be
> more precise, I have to, due to customer requirement. So in any case,
> I'll hack something the next few days. In case I'll hook it into the
> ptxdist scripts, I'll CC the list, as a RFC.
> Of course, I'm happy about any pointers where to put my fingers on in
> the scripts ;-)

You will probably need to overwrite ptxd_make_world_license() and expand it
to generate the needed per package data.
Then you can use ptxd_make_license_report() as a template to generate the
final file.

> Out of my experience with these topics (license compliance, due
> diligence by laywer), the todays report is wonderful !
> And sufficient here. Maybe it could be condensed in some points (like
> referencing fixed texts as the GPL instead of copying each time), but
> even this is not necessary. And/or converted to some other format (text,
> html), to be included in a product delivery itself (menu item, web
> interface, whatever).
> 
> But to ease everyones task, and esp. have both developers and lawyers
> easily check what changed, we need a simple list. In whatever format,
> and included in the report or not.

The current latex/pdf result was always just a first step for me.

> > The main issue with PTXdist is that we cannot just iterate over all
> > packages is one script. So what I'd like to see is something like the
> > current report stage to aggregate all possibly interesting data about the
> > packages and later convert an filter that into the required output.
> > What would be a good intermediate format for this?
> 
> Sounds like the best way to do it. About an intermediate format, hmmm,
> good question. Not sure if there is one per se. Or from an pragmatic
> point of view, we already have one, the information in the report
> generation scripts.

I meant a file format that's easy to parse later.

One of the problems here is, that the data structures get rather complex
and are not something that can really be handled in shell scripts.

> >  I'd like to avoid escaping problems if possible.
> 
> Me too :-)
> CSV is always fun...
> 
> Looking at the minimal information (package name, version, license tags,
> maybe flags), I think we will not run into encoding and escaping issues
> here. As long as we leave out the URL. And stick with western package
> names.
> Just take everything as a string, and use | as separator. Should be good
> for now. At least, this would be my first try.
> And even if the format changes some day, this could be converted
> externally with minimal effort.

Two points here:
1. just CSV or similar makes it hard to add or remove fields.
2. Depending on what you use to parse the files, 0x1F (ascii unit
separator) or something like that, might be a good alternative. We use this
in the permission files to avoid characters that might appear in file
names.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de