Re: [ptxdist] [PATCH v3 1/3] config/setup: make reproducible builds configurable

[Michael Olbrich <m.olbrich@pengutronix.de>]

On Fri, Dec 21, 2018 at 10:06:59AM +0000, Baeuerle, Florian wrote:
> Some of ptxdist's packages use SOURCE_DATE_EPOCH to make the build
> results predictable. Make this behaviour more configurable via a newly
> introduced ptxdist setup options.
> 
> By default, this will set SOURCE_DATE_EPOCH to the year and month of the
> used OSELAS Toolchain version. If the used toolchain is not an
> OSELAS-Toolchain, the PTXdist version is used as a fallback.
> 
> Signed-off-by: Florian Bäuerle <florian.baeuerle@allegion.com>

Sorry for the late reply. This got lost on my side.

> ---
>  config/setup/Kconfig                 | 47 ++++++++++++++++++++++++++
>  config/setup/ptxdistrc.default       |  5 +++
>  scripts/lib/ptxd_lib_reproducible.sh | 49 +++++++++++++++++++++++++++-
>  3 files changed, 100 insertions(+), 1 deletion(-)
> 
> diff --git a/config/setup/Kconfig b/config/setup/Kconfig
> index 990da03a1..10adb970d 100644
> --- a/config/setup/Kconfig
> +++ b/config/setup/Kconfig
> @@ -264,6 +264,53 @@ config SETUP_DISABLE_LOCAL_CHECK
>  	  may fail under certain circumstances.
>  	  Disable this check at your own risk.
>  
> +config SETUP_DISABLE_REPRODUCIBLE
> +	bool
> +	prompt "disable reproducible builds"
> +	help
> +	  By default ptxdist will build some packages in a
> +	  reproducible way by injecting fake timestamps, user and
> +	  host name into the build.
> +

The switch to disable this should be in setup ...

> +if !SETUP_DISABLE_REPRODUCIBLE
> +
> +choice
> +	prompt "Fake timestamps source"
> +	default SETUP_REPRODUCIBLE_TIMESTAMP_TOOLCHAIN
> +
> +	config SETUP_REPRODUCIBLE_TIMESTAMP_TOOLCHAIN
> +		bool
> +		prompt "toolchain version"
> +
> +	config SETUP_REPRODUCIBLE_TIMESTAMP_PTXDIST
> +		bool
> +		prompt "ptxdist version"
> +
> +	config SETUP_REPRODUCIBLE_TIMESTAMP_CUSTOM
> +		bool
> +		prompt "custom timestamp"
> +
> +endchoice

... but the choice should be part of the pxconfig. I would accept a option
to disable this here too, but I don't think that's necessary.
In my opinion disabling reproducible builds is strictly a development
feature.

> +
> +config SETUP_REPRODUCIBLE_TIMESTAMP
> +	string
> +	default "toolchain" if SETUP_REPRODUCIBLE_TIMESTAMP_TOOLCHAIN
> +	default "ptxdist" if SETUP_REPRODUCIBLE_TIMESTAMP_PTXDIST
> +	default "custom" if SETUP_REPRODUCIBLE_TIMESTAMP_CUSTOM
> +
> +if SETUP_REPRODUCIBLE_TIMESTAMP_CUSTOM
> +
> +config SETUP_REPRODUCIBLE_TIMESTAMP_STRING
> +	string
> +	prompt "Fake timestamp"
> +	default "2018-11-01 UTC"

I don't think we should provide a default. This should be an explicit
choice. But maybe put this as an example in the help text.

> +	help
> +	  Supply a custom fake timestamp to be injected to the build.
> +	  The timestamp is passed to 'date --date'.
> +
> +endif
> +endif
> +
>  config SETUP_ENV_WHITELIST
>  	string "environment variable whitelist (space separated)"
>  	help
> diff --git a/config/setup/ptxdistrc.default b/config/setup/ptxdistrc.default
> index 397b78eb7..8aa7dcc9a 100644
> --- a/config/setup/ptxdistrc.default
> +++ b/config/setup/ptxdistrc.default
> @@ -61,6 +61,11 @@ PTXCONF_SETUP_JAVA_SDK="/usr/lib/jvm/default-java"
>  # Developer Options 
>  #
>  # PTXCONF_SETUP_DISABLE_LOCAL_CHECK is not set
> +# PTXCONF_SETUP_DISABLE_REPRODUCIBLE is not set
> +PTXCONF_SETUP_REPRODUCIBLE_TIMESTAMP_TOOLCHAIN=y
> +# PTXCONF_SETUP_REPRODUCIBLE_TIMESTAMP_PTXDIST is not set
> +# PTXCONF_SETUP_REPRODUCIBLE_TIMESTAMP_CUSTOM is not set
> +PTXCONF_SETUP_REPRODUCIBLE_TIMESTAMP="toolchain"
>  PTXCONF_SETUP_ENV_WHITELIST=""
>  # PTXCONF_SETUP_COMMON_CACHE is not set
>  # PTXCONF_SETUP_GEN_DEP_TREE is not set
> diff --git a/scripts/lib/ptxd_lib_reproducible.sh b/scripts/lib/ptxd_lib_reproducible.sh
> index e2e664ba8..98c528ac2 100644
> --- a/scripts/lib/ptxd_lib_reproducible.sh
> +++ b/scripts/lib/ptxd_lib_reproducible.sh
> @@ -8,8 +8,55 @@
>  # see the README file.
>  #
>  
> +ptxd_timestamp_ptxdist() {
> +    ptxd_reply="${PTXDIST_VERSION_YEAR}-${PTXDIST_VERSION_MONTH}-01 UTC"
> +}
> +
> +ptxd_timestamp_toolchain() {
> +    local oselas_ptxconfig="$(readlink -f "${PTXDIST_TOOLCHAIN}/ptxconfig")"
> +
> +    if [ -e "${oselas_ptxconfig}" ]; then
> +        local oselas_version="$(source "${oselas_ptxconfig}" && echo ${PTXCONF_CONFIGFILE_VERSION})"

           local oselas_version="$(ptxd_get_kconfig "${oselas_ptxconfig}" PTXCONF_CONFIGFILE_VERSION)"

> +        local orig_IFS="${IFS}"
> +        local IFS="."
> +        set -- ${oselas_version}
> +        IFS="${orig_IFS}"
> +        ptxd_reply="${1}-${2}-01 UTC"
> +    else
> +        echo "${PTXDIST_LOG_PROMPT}warning: cannot deduce timestamp from toolchain, falling back to PTXdist version for reproducible timestamp"
> +        ptxd_timestamp_ptxdist

No. Just fail here. With the options in the ptxconfig that should be ok.

> +    fi
> +}
> +
> +ptxd_timestamp_custom() {
> +    local ts="${PTXCONF_SETUP_REPRODUCIBLE_TIMESTAMP_STRING}"

With the change I requested above, you'll need to use ptxd_get_ptxconf
here (and for the other PTXCONF_* variables from ptxconfig.

> +
> +    if ! date --date "${ts}" > /dev/null 2>&1; then
> +        echo "${PTXDIST_LOG_PROMPT}warning: '${ts}' is not a valid timestamp, falling back to toolchain for reproducible timestamp"
> +        ptxd_timestamp_toolchain
> +    else
> +        ptxd_reply="${ts}"
> +    fi
> +}
> +
>  ptxd_lib_reproducible() {
> -    SOURCE_DATE_EPOCH="$(echo $(date --date="${PTXDIST_VERSION_YEAR}-${PTXDIST_VERSION_MONTH}-01 UTC" "+%s"))"
> +    if [ "${PTXCONF_SETUP_DISABLE_REPRODUCIBLE}" = "y" ]; then
> +        ptxd_timestamp_ptxdist

I don't think that this is correct. Maybe:

	ptxd_reply="$(date "+@%s")"

> +    else
           ptxd_timestamp_$(ptxd_get_ptxconf PTXCONF_SETUP_REPRODUCIBLE_TIMESTAMP)

should work as expected.

Michael

> +        case "${PTXCONF_SETUP_REPRODUCIBLE_TIMESTAMP}" in
> +            "custom")
> +                ptxd_timestamp_custom
> +                ;;
> +            "ptxdist")
> +                ptxd_timestamp_ptxdist
> +                ;;
> +            *)
> +                ptxd_timestamp_toolchain
> +                ;;
> +        esac
> +    fi
> +
> +    SOURCE_DATE_EPOCH="$(echo $(date --date="${ptxd_reply}" "+%s"))"
>      export SOURCE_DATE_EPOCH
>  
>      PTXDIST_BUILD_TIMESTAMP="$(echo $(date --utc --date @${SOURCE_DATE_EPOCH} +%Y-%m-%dT%H:%M+0000))"
> -- 
> 2.19.2
> 
> _______________________________________________
> ptxdist mailing list
> ptxdist@pengutronix.de

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de