[ptxdist] git ptxdist best practices

[ptxdist] git ptxdist best practices

[Jon Ringle]

[-- Attachment #1.1: Type: text/plain, Size: 1087 bytes --]

I'm looking for some best practices ideas as it relates to using ptxdist to
build a complete system that pulls from various git repos.

I have a clone Linux git repo that has local commits specific to our
embedded hardware.
I build an initramfs filesystem with ptxdist that gets embedded within the
Linux kernel image.
I build a rootfs with ptxdist for the real rootfs
I also have a couple of libs and apps that have their own git repos as well
that I'm getting ptxdist to build as well.

I'm now at the point where I now need to make this available to other
developers and also get setup for a build bot to build (bamboo).

My initial thought is to create a git repo that holds the ptxconfig files
and local rules/, and then configure the rules to somehow get the git repos
by tag for building (as it seems that there is some sort of support for
this in scripts/lib/ptxd_make_get.sh)... but I don't quite understand how
to integrate this in my rules/*.make file.

Perhaps there is a better way of doing this. I'd be interested to hear what
others are doing in this regard.

Thanks,
Jon

[-- Attachment #1.2: Type: text/html, Size: 1247 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Alexander Dahl]

[-- Attachment #1.1: Type: text/plain, Size: 1250 bytes --]

Hei hei,

On 08.01.2014 03:32, Jon Ringle wrote:
> My initial thought is to create a git repo that holds the ptxconfig files
> and local rules/, and then configure the rules to somehow get the git repos
> by tag for building (as it seems that there is some sort of support for
> this in scripts/lib/ptxd_make_get.sh)... but I don't quite understand how
> to integrate this in my rules/*.make file.

This is exactly what we do, alas we do not use any ptxdist magic but two
self written shell scripts residing in another git repository "scripts"
included as submodule. Our rules/foo.make use one script for the get and
one for the extract stage. Those scripts make are called with two extra
variables from rules/foo.in as parameter. For each of our own packages
those describe clone path and which git entity (branch, tag, changeset)
to checkout after cloning. All these packages are cloned to
local_src/foo and build with cmake in local_src/foo-git. I assume all
this specific to our needs, but that's the way we do it. If you're
interested in the scripts, tell me.

However I'm also curious about other ways. Is there a nice ptxdist way
to handle packages without tarball coming directly from some git repo?

Greets
Alex



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Jon Ringle]

[-- Attachment #1.1: Type: text/plain, Size: 1506 bytes --]

On Wed, Jan 8, 2014 at 12:56 AM, Alexander Dahl <post@lespocky.de> wrote:

> Hei hei,
>
> On 08.01.2014 03:32, Jon Ringle wrote:
> > My initial thought is to create a git repo that holds the ptxconfig files
> > and local rules/, and then configure the rules to somehow get the git
> repos
> > by tag for building (as it seems that there is some sort of support for
> > this in scripts/lib/ptxd_make_get.sh)... but I don't quite understand how
> > to integrate this in my rules/*.make file.
>
> This is exactly what we do, alas we do not use any ptxdist magic but two
> self written shell scripts residing in another git repository "scripts"
> included as submodule. Our rules/foo.make use one script for the get and
> one for the extract stage. Those scripts make are called with two extra
> variables from rules/foo.in as parameter. For each of our own packages
> those describe clone path and which git entity (branch, tag, changeset)
> to checkout after cloning. All these packages are cloned to
> local_src/foo and build with cmake in local_src/foo-git. I assume all
> this specific to our needs, but that's the way we do it. If you're
> interested in the scripts, tell me.
>
>
Yes, I would be interested to see your scripts and how you use them in your
*.make and *.in :)


> However I'm also curious about other ways. Is there a nice ptxdist way
> to handle packages without tarball coming directly from some git repo?
>
> Greets
> Alex
>
>
>
> --
> ptxdist mailing list
> ptxdist@pengutronix.de
>
>

[-- Attachment #1.2: Type: text/html, Size: 2180 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Jürgen Beisert]

Hi Jon,

On Wednesday 08 January 2014 03:32:41 Jon Ringle wrote:
> I'm looking for some best practices ideas as it relates to using ptxdist to
> build a complete system that pulls from various git repos.
>
> I have a clone Linux git repo that has local commits specific to our
> embedded hardware.
> I build an initramfs filesystem with ptxdist that gets embedded within the
> Linux kernel image.
> I build a rootfs with ptxdist for the real rootfs
> I also have a couple of libs and apps that have their own git repos as well
> that I'm getting ptxdist to build as well.
>
> I'm now at the point where I now need to make this available to other
> developers and also get setup for a build bot to build (bamboo).
>
> My initial thought is to create a git repo that holds the ptxconfig files
> and local rules/, and then configure the rules to somehow get the git repos
> by tag for building (as it seems that there is some sort of support for
> this in scripts/lib/ptxd_make_get.sh)... but I don't quite understand how
> to integrate this in my rules/*.make file.
>
> Perhaps there is a better way of doing this. I'd be interested to hear what
> others are doing in this regard.

Do you want to use this BSP only by your own, or is it intended to provide it 
to other developers who use it as a base for their own work?

In the latter case I would "release" the various components and create archives 
of it. And these archives can simply be part of the (finally) released BSP you 
provide to other developers (in contrast to the downloadable archives PTXdist 
uses for the remaining packages like Busybox, Qt and so on).

If you still want to use git repositories for the external components we should 
wait for Michael, because he can explain how to use the PTXdist feature which 
deals with such repositories.

Regards,
jbe

-- 
Pengutronix e.K.                              | Juergen Beisert             |
Linux Solutions for Science and Industry      | http://www.pengutronix.de/  |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Alexander Dahl]

Hei hei, 

Am 2014-01-08 07:12, schrieb Jon Ringle:
> On Wed, Jan 8, 2014 at 12:56 AM, Alexander Dahl <post@lespocky.de> wrote:
>> This is exactly what we do, alas we do not use any ptxdist magic but two
>> self written shell scripts residing in another git repository "scripts"
>> included as submodule. Our rules/foo.make use one script for the get and
>> one for the extract stage. Those scripts make are called with two extra
>> variables from rules/foo.in as parameter. For each of our own packages
>> those describe clone path and which git entity (branch, tag, changeset)
>> to checkout after cloning. All these packages are cloned to
>> local_src/foo and build with cmake in local_src/foo-git. I assume all
>> this specific to our needs, but that's the way we do it. If you're
>> interested in the scripts, tell me.
>>
>>
> Yes, I would be interested to see your scripts and how you use them in your
> *.make and *.in :)

Okay I edited them a little and attached it. Note: the scripts itself
are put in a subfolder of the BSP here named "scripts".

I decided to use the get stage for cloning a git repo or checking if
it's correctly cloned and if fetching all branches and changesets
without updating the working copy at this stage. The other script for
the extract stage does the actual checkout then. It works. ;-)

Greets
Alex

-- 
»With the first link, the chain is forged. The first speech censured,
the first thought forbidden, the first freedom denied, chains us all
irrevocably.« (Jean-Luc Picard, quoting Judge Aaron Satie)
*** GnuPG-FP: 02C8 A590 7FE5 CA5F 3601  D1D5 8FBA 7744 CC87 10D0 ***

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Alexander Dahl]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

Am 2014-01-08 10:21, schrieb Alexander Dahl:
> Okay I edited them a little and attached it. Note: the scripts itself
> are put in a subfolder of the BSP here named "scripts".

Guess what I forgot to attach?!

Alex

-- 
»With the first link, the chain is forged. The first speech censured,
the first thought forbidden, the first freedom denied, chains us all
irrevocably.« (Jean-Luc Picard, quoting Judge Aaron Satie)
*** GnuPG-FP: 02C8 A590 7FE5 CA5F 3601  D1D5 8FBA 7744 CC87 10D0 ***

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: get-git.sh --]
[-- Type: text/x-shellscript; name=get-git.sh, Size: 1747 bytes --]

#!/bin/sh
# ----------------------------------------------------------------------
# Clone a git repository in get stage or check if it's the right one
# in case already cloned.
#
# Authors:  Alexander Dahl <post@lespocky.de>
#
# Copyright 2011,2012,2013 Alexander Dahl
# ----------------------------------------------------------------------
#set -x
#set -e

_RED="\033[31m"
_YELLOW="\033[33m\033[40m\033[1m"
_NOCOLOR="\033[0m\033[49m"

GITURL="$1"
DESTDIR="$2"

print_usage() {
    echo "Usage: $0 giturl destdir"
}

if [ -z "$1" -o -z "$2" ]
then
    print_usage
    exit 1
fi

CHECKOUTDIR="$(basename ${DESTDIR})"
SRCDIR="$(dirname ${DESTDIR})"

if [ ! -d "${SRCDIR}" ]
then
    echo "folder ${_YELLOW}local_src${_NOCOLOR} does not exist, please create it!"
    exit 4
fi

cd "${SRCDIR}"
if [ ! -d "${CHECKOUTDIR}" ]
then
    if [ -e "${CHECKOUTDIR}" ]
    then
        echo "${CHECKOUTDIR} exists but is no directory!"
        exit 2
    else
        echo "${CHECKOUTDIR} does not exist."
    fi

    if git clone "${GITURL}" "${CHECKOUTDIR}"
    then
        echo 'git clone successful ...'
    else
        exit $?
    fi

    cd "${CHECKOUTDIR}"
    if git fetch --all
    then
        echo 'git fetch successful ...'
    else
        exit $?
    fi
else
    if [ -L "${CHECKOUTDIR}" ]
    then
        echo "${_RED}WARNING: ${_YELLOW}${CHECKOUTDIR} is a symbolic link!${_NOCOLOR}"
        exit 0
    fi

    cd "${CHECKOUTDIR}"
    REMOTEURL="$(git remote -v | awk '/^origin.+(fetch)/ {print $2;}')"
    if [ "${REMOTEURL}" = "${GITURL}" ]
    then
        echo "This already cloned repo is the right one."
        exit 0
    else
        echo "Cloned repo remote is '${REMOTEURL}' but '${GITURL}' was expected!"
        exit 3
    fi
fi

[-- Attachment #3: foo.make --]
[-- Type: text/plain, Size: 2164 bytes --]

# -*-makefile-*-
#
# Copyright (C) 2012 by Alexander Dahl <post@lespocky.de>
#
# See CREDITS for details about who has contributed to this project.
#
# For further information about the PTXdist project and license conditions
# see the README file.
#

#
# We provide this package
#
PACKAGES-$(PTXCONF_FOO) += foo

#
# Paths and names
#
FOO_VERSION	:= git
#FOO_MD5		:= 
FOO			:= foo-$(FOO_VERSION)
#FOO_SUFFIX	:= 
#FOO_URL		:= /$(FOO).$(FOO_SUFFIX)
#FOO_SOURCE	:= $(SRCDIR)/$(FOO).$(FOO_SUFFIX)
#FOO_DIR		:= $(BUILDDIR)/$(FOO)
FOO_LICENSE	:= proprietary

FOO_SOURCE_LOCAL		:= $(PTXDIST_WORKSPACE)/local_src/$(FOO)
FOO_GIT_URL			:= $(PTXCONF_FOO_CLONE_URL)
FOO_CHECKOUT_ENTITY	:= $(PTXCONF_FOO_CHECKOUT_ENTITY)
FOO_DIR				:= $(FOO_SOURCE_LOCAL)

# ----------------------------------------------------------------------------
# Get
# ----------------------------------------------------------------------------
$(STATEDIR)/foo.get:
	@$(call targetinfo)
	@$(PTXDIST_WORKSPACE)/scripts/get-git.sh "$(FOO_GIT_URL)" "$(FOO_SOURCE_LOCAL)"
	@$(call touch)

# ----------------------------------------------------------------------------
# Extract
# ----------------------------------------------------------------------------
$(STATEDIR)/foo.extract:
	@$(call targetinfo)
	@cd $(FOO_SOURCE_LOCAL) && \
		$(PTXDIST_WORKSPACE)/scripts/extract-git.sh "$(FOO_CHECKOUT_ENTITY)"
	@$(call touch)

# ----------------------------------------------------------------------------
# Prepare
# ----------------------------------------------------------------------------
FOO_CONF_TOOL	:= cmake

# ----------------------------------------------------------------------------
# Target-Install
# ----------------------------------------------------------------------------
$(STATEDIR)/foo.targetinstall:
	@$(call targetinfo)

	@$(call install_init, foo)
	@$(call install_fixup, foo,PRIORITY,optional)
	@$(call install_fixup, foo,SECTION,base)
	@$(call install_fixup, foo,AUTHOR,"Alexander Dahl <post@lespocky.de>")
	@$(call install_fixup, foo,DESCRIPTION,missing)

	@$(call install_copy, foo, 0, 0, 0755, -, /usr/bin/foo)

	@$(call install_finish, foo)

	@$(call touch)

# vim: ft=make noet

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: extract-git.sh --]
[-- Type: text/x-shellscript; name=extract-git.sh, Size: 1104 bytes --]

#!/bin/sh
# ----------------------------------------------------------------------
# In a git "working copy" checkout the given entity (branch, tag,
# changeset). In case of a branch pull the latest changes, in every
# other case leave it in detached HEAD state. This script performs no
# directory changes, so start it in the target directory!
#
# Authors:  Alexander Dahl <post@lespocky.de>
#
# Copyright 2011,2013 Alexander Dahl
# ----------------------------------------------------------------------

ENTITY="$1"
BRANCH_EXISTS="no"

print_usage() {
    echo "Usage: $0 checkoutentity"
}

if [ -z "$1" ]
then
    print_usage
    exit 1
fi

if git fetch
then
    echo 'Git fetch successful!'
else
    exit $?
fi

if git checkout "$1"
then
    echo 'Git checkout successful!'
else
    exit $?
fi

for branch in $(git branch | cut -c3-)
do
    if [ "${ENTITY}" = "${branch}" ]
    then
        BRANCH_EXISTS="yes"
    fi
done

if [ "${BRANCH_EXISTS}" = "yes" ]
then
    git pull
fi

if git submodule update --init --recursive
then
    echo 'Git submodule update successful!'
else
    exit $?
fi

exit 0

[-- Attachment #5: foo.in --]
[-- Type: text/plain, Size: 578 bytes --]

## SECTION=project_specific

menuconfig FOO
	tristate
	prompt "foo                           "
	select HOST_CMAKE
	default y
	help
	  Foo like in bar.

config FOO_CLONE_URL
	string
	depends on FOO
	prompt "Git url to clone from"
	default "git://git.example.com/git/foo.git"
	help
	  Git URL to clone from. Defaults to local server, but can be
	  overwritten with any local or remote URL.

config FOO_CHECKOUT_ENTITY
	string
	depends on FOO
	prompt "Git entity to checkout."
	default "stable"
	help
	  Check out a specific branch, tag or changeset.

# vim: ft=kconfig noet tw=72

[-- Attachment #6: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Jürgen Beisert]

On Wednesday 08 January 2014 10:21:53 Alexander Dahl wrote:
> Am 2014-01-08 07:12, schrieb Jon Ringle:
> > On Wed, Jan 8, 2014 at 12:56 AM, Alexander Dahl <post@lespocky.de> wrote:
> >> This is exactly what we do, alas we do not use any ptxdist magic but two
> >> self written shell scripts residing in another git repository "scripts"
> >> included as submodule. Our rules/foo.make use one script for the get and
> >> one for the extract stage. Those scripts make are called with two extra
> >> variables from rules/foo.in as parameter. For each of our own packages
> >> those describe clone path and which git entity (branch, tag, changeset)
> >> to checkout after cloning. All these packages are cloned to
> >> local_src/foo and build with cmake in local_src/foo-git. I assume all
> >> this specific to our needs, but that's the way we do it. If you're
> >> interested in the scripts, tell me.
> >
> > Yes, I would be interested to see your scripts and how you use them in
> > your *.make and *.in :)
>
> Okay I edited them a little and attached it.

<nothing>

More coffee? :))

jbe

-- 
Pengutronix e.K.                              | Juergen Beisert             |
Linux Solutions for Science and Industry      | http://www.pengutronix.de/  |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Jon Ringle]

[-- Attachment #1.1: Type: text/plain, Size: 1065 bytes --]

On Wed, Jan 8, 2014 at 2:59 AM, Jürgen Beisert <jbe@pengutronix.de> wrote:

> Hi Jon,
> Do you want to use this BSP only by your own, or is it intended to provide
> it
> to other developers who use it as a base for their own work?
>
> In the latter case I would "release" the various components and create
> archives
> of it. And these archives can simply be part of the (finally) released BSP
> you
> provide to other developers (in contrast to the downloadable archives
> PTXdist
> uses for the remaining packages like Busybox, Qt and so on).
>
> Yes, this will be for other developers to base their work off of.
So, what exactly would be included in the "release" BSP?
platform/sysroot-*/ and platform/root*/ ??


> If you still want to use git repositories for the external components we
> should
> wait for Michael, because he can explain how to use the PTXdist feature
> which
> deals with such repositories.
>
> Yes, I'd like to understand better ptxdist features to support externat
components maintained by our own git repos :)

[-- Attachment #1.2: Type: text/html, Size: 1573 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Jon Ringle]

[-- Attachment #1.1: Type: text/plain, Size: 1625 bytes --]

On Wed, Jan 8, 2014 at 4:21 AM, Alexander Dahl <post@lespocky.de> wrote:

> Hei hei,
>
> Am 2014-01-08 07:12, schrieb Jon Ringle:
> > On Wed, Jan 8, 2014 at 12:56 AM, Alexander Dahl <post@lespocky.de>
> wrote:
> >> This is exactly what we do, alas we do not use any ptxdist magic but two
> >> self written shell scripts residing in another git repository "scripts"
> >> included as submodule. Our rules/foo.make use one script for the get and
> >> one for the extract stage. Those scripts make are called with two extra
> >> variables from rules/foo.in as parameter. For each of our own packages
> >> those describe clone path and which git entity (branch, tag, changeset)
> >> to checkout after cloning. All these packages are cloned to
> >> local_src/foo and build with cmake in local_src/foo-git. I assume all
> >> this specific to our needs, but that's the way we do it. If you're
> >> interested in the scripts, tell me.
> >>
> >>
> > Yes, I would be interested to see your scripts and how you use them in
> your
> > *.make and *.in :)
>
> Okay I edited them a little and attached it. Note: the scripts itself
> are put in a subfolder of the BSP here named "scripts".
>
> I decided to use the get stage for cloning a git repo or checking if
> it's correctly cloned and if fetching all branches and changesets
> without updating the working copy at this stage. The other script for
> the extract stage does the actual checkout then. It works. ;-)
>
> Thanks!

How do you deal with creating a rule/kernel.make file that does this if I
build Linux using my own tag? It seems that kernel may need some special
handling

Jon

[-- Attachment #1.2: Type: text/html, Size: 2286 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Alexander Dahl]

[-- Attachment #1.1: Type: text/plain, Size: 727 bytes --]

Hei Jon,

On Wed, Jan 08, 2014 at 12:56:37PM -0500, Jon Ringle wrote:
> How do you deal with creating a rule/kernel.make file that does this if I
> build Linux using my own tag? It seems that kernel may need some special
> handling

We use a vanilla kernel and just compile some additional modules for
our own hardware. I can check tomorrow, how this is done, but as far
as I remember just a simple additional make rule.

Greets
Alex

-- 
»With the first link, the chain is forged. The first speech censured, 
the first thought forbidden, the first freedom denied, chains us all 
irrevocably.« (Jean-Luc Picard, quoting Judge Aaron Satie)
*** GnuPG-FP: 02C8 A590 7FE5 CA5F 3601  D1D5 8FBA 7744 CC87 10D0 ***

[-- Attachment #1.2: Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Jürgen Beisert]

Hi Jon,

On Wednesday 08 January 2014 15:14:26 Jon Ringle wrote:
> On Wed, Jan 8, 2014 at 2:59 AM, Jürgen Beisert <jbe@pengutronix.de> wrote:
> > Do you want to use this BSP only by your own, or is it intended to
> > provide it to other developers who use it as a base for their own work?
> > In the latter case I would "release" the various components and create
> > archives of it. And these archives can simply be part of the (finally)
> > released BSP you provide to other developers (in contrast to the
> > downloadable archives PTXdist uses for the remaining packages like Busybox,
> > Qt and so on). 
> >
> Yes, this will be for other developers to base their work off of.
>
> So, what exactly would be included in the "release" BSP?
> platform/sysroot-*/ and platform/root*/ ??

No. These directories exist at buildtime only and will be recreated and filled 
up at each PTXdist run. Their content depends on the selected toolchain and 
package selection. You should keep this behaviour for the user of the BSP.

Maybe I read your first question wrong: You have some development made and the 
results are part of a few external git repositories, haven't you?
Why not tar them (not the whole repository, just a released variant: git 
archive --format=tar ...) and include the archive in the BSP? The 
corresponding rule file uses a local URL pointing to this archive and you are 
done.

> > If you still want to use git repositories for the external components we
> > should wait for Michael, because he can explain how to use the PTXdist
> > feature which deals with such repositories.
> >
> Yes, I'd like to understand better ptxdist features to support externat
> components maintained by our own git repos :)

But this can work only, if everyone using your BSP has access to these 
repositories.

Regards,
Juergen

-- 
Pengutronix e.K.                              | Juergen Beisert             |
Linux Solutions for Science and Industry      | http://www.pengutronix.de/  |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Olbrich, Michael]

On Wed, Jan 08, 2014 at 08:59:43AM +0100, Jürgen Beisert wrote:
> On Wednesday 08 January 2014 03:32:41 Jon Ringle wrote:
> > I'm looking for some best practices ideas as it relates to using ptxdist to
> > build a complete system that pulls from various git repos.
> >
> > I have a clone Linux git repo that has local commits specific to our
> > embedded hardware.
> > I build an initramfs filesystem with ptxdist that gets embedded within the
> > Linux kernel image.
> > I build a rootfs with ptxdist for the real rootfs
> > I also have a couple of libs and apps that have their own git repos as well
> > that I'm getting ptxdist to build as well.
> >
> > I'm now at the point where I now need to make this available to other
> > developers and also get setup for a build bot to build (bamboo).
> >
> > My initial thought is to create a git repo that holds the ptxconfig files
> > and local rules/, and then configure the rules to somehow get the git repos
> > by tag for building (as it seems that there is some sort of support for
> > this in scripts/lib/ptxd_make_get.sh)... but I don't quite understand how
> > to integrate this in my rules/*.make file.
> >
> > Perhaps there is a better way of doing this. I'd be interested to hear what
> > others are doing in this regard.
> 
> Do you want to use this BSP only by your own, or is it intended to provide it 
> to other developers who use it as a base for their own work?
> 
> In the latter case I would "release" the various components and create archives 
> of it. And these archives can simply be part of the (finally) released BSP you 
> provide to other developers (in contrast to the downloadable archives PTXdist 
> uses for the remaining packages like Busybox, Qt and so on).
> 
> If you still want to use git repositories for the external components we should 
> wait for Michael, because he can explain how to use the PTXdist feature which 
> deals with such repositories.

It really depends on how you want to work with this. If the external
repositories are integrated often, then add them as git submodules in
local_src and use "file://$(PTXDIST_WORKSPACE)/local_src/foo/" as URL. If
not then create a tag when integrating and use
git://myhost.de/foo.git;tag=mytag1" as URL and foo-mytag1.tar.bz2 as SOURCE
and update the tag as needed.  This will clone the repository and create a
tarball for the tag. Note: you cannot follow a branch here! It will just
use the tarball and ignore any changes to the branch. This is only for
projects that only provide git tags for releases but no tarballs.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Jon Ringle]

[-- Attachment #1.1: Type: text/plain, Size: 1456 bytes --]

On Thu, Jan 9, 2014 at 4:03 AM, Olbrich, Michael
<m.olbrich@pengutronix.de>wrote:

> It really depends on how you want to work with this. If the external
> repositories are integrated often, then add them as git submodules in
> local_src and use "file://$(PTXDIST_WORKSPACE)/local_src/foo/" as URL. If
>

I've not used git submodules before. Can I bind a git submodule to a
specific tag?


> not then create a tag when integrating and use
> git://myhost.de/foo.git;tag=mytag1" as URL and foo-mytag1.tar.bz2 as
> SOURCE
> and update the tag as needed.  This will clone the repository and create a
> tarball for the tag. Note: you cannot follow a branch here! It will just
> use the tarball and ignore any changes to the branch. This is only for
> projects that only provide git tags for releases but no tarballs.
>
>
One issue that I ran across with this is that I have:
PTXCONF_SETUP_PTXMIRROR_ONLY=y
PTXCONF_SETUP_PTXMIRROR="http://opensource/pool/"

Where opensource is an internal server that keeps all tar balls used to
build. I'm paranoid that someday a upstream tar ball won't be available and
would cause a build break if someone tried to build and they didn't have
that tar ball in ${PTXCONF_SETUP_SRCDIR}

But for my purposes, if I wanted to use the git URL in a rule file I'd want
it to use the URL as is. But it rewrites it trying to use
${PTXCONF_SETUP_PTXMIRROR} instead in scripts/lib/ptxd_make_get.sh:295

Is there away to work around this?

[-- Attachment #1.2: Type: text/html, Size: 2301 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Michael Olbrich]

On Thu, Jan 09, 2014 at 10:23:58AM -0500, Jon Ringle wrote:
> On Thu, Jan 9, 2014 at 4:03 AM, Olbrich, Michael
> <m.olbrich@pengutronix.de>wrote:
> 
> > It really depends on how you want to work with this. If the external
> > repositories are integrated often, then add them as git submodules in
> > local_src and use "file://$(PTXDIST_WORKSPACE)/local_src/foo/" as URL. If
> >
> 
> I've not used git submodules before. Can I bind a git submodule to a
> specific tag?

git submodules are always bound to a specific commit.

> > not then create a tag when integrating and use
> > git://myhost.de/foo.git;tag=mytag1" as URL and foo-mytag1.tar.bz2 as
> > SOURCE
> > and update the tag as needed.  This will clone the repository and create a
> > tarball for the tag. Note: you cannot follow a branch here! It will just
> > use the tarball and ignore any changes to the branch. This is only for
> > projects that only provide git tags for releases but no tarballs.
> >
> >
> One issue that I ran across with this is that I have:
> PTXCONF_SETUP_PTXMIRROR_ONLY=y
> PTXCONF_SETUP_PTXMIRROR="http://opensource/pool/"
> 
> Where opensource is an internal server that keeps all tar balls used to
> build. I'm paranoid that someday a upstream tar ball won't be available and
> would cause a build break if someone tried to build and they didn't have
> that tar ball in ${PTXCONF_SETUP_SRCDIR}
> 
> But for my purposes, if I wanted to use the git URL in a rule file I'd want
> it to use the URL as is. But it rewrites it trying to use
> ${PTXCONF_SETUP_PTXMIRROR} instead in scripts/lib/ptxd_make_get.sh:295
> 
> Is there away to work around this?

put your git repositories in the mirror:
FOO_URL := http://opensource/pool/foo.git;tag=bar

The mechanism whitelists any URL starting with ${PTXCONF_SETUP_PTXMIRROR}

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Jon Ringle]

[-- Attachment #1.1: Type: text/plain, Size: 1753 bytes --]

On Thu, Jan 9, 2014 at 11:06 AM, Michael Olbrich
<m.olbrich@pengutronix.de>wrote:

> On Thu, Jan 09, 2014 at 10:23:58AM -0500, Jon Ringle wrote:> One issue
> that I ran across with this is that I have:
> > PTXCONF_SETUP_PTXMIRROR_ONLY=y
> > PTXCONF_SETUP_PTXMIRROR="http://opensource/pool/"
> >
> > Where opensource is an internal server that keeps all tar balls used to
> > build. I'm paranoid that someday a upstream tar ball won't be available
> and
> > would cause a build break if someone tried to build and they didn't have
> > that tar ball in ${PTXCONF_SETUP_SRCDIR}
> >
> > But for my purposes, if I wanted to use the git URL in a rule file I'd
> want
> > it to use the URL as is. But it rewrites it trying to use
> > ${PTXCONF_SETUP_PTXMIRROR} instead in scripts/lib/ptxd_make_get.sh:295
> >
> > Is there away to work around this?
>
> put your git repositories in the mirror:
> FOO_URL := http://opensource/pool/foo.git;tag=bar
>
> Unfortunately, I can't do that. The opensource server is under IT control
and they've locked it down so that I can only write to it using scp or sftp
and have no ssh shell access to it.

The mechanism whitelists any URL starting with ${PTXCONF_SETUP_PTXMIRROR}
>

Would the following approach work:

1. In ptxdist/rules/post/ptxd_make_world_common.make add to world/env/impl:
    pkg_url_safe="$(call ptx/escape,$($(1)_URL_SAFE))"

2. In ptxdist/scripts/lib/ptxd_make_get.sh function ptxd_make_get
change all occurances of:
    if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" ]; then
with something like:
    if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" -o -n "${pkg_url_safe}" ];
then

Then in the rule file you can do:
FOO_URL_SAFE := y

So that you can whitelist a specific URL if PTXCONF_SETUP_PTXMIRROR_ONLY is
set

[-- Attachment #1.2: Type: text/html, Size: 2705 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Michael Olbrich]

On Thu, Jan 09, 2014 at 11:37:15AM -0500, Jon Ringle wrote:
> On Thu, Jan 9, 2014 at 11:06 AM, Michael Olbrich
> <m.olbrich@pengutronix.de>wrote:
> 
> > On Thu, Jan 09, 2014 at 10:23:58AM -0500, Jon Ringle wrote:> One issue
> > that I ran across with this is that I have:
> > > PTXCONF_SETUP_PTXMIRROR_ONLY=y
> > > PTXCONF_SETUP_PTXMIRROR="http://opensource/pool/"
> > >
> > > Where opensource is an internal server that keeps all tar balls used to
> > > build. I'm paranoid that someday a upstream tar ball won't be available
> > and
> > > would cause a build break if someone tried to build and they didn't have
> > > that tar ball in ${PTXCONF_SETUP_SRCDIR}
> > >
> > > But for my purposes, if I wanted to use the git URL in a rule file I'd
> > want
> > > it to use the URL as is. But it rewrites it trying to use
> > > ${PTXCONF_SETUP_PTXMIRROR} instead in scripts/lib/ptxd_make_get.sh:295
> > >
> > > Is there away to work around this?
> >
> > put your git repositories in the mirror:
> > FOO_URL := http://opensource/pool/foo.git;tag=bar
> >
> > Unfortunately, I can't do that. The opensource server is under IT control
> and they've locked it down so that I can only write to it using scp or sftp
> and have no ssh shell access to it.
> 
> The mechanism whitelists any URL starting with ${PTXCONF_SETUP_PTXMIRROR}
> >
> 
> Would the following approach work:
> 
> 1. In ptxdist/rules/post/ptxd_make_world_common.make add to world/env/impl:
>     pkg_url_safe="$(call ptx/escape,$($(1)_URL_SAFE))"
> 
> 2. In ptxdist/scripts/lib/ptxd_make_get.sh function ptxd_make_get
> change all occurances of:
>     if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" ]; then
> with something like:
>     if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" -o -n "${pkg_url_safe}" ];
> then
> 
> Then in the rule file you can do:
> FOO_URL_SAFE := y
> 
> So that you can whitelist a specific URL if PTXCONF_SETUP_PTXMIRROR_ONLY is
> set

Hmmm, this is not so nice. How about making PTXCONF_SETUP_PTXMIRROR a list?
Then you can add http://opensource/pool/ and your git server to the
'whitelist'.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Jon Ringle]

[-- Attachment #1.1: Type: text/plain, Size: 2346 bytes --]

On Fri, Jan 10, 2014 at 10:37 AM, Michael Olbrich
<m.olbrich@pengutronix.de>wrote:

> On Thu, Jan 09, 2014 at 11:37:15AM -0500, Jon Ringle wrote:
> > On Thu, Jan 9, 2014 at 11:06 AM, Michael Olbrich
> > <m.olbrich@pengutronix.de>wrote:
> >
> > > On Thu, Jan 09, 2014 at 10:23:58AM -0500, Jon Ringle wrote:> One issue
> > > that I ran across with this is that I have:
> > > > PTXCONF_SETUP_PTXMIRROR_ONLY=y
> > > > PTXCONF_SETUP_PTXMIRROR="http://opensource/pool/"
> > > >
> > > > Where opensource is an internal server that keeps all tar balls used
> to
> > > > build. I'm paranoid that someday a upstream tar ball won't be
> available
> > > and
> > > > would cause a build break if someone tried to build and they didn't
> have
> > > > that tar ball in ${PTXCONF_SETUP_SRCDIR}
> > > >
> > > > But for my purposes, if I wanted to use the git URL in a rule file
> I'd
> > > want
> > > > it to use the URL as is. But it rewrites it trying to use
> > > > ${PTXCONF_SETUP_PTXMIRROR} instead in
> scripts/lib/ptxd_make_get.sh:295
> > > >
> > > > Is there away to work around this?
> > >
> > > put your git repositories in the mirror:
> > > FOO_URL := http://opensource/pool/foo.git;tag=bar
> > >
> > > Unfortunately, I can't do that. The opensource server is under IT
> control
> > and they've locked it down so that I can only write to it using scp or
> sftp
> > and have no ssh shell access to it.
> >
> > The mechanism whitelists any URL starting with ${PTXCONF_SETUP_PTXMIRROR}
> > >
> >
> > Would the following approach work:
> >
> > 1. In ptxdist/rules/post/ptxd_make_world_common.make add to
> world/env/impl:
> >     pkg_url_safe="$(call ptx/escape,$($(1)_URL_SAFE))"
> >
> > 2. In ptxdist/scripts/lib/ptxd_make_get.sh function ptxd_make_get
> > change all occurances of:
> >     if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" ]; then
> > with something like:
> >     if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" -o -n "${pkg_url_safe}" ];
> > then
> >
> > Then in the rule file you can do:
> > FOO_URL_SAFE := y
> >
> > So that you can whitelist a specific URL if PTXCONF_SETUP_PTXMIRROR_ONLY
> is
> > set
>
> Hmmm, this is not so nice. How about making PTXCONF_SETUP_PTXMIRROR a list?
> Then you can add http://opensource/pool/ and your git server to the
> 'whitelist'.
>
> Does PTXCONF_SETUP_PTXMIRROR currently support supplying a list of URLs?--

[-- Attachment #1.2: Type: text/html, Size: 3475 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Jon Ringle]

[-- Attachment #1.1: Type: text/plain, Size: 2145 bytes --]

On Fri, Jan 10, 2014 at 10:37 AM, Michael Olbrich
<m.olbrich@pengutronix.de>wrote:

> On Thu, Jan 09, 2014 at 11:37:15AM -0500, Jon Ringle wrote:
> > Would the following approach work:
> >
> > 1. In ptxdist/rules/post/ptxd_make_world_common.make add to
> world/env/impl:
> >     pkg_url_safe="$(call ptx/escape,$($(1)_URL_SAFE))"
> >
> > 2. In ptxdist/scripts/lib/ptxd_make_get.sh function ptxd_make_get
> > change all occurances of:
> >     if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" ]; then
> > with something like:
> >     if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" -o -n "${pkg_url_safe}" ];
> > then
> >
> > Then in the rule file you can do:
> > FOO_URL_SAFE := y
> >
> > So that you can whitelist a specific URL if PTXCONF_SETUP_PTXMIRROR_ONLY
> is
> > set
>
> Hmmm, this is not so nice. How about making PTXCONF_SETUP_PTXMIRROR a list?
> Then you can add http://opensource/pool/ and your git server to the
> 'whitelist'.
>
> The approach you are proposing has several issue for which I don't have
the necessary bash chops to figure out how to tackle:

1. In the 1st while loop in ptxd_make_get(), I don't know how to get the
case to accept multiple array values for ${PTXCONF_SETUP_PTXMIRROR}:
        case "${url}" in
        ${PTXCONF_SETUP_PTXMIRROR}/*/*)
            # keep original URL, for stuff like glibc
            argv[${#argv[@]}]="${url}"
            mrd=true
            ;;
        ${PTXCONF_SETUP_PTXMIRROR}/*)
            # if mirror is given us to download, add it, but only once
            if ! ${mrd}; then
                argv[${#argv[@]}]="${url}"
                mrd=true
            fi
            ;;

Replacing with ${PTXCONF_SETUP_PTXMIRROR[@]} clear won't help as the
various options aren't separate by PIPE character |.

2. Where ptxmirror_url gets assigned, it would also need to be an array too
3. And then also where ${ptxmirror_url} gets added to the argv array

Quite frankly, I really like my approach much better... and it does not
preclude you from implementing your solution as well.
If I created and tested a patch that allows FOO_URL_SAFE, would you
consider it for inclusion to ptxdist?

Thanks,
Jon

[-- Attachment #1.2: Type: text/html, Size: 3116 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Michael Olbrich]

On Fri, Jan 10, 2014 at 08:47:52PM -0500, Jon Ringle wrote:
> On Fri, Jan 10, 2014 at 10:37 AM, Michael Olbrich
> <m.olbrich@pengutronix.de>wrote:
> 
> > On Thu, Jan 09, 2014 at 11:37:15AM -0500, Jon Ringle wrote:
> > > Would the following approach work:
> > >
> > > 1. In ptxdist/rules/post/ptxd_make_world_common.make add to
> > world/env/impl:
> > >     pkg_url_safe="$(call ptx/escape,$($(1)_URL_SAFE))"
> > >
> > > 2. In ptxdist/scripts/lib/ptxd_make_get.sh function ptxd_make_get
> > > change all occurances of:
> > >     if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" ]; then
> > > with something like:
> > >     if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" -o -n "${pkg_url_safe}" ];
> > > then
> > >
> > > Then in the rule file you can do:
> > > FOO_URL_SAFE := y
> > >
> > > So that you can whitelist a specific URL if PTXCONF_SETUP_PTXMIRROR_ONLY
> > is
> > > set
> >
> > Hmmm, this is not so nice. How about making PTXCONF_SETUP_PTXMIRROR a list?
> > Then you can add http://opensource/pool/ and your git server to the
> > 'whitelist'.
> >
> > The approach you are proposing has several issue for which I don't have
> the necessary bash chops to figure out how to tackle:
> 
> 1. In the 1st while loop in ptxd_make_get(), I don't know how to get the
> case to accept multiple array values for ${PTXCONF_SETUP_PTXMIRROR}:
>         case "${url}" in
>         ${PTXCONF_SETUP_PTXMIRROR}/*/*)
>             # keep original URL, for stuff like glibc
>             argv[${#argv[@]}]="${url}"
>             mrd=true
>             ;;
>         ${PTXCONF_SETUP_PTXMIRROR}/*)
>             # if mirror is given us to download, add it, but only once
>             if ! ${mrd}; then
>                 argv[${#argv[@]}]="${url}"
>                 mrd=true
>             fi
>             ;;
> 
> Replacing with ${PTXCONF_SETUP_PTXMIRROR[@]} clear won't help as the
> various options aren't separate by PIPE character |.
> 
> 2. Where ptxmirror_url gets assigned, it would also need to be an array too
> 3. And then also where ${ptxmirror_url} gets added to the argv array
> 
> Quite frankly, I really like my approach much better... and it does not
> preclude you from implementing your solution as well.
> If I created and tested a patch that allows FOO_URL_SAFE, would you
> consider it for inclusion to ptxdist?

I've been thinking about this some more. The choice to only download from
the mirror comes from the BSP user via 'setup'. I don't think allowing the
BSP / ptxdist developer to overwrite that is a good idea.
What about an explicit URL whitelist in 'setup'?

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] git ptxdist best practices

[Jon Ringle]

On Fri, Jan 24, 2014 at 4:21 AM, Michael Olbrich
<m.olbrich@pengutronix.de> wrote:
> On Fri, Jan 10, 2014 at 08:47:52PM -0500, Jon Ringle wrote:
>> On Fri, Jan 10, 2014 at 10:37 AM, Michael Olbrich
>> <m.olbrich@pengutronix.de>wrote:
>>
>> > On Thu, Jan 09, 2014 at 11:37:15AM -0500, Jon Ringle wrote:
>> > > Would the following approach work:
>> > >
>> > > 1. In ptxdist/rules/post/ptxd_make_world_common.make add to
>> > world/env/impl:
>> > >     pkg_url_safe="$(call ptx/escape,$($(1)_URL_SAFE))"
>> > >
>> > > 2. In ptxdist/scripts/lib/ptxd_make_get.sh function ptxd_make_get
>> > > change all occurances of:
>> > >     if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" ]; then
>> > > with something like:
>> > >     if [ -z "${PTXCONF_SETUP_PTXMIRROR_ONLY}" -o -n "${pkg_url_safe}" ];
>> > > then
>> > >
>> > > Then in the rule file you can do:
>> > > FOO_URL_SAFE := y
>> > >
>> > > So that you can whitelist a specific URL if PTXCONF_SETUP_PTXMIRROR_ONLY
>> > is
>> > > set
>> >
>> > Hmmm, this is not so nice. How about making PTXCONF_SETUP_PTXMIRROR a list?
>> > Then you can add http://opensource/pool/ and your git server to the
>> > 'whitelist'.
>> >
>> > The approach you are proposing has several issue for which I don't have
>> the necessary bash chops to figure out how to tackle:
>>
>> 1. In the 1st while loop in ptxd_make_get(), I don't know how to get the
>> case to accept multiple array values for ${PTXCONF_SETUP_PTXMIRROR}:
>>         case "${url}" in
>>         ${PTXCONF_SETUP_PTXMIRROR}/*/*)
>>             # keep original URL, for stuff like glibc
>>             argv[${#argv[@]}]="${url}"
>>             mrd=true
>>             ;;
>>         ${PTXCONF_SETUP_PTXMIRROR}/*)
>>             # if mirror is given us to download, add it, but only once
>>             if ! ${mrd}; then
>>                 argv[${#argv[@]}]="${url}"
>>                 mrd=true
>>             fi
>>             ;;
>>
>> Replacing with ${PTXCONF_SETUP_PTXMIRROR[@]} clear won't help as the
>> various options aren't separate by PIPE character |.
>>
>> 2. Where ptxmirror_url gets assigned, it would also need to be an array too
>> 3. And then also where ${ptxmirror_url} gets added to the argv array
>>
>> Quite frankly, I really like my approach much better... and it does not
>> preclude you from implementing your solution as well.
>> If I created and tested a patch that allows FOO_URL_SAFE, would you
>> consider it for inclusion to ptxdist?
>
> I've been thinking about this some more. The choice to only download from
> the mirror comes from the BSP user via 'setup'. I don't think allowing the
> BSP / ptxdist developer to overwrite that is a good idea.
> What about an explicit URL whitelist in 'setup'?

I wouldn't mind that. This isn't any different than what you suggested
before. But as I mentioned before, I don't know how to implement this.

Jon

-- 
ptxdist mailing list
ptxdist@pengutronix.de