[ptxdist] tarballs from git URLs: git archive vs. git submodule

[ptxdist] tarballs from git URLs: git archive vs. git submodule

[Alexander Dahl]

Hei hei, 

while trying the local_src feature (local_src/${PACKAGE}.{PLATFORM})
which is actually really cool, I came across a problem with the
automatic tarball creation from Git URLs. ptxd_make_get.sh uses `git
archive` for this purpose which does not support git submodules. There
are several scripts around the web by people who needed this in the past
and although there were feature requests and even patches years ago in
git, it's still not included mainline. :-(

The possibilities I see are these:

* waiting probably very long for a new git release with git archive
supporting submodules
* replace the use of submodules in the repositories I want to use with
something else, maybe git subtree
* scripting this and change ptxd_make_get.sh which could be quite
complicated

For the third possibility: I would suggest cloning the repo with the
--mirror option into a --bare repository almost like it's done now. Then
do a `git clone --recursive` to a temporary folder followed by a
checkout to the desired tag. Make a tarball from this clone and remove
the directory right afterwards. The git url support was introduced in
June 2013 by Michael Olbrich and not touch since. Did anyone try to
improve it in any way? Would a change in the proposed direction have
chances to be accepted by the ptxdist developers? It would make the code
more complex I guess.

The main and almost only use for git submodules here is pulling in some
cmake modules not included by CMake upstream but necessary for our
builds. I'm open for any other suggestions. ;-)

Greets
Alex

-- 
»With the first link, the chain is forged. The first speech censured,
the first thought forbidden, the first freedom denied, chains us all
irrevocably.« (Jean-Luc Picard, quoting Judge Aaron Satie)
*** GnuPG-FP: 02C8 A590 7FE5 CA5F 3601  D1D5 8FBA 7744 CC87 10D0 ***

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] tarballs from git URLs: git archive vs. git submodule

[Michael Olbrich]

Hi,

On Mon, Mar 03, 2014 at 01:13:00PM +0100, Alexander Dahl wrote:
> while trying the local_src feature (local_src/${PACKAGE}.{PLATFORM})
> which is actually really cool, I came across a problem with the
> automatic tarball creation from Git URLs. ptxd_make_get.sh uses `git
> archive` for this purpose which does not support git submodules. There
> are several scripts around the web by people who needed this in the past
> and although there were feature requests and even patches years ago in
> git, it's still not included mainline. :-(
> 
> The possibilities I see are these:
> 
> * waiting probably very long for a new git release with git archive
> supporting submodules
> * replace the use of submodules in the repositories I want to use with
> something else, maybe git subtree
> * scripting this and change ptxd_make_get.sh which could be quite
> complicated

 * create a tarball for each repository and extract all in a custom extract
   stage.

Take a look at e.g. host-tz-database on how to handle multiple tarballs.

> For the third possibility: I would suggest cloning the repo with the
> --mirror option into a --bare repository almost like it's done now. Then
> do a `git clone --recursive` to a temporary folder followed by a
> checkout to the desired tag. Make a tarball from this clone and remove
> the directory right afterwards. The git url support was introduced in
> June 2013 by Michael Olbrich and not touch since. Did anyone try to
> improve it in any way? Would a change in the proposed direction have
> chances to be accepted by the ptxdist developers? It would make the code
> more complex I guess.

'git archive' always creates the exact same tarball. It's not that easy to
do the same thing manually. I'd like to avoid too much complexity here.
But I'd be open for a patch that puts the 'git archive' call in a separate
function. Then you only need to overwrite this function locally in your
BSP with a special version that handles submodules.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] tarballs from git URLs: git archive vs. git submodule

[wim vinckier]

Hi all,

I'm having the same issue as described.  Is there meanwhile any other
solution but the ones subscribed?

Kind regards,

wimpunk.

On 6 March 2014 at 19:03, Michael Olbrich <m.olbrich@pengutronix.de> wrote:
> Hi,
>
> On Mon, Mar 03, 2014 at 01:13:00PM +0100, Alexander Dahl wrote:
>> while trying the local_src feature (local_src/${PACKAGE}.{PLATFORM})
>> which is actually really cool, I came across a problem with the
>> automatic tarball creation from Git URLs. ptxd_make_get.sh uses `git
>> archive` for this purpose which does not support git submodules. There
>> are several scripts around the web by people who needed this in the past
>> and although there were feature requests and even patches years ago in
>> git, it's still not included mainline. :-(
>>
>> The possibilities I see are these:
>>
>> * waiting probably very long for a new git release with git archive
>> supporting submodules
>> * replace the use of submodules in the repositories I want to use with
>> something else, maybe git subtree
>> * scripting this and change ptxd_make_get.sh which could be quite
>> complicated
>
>  * create a tarball for each repository and extract all in a custom extract
>    stage.
>
> Take a look at e.g. host-tz-database on how to handle multiple tarballs.
>
>> For the third possibility: I would suggest cloning the repo with the
>> --mirror option into a --bare repository almost like it's done now. Then
>> do a `git clone --recursive` to a temporary folder followed by a
>> checkout to the desired tag. Make a tarball from this clone and remove
>> the directory right afterwards. The git url support was introduced in
>> June 2013 by Michael Olbrich and not touch since. Did anyone try to
>> improve it in any way? Would a change in the proposed direction have
>> chances to be accepted by the ptxdist developers? It would make the code
>> more complex I guess.
>
> 'git archive' always creates the exact same tarball. It's not that easy to
> do the same thing manually. I'd like to avoid too much complexity here.
> But I'd be open for a patch that puts the 'git archive' call in a separate
> function. Then you only need to overwrite this function locally in your
> BSP with a special version that handles submodules.
>
> Michael
>
> --
> Pengutronix e.K.                           |                             |
> Industrial Linux Solutions                 | http://www.pengutronix.de/  |
> Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |
>
> --
> ptxdist mailing list
> ptxdist@pengutronix.de

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] tarballs from git URLs: git archive vs. git submodule

[Wim Vinckier]

Hi all,

I'm having the same issue as described.  Is there meanwhile any other
solution but the ones subscribed?

Kind regards,

wimpunk.

On 6 March 2014 at 19:03, Michael Olbrich <m.olbrich@pengutronix.de> wrote:
> Hi,
>
> On Mon, Mar 03, 2014 at 01:13:00PM +0100, Alexander Dahl wrote:
>> while trying the local_src feature (local_src/${PACKAGE}.{PLATFORM})
>> which is actually really cool, I came across a problem with the
>> automatic tarball creation from Git URLs. ptxd_make_get.sh uses `git
>> archive` for this purpose which does not support git submodules. There
>> are several scripts around the web by people who needed this in the past
>> and although there were feature requests and even patches years ago in
>> git, it's still not included mainline. :-(
>>
>> The possibilities I see are these:
>>
>> * waiting probably very long for a new git release with git archive
>> supporting submodules
>> * replace the use of submodules in the repositories I want to use with
>> something else, maybe git subtree
>> * scripting this and change ptxd_make_get.sh which could be quite
>> complicated
>
>  * create a tarball for each repository and extract all in a custom extract
>    stage.
>
> Take a look at e.g. host-tz-database on how to handle multiple tarballs.
>
>> For the third possibility: I would suggest cloning the repo with the
>> --mirror option into a --bare repository almost like it's done now. Then
>> do a `git clone --recursive` to a temporary folder followed by a
>> checkout to the desired tag. Make a tarball from this clone and remove
>> the directory right afterwards. The git url support was introduced in
>> June 2013 by Michael Olbrich and not touch since. Did anyone try to
>> improve it in any way? Would a change in the proposed direction have
>> chances to be accepted by the ptxdist developers? It would make the code
>> more complex I guess.
>
> 'git archive' always creates the exact same tarball. It's not that easy to
> do the same thing manually. I'd like to avoid too much complexity here.
> But I'd be open for a patch that puts the 'git archive' call in a separate
> function. Then you only need to overwrite this function locally in your
> BSP with a special version that handles submodules.
>
> Michael
>
> --
> Pengutronix e.K.                           |                             |
> Industrial Linux Solutions                 | http://www.pengutronix.de/  |
> Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |
>
> --
> ptxdist mailing list
> ptxdist@pengutronix.de

-- 
ptxdist mailing list
ptxdist@pengutronix.de