[ptxdist] lndir'ing sysroot-target

[ptxdist] lndir'ing sysroot-target

[Erwin Rol]

Hallo all,

in my search to find a way to split ptxdist into an OS build and 
application build process I tried the following.

1) create and build an ptxdist project with everything the OS needs like 
glib, gtk, dbus, cairo, etc.
2) create an ptxdist project with applications. And don't have any 
"select"s to OS parts like gtk or cairo in the .in files of the 
applications.
3) lndir sysroot-target from the OS build into the sysroot-target of the 
application build.
4) build the application images.


That way I was able to build my applications apart from the OS, but 
still use ptxdist to build.

Would this be a feature that others are interested in ?

And any ideas or warnings on what could go wrong ?

- Erwin

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] lndir'ing sysroot-target

[Robert Schwebel]

On Fri, Nov 04, 2011 at 08:46:41PM +0100, Erwin Rol wrote:
> Would this be a feature that others are interested in ?
> 
> And any ideas or warnings on what could go wrong ?

Sounds interesging - however, you should synchronize this with Michael
Olbrich. I'd like to have only one method for "SDK style" development in
ptxdist in the end.

rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] lndir'ing sysroot-target

[Michael Olbrich]

Hi,

On Fri, Nov 04, 2011 at 08:46:41PM +0100, Erwin Rol wrote:
> in my search to find a way to split ptxdist into an OS build and
> application build process I tried the following.
> 
> 1) create and build an ptxdist project with everything the OS needs
> like glib, gtk, dbus, cairo, etc.
> 2) create an ptxdist project with applications. And don't have any
> "select"s to OS parts like gtk or cairo in the .in files of the
> applications.
> 3) lndir sysroot-target from the OS build into the sysroot-target of
> the application build.
> 4) build the application images.
> 
> 
> That way I was able to build my applications apart from the OS, but
> still use ptxdist to build.
> 
> Would this be a feature that others are interested in ?
> 
> And any ideas or warnings on what could go wrong ?

Something like that already exists. With "build type" == "production" you
can put the build in some global location. Then with "use pre-built
production release" reuse it and add packages without dependencies on the
core system like you suggested.
Both sources are then used to create the images. I think it's even possible
to combine that with collections.
"built on top of another platform" is a variation of this where you use a
different local platform in the same BSP build, e.g. a generic ARMv5 core
platform + several variants with extra local packages.

This works well enough if the base platform is stable. However, because of
the missing dependencies you should rebuilt everything when it changes.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] lndir'ing sysroot-target

[Erwin Rol]

On 5-11-2011 12:23, Michael Olbrich wrote:
> Hi,
>
> On Fri, Nov 04, 2011 at 08:46:41PM +0100, Erwin Rol wrote:
>> in my search to find a way to split ptxdist into an OS build and
>> application build process I tried the following.
>>
>> 1) create and build an ptxdist project with everything the OS needs
>> like glib, gtk, dbus, cairo, etc.
>> 2) create an ptxdist project with applications. And don't have any
>> "select"s to OS parts like gtk or cairo in the .in files of the
>> applications.
>> 3) lndir sysroot-target from the OS build into the sysroot-target of
>> the application build.
>> 4) build the application images.
>>
>>
>> That way I was able to build my applications apart from the OS, but
>> still use ptxdist to build.
>>
>> Would this be a feature that others are interested in ?
>>
>> And any ideas or warnings on what could go wrong ?
>
> Something like that already exists. With "build type" == "production" you
> can put the build in some global location. Then with "use pre-built
> production release" reuse it and add packages without dependencies on the
> core system like you suggested.
> Both sources are then used to create the images. I think it's even possible
> to combine that with collections.
> "built on top of another platform" is a variation of this where you use a
> different local platform in the same BSP build, e.g. a generic ARMv5 core
> platform + several variants with extra local packages.
>
> This works well enough if the base platform is stable. However, because of
> the missing dependencies you should rebuilt everything when it changes.
>

Is that really the same. When I understand you correctly the Production 
build only builds part of the system and the "use pre-built" creates the 
final images out of the two parts.

That is not the same as generating an OS image that runs and works fine. 
And than use a second ptxdist project to create applications that you 
can be installed on that OS.

Or did I misunderstand something?

- Erwin

PS: is there any documentation on how those mentioned feature should 
work, cause the "Help" in ptxdist isn't really helpful.






-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] lndir'ing sysroot-target

[Michael Olbrich]

On Sun, Nov 06, 2011 at 01:37:29AM +0100, Erwin Rol wrote:
> On 5-11-2011 12:23, Michael Olbrich wrote:
> >Hi,
> >
> >On Fri, Nov 04, 2011 at 08:46:41PM +0100, Erwin Rol wrote:
> >>in my search to find a way to split ptxdist into an OS build and
> >>application build process I tried the following.
> >>
> >>1) create and build an ptxdist project with everything the OS needs
> >>like glib, gtk, dbus, cairo, etc.
> >>2) create an ptxdist project with applications. And don't have any
> >>"select"s to OS parts like gtk or cairo in the .in files of the
> >>applications.
> >>3) lndir sysroot-target from the OS build into the sysroot-target of
> >>the application build.
> >>4) build the application images.
> >>
> >>
> >>That way I was able to build my applications apart from the OS, but
> >>still use ptxdist to build.
> >>
> >>Would this be a feature that others are interested in ?
> >>
> >>And any ideas or warnings on what could go wrong ?
> >
> >Something like that already exists. With "build type" == "production" you
> >can put the build in some global location. Then with "use pre-built
> >production release" reuse it and add packages without dependencies on the
> >core system like you suggested.
> >Both sources are then used to create the images. I think it's even possible
> >to combine that with collections.
> >"built on top of another platform" is a variation of this where you use a
> >different local platform in the same BSP build, e.g. a generic ARMv5 core
> >platform + several variants with extra local packages.
> >
> >This works well enough if the base platform is stable. However, because of
> >the missing dependencies you should rebuilt everything when it changes.
> >
> 
> Is that really the same. When I understand you correctly the
> Production build only builds part of the system and the "use
> pre-built" creates the final images out of the two parts.
> 
> That is not the same as generating an OS image that runs and works
> fine. And than use a second ptxdist project to create applications
> that you can be installed on that OS.
> 
> Or did I misunderstand something?

But the only difference is 2 images vs. 1, right? That's not quite that
simple. We can easily add an option to not include the ipkgs from the base
system. However the resulting image needs to be mounted somewhere. To do
this right we would need to strip the mount point from the paths when
creating the image. Otherwise the prefix when building and running wont
match.
The question is why do you want 2 images? Unless you add some ugly hacks
you probably need to add some stuff to /etc anyway (e.g. init scripts).

> PS: is there any documentation on how those mentioned feature should
> work, cause the "Help" in ptxdist isn't really helpful.

Unfortunately not. That would require several pages od documentation and
and nobody is really happy with how these features work anyways...
The main problem is, that I have yet to find two people that agree on how
something like this is supposed to work. And while we could probably
implement a lot of these variations, it would create (or rather already is)
a configuration nightmare that only a few experts can handle.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] lndir'ing sysroot-target

[Erwin Rol]

Hey all,

On 6-11-2011 11:33, Michael Olbrich wrote:

>> Or did I misunderstand something?
>
> But the only difference is 2 images vs. 1, right? That's not quite that
> simple. We can easily add an option to not include the ipkgs from the base
> system. However the resulting image needs to be mounted somewhere. To do
> this right we would need to strip the mount point from the paths when
> creating the image. Otherwise the prefix when building and running wont
> match.
> The question is why do you want 2 images? Unless you add some ugly hacks
> you probably need to add some stuff to /etc anyway (e.g. init scripts).

The two images is not only about "two images" it is also a 
organizational boundary.  The OS is created, installed on the hardware 
and than that finished product is handed to someone that will develop 
applications that run on it.  How those applications are installed is a 
solvable detail.

>> PS: is there any documentation on how those mentioned feature should
>> work, cause the "Help" in ptxdist isn't really helpful.
>
> Unfortunately not. That would require several pages od documentation and
> and nobody is really happy with how these features work anyways...
> The main problem is, that I have yet to find two people that agree on how
> something like this is supposed to work. And while we could probably
> implement a lot of these variations, it would create (or rather already is)
> a configuration nightmare that only a few experts can handle.

This sounds like an point where ptxdist could/should be improved. 
Ptxdist is in my eyes a bit to monolithic  (source -> magic -> images). 
It would be really nice to be able to hook into certain parts, like the 
image generation, without to much black magic.  I really would like to 
see Ptxdist move forwards (and not just under the hood).  But that 
discussion could become a very large thread I guess.

- Erwin

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] lndir'ing sysroot-target

[Michael Olbrich]

On Sun, Nov 06, 2011 at 02:16:56PM +0100, Erwin Rol wrote:
> >But the only difference is 2 images vs. 1, right? That's not quite that
> >simple. We can easily add an option to not include the ipkgs from the base
> >system. However the resulting image needs to be mounted somewhere. To do
> >this right we would need to strip the mount point from the paths when
> >creating the image. Otherwise the prefix when building and running wont
> >match.
> >The question is why do you want 2 images? Unless you add some ugly hacks
> >you probably need to add some stuff to /etc anyway (e.g. init scripts).
> 
> The two images is not only about "two images" it is also a
> organizational boundary.  The OS is created, installed on the
> hardware and than that finished product is handed to someone that
> will develop applications that run on it.  How those applications
> are installed is a solvable detail.

Well, you can get the 'OS' image from the production build, and the ipkgs
for you applications. What else do you need then to pull everything
together?

> >>PS: is there any documentation on how those mentioned feature should
> >>work, cause the "Help" in ptxdist isn't really helpful.
> >
> >Unfortunately not. That would require several pages od documentation and
> >and nobody is really happy with how these features work anyways...
> >The main problem is, that I have yet to find two people that agree on how
> >something like this is supposed to work. And while we could probably
> >implement a lot of these variations, it would create (or rather already is)
> >a configuration nightmare that only a few experts can handle.
> 
> This sounds like an point where ptxdist could/should be improved.
> Ptxdist is in my eyes a bit to monolithic  (source -> magic ->
> images). It would be really nice to be able to hook into certain
> parts, like the image generation, without to much black magic.  I
> really would like to see Ptxdist move forwards (and not just under
> the hood).  But that discussion could become a very large thread I
> guess.

This won't be a short discussion, but a very welcome one. I've talked a lot
about separating the image generation a bit more. But that's not that easy.
In the end running "ptxdist images" to get on final blob must still be
possible.
We're actually working on some separate tool to create the images, that can
be used separately or by ptxdist to create the images. The main problem we
have here is configuration. Defining the images in kconfig is rather
awkward and not really flexible. However, while we think, we now have
something nice to define images, I still need to work out how to integrate
that with ptxdist...
Any comments on what you would like to see there are welcome.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] lndir'ing sysroot-target

[Jon Ringle]

[-- Attachment #1.1: Type: text/plain, Size: 1176 bytes --]

On Sun, Nov 6, 2011 at 9:40 AM, Michael Olbrich <m.olbrich@pengutronix.de>wrote:

> This won't be a short discussion, but a very welcome one. I've talked a lot
> about separating the image generation a bit more. But that's not that easy.
> In the end running "ptxdist images" to get on final blob must still be
> possible.
> We're actually working on some separate tool to create the images, that can
> be used separately or by ptxdist to create the images. The main problem we
> have here is configuration. Defining the images in kconfig is rather
> awkward and not really flexible. However, while we think, we now have
> something nice to define images, I still need to work out how to integrate
> that with ptxdist...
> Any comments on what you would like to see there are welcome.
>
> Something that I'd like to see is a way to have a kernel+initramfs image
to be created as a package. As it stands now, I create a kernel platform
image and a rootfs platform image, and because the rootfs image contains
kernel modules not contained within the initramfs image, I end up having
the kernel built twice with two separate kernel source trees that are
basically the same.

Jon

[-- Attachment #1.2: Type: text/html, Size: 1566 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] lndir'ing sysroot-target

[Erwin Rol]

On 6-11-2011 20:10, Jon Ringle wrote:

> Something that I'd like to see is a way to have a kernel+initramfs image
> to be created as a package. As it stands now, I create a kernel platform
> image and a rootfs platform image, and because the rootfs image contains
> kernel modules not contained within the initramfs image, I end up having
> the kernel built twice with two separate kernel source trees that are
> basically the same.

I have that with one project too. But that also boils down to multiple 
images I guess, one hardisk rootfs and one ram rootfs.

- Erwin


-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] lndir'ing sysroot-target

[Michael Olbrich]

On Mon, Nov 07, 2011 at 11:15:23AM +0100, Erwin Rol wrote:
> On 6-11-2011 20:10, Jon Ringle wrote:
> >Something that I'd like to see is a way to have a kernel+initramfs image
> >to be created as a package. As it stands now, I create a kernel platform
> >image and a rootfs platform image, and because the rootfs image contains
> >kernel modules not contained within the initramfs image, I end up having
> >the kernel built twice with two separate kernel source trees that are
> >basically the same.
> 
> I have that with one project too. But that also boils down to
> multiple images I guess, one hardisk rootfs and one ram rootfs.

What are the requirements for these images?

What we currently do to create images is:
- take some ipkgs (currently all selected)
- extract them to a tmpdir
- create all selected images from this tmpdir

If we expand this to do this multiple times for different sets of ipkgs and
different images, would that cover your use-case?

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

-- 
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] lndir'ing sysroot-target

[Jon Ringle]

[-- Attachment #1.1: Type: text/plain, Size: 1989 bytes --]

On Mon, Nov 7, 2011 at 5:49 AM, Michael Olbrich <m.olbrich@pengutronix.de>wrote:

> On Mon, Nov 07, 2011 at 11:15:23AM +0100, Erwin Rol wrote:
> > On 6-11-2011 20:10, Jon Ringle wrote:
> > >Something that I'd like to see is a way to have a kernel+initramfs image
> > >to be created as a package. As it stands now, I create a kernel platform
> > >image and a rootfs platform image, and because the rootfs image contains
> > >kernel modules not contained within the initramfs image, I end up having
> > >the kernel built twice with two separate kernel source trees that are
> > >basically the same.
> >
> > I have that with one project too. But that also boils down to
> > multiple images I guess, one hardisk rootfs and one ram rootfs.
>
> What are the requirements for these images?
>
> What we currently do to create images is:
> - take some ipkgs (currently all selected)
> - extract them to a tmpdir
> - create all selected images from this tmpdir
>
> If we expand this to do this multiple times for different sets of ipkgs and
> different images, would that cover your use-case?
>
>
In the case of kernel+initramfs, it would not be as simple as that, since
in my case the initramfs image is built using uclibc toolchain and rootfs
image is built with glibc toolchain. So, I may have busybox for initramfs
using uclibc and busybox for rootfs using glibc.

I was envisioning something along the lines of a "nested" ptx project for
the kernel+initramfs where kernel and packages for initramfs are built
using PTXCONF_COMPILER_PREFIX_KERNEL as the toolchain.

Menu to select packages for kernel+initramfs "nested" ptx project:
$ ptxdist initramfs menuconfig

The following would build all selected packages and if kernel+initramfs
package is selected then to build it, it would recurse into calling a
nested ptxdist to build the kernel+initramfs and set the nested
PTXCONF_COMPILER_PREFIX to the parent's PTXCONF_COMPILER_PREFIX_KERNEL as
the toolchain:
$ ptxdist go
$ ptxdist images

Jon

[-- Attachment #1.2: Type: text/html, Size: 2613 bytes --]

[-- Attachment #2: Type: text/plain, Size: 48 bytes --]

-- 
ptxdist mailing list
ptxdist@pengutronix.de