[ptxdist] root.ext2

[ptxdist] root.ext2

[Clemens Gruber]

Hi,

I noticed that the image-root-ext and image_ext2 rule still uses
genext2fs to first create an ext2 filesystem and then uses tune2fs to
convert it to ext3 and subsequently to ext4.

I think it is time to create ext4 images directly, otherwise we can't
use features like flex_bg and extra_isize because they need to be
specified at the time of creation.

To create the images, we could use make_ext4fs from the Android project
or (probably better) the standalone fork from the LEDE project:
https://git.lede-project.org/?p=project/make_ext4fs.git;a=summary

What do you think about this?

Thanks,
Clemens

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] root.ext2

[Michael Olbrich]

Hi,

On Sun, May 07, 2017 at 05:56:40PM +0200, Clemens Gruber wrote:
> I noticed that the image-root-ext and image_ext2 rule still uses
> genext2fs to first create an ext2 filesystem and then uses tune2fs to
> convert it to ext3 and subsequently to ext4.
> 
> I think it is time to create ext4 images directly, otherwise we can't
> use features like flex_bg and extra_isize because they need to be
> specified at the time of creation.
> 
> To create the images, we could use make_ext4fs from the Android project
> or (probably better) the standalone fork from the LEDE project:
> https://git.lede-project.org/?p=project/make_ext4fs.git;a=summary
> 
> What do you think about this?

mkfs.ext4 in the current version can add content to the filesystem. So that
is the way to go. I started a patch for genimage to use it, but so far I'm
lacking the time to finish this.
The main problem is, that mkfs.ext4 checks /etc/mke2fs.conf by default and
looks up some stuff in /proc when deciding what to do. So it's a bit tricky
to make sure that the build host does not influence the creation of the
image.

Regards,
Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] root.ext2

[Clemens Gruber]

Hi,

On Mon, May 08, 2017 at 10:02:10AM +0200, Michael Olbrich wrote:
> mkfs.ext4 in the current version can add content to the filesystem. So that
> is the way to go. I started a patch for genimage to use it, but so far I'm
> lacking the time to finish this.
> The main problem is, that mkfs.ext4 checks /etc/mke2fs.conf by default and
> looks up some stuff in /proc when deciding what to do. So it's a bit tricky
> to make sure that the build host does not influence the creation of the
> image.

Ok, so considering these problems of host influences when using
mkfs.ext4, even though -d is now a supported option, shouldn't we then
give make_ext4fs a try?

If mkfs.ext4 reads from /proc, we would have to patch this out or work
with upstream e2fsprogs adding a flag to skip these host system lookups?

If you still prefer mkfs.ext4, please send the current state of your
patch.

While we're at it, I thought we could also add support for f2fs image
creation. The problem there however is, that neither the mkfs.f2fs tool
nor any other 3rd party tool I know of can create a f2fs filesystem with
a pre-defined root directory. But maybe I missed something.
And loop-mounting to copy stuff to it, is probably not an option.

--

By the way:
In my platformconfig, I currently use IMAGE_ROOT_EXT from section image2
and not IMAGE_EXT2 from section image, so in my case, genext2fs is
called from rules/post/image_ext2.make and not via genimage from
rules/image-root-ext.make.
What is the purpose of there being two image sections image and image2
and is one of them deprecated or discouraged?

Thanks,
Clemens

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] root.ext2

[Michael Olbrich]

[-- Attachment #1: Type: text/plain, Size: 2912 bytes --]

Hi,

On Thu, May 11, 2017 at 06:14:21PM +0200, Clemens Gruber wrote:
> On Mon, May 08, 2017 at 10:02:10AM +0200, Michael Olbrich wrote:
> > mkfs.ext4 in the current version can add content to the filesystem. So that
> > is the way to go. I started a patch for genimage to use it, but so far I'm
> > lacking the time to finish this.
> > The main problem is, that mkfs.ext4 checks /etc/mke2fs.conf by default and
> > looks up some stuff in /proc when deciding what to do. So it's a bit tricky
> > to make sure that the build host does not influence the creation of the
> > image.
> 
> Ok, so considering these problems of host influences when using
> mkfs.ext4, even though -d is now a supported option, shouldn't we then
> give make_ext4fs a try?

No, I want the generated filesystem to be as close as possible to something
created on the target.

> If mkfs.ext4 reads from /proc, we would have to patch this out or work
> with upstream e2fsprogs adding a flag to skip these host system lookups?

I don't think patching is necessary. What I've seen was just defaults for
some options. So this just needs review and some options.

> If you still prefer mkfs.ext4, please send the current state of your
> patch.

I've attached my genimage patch. I think the lazy_* stuff was related to
the /proc issue. I'm not sure I like the tool selection, but I don't want
to break old setups either.

> While we're at it, I thought we could also add support for f2fs image
> creation. The problem there however is, that neither the mkfs.f2fs tool
> nor any other 3rd party tool I know of can create a f2fs filesystem with
> a pre-defined root directory. But maybe I missed something.

I never used f2fs, so I have no idea.

> And loop-mounting to copy stuff to it, is probably not an option.

That requires root access and is not easily integrated.

> By the way:
> In my platformconfig, I currently use IMAGE_ROOT_EXT from section image2
> and not IMAGE_EXT2 from section image, so in my case, genext2fs is
> called from rules/post/image_ext2.make and not via genimage from
> rules/image-root-ext.make.

That makes no sense. rules/image-root-ext.make is used by IMAGE_EXT2.

> What is the purpose of there being two image sections image and image2
> and is one of them deprecated or discouraged?

Only use the new image stuff. It's a rewrite of the other rules using
genimage and other internal changes. It should also be easier to use.

I just didn't want to remove the old rules until we had all the existing
features. I should probably remove the rules that have a matching in the
new image section.

Regards,
Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

[-- Attachment #2: 0001-image-ext-make-it-possible-to-create-the-image-with-.patch --]
[-- Type: text/x-diff, Size: 3422 bytes --]

From d5a155dfa2dfbbf42059697007e304326ffd4331 Mon Sep 17 00:00:00 2001
From: Michael Olbrich <m.olbrich@pengutronix.de>
Date: Thu, 16 Jun 2016 17:58:35 +0200
Subject: [PATCH] image-ext: make it possible to create the image with mke2fs

Signed-off-by: Michael Olbrich <m.olbrich@pengutronix.de>
---
 config.c     |  5 +++++
 image-ext2.c | 46 +++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 50 insertions(+), 1 deletion(-)

diff --git a/config.c b/config.c
index 7ba515756252..052553578f84 100644
--- a/config.c
+++ b/config.c
@@ -307,6 +307,11 @@ static struct config opts[] = {
 		.env = "GENIMAGE_MKDOSFS",
 		.def = "mkdosfs",
 	}, {
+		.name = "mke2fs",
+		.opt = CFG_STR("mke2fs", NULL, CFGF_NONE),
+		.env = "GENIMAGE_MKE2FS",
+		.def = "mke2fs",
+	}, {
 		.name = "mkfsjffs2",
 		.opt = CFG_STR("mkfsjffs2", NULL, CFGF_NONE),
 		.env = "GENIMAGE_MKFJFFS2",
diff --git a/image-ext2.c b/image-ext2.c
index 1440e1cd6b10..8f7cff0364a3 100644
--- a/image-ext2.c
+++ b/image-ext2.c
@@ -22,7 +22,7 @@
 
 #include "genimage.h"
 
-static int ext2_generate(struct image *image)
+static int ext2_generate_genext2fs(struct image *image)
 {
 	int ret;
 	char *extraargs = cfg_getstr(image->imagesec, "extraargs");
@@ -49,6 +49,44 @@ static int ext2_generate(struct image *image)
 		if (ret)
 			return ret;
 	}
+	return 0;
+}
+
+static int ext2_generate_mke2fs(struct image *image)
+{
+	char *extraargs = cfg_getstr(image->imagesec, "extraargs");
+	char *label = cfg_getstr(image->imagesec, "label");
+	char *conf = cfg_getstr(image->imagesec, "mke2fs_conf");
+
+	if (label &&label[0] == '\0')
+		label = NULL;
+
+	return systemp(image, "%s%s%s%s -t %s -E root_owner=0:0 -E lazy_itable_init=0,lazy_journal_init=0 %s -d '%s' %s %s%s '%s' %lld",
+			conf ? "MKE2FS_CONFIG=" : "", conf ? conf : "", conf ? " " : "",
+			get_opt("mke2fs"), image->handler->type,
+			image->size < 0x20000000000 ? "-O ^huge_file" : "",
+			mountpath(image), extraargs,
+			label ? "-L " : "", label ? label : "",
+			imageoutfile(image), image->size / 1024);
+}
+
+static int ext2_generate(struct image *image)
+{
+	int ret;
+	const char *tool = cfg_getstr(image->imagesec, "tool");
+
+	if (strcmp(tool, "mke2fs") == 0)
+		ret = ext2_generate_mke2fs(image);
+	else if (strcmp(tool, "genext2fs") == 0)
+		ret = ext2_generate_genext2fs(image);
+	else {
+		image_error(image, "unknown tool to create %s images: %s",
+			image->handler->type, tool);
+		ret = -EINVAL;
+	}
+
+	if (ret)
+		return ret;
 
 	ret = systemp(image, "%s -pvfD %s", get_opt("e2fsck"),
 			imageoutfile(image));
@@ -61,6 +99,8 @@ static cfg_opt_t ext2_opts[] = {
 	CFG_STR("extraargs", "", CFGF_NONE),
 	CFG_STR("features", 0, CFGF_NONE),
 	CFG_STR("label", 0, CFGF_NONE),
+	CFG_STR("tool", "genext2fs", CFGF_NONE),
+	CFG_STR("mke2fs_conf", 0, CFGF_NONE),
 	CFG_END()
 };
 
@@ -74,6 +114,8 @@ static cfg_opt_t ext3_opts[] = {
 	CFG_STR("extraargs", "", CFGF_NONE),
 	CFG_STR("features", "has_journal", CFGF_NONE),
 	CFG_STR("label", 0, CFGF_NONE),
+	CFG_STR("tool", "genext2fs", CFGF_NONE),
+	CFG_STR("mke2fs_conf", 0, CFGF_NONE),
 	CFG_END()
 };
 
@@ -87,6 +129,8 @@ static cfg_opt_t ext4_opts[] = {
 	CFG_STR("extraargs", "", CFGF_NONE),
 	CFG_STR("features", "extents,uninit_bg,dir_index,has_journal", CFGF_NONE),
 	CFG_STR("label", 0, CFGF_NONE),
+	CFG_STR("tool", "genext2fs", CFGF_NONE),
+	CFG_STR("mke2fs_conf", 0, CFGF_NONE),
 	CFG_END()
 };
 
-- 
2.11.0


[-- Attachment #3: Type: text/plain, Size: 91 bytes --]

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de