[ptxdist] systemd: Using second offline update service beside rc-once in PTXDIST

[ptxdist] systemd: Using second offline update service beside rc-once in PTXDIST

[Schenk, Gavin]

Hi,

before continuing to hack ptxdist and rc-once, I would appreciate some feedback from the gurus.
 
I want to use systemd.offline-updated as introduced here and already used in PTXDIST by rc-once:
https://www.freedesktop.org/software/systemd/man/systemd.offline-updates.html#

In my use case, I have to do some data-migration stuff on an already delivered target. The data-migration has some deep impacts on the targets storages and I want to ensure no other stuff is running on the machine. Additionally I would like to provide my customers some gui when transforming the targets data. As the migration is done via shell scripts I use pv and dialog. To be able to use dialog a VT on the framebuffer I have to do some settings in the service file e.g.:

...
StandardOutput=tty                                                                                                                                                                             
StandardInput=tty                                                                                                                                                                              
TTYPath=/dev/tty1                                                                                                                                                                              
TTYReset=yes                                                                                                                                                                                   
TTYVHangup=yes
...

One of my problems is, that rc-once does not check where /system-update points to, as suggested. My idea is to install a second update-service that does the migration stuff instead of hacking rc-once service file and do it there. Running both in parallel is a bad idea!
 
What do you think about using readlink in systemd-rc-once and just exit without error if /system-update does point to another location than /etc/rc-once?
Another idea is to add a second service that doesn´t check /system-update either and use a after directive in its service file?
Is this something we want a common solution for?

Best regards
Gavin Schenk

Eckelmann AG
Vorstand: Dipl.-Ing. Peter Frankenbach (Sprecher) Dipl.-Wi.-Ing. Philipp Eckelmann
Dr.-Ing. Marco Münchhof Dr.-Ing. Frank Uhlemann
Vorsitzender des Aufsichtsrats: Hubertus G. Krossa
Stv. Vorsitzender des Aufsichtsrats: Dr.-Ing. Gerd Eckelmann
Sitz der Gesellschaft: Berliner Str. 161, 65205 Wiesbaden, Amtsgericht Wiesbaden HRB 12636
http://www.eckelmann.de 


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] systemd: Using second offline update service beside rc-once in PTXDIST

[Michael Olbrich]

Hi,

On Thu, Jun 01, 2017 at 02:51:46PM +0000, Schenk, Gavin wrote:
> before continuing to hack ptxdist and rc-once, I would appreciate some
> feedback from the gurus.
>  
> I want to use systemd.offline-updated as introduced here and already used in PTXDIST by rc-once:
> https://www.freedesktop.org/software/systemd/man/systemd.offline-updates.html#
> 
> In my use case, I have to do some data-migration stuff on an already
> delivered target. The data-migration has some deep impacts on the targets
> storages and I want to ensure no other stuff is running on the machine.
> Additionally I would like to provide my customers some gui when transforming
> the targets data. As the migration is done via shell scripts I use pv and
> dialog. To be able to use dialog a VT on the framebuffer I have to do some
> settings in the service file e.g.:
> 
> ...
> StandardOutput=tty
> StandardInput=tty
> TTYPath=/dev/tty1
> TTYReset=yes
> TTYVHangup=yes
> ...
> 
> One of my problems is, that rc-once does not check where /system-update
> points to, as suggested. My idea is to install a second update-service that
> does the migration stuff instead of hacking rc-once service file and do it
> there. Running both in parallel is a bad idea!
>  
> What do you think about using readlink in systemd-rc-once and just exit
> without error if /system-update does point to another location than
> /etc/rc-once?
> Another idea is to add a second service that doesn´t check /system-update
> either and use a after directive in its service file?
> Is this something we want a common solution for?

I think checking the link target for /system-update is a good solution.
This will keep things separated. But make sure the check is correct:
Currently /usr/sbin/enable-rc-once uses 'etc/rc.once.d' as link target and
/usr/lib/systemd/systemd-rc-once uses '/etc/rc.once.d' (in the fallback
case). I think that should be fixed, but both versions should be accepted
by systemd-rc-once.

Regards,
Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] systemd: Using second offline update service beside rc-once in PTXDIST

[Schenk, Gavin]

Hi Michael,

> >
> > What do you think about using readlink in systemd-rc-once and just
> > exit without error if /system-update does point to another location
> > than /etc/rc-once?
> > Another idea is to add a second service that doesn´t check
> > /system-update either and use a after directive in its service file?
> > Is this something we want a common solution for?
> 
> I think checking the link target for /system-update is a good solution.
> This will keep things separated. But make sure the check is correct:
> Currently /usr/sbin/enable-rc-once uses 'etc/rc.once.d' as link target
> and /usr/lib/systemd/systemd-rc-once uses '/etc/rc.once.d' (in the
> fallback case). I think that should be fixed, but both versions should
> be accepted by systemd-rc-once.
> 

Thank you for the fast feedback, I will send a patch.

Some Questions:
Using readlink is ok? (additional dependency rc-once -> readlink ) 
Is there anything more lightweight?
Do we need this stuff configurable?

Regards
Gavin



Eckelmann AG
Vorstand: Dipl.-Ing. Peter Frankenbach (Sprecher) Dipl.-Wi.-Ing. Philipp Eckelmann
Dr.-Ing. Marco Münchhof Dr.-Ing. Frank Uhlemann
Vorsitzender des Aufsichtsrats: Hubertus G. Krossa
Stv. Vorsitzender des Aufsichtsrats: Dr.-Ing. Gerd Eckelmann
Sitz der Gesellschaft: Berliner Str. 161, 65205 Wiesbaden, Amtsgericht Wiesbaden HRB 12636
http://www.eckelmann.de 

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] systemd: Using second offline update service beside rc-once in PTXDIST

[Michael Olbrich]

Hi,

On Fri, Jun 02, 2017 at 11:15:37AM +0000, Schenk, Gavin wrote:
> > > What do you think about using readlink in systemd-rc-once and just
> > > exit without error if /system-update does point to another location
> > > than /etc/rc-once?
> > > Another idea is to add a second service that doesn´t check
> > > /system-update either and use a after directive in its service file?
> > > Is this something we want a common solution for?
> > 
> > I think checking the link target for /system-update is a good solution.
> > This will keep things separated. But make sure the check is correct:
> > Currently /usr/sbin/enable-rc-once uses 'etc/rc.once.d' as link target
> > and /usr/lib/systemd/systemd-rc-once uses '/etc/rc.once.d' (in the
> > fallback case). I think that should be fixed, but both versions should
> > be accepted by systemd-rc-once.
> > 
> 
> Thank you for the fast feedback, I will send a patch.
> 
> Some Questions:
> Using readlink is ok? (additional dependency rc-once -> readlink ) 
> Is there anything more lightweight?

Not that I know. Note: if you use 'readlink -f' then the busybox version
has a option for this, so maybe:

	select BUSYBOX_FEATURE_READLINK_FOLLOW	if BUSYBOX_READLINK
	select COREUTILS			if !BUSYBOX_READLINK
	select COREUTILS_READLINK		if !BUSYBOX_READLINK

> Do we need this stuff configurable?

No. But I think we should have a sanity check. To avoid getting stuck in
the system-update.target: A service that is part of system-update.target
and starts the rescue target. rc-once.service and your service should both
be sorted 'Before' that service. If no other service starts a different
target we automatically fall into the rescue target.

Regards,
Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] systemd: Using second offline update service beside rc-once in PTXDIST

[Schenk, Gavin]

Hi Michael,

> >
> > Thank you for the fast feedback, I will send a patch.
> >
> > Some Questions:
> > Using readlink is ok? (additional dependency rc-once -> readlink ) Is
> > there anything more lightweight?
> 
> Not that I know. Note: if you use 'readlink -f' then the busybox version
> has a option for this, so maybe:
> 
> 	select BUSYBOX_FEATURE_READLINK_FOLLOW	if BUSYBOX_READLINK
> 	select COREUTILS			if !BUSYBOX_READLINK
> 	select COREUTILS_READLINK		if !BUSYBOX_READLINK
> 
Ok, done.

> > Do we need this stuff configurable?
> 
> No. But I think we should have a sanity check. To avoid getting stuck in
> the system-update.target: A service that is part of system-update.target
> and starts the rescue target. rc-once.service and your service should
> both be sorted 'Before' that service. If no other service starts a
> different target we automatically fall into the rescue target.
> 
Is "FailureAction=reboot" or "FailureAction=rescue.target" in service file sufficient here?

I am not happy with my implementation today :-(. The main reason is, it does not cover my usecase where I want to do something:
   * Once at first boot
   * With UI on tty
   * Without conflicting rc-once

For this usecase it seems to be enough to add a service to system-update.target.wants with "After=rc-once.service". This is not flawless, because the system continues booting after rc-once is done e.g. getty.target. It seems to be related to "systemctl daemon-reexec" in systemd-rc-once I am not sure. The handling of bbinit beside systemd and managing write protection in this scripts scares me. I am afraid to break stuff when doing changes here.

I reverted my last commit where I added some recommendations of freedesktop.org:
   * Requires=sysinit.target
   * After=sysinit.target

The result was a broken system. I reverted this to your implementation
   * After=local-fs.target
   * Requires=local-fs.target
   * Before=system-update-cleanup.service
   * After=system-update.target

I will send my three small patches after this mail. It adds a bugfix (etc/rc-once.d -> /etc/rc-once.d) and the readlink check. I moved the deletion of /system-update into run_rc_once function. Not sure if you like this, I followed recommendation 2.

Best regards
Gavin Schenk


Eckelmann AG
Vorstand: Dipl.-Ing. Peter Frankenbach (Sprecher) Dipl.-Wi.-Ing. Philipp Eckelmann
Dr.-Ing. Marco Münchhof Dr.-Ing. Frank Uhlemann
Vorsitzender des Aufsichtsrats: Hubertus G. Krossa
Stv. Vorsitzender des Aufsichtsrats: Dr.-Ing. Gerd Eckelmann
Sitz der Gesellschaft: Berliner Str. 161, 65205 Wiesbaden, Amtsgericht Wiesbaden HRB 12636
http://www.eckelmann.de 

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] systemd: Using second offline update service beside rc-once in PTXDIST

[Michael Olbrich]

On Tue, Jun 06, 2017 at 03:28:47PM +0000, Schenk, Gavin wrote:
> Hi Michael,
> 
> > >
> > > Thank you for the fast feedback, I will send a patch.
> > >
> > > Some Questions:
> > > Using readlink is ok? (additional dependency rc-once -> readlink ) Is
> > > there anything more lightweight?
> > 
> > Not that I know. Note: if you use 'readlink -f' then the busybox version
> > has a option for this, so maybe:
> > 
> > 	select BUSYBOX_FEATURE_READLINK_FOLLOW	if BUSYBOX_READLINK
> > 	select COREUTILS			if !BUSYBOX_READLINK
> > 	select COREUTILS_READLINK		if !BUSYBOX_READLINK
> > 
> Ok, done.
> 
> > > Do we need this stuff configurable?
> > 
> > No. But I think we should have a sanity check. To avoid getting stuck in
> > the system-update.target: A service that is part of system-update.target
> > and starts the rescue target. rc-once.service and your service should
> > both be sorted 'Before' that service. If no other service starts a
> > different target we automatically fall into the rescue target.
> > 
> Is "FailureAction=reboot" or "FailureAction=rescue.target" in service
> file sufficient here?

Actually, I noticed that system-update-cleanup.service is supposed to
handle this case. I'm currently unsure if 'remove the link and reboot' is
ok, of if I want to drop into the rescue target.

> I am not happy with my implementation today :-(. The main reason is, it does not cover my usecase where I want to do something:
>    * Once at first boot
>    * With UI on tty
>    * Without conflicting rc-once
> 
> For this usecase it seems to be enough to add a service to
> system-update.target.wants with "After=rc-once.service". This is not
> flawless, because the system continues booting after rc-once is done e.g.
> getty.target. It seems to be related to "systemctl daemon-reexec" in
> systemd-rc-once I am not sure. The handling of bbinit beside systemd and
> managing write protection in this scripts scares me. I am afraid to break
> stuff when doing changes here.

This stuff is a bit tricky. There are several things to consider:
- The rootfs may or may not be mounted read-only. The remount stuff
  basically makes sure the rootfs is writable and returns to the previous
  state when we're done
- The rc-once may modify the currently running executables and libraries.
  'exec "$0" ...' and daemon-reexec try to make sure the new versions are
  used before the rootfs is mounted read-only. Otherwise this may fail.
- Deleting /system-update + daemon-reexec (or daemon-reload) results in a
  new 'default.target'. This is then activated by 'systemctl default'.
- If mounting read-only fails, we reboot and run rc-once again. All scripts
  are done at this point so that's just some remounting. This is needed to
  make sure the journal is flushed and the filesystem is clean.
- If any rc-once script fails, we want to drop into the rescue target. This
  is already used for other fatal errors during startup, so it's a central
  place to handle a broken rootfs.

> I reverted my last commit where I added some recommendations of freedesktop.org:
>    * Requires=sysinit.target
>    * After=sysinit.target
> 
> The result was a broken system. I reverted this to your implementation
>    * After=local-fs.target
>    * Requires=local-fs.target
>    * Before=system-update-cleanup.service
>    * After=system-update.target
> 
> I will send my three small patches after this mail. It adds a bugfix
> (etc/rc-once.d -> /etc/rc-once.d) and the readlink check. I moved the
> deletion of /system-update into run_rc_once function. Not sure if you like
> this, I followed recommendation 2.

I'm not sure what you want to do in your service. What should happen once
it's done?
We could split out the 'systemctl default' into a
'system-update-done.service'. Then you could sort your service between
rc-once an this new service. Would that work for you?

Do we need to check the link target of /system-update with your current
requirements? Is there a use-case where your service should be started, but
not rc-once? If not, then I'd say we leave this as is for now.

Michael

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de

Re: [ptxdist] systemd: Using second offline update service beside rc-once in PTXDIST

[Schenk, Gavin]

Hi,

> > > > Do we need this stuff configurable?
> > >
> > > No. But I think we should have a sanity check. To avoid getting
> > > stuck in the system-update.target: A service that is part of
> > > system-update.target and starts the rescue target. rc-once.service
> > > and your service should both be sorted 'Before' that service. If no
> > > other service starts a different target we automatically fall into
> the rescue target.
> > >
> > Is "FailureAction=reboot" or "FailureAction=rescue.target" in service
> > file sufficient here?
> 
> Actually, I noticed that system-update-cleanup.service is supposed to
> handle this case. I'm currently unsure if 'remove the link and reboot'
> is ok, of if I want to drop into the rescue target.
>
I am unsure too. Because it depends on the system. On one hand, who knows why the update fails and a reboot helps here? A reboot is simple, but does not solve every problem. On the other hand, if you have e.g. an embedded system in an industrial environment, you need a good rescue target that put the plant into a safe mode :). I trend to rescue target (gut instinct).

> > I am not happy with my implementation today :-(. The main reason is,
> it does not cover my usecase where I want to do something:
> >    * Once at first boot
> >    * With UI on tty
> >    * Without conflicting rc-once
> >
> > For this usecase it seems to be enough to add a service to
> > system-update.target.wants with "After=rc-once.service". This is not
> > flawless, because the system continues booting after rc-once is done
> e.g.
> > getty.target. It seems to be related to "systemctl daemon-reexec" in
> > systemd-rc-once I am not sure. The handling of bbinit beside systemd
> > and managing write protection in this scripts scares me. I am afraid
> > to break stuff when doing changes here.
> 
> This stuff is a bit tricky. There are several things to consider:
> - The rootfs may or may not be mounted read-only. The remount stuff
>   basically makes sure the rootfs is writable and returns to the
> previous
>   state when we're done

I understand, and yesterday I thought that this (rw/ro) should be the job of systemd update mode. But now I have a better understanding that you have to consider bbinit, too.

> - The rc-once may modify the currently running executables and
> libraries.
>   'exec "$0" ...' and daemon-reexec try to make sure the new versions
> are
>   used before the rootfs is mounted read-only. Otherwise this may fail.

Puhh, this is really tricky.

> - Deleting /system-update + daemon-reexec (or daemon-reload) results in
> a
>   new 'default.target'. This is then activated by 'systemctl default'.
> - If mounting read-only fails, we reboot and run rc-once again. All
> scripts
>   are done at this point so that's just some remounting. This is needed
> to
>   make sure the journal is flushed and the filesystem is clean.
> - If any rc-once script fails, we want to drop into the rescue target.
> This
>   is already used for other fatal errors during startup, so it's a
> central
>   place to handle a broken rootfs.
> 
Good explanation, I agree.

> I'm not sure what you want to do in your service. What should happen
> once it's done?
> We could split out the 'systemctl default' into a 'system-update-
> done.service'. Then you could sort your service between rc-once an this
> new service. Would that work for you?
> 
Little background of what I am doing at the moment:
I use this stuff for a migration tool to transform older devices into new RAUC based A+B system without losing user data. In my case the system boots from a USB stick and I have a package 'system-migration-part1' that is part of the userland of the USB stick (collections are cool!). This packages saves all user data from the internal memory onto the USB-stick. Depending on the amount of data the process takes up to 8 minutes. While this is running I want to show the user a kind of UI, without kernel messages. Currently I am using pipeviewer. In combination with dialog it looks ok for me. Next step is to reorganize all internal storages (partition and filesystems) and put a cool RAUC based image (recovery) into SLOT A installing another bootloader and do a reboot into recovery starting migration-part-2 where the user data is restored and the final firmware gets installed into SLOT B

I will try your idea with system-update-done.service.

> Do we need to check the link target of /system-update with your current
> requirements? Is there a use-case where your service should be started,
> but not rc-once? If not, then I'd say we leave this as is for now.
> 
No, I currently do not need it for my requirements. The only thing left is the Bugfix (/etc/rc.once.d -> etc/rc-once.d). And, because nobody really checks the link target, we do not need this either. I suggest to leave things as is, until a real requirement spawns?

Thank you both for detailed explanations and your help.

Regards Gavin

Eckelmann AG
Vorstand: Dipl.-Ing. Peter Frankenbach (Sprecher) Dipl.-Wi.-Ing. Philipp Eckelmann
Dr.-Ing. Marco Münchhof Dr.-Ing. Frank Uhlemann
Vorsitzender des Aufsichtsrats: Hubertus G. Krossa
Stv. Vorsitzender des Aufsichtsrats: Dr.-Ing. Gerd Eckelmann
Sitz der Gesellschaft: Berliner Str. 161, 65205 Wiesbaden, Amtsgericht Wiesbaden HRB 12636
http://www.eckelmann.de 

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de