mailarchive of the ptxdist mailing list
 help / color / mirror / Atom feed
* [ptxdist] [PATCH 1/2] doc: dev manual: propagate two subsections to sections
@ 2021-06-16 15:56 Roland Hieber
  2021-06-16 15:56 ` [ptxdist] [PATCH 2/2] doc: dev manual: how to recover from patch merge conflicts with git Roland Hieber
  2021-06-29  5:09 ` [ptxdist] [APPLIED] doc: dev manual: propagate two subsections to sections Michael Olbrich
  0 siblings, 2 replies; 4+ messages in thread
From: Roland Hieber @ 2021-06-16 15:56 UTC (permalink / raw)
  To: ptxdist; +Cc: Roland Hieber

The "Adding new packages" section has become quite long already, to the
point that some headings have reached level five and no longer make it
into the table of contents. The subsections "Advanced rule files" and
"Patching packages" are also not only relevant for new packages, so
propagate them to their own sections.

Signed-off-by: Roland Hieber <rhi@pengutronix.de>
---
 doc/dev_add_new_pkgs.rst        | 689 --------------------------------
 doc/dev_advanced_rule_files.rst | 463 +++++++++++++++++++++
 doc/dev_manual.rst              |   2 +
 doc/dev_patching.rst            | 226 +++++++++++
 4 files changed, 691 insertions(+), 689 deletions(-)
 create mode 100644 doc/dev_advanced_rule_files.rst
 create mode 100644 doc/dev_patching.rst

diff --git a/doc/dev_add_new_pkgs.rst b/doc/dev_add_new_pkgs.rst
index 0dbc7a266ea8..3506436d78ec 100644
--- a/doc/dev_add_new_pkgs.rst
+++ b/doc/dev_add_new_pkgs.rst
@@ -648,692 +648,3 @@ repository and its content. It contains a list of known licenses based on their
 SPDX identifier. The content is without formatting to simplify text search.
 
    https://github.com/spdx/license-list-data/tree/master/text
-
-Advanced Rule Files
-~~~~~~~~~~~~~~~~~~~
-
-The previous example on how to create a rule file sometimes works as
-shown above. But most of the time source archives are not that simple.
-In this section we want to give the user a more detailed selection how
-the package will be built.
-
-Adding Static Configure Parameters
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The ``configure`` scripts of various source archives provide additional
-parameters to enable or disable features, or to configure them in a
-specific way.
-
-We assume the ``configure`` script of our ``foo`` example (refer to
-section :ref:`foo_example`) supports two additional parameters:
-
--  **--enable-debug**: Make the program more noisy. It’s disabled by
-   default.
-
--  **--with-bar**: Also build the special executable **bar**. Building
-   this executable is also disabled by default.
-
-We now want to forward these options to the ``configure`` script when it
-runs in the *prepare* stage. To do so, we must again open the rule file
-with our favourite editor and navigate to the *prepare* stage entry.
-
-PTXdist uses the variable ``FOO_CONF_OPT`` as the list of parameters to
-be given to ``configure``.
-
-Currently this variable is commented out and defined to:
-
-.. code-block:: make
-
-    # FOO_CONF_OPT := $(CROSS_AUTOCONF_USR)
-
-The variable ``CROSS_AUTOCONF_USR`` is predefined by PTXdist and
-contains all basic parameters to instruct ``configure`` to prepare for a
-**cross** compile environment.
-
-To use the two additional mentioned ``configure`` parameters, we comment
-in this line and supplement this expression as follows:
-
-.. code-block:: make
-
-    FOO_CONF_OPT := \
-        $(CROSS_AUTOCONF_USR) \
-        --enable-debug \
-        --with-bar
-
-.. note:: We recommend to use this format with each parameter on a line of
- its own. This format is easier to read and a diff shows more exactly any
- change.
-
-To do a fast check if this addition was successful, we run:
-
-.. code-block:: text
-
-    $ ptxdist print FOO_CONF_OPT
-    --prefix=/usr --sysconfdir=/etc --host=|ptxdistCompilerName| --build=i686-host-linux-gnu --enable-debug --with-bar
-
-.. note:: It depends on the currently selected platform and its architecture
- what content this variable will have. The content shown above is an
- example for a target.
-
-Or re-build the package with the new settings:
-
-.. code-block:: text
-
-    $ ptxdist drop foo prepare
-    $ ptxdist targetinstall foo
-
-Adding Dynamic Configure Parameters
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Sometimes it makes sense to add this kind of parameters on demand only;
-especially a parameter like ``--enable-debug``. To let the user decide
-if this parameter is to be used or not, we must add a menu entry. So,
-let’s expand our menu. Here is its current content:
-
-.. code-block:: kconfig
-
-    ## SECTION=project_specific
-
-    config FOO
-            tristate
-            prompt "foo"
-            help
-              FIXME
-
-We’ll add two menu entries, one for each optional parameter we want to
-add on demand to the ``configure`` parameters:
-
-.. code-block:: kconfig
-
-    ## SECTION=project_specific
-
-    config FOO
-           tristate
-           prompt "foo"
-           help
-             FIXME
-
-    if FOO
-    config FOO_DEBUG
-           bool
-           prompt "add debug noise"
-
-    config FOO_BAR
-           bool
-           prompt "build bar"
-
-    endif
-
-.. important:: Always follow the rule to extend the base name by a suboption
-  name as the trailing part of the variable name. This gives PTXdist the ability
-  to detect a change in the package’s settings (via menuconfig) to force its
-  rebuild on demand.
-
-To make usage of the new menu entries, we must check them in the rule
-file and add the correct parameters:
-
-.. code-block:: make
-
-    #
-    # autoconf
-    #
-    FOO_CONF_OPT := \
-        $(CROSS_AUTOCONF_USR) \
-        --$(call ptx/endis, PTXCONF_FOO_DEBUG)-debug \
-        --$(call ptx/wwo, PTXCONF_FOO_BAR)-bar
-
-.. important:: Please note the leading ``PTXCONF_`` for each define. While Kconfig is
-  using ``FOO_BAR``, the rule file must use ``PTXCONF_FOO_BAR`` instead.
-
-.. note:: Refer :ref:`Rule File Macro Reference <param_macros>` for further
-   details about these special kind of option macros (e.g. ``ptx/...``).
-
-It is a good practice to always add both settings, e.g. ``--disable-debug``
-even if this is the default case. Sometimes ``configure`` tries to guess
-something and the binary result might differ depending on the build
-order. For example some kind of package would also build some X related
-tools, if X libraries are found. In this case it depends on the build
-order, if the X related tools are built or not. All the autocheck
-features are problematic here. So, if we do not want ``configure`` to
-guess its settings we **must disable everything we do not want**.
-
-To support this process, PTXdist supplies a helper script, located at
-``/path/to/ptxdist/scripts/configure-helper.py`` that compares the configure
-output with the settings from ``FOO_CONF_OPT``:
-
-.. code-block:: text
-
-    $ /opt/ptxdist-2017.06.0/scripts/configure-helper.py -p libsigrok
-    --- rules/libsigrok.make
-    +++ libsigrok-0.5.0
-    @@ -4,3 +4,74 @@
-     	--libdir=/usr/lib
-     	--build=x86_64-host-linux-gnu
-     	--host=arm-v7a-linux-gnueabihf
-    +	--enable-warnings=min|max|fatal|no
-    +	--disable-largefile
-    +	--enable-all-drivers
-    +	--enable-agilent-dmm
-    [...]
-    +	--enable-ruby
-    +	--enable-java
-    +	--without-libserialport
-    +	--without-libftdi
-    +	--without-libusb
-    +	--without-librevisa
-    +	--without-libgpib
-    +	--without-libieee1284
-    +	--with-jni-include-path=DIR-LIST
-
-In this example, many configure options from libsigrok (marked with ``+``)
-are not yet present in ``LIBSIGROK_CONF_OPT`` and must be added, possibly also
-by providing more dynamic options in the package definition.
-
-If some parts of a package are built on demand only, they must also be
-installed on demand only. Besides the *prepare* stage, we also must
-modify our *targetinstall* stage:
-
-.. code-block:: make
-
-    	@$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/foo, /usr/bin/foo)
-
-    ifdef PTXCONF_FOO_BAR
-    	@$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/bar, /usr/bin/bar)
-    endif
-
-    	@$(call install_finish, foo)
-    	@$(call touch)
-
-Now we can play with our new menu entries and check if they are working
-as expected:
-
-.. code-block:: text
-
-    $ ptxdist menuconfig
-    $ ptxdist targetinstall foo
-
-Whenever we change a *FOO* related menu entry, PTXdist should detect it
-and re-build the package when a new build is started.
-
-.. _external_dependencies:
-
-Managing External Compile Time Dependencies
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-While running the prepare stage, it could happen that it fails due to a
-missing external dependency.
-
-For example:
-
-.. code-block:: text
-
-    checking whether zlib exists....failed
-
-In this example, our new package depends on the compression library
-*zlib*. PTXdist comes with a target *zlib*. All we need to do in this
-case is to declare that our new package *foo* depends on *zlib*. This
-kind of dependency is managed in the menu file of our new package by
-simply adding the ``select ZLIB`` line. After this addition our menu
-file looks like:
-
-.. code-block:: kconfig
-
-    ## SECTION=project_specific
-
-    config FOO
-           tristate
-           select ZLIB
-           prompt "foo"
-           help
-             FIXME
-
-    if FOO
-    config FOO_DEBUG
-           bool
-           prompt "add debug noise"
-
-    config FOO_BAR
-           bool
-           prompt "build bar"
-
-    endif
-
-PTXdist now builds the *zlib* first and our new package thereafter.
-
-Refer :ref:`external_dependencies_variants` for more specific dependency
-description.
-
-Managing External Compile Time Dependencies on Demand
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-It is good practice to add only those dependencies that are really
-required for the current configuration of the package. If the package
-provides the features *foo* and *bar* and its ``configure`` provides
-switches to enable/disable them independently, we can also add
-dependencies on demand. Let’s assume feature *foo* needs the compression
-library *libz* and *bar* needs the XML2 library *libxml2*. These
-libraries are only required at run-time if the corresponding feature is
-enabled. To add these dependencies on demand, the menu file looks like:
-
-.. code-block:: kconfig
-
-    ## SECTION=project_specific
-
-    config FOO
-           tristate
-           select ZLIB if FOO_FOO
-           select LIBXML2 if FOO_BAR
-           prompt "foo"
-           help
-             FIXME
-
-    if FOO
-    config FOO_DEBUG
-           bool
-           prompt "add debug noise"
-
-    config FOO_FOO
-           bool
-           prompt "build foo"
-
-    config FOO_BAR
-           bool
-           prompt "build bar"
-
-    endif
-
-.. important:: Do not add these ``select`` statements to the corresponding menu entry.
-  They must belong to the main menu entry of the package to ensure that
-  the calculation of the dependencies between the packages is done in a
-  correct manner.
-
-Managing External Runtime Dependencies
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Some packages are building all of their components and also installing
-them into the target’s sysroot. But only their *targetinstall* stage
-decides which parts are copied to the root filesystem. So, compiling and
-linking of our package will work, because everything required is found
-in the target’s sysroot.
-
-In our example there is a hidden dependency to the math library
-``libm``. Our new package was built successfully, because the linker was
-able to link our binaries against the ``libm`` from the toolchain. But
-in this case the ``libm`` must also be available in the target’s root
-filesystem to fulfill the run-time dependency: We have to force PTXdist to
-install ``libm``. ``libm`` is part of the *glibc* package, but is not
-installed by default (to keep the root filesystem small). So, it **does
-not** help to select the ``GLIBC`` symbol, to get a ``libm`` at run-time.
-
-The correct solution here is to add a ``select LIBC_M`` to our menu
-file. With all the additions above it now looks like:
-
-.. code-block:: kconfig
-
-    ## SECTION=project_specific
-
-    config FOO
-           tristate
-           select ZLIB if FOO_FOO
-           select LIBXML2 if FOO_BAR
-           select LIBC_M
-           prompt "foo"
-           help
-             FIXME
-
-    if FOO
-    config FOO_DEBUG
-           bool
-           prompt "add debug noise"
-
-    config FOO_FOO
-           bool
-           prompt "build foo"
-
-    config FOO_BAR
-           bool
-           prompt "build bar"
-
-    endif
-
-.. note:: There are other packages around, that do not install everything by
-  default. If our new package needs something special, we must take a look
-  into the menu of the other package how to force the required components
-  to be installed and add the corresponding ``selects`` to our own menu
-  file. In this case it does not help to enable the required parts in our
-  project configuration, because this has no effect on the build order!
-
-Managing Plain Makefile Packages
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Many packages are still coming with a plain ``Makefile``. The user has
-to adapt it to make it work in a cross compile environment as well.
-PTXdist can also handle this kind of packages. We only have to specify
-a special *prepare* and *compile* stage.
-
-Such packages often have no special need for any kind of preparation. In
-this we must instruct PTXdist to do nothing in the *prepare* stage:
-
-.. code-block:: make
-
-    FOO_CONF_TOOL := NO
-
-To compile the package, we can use ``make``\ ’s feature to overwrite
-variables used in the ``Makefile``. With this feature we can still use
-the original ``Makefile`` but with our own (cross compile) settings.
-
-Most of the time the generic compile rule can be used, only a few
-settings are required. For a well defined ``Makefile`` it is sufficient to
-set up the correct cross compile environment for the *compile* stage:
-
-.. code-block:: make
-
-    FOO_MAKE_ENV := $(CROSS_ENV)
-
-``make`` will be called in this case with:
-
-``$(FOO_MAKE_ENV) $(MAKE) -C $(FOO_DIR) $(FOO_MAKE_OPT)``
-
-So, in the rule file only the two variables ``FOO_MAKE_ENV`` and
-``FOO_MAKE_OPT`` must be set, to forward the required settings to the
-package’s buildsystem. If the package cannot be built in parallel, we
-can also add the ``FOO_MAKE_PAR := NO``. ``YES`` is the default.
-
-Managing CMake/QMake/Meson Packages
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Building packages that use ``cmake``, ``qmake`` or ``meson`` is much like
-building packages with an autotools based buildsystem. We need to specify
-the configuration tool:
-
-.. code-block:: make
-
-    FOO_CONF_TOOL := cmake
-
-or
-
-.. code-block:: make
-
-    FOO_CONF_TOOL := qmake
-
-or respectively
-
-.. code-block:: make
-
-    FOO_CONF_TOOL := meson
-
-And provide the correct configuration options. The syntax is different so
-PTXdist provides additional macros to simplify configurable features.
-For ``cmake`` the configuration options typically look like this:
-
-.. code-block:: make
-
-    FOO_CONF_OPT := \
-    	$(CROSS_CMAKE_USR) \
-    	-DBUILD_TESTS:BOOL=OFF \
-    	-DENABLE_BAR:BOOL=$(call ptx/onoff, PTXCONF_FOO_BAR)
-
-For ``qmake`` the configuration options typically look like this:
-
-.. code-block:: make
-
-    FOO_CONF_OPT := \
-    	$(CROSS_QMAKE_OPT) \
-    	PREFIX=/usr
-
-And for ``meson`` the configuration options typically look like this:
-
-.. code-block:: make
-
-    FOO_CONF_OPT := \
-    	$(CROSS_MESON_USR) \
-    	-Dbar=$(call ptx/truefalse,PTXCONF_FOO_BAR)
-
-Please note that currently only host and target ``cmake``\/``meson`` packages
-and only target ``qmake`` packages are supported.
-
-Managing Python Packages
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-As with any other package, the correct configuration tool must be selected
-for Python packages:
-
-.. code-block:: make
-
-    FOO_CONF_TOOL := python
-
-.. note:: For Python3 packages the value must be ``python3``.
-
-No Makefiles are used when building Python packages so the usual ``make``
-and ``make install`` for the *compile* and *install* stages cannot be used.
-PTXdist will call ``python setup.py build`` and ``python setup.py install``
-instead.
-
-.. note:: *FOO* is still the name of our example package. It must be
-  replaced by the real package name.
-
-
-.. _patching_packages:
-
-Patching Packages
-~~~~~~~~~~~~~~~~~
-
-There can be various reasons why a package must be patched:
-
--  Package is broken for cross compile environments
-
--  Package is broken within a specific feature
-
--  Package is vulnerable and needs some fixes
-
--  or anything else (this case is the most common one)
-
-Ideally, those problems should be addressed in the original project,
-so any patches you add to your BSP or to PTXdist should also be submitted upstream.
-The upstream project can often provide better feedback, they can integrate your
-patch into a new release, and also maintain your changes as part of the project.
-This way we make sure that all advantages of the open source idea work for us;
-and your patch can be removed again later when a new release of the project is
-integrated into your BSP or into PTXdist.
-
-PTXdist handles patching automatically.
-After extracting the archive of a package, PTXdist checks for the existence of
-a patch directory named like its ``<PKG>_PATCHES`` variable, or, if this variable
-is not set, like its ``<PKG>`` variable.
-The patch directory is then searched in all locations listed by the
-``PTXDIST_PATH_PATCHES`` variable, and the first one found is used.
-Take an exemplary package ``foo`` with version ``1.1.0``:
-The variable ``FOO`` will have the value ``foo-1.1.0``, so PTXdist will look for
-a patch directory named ``foo-1.1.0`` in the following locations:
-
-#. the current layer:
-
-   a. project (``./patches/foo-1.1.0``)
-   b. platform (``./configs/|ptxdistPlatformConfigDir|/patches/foo-1.1.0``)
-
-#. any :ref:`base layers <layers-in-ptxdist>`,
-   applying the same search order as above for each layer recursively
-
-#. ptxdist (``<ptxdist/installation/path>/patches/foo-1.1.0``)
-
-The patches from the first location found are used. Note: Due to this
-search order, a PTXdist project can replace global patches from the
-PTXdist installation. This can be useful if a project sticks to a
-specific PTXdist revision but fixes from a more recent revision of
-PTXdist should be used.
-
-PTXdist uses the utilities *git*, *patch* or *quilt* to work with
-patches or patch series. We recommend *git*, as it can manage patch
-series in a very easy way.
-
-Creating a Patch Series for a Package
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To create a patch series for the first time, we can run the following
-steps. We are still using our *foo-1.1.0* example package here:
-
-Using Quilt
-"""""""""""
-
-We create a special directory for the patch series in the local project
-directory:
-
-.. code-block:: text
-
-    $ mkdir -p patches/foo-1.1.0
-
-PTXdist expects a ``series`` file in the patch directory and at least
-one patch. Otherwise it fails. Due to the fact that we do not have any
-patch content yet, we’ll start with a dummy entry in the ``series`` file
-and an empty ``patch`` file.
-
-.. code-block:: text
-
-    $ touch patches/foo-1.1.0/dummy
-    $ echo dummy > patches/foo-1.1.0/series
-
-Next is to extract the package (if already done, we must remove it
-first):
-
-.. code-block:: text
-
-    $ ptxdist extract foo
-
-This will extract the archive and create a symbolic link in the build
-directory pointing to our local patch directory. Working this way will
-ensure that we do not lose our created patches if we enter
-``ptxdist clean foo`` by accident. In our case the patches are still
-present in ``patches/foo-1.1.0`` and can be used the next time we
-extract the package again.
-
-All we have to do now is to do the modification we need to make the
-package work. We change into the build directory and use quilt_ to
-create new patches, add files to respective patches, modify these files
-and refresh the patches to save our changes.
-See the *quilt* documentation (``man 1 quilt``) for more information.
-
-.. note:: For patches that are intended for PTXdist upstream use the git
-  workflow described below to get proper patch headers.
-
-.. _quilt: http://savannah.nongnu.org/projects/quilt
-
-Using Git
-"""""""""
-
-Create the patch directory like above for *quilt*,
-but only add an empty series file:
-
-.. code-block:: text
-
-    $ mkdir -p patches/foo-1.1.0
-    $ touch patches/foo-1.1.0/series
-
-Then extract the package with an additional command line switch:
-
-.. code-block:: text
-
-    $ ptxdist --git extract foo
-
-The empty series file makes PTXdist create a Git repository in the
-respective package build directory,
-and import the package source as the first commit.
-
-.. note:: Optionally, you can enable the setting *Developer Options →
-  use git to apply patches* in `ptxdist setup` to get this behaviour
-  as a default for every package.
-  However, note that this setting is meant for development only, and can lead
-  to failures – some packages try to determine if they are being compiled from
-  a Git source tree, and behave differently in that case.
-
-Then you can change into the package build directory
-(``platform-<name>/build-target/foo-1.1.0``),
-patch the required source files,
-and make Git commits on the way.
-The Git history should now look something like this:
-
-.. code-block:: text
-
-    $ git log --oneline --decorate
-    * df343e821851 (HEAD -> master) Makefile: don't build the tests
-    * 65a360c2bd60 strfry.c: frobnicate the excusator
-    * fdc315f6844c (tag: foobar-1.1.0, tag: base) initial commit
-
-Finally, call ``git ptx-patches`` to transform those Git commits into the patch
-series in the ``patches/foo-1.1.0`` folder.
-This way they don't get lost when cleaning the package.
-
-.. note:: PTXdist will only create a Git repository for packages with
-  patches.  To use Git to generate the first patch, create an empty series
-  file ``patches/foobar-1.1.0/series`` before extracting the packages. This
-  will tell PTXdist to use Git anyways and ``git ptx-patches`` will put the
-  patches there.
-
-Both approaches (Git and quilt) are not suitable for modifying files
-that are autogenerated in autotools-based buildsystems.
-Refer to the section :ref:`configure_rebuild` on how PTXdist can
-handle this special task.
-
-Adding More Patches to a Package
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-If we want to add more patches to an already patched package, we can use
-nearly the same way as creating patches for the first time. But if the
-patch series comes from the PTXdist main installation, we do not have
-write permissions to these directories (do NEVER work on the main
-installation directories, NEVER, NEVER, NEVER). Due to the search order
-in which PTXdist searches for patches for a specific package, we can
-copy the global patch series to our local project directory. Now we have
-the permissions to add more patches or modify the existing ones. Also
-*quilt* and *git* are our friends here to manage the patch series.
-
-If we think that our new patches are valuable also for others, or they
-fix an error, it could be a good idea to send these patches to PTXdist
-mainline, and to the upstream project too.
-
-
-.. _configure_rebuild:
-
-Modifying Autotoolized Packages
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Autotoolized packages are very picky when automatically generated files
-get patched. The patch order is very important in this case and
-sometimes it even fails and nobody knows why.
-
-To improve a package’s autotools-based build system, PTXdist comes with
-its own project local autotools to regenerate the autotools template
-files, instead of patching them. With this feature, only the template
-files must be patched, the required ``configure`` script and the
-``Makefile.in`` files are regenerated in the final stages of the
-*prepare* step.
-
-This feature works like the regular patching mechanism. The only
-difference is the additional ``autogen.sh`` file in the patch directory.
-If it exists and has execution permissions, it will be called after the
-package was patched (while the *extract* stage is running).
-
-Its content depends on developer needs; for the most simple case the
-content can be:
-
-.. code-block:: bash
-
-    #!/bin/bash
-
-    aclocal $ACLOCAL_FLAGS
-
-    libtoolize \
-            --force \
-            --copy
-
-    autoreconf \
-            --force \
-            --install \
-            --warnings=cross \
-            --warnings=syntax \
-            --warnings=obsolete \
-            --warnings=unsupported
-
-.. note:: In this way not yet autotoolized package can be autotoolized. We
-  just have to add the common autotool template files (``configure.ac``
-  and ``Makefile.am`` for example) via a patch series to the package
-  source and the ``autogen.sh`` to the patch directory.
diff --git a/doc/dev_advanced_rule_files.rst b/doc/dev_advanced_rule_files.rst
new file mode 100644
index 000000000000..794bd148cbae
--- /dev/null
+++ b/doc/dev_advanced_rule_files.rst
@@ -0,0 +1,463 @@
+*******************
+Advanced Rule Files
+*******************
+
+The previous example on how to create a rule file sometimes works as
+shown above. But most of the time source archives are not that simple.
+In this section we want to give the user a more detailed selection how
+the package will be built.
+
+Adding Static Configure Parameters
+==================================
+
+The ``configure`` scripts of various source archives provide additional
+parameters to enable or disable features, or to configure them in a
+specific way.
+
+We assume the ``configure`` script of our ``foo`` example (refer to
+section :ref:`foo_example`) supports two additional parameters:
+
+-  **--enable-debug**: Make the program more noisy. It’s disabled by
+   default.
+
+-  **--with-bar**: Also build the special executable **bar**. Building
+   this executable is also disabled by default.
+
+We now want to forward these options to the ``configure`` script when it
+runs in the *prepare* stage. To do so, we must again open the rule file
+with our favourite editor and navigate to the *prepare* stage entry.
+
+PTXdist uses the variable ``FOO_CONF_OPT`` as the list of parameters to
+be given to ``configure``.
+
+Currently this variable is commented out and defined to:
+
+.. code-block:: make
+
+    # FOO_CONF_OPT := $(CROSS_AUTOCONF_USR)
+
+The variable ``CROSS_AUTOCONF_USR`` is predefined by PTXdist and
+contains all basic parameters to instruct ``configure`` to prepare for a
+**cross** compile environment.
+
+To use the two additional mentioned ``configure`` parameters, we comment
+in this line and supplement this expression as follows:
+
+.. code-block:: make
+
+    FOO_CONF_OPT := \
+        $(CROSS_AUTOCONF_USR) \
+        --enable-debug \
+        --with-bar
+
+.. note:: We recommend to use this format with each parameter on a line of
+ its own. This format is easier to read and a diff shows more exactly any
+ change.
+
+To do a fast check if this addition was successful, we run:
+
+.. code-block:: text
+
+    $ ptxdist print FOO_CONF_OPT
+    --prefix=/usr --sysconfdir=/etc --host=|ptxdistCompilerName| --build=i686-host-linux-gnu --enable-debug --with-bar
+
+.. note:: It depends on the currently selected platform and its architecture
+ what content this variable will have. The content shown above is an
+ example for a target.
+
+Or re-build the package with the new settings:
+
+.. code-block:: text
+
+    $ ptxdist drop foo prepare
+    $ ptxdist targetinstall foo
+
+Adding Dynamic Configure Parameters
+===================================
+
+Sometimes it makes sense to add this kind of parameters on demand only;
+especially a parameter like ``--enable-debug``. To let the user decide
+if this parameter is to be used or not, we must add a menu entry. So,
+let’s expand our menu. Here is its current content:
+
+.. code-block:: kconfig
+
+    ## SECTION=project_specific
+
+    config FOO
+            tristate
+            prompt "foo"
+            help
+              FIXME
+
+We’ll add two menu entries, one for each optional parameter we want to
+add on demand to the ``configure`` parameters:
+
+.. code-block:: kconfig
+
+    ## SECTION=project_specific
+
+    config FOO
+           tristate
+           prompt "foo"
+           help
+             FIXME
+
+    if FOO
+    config FOO_DEBUG
+           bool
+           prompt "add debug noise"
+
+    config FOO_BAR
+           bool
+           prompt "build bar"
+
+    endif
+
+.. important:: Always follow the rule to extend the base name by a suboption
+  name as the trailing part of the variable name. This gives PTXdist the ability
+  to detect a change in the package’s settings (via menuconfig) to force its
+  rebuild on demand.
+
+To make usage of the new menu entries, we must check them in the rule
+file and add the correct parameters:
+
+.. code-block:: make
+
+    #
+    # autoconf
+    #
+    FOO_CONF_OPT := \
+        $(CROSS_AUTOCONF_USR) \
+        --$(call ptx/endis, PTXCONF_FOO_DEBUG)-debug \
+        --$(call ptx/wwo, PTXCONF_FOO_BAR)-bar
+
+.. important:: Please note the leading ``PTXCONF_`` for each define. While Kconfig is
+  using ``FOO_BAR``, the rule file must use ``PTXCONF_FOO_BAR`` instead.
+
+.. note:: Refer :ref:`Rule File Macro Reference <param_macros>` for further
+   details about these special kind of option macros (e.g. ``ptx/...``).
+
+It is a good practice to always add both settings, e.g. ``--disable-debug``
+even if this is the default case. Sometimes ``configure`` tries to guess
+something and the binary result might differ depending on the build
+order. For example some kind of package would also build some X related
+tools, if X libraries are found. In this case it depends on the build
+order, if the X related tools are built or not. All the autocheck
+features are problematic here. So, if we do not want ``configure`` to
+guess its settings we **must disable everything we do not want**.
+
+To support this process, PTXdist supplies a helper script, located at
+``/path/to/ptxdist/scripts/configure-helper.py`` that compares the configure
+output with the settings from ``FOO_CONF_OPT``:
+
+.. code-block:: text
+
+    $ /opt/ptxdist-2017.06.0/scripts/configure-helper.py -p libsigrok
+    --- rules/libsigrok.make
+    +++ libsigrok-0.5.0
+    @@ -4,3 +4,74 @@
+     	--libdir=/usr/lib
+     	--build=x86_64-host-linux-gnu
+     	--host=arm-v7a-linux-gnueabihf
+    +	--enable-warnings=min|max|fatal|no
+    +	--disable-largefile
+    +	--enable-all-drivers
+    +	--enable-agilent-dmm
+    [...]
+    +	--enable-ruby
+    +	--enable-java
+    +	--without-libserialport
+    +	--without-libftdi
+    +	--without-libusb
+    +	--without-librevisa
+    +	--without-libgpib
+    +	--without-libieee1284
+    +	--with-jni-include-path=DIR-LIST
+
+In this example, many configure options from libsigrok (marked with ``+``)
+are not yet present in ``LIBSIGROK_CONF_OPT`` and must be added, possibly also
+by providing more dynamic options in the package definition.
+
+If some parts of a package are built on demand only, they must also be
+installed on demand only. Besides the *prepare* stage, we also must
+modify our *targetinstall* stage:
+
+.. code-block:: make
+
+    	@$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/foo, /usr/bin/foo)
+
+    ifdef PTXCONF_FOO_BAR
+    	@$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/bar, /usr/bin/bar)
+    endif
+
+    	@$(call install_finish, foo)
+    	@$(call touch)
+
+Now we can play with our new menu entries and check if they are working
+as expected:
+
+.. code-block:: text
+
+    $ ptxdist menuconfig
+    $ ptxdist targetinstall foo
+
+Whenever we change a *FOO* related menu entry, PTXdist should detect it
+and re-build the package when a new build is started.
+
+.. _external_dependencies:
+
+Managing External Compile Time Dependencies
+===========================================
+
+While running the prepare stage, it could happen that it fails due to a
+missing external dependency.
+
+For example:
+
+.. code-block:: text
+
+    checking whether zlib exists....failed
+
+In this example, our new package depends on the compression library
+*zlib*. PTXdist comes with a target *zlib*. All we need to do in this
+case is to declare that our new package *foo* depends on *zlib*. This
+kind of dependency is managed in the menu file of our new package by
+simply adding the ``select ZLIB`` line. After this addition our menu
+file looks like:
+
+.. code-block:: kconfig
+
+    ## SECTION=project_specific
+
+    config FOO
+           tristate
+           select ZLIB
+           prompt "foo"
+           help
+             FIXME
+
+    if FOO
+    config FOO_DEBUG
+           bool
+           prompt "add debug noise"
+
+    config FOO_BAR
+           bool
+           prompt "build bar"
+
+    endif
+
+PTXdist now builds the *zlib* first and our new package thereafter.
+
+Refer :ref:`external_dependencies_variants` for more specific dependency
+description.
+
+Managing External Compile Time Dependencies on Demand
+=====================================================
+
+It is good practice to add only those dependencies that are really
+required for the current configuration of the package. If the package
+provides the features *foo* and *bar* and its ``configure`` provides
+switches to enable/disable them independently, we can also add
+dependencies on demand. Let’s assume feature *foo* needs the compression
+library *libz* and *bar* needs the XML2 library *libxml2*. These
+libraries are only required at run-time if the corresponding feature is
+enabled. To add these dependencies on demand, the menu file looks like:
+
+.. code-block:: kconfig
+
+    ## SECTION=project_specific
+
+    config FOO
+           tristate
+           select ZLIB if FOO_FOO
+           select LIBXML2 if FOO_BAR
+           prompt "foo"
+           help
+             FIXME
+
+    if FOO
+    config FOO_DEBUG
+           bool
+           prompt "add debug noise"
+
+    config FOO_FOO
+           bool
+           prompt "build foo"
+
+    config FOO_BAR
+           bool
+           prompt "build bar"
+
+    endif
+
+.. important:: Do not add these ``select`` statements to the corresponding menu entry.
+  They must belong to the main menu entry of the package to ensure that
+  the calculation of the dependencies between the packages is done in a
+  correct manner.
+
+Managing External Runtime Dependencies
+======================================
+
+Some packages are building all of their components and also installing
+them into the target’s sysroot. But only their *targetinstall* stage
+decides which parts are copied to the root filesystem. So, compiling and
+linking of our package will work, because everything required is found
+in the target’s sysroot.
+
+In our example there is a hidden dependency to the math library
+``libm``. Our new package was built successfully, because the linker was
+able to link our binaries against the ``libm`` from the toolchain. But
+in this case the ``libm`` must also be available in the target’s root
+filesystem to fulfill the run-time dependency: We have to force PTXdist to
+install ``libm``. ``libm`` is part of the *glibc* package, but is not
+installed by default (to keep the root filesystem small). So, it **does
+not** help to select the ``GLIBC`` symbol, to get a ``libm`` at run-time.
+
+The correct solution here is to add a ``select LIBC_M`` to our menu
+file. With all the additions above it now looks like:
+
+.. code-block:: kconfig
+
+    ## SECTION=project_specific
+
+    config FOO
+           tristate
+           select ZLIB if FOO_FOO
+           select LIBXML2 if FOO_BAR
+           select LIBC_M
+           prompt "foo"
+           help
+             FIXME
+
+    if FOO
+    config FOO_DEBUG
+           bool
+           prompt "add debug noise"
+
+    config FOO_FOO
+           bool
+           prompt "build foo"
+
+    config FOO_BAR
+           bool
+           prompt "build bar"
+
+    endif
+
+.. note:: There are other packages around, that do not install everything by
+  default. If our new package needs something special, we must take a look
+  into the menu of the other package how to force the required components
+  to be installed and add the corresponding ``selects`` to our own menu
+  file. In this case it does not help to enable the required parts in our
+  project configuration, because this has no effect on the build order!
+
+Managing Plain Makefile Packages
+================================
+
+Many packages are still coming with a plain ``Makefile``. The user has
+to adapt it to make it work in a cross compile environment as well.
+PTXdist can also handle this kind of packages. We only have to specify
+a special *prepare* and *compile* stage.
+
+Such packages often have no special need for any kind of preparation. In
+this we must instruct PTXdist to do nothing in the *prepare* stage:
+
+.. code-block:: make
+
+    FOO_CONF_TOOL := NO
+
+To compile the package, we can use ``make``\ ’s feature to overwrite
+variables used in the ``Makefile``. With this feature we can still use
+the original ``Makefile`` but with our own (cross compile) settings.
+
+Most of the time the generic compile rule can be used, only a few
+settings are required. For a well defined ``Makefile`` it is sufficient to
+set up the correct cross compile environment for the *compile* stage:
+
+.. code-block:: make
+
+    FOO_MAKE_ENV := $(CROSS_ENV)
+
+``make`` will be called in this case with:
+
+``$(FOO_MAKE_ENV) $(MAKE) -C $(FOO_DIR) $(FOO_MAKE_OPT)``
+
+So, in the rule file only the two variables ``FOO_MAKE_ENV`` and
+``FOO_MAKE_OPT`` must be set, to forward the required settings to the
+package’s buildsystem. If the package cannot be built in parallel, we
+can also add the ``FOO_MAKE_PAR := NO``. ``YES`` is the default.
+
+Managing CMake/QMake/Meson Packages
+===================================
+
+Building packages that use ``cmake``, ``qmake`` or ``meson`` is much like
+building packages with an autotools based buildsystem. We need to specify
+the configuration tool:
+
+.. code-block:: make
+
+    FOO_CONF_TOOL := cmake
+
+or
+
+.. code-block:: make
+
+    FOO_CONF_TOOL := qmake
+
+or respectively
+
+.. code-block:: make
+
+    FOO_CONF_TOOL := meson
+
+And provide the correct configuration options. The syntax is different so
+PTXdist provides additional macros to simplify configurable features.
+For ``cmake`` the configuration options typically look like this:
+
+.. code-block:: make
+
+    FOO_CONF_OPT := \
+    	$(CROSS_CMAKE_USR) \
+    	-DBUILD_TESTS:BOOL=OFF \
+    	-DENABLE_BAR:BOOL=$(call ptx/onoff, PTXCONF_FOO_BAR)
+
+For ``qmake`` the configuration options typically look like this:
+
+.. code-block:: make
+
+    FOO_CONF_OPT := \
+    	$(CROSS_QMAKE_OPT) \
+    	PREFIX=/usr
+
+And for ``meson`` the configuration options typically look like this:
+
+.. code-block:: make
+
+    FOO_CONF_OPT := \
+    	$(CROSS_MESON_USR) \
+    	-Dbar=$(call ptx/truefalse,PTXCONF_FOO_BAR)
+
+Please note that currently only host and target ``cmake``\/``meson`` packages
+and only target ``qmake`` packages are supported.
+
+Managing Python Packages
+========================
+
+As with any other package, the correct configuration tool must be selected
+for Python packages:
+
+.. code-block:: make
+
+    FOO_CONF_TOOL := python
+
+.. note:: For Python3 packages the value must be ``python3``.
+
+No Makefiles are used when building Python packages so the usual ``make``
+and ``make install`` for the *compile* and *install* stages cannot be used.
+PTXdist will call ``python setup.py build`` and ``python setup.py install``
+instead.
+
+.. note:: *FOO* is still the name of our example package. It must be
+  replaced by the real package name.
diff --git a/doc/dev_manual.rst b/doc/dev_manual.rst
index c232cc91428a..e9a88c1a97f5 100644
--- a/doc/dev_manual.rst
+++ b/doc/dev_manual.rst
@@ -11,6 +11,8 @@ This chapter shows all (or most) of the details of how PTXdist works.
 
    dev_dir_hierarchy
    dev_add_new_pkgs
+   dev_advanced_rule_files
+   dev_patching
    dev_add_bin_only_files
    dev_create_new_pkg_templates
    dev_layers_in_ptxdist
diff --git a/doc/dev_patching.rst b/doc/dev_patching.rst
new file mode 100644
index 000000000000..e3690dfe1b48
--- /dev/null
+++ b/doc/dev_patching.rst
@@ -0,0 +1,226 @@
+.. _patching_packages:
+
+*****************
+Patching Packages
+*****************
+
+There can be various reasons why a package must be patched:
+
+-  Package is broken for cross compile environments
+
+-  Package is broken within a specific feature
+
+-  Package is vulnerable and needs some fixes
+
+-  or anything else (this case is the most common one)
+
+Ideally, those problems should be addressed in the original project,
+so any patches you add to your BSP or to PTXdist should also be submitted upstream.
+The upstream project can often provide better feedback, they can integrate your
+patch into a new release, and also maintain your changes as part of the project.
+This way we make sure that all advantages of the open source idea work for us;
+and your patch can be removed again later when a new release of the project is
+integrated into your BSP or into PTXdist.
+
+PTXdist handles patching automatically.
+After extracting the archive of a package, PTXdist checks for the existence of
+a patch directory named like its ``<PKG>_PATCHES`` variable, or, if this variable
+is not set, like its ``<PKG>`` variable.
+The patch directory is then searched in all locations listed by the
+``PTXDIST_PATH_PATCHES`` variable, and the first one found is used.
+Take an exemplary package ``foo`` with version ``1.1.0``:
+The variable ``FOO`` will have the value ``foo-1.1.0``, so PTXdist will look for
+a patch directory named ``foo-1.1.0`` in the following locations:
+
+#. the current layer:
+
+   a. project (``./patches/foo-1.1.0``)
+   b. platform (``./configs/|ptxdistPlatformConfigDir|/patches/foo-1.1.0``)
+
+#. any :ref:`base layers <layers-in-ptxdist>`,
+   applying the same search order as above for each layer recursively
+
+#. ptxdist (``<ptxdist/installation/path>/patches/foo-1.1.0``)
+
+The patches from the first location found are used. Note: Due to this
+search order, a PTXdist project can replace global patches from the
+PTXdist installation. This can be useful if a project sticks to a
+specific PTXdist revision but fixes from a more recent revision of
+PTXdist should be used.
+
+PTXdist uses the utilities *git*, *patch* or *quilt* to work with
+patches or patch series. We recommend *git*, as it can manage patch
+series in a very easy way.
+
+Creating a Patch Series for a Package
+=====================================
+
+To create a patch series for the first time, we can run the following
+steps. We are still using our *foo-1.1.0* example package here:
+
+Using Quilt
+-----------
+
+We create a special directory for the patch series in the local project
+directory:
+
+.. code-block:: text
+
+    $ mkdir -p patches/foo-1.1.0
+
+PTXdist expects a ``series`` file in the patch directory and at least
+one patch. Otherwise it fails. Due to the fact that we do not have any
+patch content yet, we’ll start with a dummy entry in the ``series`` file
+and an empty ``patch`` file.
+
+.. code-block:: text
+
+    $ touch patches/foo-1.1.0/dummy
+    $ echo dummy > patches/foo-1.1.0/series
+
+Next is to extract the package (if already done, we must remove it
+first):
+
+.. code-block:: text
+
+    $ ptxdist extract foo
+
+This will extract the archive and create a symbolic link in the build
+directory pointing to our local patch directory. Working this way will
+ensure that we do not lose our created patches if we enter
+``ptxdist clean foo`` by accident. In our case the patches are still
+present in ``patches/foo-1.1.0`` and can be used the next time we
+extract the package again.
+
+All we have to do now is to do the modification we need to make the
+package work. We change into the build directory and use quilt_ to
+create new patches, add files to respective patches, modify these files
+and refresh the patches to save our changes.
+See the *quilt* documentation (``man 1 quilt``) for more information.
+
+.. note:: For patches that are intended for PTXdist upstream use the git
+  workflow described below to get proper patch headers.
+
+.. _quilt: http://savannah.nongnu.org/projects/quilt
+
+Using Git
+---------
+
+Create the patch directory like above for *quilt*,
+but only add an empty series file:
+
+.. code-block:: text
+
+    $ mkdir -p patches/foo-1.1.0
+    $ touch patches/foo-1.1.0/series
+
+Then extract the package with an additional command line switch:
+
+.. code-block:: text
+
+    $ ptxdist --git extract foo
+
+The empty series file makes PTXdist create a Git repository in the
+respective package build directory,
+and import the package source as the first commit.
+
+.. note:: Optionally, you can enable the setting *Developer Options →
+  use git to apply patches* in `ptxdist setup` to get this behaviour
+  as a default for every package.
+  However, note that this setting is meant for development only, and can lead
+  to failures – some packages try to determine if they are being compiled from
+  a Git source tree, and behave differently in that case.
+
+Then you can change into the package build directory
+(``platform-<name>/build-target/foo-1.1.0``),
+patch the required source files,
+and make Git commits on the way.
+The Git history should now look something like this:
+
+.. code-block:: text
+
+    $ git log --oneline --decorate
+    * df343e821851 (HEAD -> master) Makefile: don't build the tests
+    * 65a360c2bd60 strfry.c: frobnicate the excusator
+    * fdc315f6844c (tag: foobar-1.1.0, tag: base) initial commit
+
+Finally, call ``git ptx-patches`` to transform those Git commits into the patch
+series in the ``patches/foo-1.1.0`` folder.
+This way they don't get lost when cleaning the package.
+
+.. note:: PTXdist will only create a Git repository for packages with
+  patches.  To use Git to generate the first patch, create an empty series
+  file ``patches/foobar-1.1.0/series`` before extracting the packages. This
+  will tell PTXdist to use Git anyways and ``git ptx-patches`` will put the
+  patches there.
+
+Both approaches (Git and quilt) are not suitable for modifying files
+that are autogenerated in autotools-based buildsystems.
+Refer to the section :ref:`configure_rebuild` on how PTXdist can
+handle this special task.
+
+
+Adding More Patches to a Package
+================================
+
+If we want to add more patches to an already patched package, we can use
+nearly the same way as creating patches for the first time. But if the
+patch series comes from the PTXdist main installation, we do not have
+write permissions to these directories (do NEVER work on the main
+installation directories, NEVER, NEVER, NEVER). Due to the search order
+in which PTXdist searches for patches for a specific package, we can
+copy the global patch series to our local project directory. Now we have
+the permissions to add more patches or modify the existing ones. Also
+*quilt* and *git* are our friends here to manage the patch series.
+
+If we think that our new patches are valuable also for others, or they
+fix an error, it could be a good idea to send these patches to PTXdist
+mainline, and to the upstream project too.
+
+
+.. _configure_rebuild:
+
+Modifying Autotoolized Packages
+===============================
+
+Autotoolized packages are very picky when automatically generated files
+get patched. The patch order is very important in this case and
+sometimes it even fails and nobody knows why.
+
+To improve a package’s autotools-based build system, PTXdist comes with
+its own project local autotools to regenerate the autotools template
+files, instead of patching them. With this feature, only the template
+files must be patched, the required ``configure`` script and the
+``Makefile.in`` files are regenerated in the final stages of the
+*prepare* step.
+
+This feature works like the regular patching mechanism. The only
+difference is the additional ``autogen.sh`` file in the patch directory.
+If it exists and has execution permissions, it will be called after the
+package was patched (while the *extract* stage is running).
+
+Its content depends on developer needs; for the most simple case the
+content can be:
+
+.. code-block:: bash
+
+    #!/bin/bash
+
+    aclocal $ACLOCAL_FLAGS
+
+    libtoolize \
+            --force \
+            --copy
+
+    autoreconf \
+            --force \
+            --install \
+            --warnings=cross \
+            --warnings=syntax \
+            --warnings=obsolete \
+            --warnings=unsupported
+
+.. note:: In this way not yet autotoolized package can be autotoolized. We
+  just have to add the common autotool template files (``configure.ac``
+  and ``Makefile.am`` for example) via a patch series to the package
+  source and the ``autogen.sh`` to the patch directory.
-- 
2.29.2


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [ptxdist] [PATCH 2/2] doc: dev manual: how to recover from patch merge conflicts with git
  2021-06-16 15:56 [ptxdist] [PATCH 1/2] doc: dev manual: propagate two subsections to sections Roland Hieber
@ 2021-06-16 15:56 ` Roland Hieber
  2021-06-29  5:09   ` [ptxdist] [APPLIED] " Michael Olbrich
  2021-06-29  5:09 ` [ptxdist] [APPLIED] doc: dev manual: propagate two subsections to sections Michael Olbrich
  1 sibling, 1 reply; 4+ messages in thread
From: Roland Hieber @ 2021-06-16 15:56 UTC (permalink / raw)
  To: ptxdist; +Cc: Roland Hieber

Signed-off-by: Roland Hieber <rhi@pengutronix.de>
---
If someone could write a similar section for the quilt workflow, that
would be great, as I don't know too much about quilt :-)

 doc/dev_patching.rst | 72 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)

diff --git a/doc/dev_patching.rst b/doc/dev_patching.rst
index e3690dfe1b48..a17983aa826a 100644
--- a/doc/dev_patching.rst
+++ b/doc/dev_patching.rst
@@ -159,6 +159,78 @@ that are autogenerated in autotools-based buildsystems.
 Refer to the section :ref:`configure_rebuild` on how PTXdist can
 handle this special task.
 
+Recovering from merge conflicts
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When updating packages, it can happen that older patches no longer apply.
+In this case, the *extract* stage will throw errors like this::
+
+   -----------------------------
+   target: ima-evm-utils.extract
+   -----------------------------
+
+   extract: pkg_src=/ptx/src/ima-evm-utils-1.3.2.tar.gz
+   extract: pkg_extract_dir=DistroKit/platform-v7a/build-target/ima-evm-utils-1.3.2
+   patchin: git: initializing repository
+   patchin: git: done
+
+   pkg_patch_dir:     'ptxdist/patches/ima-evm-utils-1.3.2'
+   pkg_patch_series:  'ptxdist/patches/ima-evm-utils-1.3.2/series'
+
+   patchin: git: apply 'ptxdist/patches/ima-evm-utils-1.3.2/series'
+   tagging -> base
+   0001-INSTALL-remove-file-at-it-s-autogenerated-by-autotoo.patch
+   0002-Makefile.am-rename-INCLUDES-AM_CPPFLAGS.patch
+   error: patch failed: src/Makefile.am:1
+   error: src/Makefile.am: patch does not apply
+   make: *** […/ptxdist/rules/post/ptxd_make_world_extract.make:41: …/DistroKit/platform-v7a/state/ima-evm-utils.extract] Error 4
+
+In the example above, the first patch was applied cleanly, but the second one
+was not because src/Makefile.am contained different lines than expected.
+If this happens, you have to clean up the merge conflict,
+and apply the remaining patches manually.
+
+First, change into the package's build directory, and abort the current patch::
+
+   …/distrokit/master (master) $  cd platform-v7a/build-target/ima-evm-utils-1.3.2/
+
+   …/build-target/ima-evm-utils-1.3.2 (master|AM/REBASE) $  git log --oneline --graph
+   * 6687ab46087c (HEAD -> master) INSTALL: remove file, as it's autogenerated by autotools
+   * 01a52624237f (tag: ima-evm-utils-1.3.2, tag: base) initial commit
+
+   …/build-target/ima-evm-utils-1.3.2 (master|AM/REBASE) $  git am --abort
+
+   …/build-target/ima-evm-utils-1.3.2 (master) $
+
+(Notice how the Git integration in the shell prompt still shows ``AM/REBASE``
+before the cleanup, signaling the ongoing conflict resolution.)
+
+The remaining patches are still available in the ``./patches`` directory
+relative to your current location.
+Try to apply each of them in turn using ``git am``.
+If a patch fails to apply, Git will not change any files, but will still
+remember the patch's author and commit message, and prompt you to resolve
+the conflicts::
+
+   …/build-target/ima-evm-utils-1.3.2 (master) $  git am patches/0005-evmctl-add-fallback-definitions-for-XATTR_NAME_IMA.patch
+
+   Applying: evmctl: add fallback definitions for XATTR_NAME_IMA
+   Patch failed at 0005 evmctl: add fallback definitions for XATTR_NAME_IMA
+   hint: Use 'git am --show-current-patch=diff' to see the failed patch
+   When you have resolved this problem, run "git am --continue".
+   If you prefer to skip this patch, run "git am --skip" instead.
+   To restore the original branch and stop patching, run "git am --abort".
+
+* If you find that the patch is no longer necessary (e.g. because it was
+  already merged upstream in the new package version), skip it with
+  ``git am --skip``.
+* Otherwise, apply the same patch again manually via ``patch --merge -p1 <
+  patches/filename.patch``, and resolve any resulting conflicts using your
+  favourite editor.
+* Finally, ``git am --continue`` commits your conflict resolution with the
+  patch's original author and log message.
+
+After porting all patches, update the package's patch queue with ``git ptx-patches``.
 
 Adding More Patches to a Package
 ================================
-- 
2.29.2


_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [ptxdist] [APPLIED] doc: dev manual: propagate two subsections to sections
  2021-06-16 15:56 [ptxdist] [PATCH 1/2] doc: dev manual: propagate two subsections to sections Roland Hieber
  2021-06-16 15:56 ` [ptxdist] [PATCH 2/2] doc: dev manual: how to recover from patch merge conflicts with git Roland Hieber
@ 2021-06-29  5:09 ` Michael Olbrich
  1 sibling, 0 replies; 4+ messages in thread
From: Michael Olbrich @ 2021-06-29  5:09 UTC (permalink / raw)
  To: ptxdist; +Cc: Roland Hieber

Thanks, applied as 9a631d7435ef7d012bdccc4354d37c446873f4e4.

Michael

[sent from post-receive hook]

On Tue, 29 Jun 2021 07:09:16 +0200, Roland Hieber <rhi@pengutronix.de> wrote:
> The "Adding new packages" section has become quite long already, to the
> point that some headings have reached level five and no longer make it
> into the table of contents. The subsections "Advanced rule files" and
> "Patching packages" are also not only relevant for new packages, so
> propagate them to their own sections.
> 
> Signed-off-by: Roland Hieber <rhi@pengutronix.de>
> Message-Id: <20210616155631.12180-1-rhi@pengutronix.de>
> Signed-off-by: Michael Olbrich <m.olbrich@pengutronix.de>
> 
> diff --git a/doc/dev_add_new_pkgs.rst b/doc/dev_add_new_pkgs.rst
> index 0dbc7a266ea8..3506436d78ec 100644
> --- a/doc/dev_add_new_pkgs.rst
> +++ b/doc/dev_add_new_pkgs.rst
> @@ -648,692 +648,3 @@ repository and its content. It contains a list of known licenses based on their
>  SPDX identifier. The content is without formatting to simplify text search.
>  
>     https://github.com/spdx/license-list-data/tree/master/text
> -
> -Advanced Rule Files
> -~~~~~~~~~~~~~~~~~~~
> -
> -The previous example on how to create a rule file sometimes works as
> -shown above. But most of the time source archives are not that simple.
> -In this section we want to give the user a more detailed selection how
> -the package will be built.
> -
> -Adding Static Configure Parameters
> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -The ``configure`` scripts of various source archives provide additional
> -parameters to enable or disable features, or to configure them in a
> -specific way.
> -
> -We assume the ``configure`` script of our ``foo`` example (refer to
> -section :ref:`foo_example`) supports two additional parameters:
> -
> --  **--enable-debug**: Make the program more noisy. It’s disabled by
> -   default.
> -
> --  **--with-bar**: Also build the special executable **bar**. Building
> -   this executable is also disabled by default.
> -
> -We now want to forward these options to the ``configure`` script when it
> -runs in the *prepare* stage. To do so, we must again open the rule file
> -with our favourite editor and navigate to the *prepare* stage entry.
> -
> -PTXdist uses the variable ``FOO_CONF_OPT`` as the list of parameters to
> -be given to ``configure``.
> -
> -Currently this variable is commented out and defined to:
> -
> -.. code-block:: make
> -
> -    # FOO_CONF_OPT := $(CROSS_AUTOCONF_USR)
> -
> -The variable ``CROSS_AUTOCONF_USR`` is predefined by PTXdist and
> -contains all basic parameters to instruct ``configure`` to prepare for a
> -**cross** compile environment.
> -
> -To use the two additional mentioned ``configure`` parameters, we comment
> -in this line and supplement this expression as follows:
> -
> -.. code-block:: make
> -
> -    FOO_CONF_OPT := \
> -        $(CROSS_AUTOCONF_USR) \
> -        --enable-debug \
> -        --with-bar
> -
> -.. note:: We recommend to use this format with each parameter on a line of
> - its own. This format is easier to read and a diff shows more exactly any
> - change.
> -
> -To do a fast check if this addition was successful, we run:
> -
> -.. code-block:: text
> -
> -    $ ptxdist print FOO_CONF_OPT
> -    --prefix=/usr --sysconfdir=/etc --host=|ptxdistCompilerName| --build=i686-host-linux-gnu --enable-debug --with-bar
> -
> -.. note:: It depends on the currently selected platform and its architecture
> - what content this variable will have. The content shown above is an
> - example for a target.
> -
> -Or re-build the package with the new settings:
> -
> -.. code-block:: text
> -
> -    $ ptxdist drop foo prepare
> -    $ ptxdist targetinstall foo
> -
> -Adding Dynamic Configure Parameters
> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -Sometimes it makes sense to add this kind of parameters on demand only;
> -especially a parameter like ``--enable-debug``. To let the user decide
> -if this parameter is to be used or not, we must add a menu entry. So,
> -let’s expand our menu. Here is its current content:
> -
> -.. code-block:: kconfig
> -
> -    ## SECTION=project_specific
> -
> -    config FOO
> -            tristate
> -            prompt "foo"
> -            help
> -              FIXME
> -
> -We’ll add two menu entries, one for each optional parameter we want to
> -add on demand to the ``configure`` parameters:
> -
> -.. code-block:: kconfig
> -
> -    ## SECTION=project_specific
> -
> -    config FOO
> -           tristate
> -           prompt "foo"
> -           help
> -             FIXME
> -
> -    if FOO
> -    config FOO_DEBUG
> -           bool
> -           prompt "add debug noise"
> -
> -    config FOO_BAR
> -           bool
> -           prompt "build bar"
> -
> -    endif
> -
> -.. important:: Always follow the rule to extend the base name by a suboption
> -  name as the trailing part of the variable name. This gives PTXdist the ability
> -  to detect a change in the package’s settings (via menuconfig) to force its
> -  rebuild on demand.
> -
> -To make usage of the new menu entries, we must check them in the rule
> -file and add the correct parameters:
> -
> -.. code-block:: make
> -
> -    #
> -    # autoconf
> -    #
> -    FOO_CONF_OPT := \
> -        $(CROSS_AUTOCONF_USR) \
> -        --$(call ptx/endis, PTXCONF_FOO_DEBUG)-debug \
> -        --$(call ptx/wwo, PTXCONF_FOO_BAR)-bar
> -
> -.. important:: Please note the leading ``PTXCONF_`` for each define. While Kconfig is
> -  using ``FOO_BAR``, the rule file must use ``PTXCONF_FOO_BAR`` instead.
> -
> -.. note:: Refer :ref:`Rule File Macro Reference <param_macros>` for further
> -   details about these special kind of option macros (e.g. ``ptx/...``).
> -
> -It is a good practice to always add both settings, e.g. ``--disable-debug``
> -even if this is the default case. Sometimes ``configure`` tries to guess
> -something and the binary result might differ depending on the build
> -order. For example some kind of package would also build some X related
> -tools, if X libraries are found. In this case it depends on the build
> -order, if the X related tools are built or not. All the autocheck
> -features are problematic here. So, if we do not want ``configure`` to
> -guess its settings we **must disable everything we do not want**.
> -
> -To support this process, PTXdist supplies a helper script, located at
> -``/path/to/ptxdist/scripts/configure-helper.py`` that compares the configure
> -output with the settings from ``FOO_CONF_OPT``:
> -
> -.. code-block:: text
> -
> -    $ /opt/ptxdist-2017.06.0/scripts/configure-helper.py -p libsigrok
> -    --- rules/libsigrok.make
> -    +++ libsigrok-0.5.0
> -    @@ -4,3 +4,74 @@
> -     	--libdir=/usr/lib
> -     	--build=x86_64-host-linux-gnu
> -     	--host=arm-v7a-linux-gnueabihf
> -    +	--enable-warnings=min|max|fatal|no
> -    +	--disable-largefile
> -    +	--enable-all-drivers
> -    +	--enable-agilent-dmm
> -    [...]
> -    +	--enable-ruby
> -    +	--enable-java
> -    +	--without-libserialport
> -    +	--without-libftdi
> -    +	--without-libusb
> -    +	--without-librevisa
> -    +	--without-libgpib
> -    +	--without-libieee1284
> -    +	--with-jni-include-path=DIR-LIST
> -
> -In this example, many configure options from libsigrok (marked with ``+``)
> -are not yet present in ``LIBSIGROK_CONF_OPT`` and must be added, possibly also
> -by providing more dynamic options in the package definition.
> -
> -If some parts of a package are built on demand only, they must also be
> -installed on demand only. Besides the *prepare* stage, we also must
> -modify our *targetinstall* stage:
> -
> -.. code-block:: make
> -
> -    	@$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/foo, /usr/bin/foo)
> -
> -    ifdef PTXCONF_FOO_BAR
> -    	@$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/bar, /usr/bin/bar)
> -    endif
> -
> -    	@$(call install_finish, foo)
> -    	@$(call touch)
> -
> -Now we can play with our new menu entries and check if they are working
> -as expected:
> -
> -.. code-block:: text
> -
> -    $ ptxdist menuconfig
> -    $ ptxdist targetinstall foo
> -
> -Whenever we change a *FOO* related menu entry, PTXdist should detect it
> -and re-build the package when a new build is started.
> -
> -.. _external_dependencies:
> -
> -Managing External Compile Time Dependencies
> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -While running the prepare stage, it could happen that it fails due to a
> -missing external dependency.
> -
> -For example:
> -
> -.. code-block:: text
> -
> -    checking whether zlib exists....failed
> -
> -In this example, our new package depends on the compression library
> -*zlib*. PTXdist comes with a target *zlib*. All we need to do in this
> -case is to declare that our new package *foo* depends on *zlib*. This
> -kind of dependency is managed in the menu file of our new package by
> -simply adding the ``select ZLIB`` line. After this addition our menu
> -file looks like:
> -
> -.. code-block:: kconfig
> -
> -    ## SECTION=project_specific
> -
> -    config FOO
> -           tristate
> -           select ZLIB
> -           prompt "foo"
> -           help
> -             FIXME
> -
> -    if FOO
> -    config FOO_DEBUG
> -           bool
> -           prompt "add debug noise"
> -
> -    config FOO_BAR
> -           bool
> -           prompt "build bar"
> -
> -    endif
> -
> -PTXdist now builds the *zlib* first and our new package thereafter.
> -
> -Refer :ref:`external_dependencies_variants` for more specific dependency
> -description.
> -
> -Managing External Compile Time Dependencies on Demand
> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -It is good practice to add only those dependencies that are really
> -required for the current configuration of the package. If the package
> -provides the features *foo* and *bar* and its ``configure`` provides
> -switches to enable/disable them independently, we can also add
> -dependencies on demand. Let’s assume feature *foo* needs the compression
> -library *libz* and *bar* needs the XML2 library *libxml2*. These
> -libraries are only required at run-time if the corresponding feature is
> -enabled. To add these dependencies on demand, the menu file looks like:
> -
> -.. code-block:: kconfig
> -
> -    ## SECTION=project_specific
> -
> -    config FOO
> -           tristate
> -           select ZLIB if FOO_FOO
> -           select LIBXML2 if FOO_BAR
> -           prompt "foo"
> -           help
> -             FIXME
> -
> -    if FOO
> -    config FOO_DEBUG
> -           bool
> -           prompt "add debug noise"
> -
> -    config FOO_FOO
> -           bool
> -           prompt "build foo"
> -
> -    config FOO_BAR
> -           bool
> -           prompt "build bar"
> -
> -    endif
> -
> -.. important:: Do not add these ``select`` statements to the corresponding menu entry.
> -  They must belong to the main menu entry of the package to ensure that
> -  the calculation of the dependencies between the packages is done in a
> -  correct manner.
> -
> -Managing External Runtime Dependencies
> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -Some packages are building all of their components and also installing
> -them into the target’s sysroot. But only their *targetinstall* stage
> -decides which parts are copied to the root filesystem. So, compiling and
> -linking of our package will work, because everything required is found
> -in the target’s sysroot.
> -
> -In our example there is a hidden dependency to the math library
> -``libm``. Our new package was built successfully, because the linker was
> -able to link our binaries against the ``libm`` from the toolchain. But
> -in this case the ``libm`` must also be available in the target’s root
> -filesystem to fulfill the run-time dependency: We have to force PTXdist to
> -install ``libm``. ``libm`` is part of the *glibc* package, but is not
> -installed by default (to keep the root filesystem small). So, it **does
> -not** help to select the ``GLIBC`` symbol, to get a ``libm`` at run-time.
> -
> -The correct solution here is to add a ``select LIBC_M`` to our menu
> -file. With all the additions above it now looks like:
> -
> -.. code-block:: kconfig
> -
> -    ## SECTION=project_specific
> -
> -    config FOO
> -           tristate
> -           select ZLIB if FOO_FOO
> -           select LIBXML2 if FOO_BAR
> -           select LIBC_M
> -           prompt "foo"
> -           help
> -             FIXME
> -
> -    if FOO
> -    config FOO_DEBUG
> -           bool
> -           prompt "add debug noise"
> -
> -    config FOO_FOO
> -           bool
> -           prompt "build foo"
> -
> -    config FOO_BAR
> -           bool
> -           prompt "build bar"
> -
> -    endif
> -
> -.. note:: There are other packages around, that do not install everything by
> -  default. If our new package needs something special, we must take a look
> -  into the menu of the other package how to force the required components
> -  to be installed and add the corresponding ``selects`` to our own menu
> -  file. In this case it does not help to enable the required parts in our
> -  project configuration, because this has no effect on the build order!
> -
> -Managing Plain Makefile Packages
> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -Many packages are still coming with a plain ``Makefile``. The user has
> -to adapt it to make it work in a cross compile environment as well.
> -PTXdist can also handle this kind of packages. We only have to specify
> -a special *prepare* and *compile* stage.
> -
> -Such packages often have no special need for any kind of preparation. In
> -this we must instruct PTXdist to do nothing in the *prepare* stage:
> -
> -.. code-block:: make
> -
> -    FOO_CONF_TOOL := NO
> -
> -To compile the package, we can use ``make``\ ’s feature to overwrite
> -variables used in the ``Makefile``. With this feature we can still use
> -the original ``Makefile`` but with our own (cross compile) settings.
> -
> -Most of the time the generic compile rule can be used, only a few
> -settings are required. For a well defined ``Makefile`` it is sufficient to
> -set up the correct cross compile environment for the *compile* stage:
> -
> -.. code-block:: make
> -
> -    FOO_MAKE_ENV := $(CROSS_ENV)
> -
> -``make`` will be called in this case with:
> -
> -``$(FOO_MAKE_ENV) $(MAKE) -C $(FOO_DIR) $(FOO_MAKE_OPT)``
> -
> -So, in the rule file only the two variables ``FOO_MAKE_ENV`` and
> -``FOO_MAKE_OPT`` must be set, to forward the required settings to the
> -package’s buildsystem. If the package cannot be built in parallel, we
> -can also add the ``FOO_MAKE_PAR := NO``. ``YES`` is the default.
> -
> -Managing CMake/QMake/Meson Packages
> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -Building packages that use ``cmake``, ``qmake`` or ``meson`` is much like
> -building packages with an autotools based buildsystem. We need to specify
> -the configuration tool:
> -
> -.. code-block:: make
> -
> -    FOO_CONF_TOOL := cmake
> -
> -or
> -
> -.. code-block:: make
> -
> -    FOO_CONF_TOOL := qmake
> -
> -or respectively
> -
> -.. code-block:: make
> -
> -    FOO_CONF_TOOL := meson
> -
> -And provide the correct configuration options. The syntax is different so
> -PTXdist provides additional macros to simplify configurable features.
> -For ``cmake`` the configuration options typically look like this:
> -
> -.. code-block:: make
> -
> -    FOO_CONF_OPT := \
> -    	$(CROSS_CMAKE_USR) \
> -    	-DBUILD_TESTS:BOOL=OFF \
> -    	-DENABLE_BAR:BOOL=$(call ptx/onoff, PTXCONF_FOO_BAR)
> -
> -For ``qmake`` the configuration options typically look like this:
> -
> -.. code-block:: make
> -
> -    FOO_CONF_OPT := \
> -    	$(CROSS_QMAKE_OPT) \
> -    	PREFIX=/usr
> -
> -And for ``meson`` the configuration options typically look like this:
> -
> -.. code-block:: make
> -
> -    FOO_CONF_OPT := \
> -    	$(CROSS_MESON_USR) \
> -    	-Dbar=$(call ptx/truefalse,PTXCONF_FOO_BAR)
> -
> -Please note that currently only host and target ``cmake``\/``meson`` packages
> -and only target ``qmake`` packages are supported.
> -
> -Managing Python Packages
> -^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -As with any other package, the correct configuration tool must be selected
> -for Python packages:
> -
> -.. code-block:: make
> -
> -    FOO_CONF_TOOL := python
> -
> -.. note:: For Python3 packages the value must be ``python3``.
> -
> -No Makefiles are used when building Python packages so the usual ``make``
> -and ``make install`` for the *compile* and *install* stages cannot be used.
> -PTXdist will call ``python setup.py build`` and ``python setup.py install``
> -instead.
> -
> -.. note:: *FOO* is still the name of our example package. It must be
> -  replaced by the real package name.
> -
> -
> -.. _patching_packages:
> -
> -Patching Packages
> -~~~~~~~~~~~~~~~~~
> -
> -There can be various reasons why a package must be patched:
> -
> --  Package is broken for cross compile environments
> -
> --  Package is broken within a specific feature
> -
> --  Package is vulnerable and needs some fixes
> -
> --  or anything else (this case is the most common one)
> -
> -Ideally, those problems should be addressed in the original project,
> -so any patches you add to your BSP or to PTXdist should also be submitted upstream.
> -The upstream project can often provide better feedback, they can integrate your
> -patch into a new release, and also maintain your changes as part of the project.
> -This way we make sure that all advantages of the open source idea work for us;
> -and your patch can be removed again later when a new release of the project is
> -integrated into your BSP or into PTXdist.
> -
> -PTXdist handles patching automatically.
> -After extracting the archive of a package, PTXdist checks for the existence of
> -a patch directory named like its ``<PKG>_PATCHES`` variable, or, if this variable
> -is not set, like its ``<PKG>`` variable.
> -The patch directory is then searched in all locations listed by the
> -``PTXDIST_PATH_PATCHES`` variable, and the first one found is used.
> -Take an exemplary package ``foo`` with version ``1.1.0``:
> -The variable ``FOO`` will have the value ``foo-1.1.0``, so PTXdist will look for
> -a patch directory named ``foo-1.1.0`` in the following locations:
> -
> -#. the current layer:
> -
> -   a. project (``./patches/foo-1.1.0``)
> -   b. platform (``./configs/|ptxdistPlatformConfigDir|/patches/foo-1.1.0``)
> -
> -#. any :ref:`base layers <layers-in-ptxdist>`,
> -   applying the same search order as above for each layer recursively
> -
> -#. ptxdist (``<ptxdist/installation/path>/patches/foo-1.1.0``)
> -
> -The patches from the first location found are used. Note: Due to this
> -search order, a PTXdist project can replace global patches from the
> -PTXdist installation. This can be useful if a project sticks to a
> -specific PTXdist revision but fixes from a more recent revision of
> -PTXdist should be used.
> -
> -PTXdist uses the utilities *git*, *patch* or *quilt* to work with
> -patches or patch series. We recommend *git*, as it can manage patch
> -series in a very easy way.
> -
> -Creating a Patch Series for a Package
> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -To create a patch series for the first time, we can run the following
> -steps. We are still using our *foo-1.1.0* example package here:
> -
> -Using Quilt
> -"""""""""""
> -
> -We create a special directory for the patch series in the local project
> -directory:
> -
> -.. code-block:: text
> -
> -    $ mkdir -p patches/foo-1.1.0
> -
> -PTXdist expects a ``series`` file in the patch directory and at least
> -one patch. Otherwise it fails. Due to the fact that we do not have any
> -patch content yet, we’ll start with a dummy entry in the ``series`` file
> -and an empty ``patch`` file.
> -
> -.. code-block:: text
> -
> -    $ touch patches/foo-1.1.0/dummy
> -    $ echo dummy > patches/foo-1.1.0/series
> -
> -Next is to extract the package (if already done, we must remove it
> -first):
> -
> -.. code-block:: text
> -
> -    $ ptxdist extract foo
> -
> -This will extract the archive and create a symbolic link in the build
> -directory pointing to our local patch directory. Working this way will
> -ensure that we do not lose our created patches if we enter
> -``ptxdist clean foo`` by accident. In our case the patches are still
> -present in ``patches/foo-1.1.0`` and can be used the next time we
> -extract the package again.
> -
> -All we have to do now is to do the modification we need to make the
> -package work. We change into the build directory and use quilt_ to
> -create new patches, add files to respective patches, modify these files
> -and refresh the patches to save our changes.
> -See the *quilt* documentation (``man 1 quilt``) for more information.
> -
> -.. note:: For patches that are intended for PTXdist upstream use the git
> -  workflow described below to get proper patch headers.
> -
> -.. _quilt: http://savannah.nongnu.org/projects/quilt
> -
> -Using Git
> -"""""""""
> -
> -Create the patch directory like above for *quilt*,
> -but only add an empty series file:
> -
> -.. code-block:: text
> -
> -    $ mkdir -p patches/foo-1.1.0
> -    $ touch patches/foo-1.1.0/series
> -
> -Then extract the package with an additional command line switch:
> -
> -.. code-block:: text
> -
> -    $ ptxdist --git extract foo
> -
> -The empty series file makes PTXdist create a Git repository in the
> -respective package build directory,
> -and import the package source as the first commit.
> -
> -.. note:: Optionally, you can enable the setting *Developer Options →
> -  use git to apply patches* in `ptxdist setup` to get this behaviour
> -  as a default for every package.
> -  However, note that this setting is meant for development only, and can lead
> -  to failures – some packages try to determine if they are being compiled from
> -  a Git source tree, and behave differently in that case.
> -
> -Then you can change into the package build directory
> -(``platform-<name>/build-target/foo-1.1.0``),
> -patch the required source files,
> -and make Git commits on the way.
> -The Git history should now look something like this:
> -
> -.. code-block:: text
> -
> -    $ git log --oneline --decorate
> -    * df343e821851 (HEAD -> master) Makefile: don't build the tests
> -    * 65a360c2bd60 strfry.c: frobnicate the excusator
> -    * fdc315f6844c (tag: foobar-1.1.0, tag: base) initial commit
> -
> -Finally, call ``git ptx-patches`` to transform those Git commits into the patch
> -series in the ``patches/foo-1.1.0`` folder.
> -This way they don't get lost when cleaning the package.
> -
> -.. note:: PTXdist will only create a Git repository for packages with
> -  patches.  To use Git to generate the first patch, create an empty series
> -  file ``patches/foobar-1.1.0/series`` before extracting the packages. This
> -  will tell PTXdist to use Git anyways and ``git ptx-patches`` will put the
> -  patches there.
> -
> -Both approaches (Git and quilt) are not suitable for modifying files
> -that are autogenerated in autotools-based buildsystems.
> -Refer to the section :ref:`configure_rebuild` on how PTXdist can
> -handle this special task.
> -
> -Adding More Patches to a Package
> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -If we want to add more patches to an already patched package, we can use
> -nearly the same way as creating patches for the first time. But if the
> -patch series comes from the PTXdist main installation, we do not have
> -write permissions to these directories (do NEVER work on the main
> -installation directories, NEVER, NEVER, NEVER). Due to the search order
> -in which PTXdist searches for patches for a specific package, we can
> -copy the global patch series to our local project directory. Now we have
> -the permissions to add more patches or modify the existing ones. Also
> -*quilt* and *git* are our friends here to manage the patch series.
> -
> -If we think that our new patches are valuable also for others, or they
> -fix an error, it could be a good idea to send these patches to PTXdist
> -mainline, and to the upstream project too.
> -
> -
> -.. _configure_rebuild:
> -
> -Modifying Autotoolized Packages
> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> -
> -Autotoolized packages are very picky when automatically generated files
> -get patched. The patch order is very important in this case and
> -sometimes it even fails and nobody knows why.
> -
> -To improve a package’s autotools-based build system, PTXdist comes with
> -its own project local autotools to regenerate the autotools template
> -files, instead of patching them. With this feature, only the template
> -files must be patched, the required ``configure`` script and the
> -``Makefile.in`` files are regenerated in the final stages of the
> -*prepare* step.
> -
> -This feature works like the regular patching mechanism. The only
> -difference is the additional ``autogen.sh`` file in the patch directory.
> -If it exists and has execution permissions, it will be called after the
> -package was patched (while the *extract* stage is running).
> -
> -Its content depends on developer needs; for the most simple case the
> -content can be:
> -
> -.. code-block:: bash
> -
> -    #!/bin/bash
> -
> -    aclocal $ACLOCAL_FLAGS
> -
> -    libtoolize \
> -            --force \
> -            --copy
> -
> -    autoreconf \
> -            --force \
> -            --install \
> -            --warnings=cross \
> -            --warnings=syntax \
> -            --warnings=obsolete \
> -            --warnings=unsupported
> -
> -.. note:: In this way not yet autotoolized package can be autotoolized. We
> -  just have to add the common autotool template files (``configure.ac``
> -  and ``Makefile.am`` for example) via a patch series to the package
> -  source and the ``autogen.sh`` to the patch directory.
> diff --git a/doc/dev_advanced_rule_files.rst b/doc/dev_advanced_rule_files.rst
> new file mode 100644
> index 000000000000..794bd148cbae
> --- /dev/null
> +++ b/doc/dev_advanced_rule_files.rst
> @@ -0,0 +1,463 @@
> +*******************
> +Advanced Rule Files
> +*******************
> +
> +The previous example on how to create a rule file sometimes works as
> +shown above. But most of the time source archives are not that simple.
> +In this section we want to give the user a more detailed selection how
> +the package will be built.
> +
> +Adding Static Configure Parameters
> +==================================
> +
> +The ``configure`` scripts of various source archives provide additional
> +parameters to enable or disable features, or to configure them in a
> +specific way.
> +
> +We assume the ``configure`` script of our ``foo`` example (refer to
> +section :ref:`foo_example`) supports two additional parameters:
> +
> +-  **--enable-debug**: Make the program more noisy. It’s disabled by
> +   default.
> +
> +-  **--with-bar**: Also build the special executable **bar**. Building
> +   this executable is also disabled by default.
> +
> +We now want to forward these options to the ``configure`` script when it
> +runs in the *prepare* stage. To do so, we must again open the rule file
> +with our favourite editor and navigate to the *prepare* stage entry.
> +
> +PTXdist uses the variable ``FOO_CONF_OPT`` as the list of parameters to
> +be given to ``configure``.
> +
> +Currently this variable is commented out and defined to:
> +
> +.. code-block:: make
> +
> +    # FOO_CONF_OPT := $(CROSS_AUTOCONF_USR)
> +
> +The variable ``CROSS_AUTOCONF_USR`` is predefined by PTXdist and
> +contains all basic parameters to instruct ``configure`` to prepare for a
> +**cross** compile environment.
> +
> +To use the two additional mentioned ``configure`` parameters, we comment
> +in this line and supplement this expression as follows:
> +
> +.. code-block:: make
> +
> +    FOO_CONF_OPT := \
> +        $(CROSS_AUTOCONF_USR) \
> +        --enable-debug \
> +        --with-bar
> +
> +.. note:: We recommend to use this format with each parameter on a line of
> + its own. This format is easier to read and a diff shows more exactly any
> + change.
> +
> +To do a fast check if this addition was successful, we run:
> +
> +.. code-block:: text
> +
> +    $ ptxdist print FOO_CONF_OPT
> +    --prefix=/usr --sysconfdir=/etc --host=|ptxdistCompilerName| --build=i686-host-linux-gnu --enable-debug --with-bar
> +
> +.. note:: It depends on the currently selected platform and its architecture
> + what content this variable will have. The content shown above is an
> + example for a target.
> +
> +Or re-build the package with the new settings:
> +
> +.. code-block:: text
> +
> +    $ ptxdist drop foo prepare
> +    $ ptxdist targetinstall foo
> +
> +Adding Dynamic Configure Parameters
> +===================================
> +
> +Sometimes it makes sense to add this kind of parameters on demand only;
> +especially a parameter like ``--enable-debug``. To let the user decide
> +if this parameter is to be used or not, we must add a menu entry. So,
> +let’s expand our menu. Here is its current content:
> +
> +.. code-block:: kconfig
> +
> +    ## SECTION=project_specific
> +
> +    config FOO
> +            tristate
> +            prompt "foo"
> +            help
> +              FIXME
> +
> +We’ll add two menu entries, one for each optional parameter we want to
> +add on demand to the ``configure`` parameters:
> +
> +.. code-block:: kconfig
> +
> +    ## SECTION=project_specific
> +
> +    config FOO
> +           tristate
> +           prompt "foo"
> +           help
> +             FIXME
> +
> +    if FOO
> +    config FOO_DEBUG
> +           bool
> +           prompt "add debug noise"
> +
> +    config FOO_BAR
> +           bool
> +           prompt "build bar"
> +
> +    endif
> +
> +.. important:: Always follow the rule to extend the base name by a suboption
> +  name as the trailing part of the variable name. This gives PTXdist the ability
> +  to detect a change in the package’s settings (via menuconfig) to force its
> +  rebuild on demand.
> +
> +To make usage of the new menu entries, we must check them in the rule
> +file and add the correct parameters:
> +
> +.. code-block:: make
> +
> +    #
> +    # autoconf
> +    #
> +    FOO_CONF_OPT := \
> +        $(CROSS_AUTOCONF_USR) \
> +        --$(call ptx/endis, PTXCONF_FOO_DEBUG)-debug \
> +        --$(call ptx/wwo, PTXCONF_FOO_BAR)-bar
> +
> +.. important:: Please note the leading ``PTXCONF_`` for each define. While Kconfig is
> +  using ``FOO_BAR``, the rule file must use ``PTXCONF_FOO_BAR`` instead.
> +
> +.. note:: Refer :ref:`Rule File Macro Reference <param_macros>` for further
> +   details about these special kind of option macros (e.g. ``ptx/...``).
> +
> +It is a good practice to always add both settings, e.g. ``--disable-debug``
> +even if this is the default case. Sometimes ``configure`` tries to guess
> +something and the binary result might differ depending on the build
> +order. For example some kind of package would also build some X related
> +tools, if X libraries are found. In this case it depends on the build
> +order, if the X related tools are built or not. All the autocheck
> +features are problematic here. So, if we do not want ``configure`` to
> +guess its settings we **must disable everything we do not want**.
> +
> +To support this process, PTXdist supplies a helper script, located at
> +``/path/to/ptxdist/scripts/configure-helper.py`` that compares the configure
> +output with the settings from ``FOO_CONF_OPT``:
> +
> +.. code-block:: text
> +
> +    $ /opt/ptxdist-2017.06.0/scripts/configure-helper.py -p libsigrok
> +    --- rules/libsigrok.make
> +    +++ libsigrok-0.5.0
> +    @@ -4,3 +4,74 @@
> +     	--libdir=/usr/lib
> +     	--build=x86_64-host-linux-gnu
> +     	--host=arm-v7a-linux-gnueabihf
> +    +	--enable-warnings=min|max|fatal|no
> +    +	--disable-largefile
> +    +	--enable-all-drivers
> +    +	--enable-agilent-dmm
> +    [...]
> +    +	--enable-ruby
> +    +	--enable-java
> +    +	--without-libserialport
> +    +	--without-libftdi
> +    +	--without-libusb
> +    +	--without-librevisa
> +    +	--without-libgpib
> +    +	--without-libieee1284
> +    +	--with-jni-include-path=DIR-LIST
> +
> +In this example, many configure options from libsigrok (marked with ``+``)
> +are not yet present in ``LIBSIGROK_CONF_OPT`` and must be added, possibly also
> +by providing more dynamic options in the package definition.
> +
> +If some parts of a package are built on demand only, they must also be
> +installed on demand only. Besides the *prepare* stage, we also must
> +modify our *targetinstall* stage:
> +
> +.. code-block:: make
> +
> +    	@$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/foo, /usr/bin/foo)
> +
> +    ifdef PTXCONF_FOO_BAR
> +    	@$(call install_copy, foo, 0, 0, 0755, $(FOO_DIR)/bar, /usr/bin/bar)
> +    endif
> +
> +    	@$(call install_finish, foo)
> +    	@$(call touch)
> +
> +Now we can play with our new menu entries and check if they are working
> +as expected:
> +
> +.. code-block:: text
> +
> +    $ ptxdist menuconfig
> +    $ ptxdist targetinstall foo
> +
> +Whenever we change a *FOO* related menu entry, PTXdist should detect it
> +and re-build the package when a new build is started.
> +
> +.. _external_dependencies:
> +
> +Managing External Compile Time Dependencies
> +===========================================
> +
> +While running the prepare stage, it could happen that it fails due to a
> +missing external dependency.
> +
> +For example:
> +
> +.. code-block:: text
> +
> +    checking whether zlib exists....failed
> +
> +In this example, our new package depends on the compression library
> +*zlib*. PTXdist comes with a target *zlib*. All we need to do in this
> +case is to declare that our new package *foo* depends on *zlib*. This
> +kind of dependency is managed in the menu file of our new package by
> +simply adding the ``select ZLIB`` line. After this addition our menu
> +file looks like:
> +
> +.. code-block:: kconfig
> +
> +    ## SECTION=project_specific
> +
> +    config FOO
> +           tristate
> +           select ZLIB
> +           prompt "foo"
> +           help
> +             FIXME
> +
> +    if FOO
> +    config FOO_DEBUG
> +           bool
> +           prompt "add debug noise"
> +
> +    config FOO_BAR
> +           bool
> +           prompt "build bar"
> +
> +    endif
> +
> +PTXdist now builds the *zlib* first and our new package thereafter.
> +
> +Refer :ref:`external_dependencies_variants` for more specific dependency
> +description.
> +
> +Managing External Compile Time Dependencies on Demand
> +=====================================================
> +
> +It is good practice to add only those dependencies that are really
> +required for the current configuration of the package. If the package
> +provides the features *foo* and *bar* and its ``configure`` provides
> +switches to enable/disable them independently, we can also add
> +dependencies on demand. Let’s assume feature *foo* needs the compression
> +library *libz* and *bar* needs the XML2 library *libxml2*. These
> +libraries are only required at run-time if the corresponding feature is
> +enabled. To add these dependencies on demand, the menu file looks like:
> +
> +.. code-block:: kconfig
> +
> +    ## SECTION=project_specific
> +
> +    config FOO
> +           tristate
> +           select ZLIB if FOO_FOO
> +           select LIBXML2 if FOO_BAR
> +           prompt "foo"
> +           help
> +             FIXME
> +
> +    if FOO
> +    config FOO_DEBUG
> +           bool
> +           prompt "add debug noise"
> +
> +    config FOO_FOO
> +           bool
> +           prompt "build foo"
> +
> +    config FOO_BAR
> +           bool
> +           prompt "build bar"
> +
> +    endif
> +
> +.. important:: Do not add these ``select`` statements to the corresponding menu entry.
> +  They must belong to the main menu entry of the package to ensure that
> +  the calculation of the dependencies between the packages is done in a
> +  correct manner.
> +
> +Managing External Runtime Dependencies
> +======================================
> +
> +Some packages are building all of their components and also installing
> +them into the target’s sysroot. But only their *targetinstall* stage
> +decides which parts are copied to the root filesystem. So, compiling and
> +linking of our package will work, because everything required is found
> +in the target’s sysroot.
> +
> +In our example there is a hidden dependency to the math library
> +``libm``. Our new package was built successfully, because the linker was
> +able to link our binaries against the ``libm`` from the toolchain. But
> +in this case the ``libm`` must also be available in the target’s root
> +filesystem to fulfill the run-time dependency: We have to force PTXdist to
> +install ``libm``. ``libm`` is part of the *glibc* package, but is not
> +installed by default (to keep the root filesystem small). So, it **does
> +not** help to select the ``GLIBC`` symbol, to get a ``libm`` at run-time.
> +
> +The correct solution here is to add a ``select LIBC_M`` to our menu
> +file. With all the additions above it now looks like:
> +
> +.. code-block:: kconfig
> +
> +    ## SECTION=project_specific
> +
> +    config FOO
> +           tristate
> +           select ZLIB if FOO_FOO
> +           select LIBXML2 if FOO_BAR
> +           select LIBC_M
> +           prompt "foo"
> +           help
> +             FIXME
> +
> +    if FOO
> +    config FOO_DEBUG
> +           bool
> +           prompt "add debug noise"
> +
> +    config FOO_FOO
> +           bool
> +           prompt "build foo"
> +
> +    config FOO_BAR
> +           bool
> +           prompt "build bar"
> +
> +    endif
> +
> +.. note:: There are other packages around, that do not install everything by
> +  default. If our new package needs something special, we must take a look
> +  into the menu of the other package how to force the required components
> +  to be installed and add the corresponding ``selects`` to our own menu
> +  file. In this case it does not help to enable the required parts in our
> +  project configuration, because this has no effect on the build order!
> +
> +Managing Plain Makefile Packages
> +================================
> +
> +Many packages are still coming with a plain ``Makefile``. The user has
> +to adapt it to make it work in a cross compile environment as well.
> +PTXdist can also handle this kind of packages. We only have to specify
> +a special *prepare* and *compile* stage.
> +
> +Such packages often have no special need for any kind of preparation. In
> +this we must instruct PTXdist to do nothing in the *prepare* stage:
> +
> +.. code-block:: make
> +
> +    FOO_CONF_TOOL := NO
> +
> +To compile the package, we can use ``make``\ ’s feature to overwrite
> +variables used in the ``Makefile``. With this feature we can still use
> +the original ``Makefile`` but with our own (cross compile) settings.
> +
> +Most of the time the generic compile rule can be used, only a few
> +settings are required. For a well defined ``Makefile`` it is sufficient to
> +set up the correct cross compile environment for the *compile* stage:
> +
> +.. code-block:: make
> +
> +    FOO_MAKE_ENV := $(CROSS_ENV)
> +
> +``make`` will be called in this case with:
> +
> +``$(FOO_MAKE_ENV) $(MAKE) -C $(FOO_DIR) $(FOO_MAKE_OPT)``
> +
> +So, in the rule file only the two variables ``FOO_MAKE_ENV`` and
> +``FOO_MAKE_OPT`` must be set, to forward the required settings to the
> +package’s buildsystem. If the package cannot be built in parallel, we
> +can also add the ``FOO_MAKE_PAR := NO``. ``YES`` is the default.
> +
> +Managing CMake/QMake/Meson Packages
> +===================================
> +
> +Building packages that use ``cmake``, ``qmake`` or ``meson`` is much like
> +building packages with an autotools based buildsystem. We need to specify
> +the configuration tool:
> +
> +.. code-block:: make
> +
> +    FOO_CONF_TOOL := cmake
> +
> +or
> +
> +.. code-block:: make
> +
> +    FOO_CONF_TOOL := qmake
> +
> +or respectively
> +
> +.. code-block:: make
> +
> +    FOO_CONF_TOOL := meson
> +
> +And provide the correct configuration options. The syntax is different so
> +PTXdist provides additional macros to simplify configurable features.
> +For ``cmake`` the configuration options typically look like this:
> +
> +.. code-block:: make
> +
> +    FOO_CONF_OPT := \
> +    	$(CROSS_CMAKE_USR) \
> +    	-DBUILD_TESTS:BOOL=OFF \
> +    	-DENABLE_BAR:BOOL=$(call ptx/onoff, PTXCONF_FOO_BAR)
> +
> +For ``qmake`` the configuration options typically look like this:
> +
> +.. code-block:: make
> +
> +    FOO_CONF_OPT := \
> +    	$(CROSS_QMAKE_OPT) \
> +    	PREFIX=/usr
> +
> +And for ``meson`` the configuration options typically look like this:
> +
> +.. code-block:: make
> +
> +    FOO_CONF_OPT := \
> +    	$(CROSS_MESON_USR) \
> +    	-Dbar=$(call ptx/truefalse,PTXCONF_FOO_BAR)
> +
> +Please note that currently only host and target ``cmake``\/``meson`` packages
> +and only target ``qmake`` packages are supported.
> +
> +Managing Python Packages
> +========================
> +
> +As with any other package, the correct configuration tool must be selected
> +for Python packages:
> +
> +.. code-block:: make
> +
> +    FOO_CONF_TOOL := python
> +
> +.. note:: For Python3 packages the value must be ``python3``.
> +
> +No Makefiles are used when building Python packages so the usual ``make``
> +and ``make install`` for the *compile* and *install* stages cannot be used.
> +PTXdist will call ``python setup.py build`` and ``python setup.py install``
> +instead.
> +
> +.. note:: *FOO* is still the name of our example package. It must be
> +  replaced by the real package name.
> diff --git a/doc/dev_manual.rst b/doc/dev_manual.rst
> index c232cc91428a..e9a88c1a97f5 100644
> --- a/doc/dev_manual.rst
> +++ b/doc/dev_manual.rst
> @@ -11,6 +11,8 @@ This chapter shows all (or most) of the details of how PTXdist works.
>  
>     dev_dir_hierarchy
>     dev_add_new_pkgs
> +   dev_advanced_rule_files
> +   dev_patching
>     dev_add_bin_only_files
>     dev_create_new_pkg_templates
>     dev_layers_in_ptxdist
> diff --git a/doc/dev_patching.rst b/doc/dev_patching.rst
> new file mode 100644
> index 000000000000..e3690dfe1b48
> --- /dev/null
> +++ b/doc/dev_patching.rst
> @@ -0,0 +1,226 @@
> +.. _patching_packages:
> +
> +*****************
> +Patching Packages
> +*****************
> +
> +There can be various reasons why a package must be patched:
> +
> +-  Package is broken for cross compile environments
> +
> +-  Package is broken within a specific feature
> +
> +-  Package is vulnerable and needs some fixes
> +
> +-  or anything else (this case is the most common one)
> +
> +Ideally, those problems should be addressed in the original project,
> +so any patches you add to your BSP or to PTXdist should also be submitted upstream.
> +The upstream project can often provide better feedback, they can integrate your
> +patch into a new release, and also maintain your changes as part of the project.
> +This way we make sure that all advantages of the open source idea work for us;
> +and your patch can be removed again later when a new release of the project is
> +integrated into your BSP or into PTXdist.
> +
> +PTXdist handles patching automatically.
> +After extracting the archive of a package, PTXdist checks for the existence of
> +a patch directory named like its ``<PKG>_PATCHES`` variable, or, if this variable
> +is not set, like its ``<PKG>`` variable.
> +The patch directory is then searched in all locations listed by the
> +``PTXDIST_PATH_PATCHES`` variable, and the first one found is used.
> +Take an exemplary package ``foo`` with version ``1.1.0``:
> +The variable ``FOO`` will have the value ``foo-1.1.0``, so PTXdist will look for
> +a patch directory named ``foo-1.1.0`` in the following locations:
> +
> +#. the current layer:
> +
> +   a. project (``./patches/foo-1.1.0``)
> +   b. platform (``./configs/|ptxdistPlatformConfigDir|/patches/foo-1.1.0``)
> +
> +#. any :ref:`base layers <layers-in-ptxdist>`,
> +   applying the same search order as above for each layer recursively
> +
> +#. ptxdist (``<ptxdist/installation/path>/patches/foo-1.1.0``)
> +
> +The patches from the first location found are used. Note: Due to this
> +search order, a PTXdist project can replace global patches from the
> +PTXdist installation. This can be useful if a project sticks to a
> +specific PTXdist revision but fixes from a more recent revision of
> +PTXdist should be used.
> +
> +PTXdist uses the utilities *git*, *patch* or *quilt* to work with
> +patches or patch series. We recommend *git*, as it can manage patch
> +series in a very easy way.
> +
> +Creating a Patch Series for a Package
> +=====================================
> +
> +To create a patch series for the first time, we can run the following
> +steps. We are still using our *foo-1.1.0* example package here:
> +
> +Using Quilt
> +-----------
> +
> +We create a special directory for the patch series in the local project
> +directory:
> +
> +.. code-block:: text
> +
> +    $ mkdir -p patches/foo-1.1.0
> +
> +PTXdist expects a ``series`` file in the patch directory and at least
> +one patch. Otherwise it fails. Due to the fact that we do not have any
> +patch content yet, we’ll start with a dummy entry in the ``series`` file
> +and an empty ``patch`` file.
> +
> +.. code-block:: text
> +
> +    $ touch patches/foo-1.1.0/dummy
> +    $ echo dummy > patches/foo-1.1.0/series
> +
> +Next is to extract the package (if already done, we must remove it
> +first):
> +
> +.. code-block:: text
> +
> +    $ ptxdist extract foo
> +
> +This will extract the archive and create a symbolic link in the build
> +directory pointing to our local patch directory. Working this way will
> +ensure that we do not lose our created patches if we enter
> +``ptxdist clean foo`` by accident. In our case the patches are still
> +present in ``patches/foo-1.1.0`` and can be used the next time we
> +extract the package again.
> +
> +All we have to do now is to do the modification we need to make the
> +package work. We change into the build directory and use quilt_ to
> +create new patches, add files to respective patches, modify these files
> +and refresh the patches to save our changes.
> +See the *quilt* documentation (``man 1 quilt``) for more information.
> +
> +.. note:: For patches that are intended for PTXdist upstream use the git
> +  workflow described below to get proper patch headers.
> +
> +.. _quilt: http://savannah.nongnu.org/projects/quilt
> +
> +Using Git
> +---------
> +
> +Create the patch directory like above for *quilt*,
> +but only add an empty series file:
> +
> +.. code-block:: text
> +
> +    $ mkdir -p patches/foo-1.1.0
> +    $ touch patches/foo-1.1.0/series
> +
> +Then extract the package with an additional command line switch:
> +
> +.. code-block:: text
> +
> +    $ ptxdist --git extract foo
> +
> +The empty series file makes PTXdist create a Git repository in the
> +respective package build directory,
> +and import the package source as the first commit.
> +
> +.. note:: Optionally, you can enable the setting *Developer Options →
> +  use git to apply patches* in `ptxdist setup` to get this behaviour
> +  as a default for every package.
> +  However, note that this setting is meant for development only, and can lead
> +  to failures – some packages try to determine if they are being compiled from
> +  a Git source tree, and behave differently in that case.
> +
> +Then you can change into the package build directory
> +(``platform-<name>/build-target/foo-1.1.0``),
> +patch the required source files,
> +and make Git commits on the way.
> +The Git history should now look something like this:
> +
> +.. code-block:: text
> +
> +    $ git log --oneline --decorate
> +    * df343e821851 (HEAD -> master) Makefile: don't build the tests
> +    * 65a360c2bd60 strfry.c: frobnicate the excusator
> +    * fdc315f6844c (tag: foobar-1.1.0, tag: base) initial commit
> +
> +Finally, call ``git ptx-patches`` to transform those Git commits into the patch
> +series in the ``patches/foo-1.1.0`` folder.
> +This way they don't get lost when cleaning the package.
> +
> +.. note:: PTXdist will only create a Git repository for packages with
> +  patches.  To use Git to generate the first patch, create an empty series
> +  file ``patches/foobar-1.1.0/series`` before extracting the packages. This
> +  will tell PTXdist to use Git anyways and ``git ptx-patches`` will put the
> +  patches there.
> +
> +Both approaches (Git and quilt) are not suitable for modifying files
> +that are autogenerated in autotools-based buildsystems.
> +Refer to the section :ref:`configure_rebuild` on how PTXdist can
> +handle this special task.
> +
> +
> +Adding More Patches to a Package
> +================================
> +
> +If we want to add more patches to an already patched package, we can use
> +nearly the same way as creating patches for the first time. But if the
> +patch series comes from the PTXdist main installation, we do not have
> +write permissions to these directories (do NEVER work on the main
> +installation directories, NEVER, NEVER, NEVER). Due to the search order
> +in which PTXdist searches for patches for a specific package, we can
> +copy the global patch series to our local project directory. Now we have
> +the permissions to add more patches or modify the existing ones. Also
> +*quilt* and *git* are our friends here to manage the patch series.
> +
> +If we think that our new patches are valuable also for others, or they
> +fix an error, it could be a good idea to send these patches to PTXdist
> +mainline, and to the upstream project too.
> +
> +
> +.. _configure_rebuild:
> +
> +Modifying Autotoolized Packages
> +===============================
> +
> +Autotoolized packages are very picky when automatically generated files
> +get patched. The patch order is very important in this case and
> +sometimes it even fails and nobody knows why.
> +
> +To improve a package’s autotools-based build system, PTXdist comes with
> +its own project local autotools to regenerate the autotools template
> +files, instead of patching them. With this feature, only the template
> +files must be patched, the required ``configure`` script and the
> +``Makefile.in`` files are regenerated in the final stages of the
> +*prepare* step.
> +
> +This feature works like the regular patching mechanism. The only
> +difference is the additional ``autogen.sh`` file in the patch directory.
> +If it exists and has execution permissions, it will be called after the
> +package was patched (while the *extract* stage is running).
> +
> +Its content depends on developer needs; for the most simple case the
> +content can be:
> +
> +.. code-block:: bash
> +
> +    #!/bin/bash
> +
> +    aclocal $ACLOCAL_FLAGS
> +
> +    libtoolize \
> +            --force \
> +            --copy
> +
> +    autoreconf \
> +            --force \
> +            --install \
> +            --warnings=cross \
> +            --warnings=syntax \
> +            --warnings=obsolete \
> +            --warnings=unsupported
> +
> +.. note:: In this way not yet autotoolized package can be autotoolized. We
> +  just have to add the common autotool template files (``configure.ac``
> +  and ``Makefile.am`` for example) via a patch series to the package
> +  source and the ``autogen.sh`` to the patch directory.

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [ptxdist] [APPLIED] doc: dev manual: how to recover from patch merge conflicts with git
  2021-06-16 15:56 ` [ptxdist] [PATCH 2/2] doc: dev manual: how to recover from patch merge conflicts with git Roland Hieber
@ 2021-06-29  5:09   ` Michael Olbrich
  0 siblings, 0 replies; 4+ messages in thread
From: Michael Olbrich @ 2021-06-29  5:09 UTC (permalink / raw)
  To: ptxdist; +Cc: Roland Hieber

Thanks, applied as 0ab10db1c2d26ae4d94454aedffa6cf5073423f7.

Michael

[sent from post-receive hook]

On Tue, 29 Jun 2021 07:09:17 +0200, Roland Hieber <rhi@pengutronix.de> wrote:
> Signed-off-by: Roland Hieber <rhi@pengutronix.de>
> Message-Id: <20210616155631.12180-2-rhi@pengutronix.de>
> Signed-off-by: Michael Olbrich <m.olbrich@pengutronix.de>
> 
> diff --git a/doc/dev_patching.rst b/doc/dev_patching.rst
> index e3690dfe1b48..a17983aa826a 100644
> --- a/doc/dev_patching.rst
> +++ b/doc/dev_patching.rst
> @@ -159,6 +159,78 @@ that are autogenerated in autotools-based buildsystems.
>  Refer to the section :ref:`configure_rebuild` on how PTXdist can
>  handle this special task.
>  
> +Recovering from merge conflicts
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +When updating packages, it can happen that older patches no longer apply.
> +In this case, the *extract* stage will throw errors like this::
> +
> +   -----------------------------
> +   target: ima-evm-utils.extract
> +   -----------------------------
> +
> +   extract: pkg_src=/ptx/src/ima-evm-utils-1.3.2.tar.gz
> +   extract: pkg_extract_dir=DistroKit/platform-v7a/build-target/ima-evm-utils-1.3.2
> +   patchin: git: initializing repository
> +   patchin: git: done
> +
> +   pkg_patch_dir:     'ptxdist/patches/ima-evm-utils-1.3.2'
> +   pkg_patch_series:  'ptxdist/patches/ima-evm-utils-1.3.2/series'
> +
> +   patchin: git: apply 'ptxdist/patches/ima-evm-utils-1.3.2/series'
> +   tagging -> base
> +   0001-INSTALL-remove-file-at-it-s-autogenerated-by-autotoo.patch
> +   0002-Makefile.am-rename-INCLUDES-AM_CPPFLAGS.patch
> +   error: patch failed: src/Makefile.am:1
> +   error: src/Makefile.am: patch does not apply
> +   make: *** […/ptxdist/rules/post/ptxd_make_world_extract.make:41: …/DistroKit/platform-v7a/state/ima-evm-utils.extract] Error 4
> +
> +In the example above, the first patch was applied cleanly, but the second one
> +was not because src/Makefile.am contained different lines than expected.
> +If this happens, you have to clean up the merge conflict,
> +and apply the remaining patches manually.
> +
> +First, change into the package's build directory, and abort the current patch::
> +
> +   …/distrokit/master (master) $  cd platform-v7a/build-target/ima-evm-utils-1.3.2/
> +
> +   …/build-target/ima-evm-utils-1.3.2 (master|AM/REBASE) $  git log --oneline --graph
> +   * 6687ab46087c (HEAD -> master) INSTALL: remove file, as it's autogenerated by autotools
> +   * 01a52624237f (tag: ima-evm-utils-1.3.2, tag: base) initial commit
> +
> +   …/build-target/ima-evm-utils-1.3.2 (master|AM/REBASE) $  git am --abort
> +
> +   …/build-target/ima-evm-utils-1.3.2 (master) $
> +
> +(Notice how the Git integration in the shell prompt still shows ``AM/REBASE``
> +before the cleanup, signaling the ongoing conflict resolution.)
> +
> +The remaining patches are still available in the ``./patches`` directory
> +relative to your current location.
> +Try to apply each of them in turn using ``git am``.
> +If a patch fails to apply, Git will not change any files, but will still
> +remember the patch's author and commit message, and prompt you to resolve
> +the conflicts::
> +
> +   …/build-target/ima-evm-utils-1.3.2 (master) $  git am patches/0005-evmctl-add-fallback-definitions-for-XATTR_NAME_IMA.patch
> +
> +   Applying: evmctl: add fallback definitions for XATTR_NAME_IMA
> +   Patch failed at 0005 evmctl: add fallback definitions for XATTR_NAME_IMA
> +   hint: Use 'git am --show-current-patch=diff' to see the failed patch
> +   When you have resolved this problem, run "git am --continue".
> +   If you prefer to skip this patch, run "git am --skip" instead.
> +   To restore the original branch and stop patching, run "git am --abort".
> +
> +* If you find that the patch is no longer necessary (e.g. because it was
> +  already merged upstream in the new package version), skip it with
> +  ``git am --skip``.
> +* Otherwise, apply the same patch again manually via ``patch --merge -p1 <
> +  patches/filename.patch``, and resolve any resulting conflicts using your
> +  favourite editor.
> +* Finally, ``git am --continue`` commits your conflict resolution with the
> +  patch's original author and log message.
> +
> +After porting all patches, update the package's patch queue with ``git ptx-patches``.
>  
>  Adding More Patches to a Package
>  ================================

_______________________________________________
ptxdist mailing list
ptxdist@pengutronix.de
To unsubscribe, send a mail with subject "unsubscribe" to ptxdist-request@pengutronix.de

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2021-06-29  5:10 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-16 15:56 [ptxdist] [PATCH 1/2] doc: dev manual: propagate two subsections to sections Roland Hieber
2021-06-16 15:56 ` [ptxdist] [PATCH 2/2] doc: dev manual: how to recover from patch merge conflicts with git Roland Hieber
2021-06-29  5:09   ` [ptxdist] [APPLIED] " Michael Olbrich
2021-06-29  5:09 ` [ptxdist] [APPLIED] doc: dev manual: propagate two subsections to sections Michael Olbrich

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox