From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Tue, 08 Nov 2022 13:14:46 +0100 Received: from metis.ext.pengutronix.de ([2001:67c:670:201:290:27ff:fe1d:cc33]) by lore.white.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1osNV8-001La1-7C for lore@lore.pengutronix.de; Tue, 08 Nov 2022 13:14:46 +0100 Received: from localhost ([127.0.0.1] helo=metis.ext.pengutronix.de) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1osNV7-0007Su-0c; Tue, 08 Nov 2022 13:14:45 +0100 Received: from mail-eopbgr80051.outbound.protection.outlook.com ([40.107.8.51] helo=EUR04-VI1-obe.outbound.protection.outlook.com) by metis.ext.pengutronix.de with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1osNUH-0007Sh-WF; Tue, 08 Nov 2022 13:13:55 +0100 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XNEktsz/3FUZvTeS4lE0NwlflK5opc4m1Oi/kECjynPUesj0oNb0702PVTds0K6vM8pOlcbIJkiojDJQC6SBLl7tFNxiubA0L5qTbYfAcA4iYW0XN/Rp0b6GCkC97w4ipLhnoi1awod0km8Yq5WTbjb6sDDHD2nnu1cov9cOdwOEJJLz3vExg7RnBh3gy7l8P+dNBdmPd/A8flRknCXVSs4Nu0j11hFAOMKPx9SKB/siDj3BwUwtYsTcPmaf2Sh1A4GC5lSOcGLDv52m7DgjuFaELDCBfc4m2wL8S5pfOaJVhPjeyZkjTm5I1sCdJ/FMalIpcPuGDNhYVFbQo9zFvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rTKpkYD11ZkWSUokJeZzgNat4iLZqvSykGSVY+088V4=; b=jR2aFzlMNB9O/vaChFW0/cEqGfHq5qE/0ZQ5lgytUKUogivUYfvmICAFJJcm3mWs1m6bK5wzpaVnUjhZPG1KhdU2k+jhCWaCx/9fqUNav3c2yt/XKKRzTb0TYcGF1gyxbyXBHfEpcHMyqpDGRp9XiMq2/ceovXEPEhzJ5eyuLtGuuvNkXagDvVX5X8UN85P40WfO2L0SlwVu3A4crkJXMSgutron6vaBKmNSBDV+Ap6fuJqNj0GR1YTu5Fas1LhXNQLmbWYFEy5ge9zEe82fdFwBACYXjLvuQAS6FgRGufyHzNtDCpYEiuCwsVWB4iw2g8xpK+HNHFhKIiLteBCR4A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=t2data.com; dmarc=pass action=none header.from=t2data.com; dkim=pass header.d=t2data.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=t2datacom.onmicrosoft.com; s=selector1-t2datacom-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rTKpkYD11ZkWSUokJeZzgNat4iLZqvSykGSVY+088V4=; b=J8lWWFP6aRH9yxZ2CbdohW1wRqKy+FNP+NE73fKmFFPxPDdqF1XDQh3Q5OQq/7uMHRjv0GWCOy0Yuz3RYpYpgQsPYntrgEwz5vXJah3ehOLot2zN3LMW/Ert8FxLEYiKq4KkAHVg7N62pFC3iaB6a8Siwp3kreV0RpLcEMniC7c= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=t2data.com; Received: from DB9P251MB0618.EURP251.PROD.OUTLOOK.COM (2603:10a6:10:334::22) by DB4P251MB0902.EURP251.PROD.OUTLOOK.COM (2603:10a6:10:379::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.22; Tue, 8 Nov 2022 12:13:52 +0000 Received: from DB9P251MB0618.EURP251.PROD.OUTLOOK.COM ([fe80::fafd:ebc3:4f2d:f330]) by DB9P251MB0618.EURP251.PROD.OUTLOOK.COM ([fe80::fafd:ebc3:4f2d:f330%7]) with mapi id 15.20.5791.027; Tue, 8 Nov 2022 12:13:52 +0000 Message-ID: Date: Tue, 8 Nov 2022 13:10:32 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.1 Content-Language: en-US From: Christian Melki To: post@lespocky.de, CG@eks-engel.de, Michael Olbrich References: <75f0b7f568884d33af8f5c6bc23bd5dd@eks-engel.de> <2aa3b38a-9025-d88e-8364-fea4ada10f3a@t2data.com> In-Reply-To: <2aa3b38a-9025-d88e-8364-fea4ada10f3a@t2data.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: GV3P280CA0079.SWEP280.PROD.OUTLOOK.COM (2603:10a6:150:a::22) To DB9P251MB0618.EURP251.PROD.OUTLOOK.COM (2603:10a6:10:334::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB9P251MB0618:EE_|DB4P251MB0902:EE_ X-MS-Office365-Filtering-Correlation-Id: f57a793c-d73a-4c56-b398-08dac182b2f7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: G+pMsZwbA6GmO91f6EJ21MUVZXN8FGLN5gzc5g18KKMiJ5tY/mbCOCLRqCxdl/YlhOcSWjzAHz8RTlDaPzGEOzL4kXUeGljy6dZ1e+BxiwvQ5l98tOfFhhTMAbDKBEaGKO1aqCoqHEgrsUZgyBnV97rtHZy3AHf6DhFPx77eTlirX1n8FnmjocsKJ2WXDWJFU2mnyuxNl0MwSgwzRItZQqVl5IG+r2WZka7zTcaE0WUFbvMiikCnpqP3eTv/yljBGDVlm/HNvJAsThuRgJzSAvJI46GKRdlGXFxjMKOfaL78NOw6VUpXplbWwrcjtbX4Lpx3R8O6Zd/Wvx/wRF9sVT4RP2QjcCmxKsYd5wjlT7OLN7ti0YXEoi2KrjI91/8AP2q/bJ2vpG7Z9VRy06v5UacaLvMq+KkFPHiRakrhN+T811uyRMUw7gSuAAHZWJ8htmoJEUttlSSBmjlYHWpFS10BKO9Kdl0Lr6yEDP1sfD75PqSRmjSdjLG1TlkAc1EG+QTqPVBpZrGKXTmVWV+kHl467iZ05ppkyRxqlb3lik11kZVY90Hg2XaQcp1sOjyeGXXYAcgHF3JwRlDFyHyl5WIwaLP7PxdrCubS2tf+/EimSsG1i38niw1dleTf6M5AO0gwv0iU5+p5XlaN35cAIpK9aAFw6lJWzLso3EdkjmmJp5vboik4aG6AGH9iJWzf+O/l+B4vBD9O75qEiSIs4qc7/XchyY4gTxtSS5+ge10ZwqUKFtF9S4sX7KbWiWQdkoElXRlqZW/9vR5YuRZLeZTcTFKrnMcyjx+Z6qYw2wboZwJ8+RjuMWxc6oAxxmRx X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DB9P251MB0618.EURP251.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(13230022)(396003)(39830400003)(136003)(346002)(366004)(376002)(451199015)(6506007)(53546011)(31696002)(86362001)(36756003)(66946007)(6666004)(316002)(6916009)(41300700001)(8676002)(4326008)(66556008)(31686004)(66476007)(6512007)(26005)(966005)(478600001)(83380400001)(6486002)(66574015)(4001150100001)(2906002)(38100700002)(186003)(8936002)(2616005)(5660300002)(44832011)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TkhoQ1kzZXBxMHFEOXU5STFMVmcvRXpCdFM1bEpjNEVzUXVzT0FTb3JkZk9n?= =?utf-8?B?QUQwa1lLamZvL1JSSU1RN1Z1N0RYcUVsK2VCbFE2RHdZOHB3Vml2ZzJuR1Mw?= =?utf-8?B?WFVINU1rclM2c28rS1JRSUtWaWZvNW9SQkxySkRId0RyQnhQa3N4ZVZHMFFZ?= =?utf-8?B?ZVRqNHUweXpkVS8xRU14d3hOQnRWNC9xdEQ4UXZrSk96OXMxb3lSQ3dORG9F?= =?utf-8?B?NlpOeE9HR0x3REpQU2tLdjRLSkdIM1oyUTNsaTFkUGhsQTZWVFhDcG81RzJs?= =?utf-8?B?MmxkbG14aTNlU0taWjFybVJCUXpud05BVDZKamtubkw2anlXeGdvWjZTUXRa?= =?utf-8?B?Q01NcFpXN25UZ3pqT21naW5YN015NStxK3N0SWtNT01hTml2dkttTTE3dlhn?= =?utf-8?B?NktxekJXVUQ0ZUlBeGRJL0syZFhWWEpqRUVVeXNpVmNudmxzRkpEa0RXa2cx?= =?utf-8?B?d2lFa2JXaVBVd3FSZGE2WVJjQ2VyWU9DR2JOdldkc1J3THI1OVBqR3dBdlM4?= =?utf-8?B?T1JtVHRPVXhFYktoMWxBRmJLZVpJM1RNMUd5TnBDL2pMN2dQYTdqRXNwREZp?= =?utf-8?B?NXp4YWkvSzArL1BYc1N4WFA5bGFZVk9qL3djSXZha2FxcGxwSzVnblFGWEQ5?= =?utf-8?B?RXN1bnJhNW5lR0hLSTdwV09HWndDb2FpOENqb0hNK3FCQXI1WWRiTFYzNDRX?= =?utf-8?B?Y0ZWMFZOTVEreUx0ZUtOSlQ4azV6YkVtbUVEOWFDaXF1Sk80aGhmZDhyTnow?= =?utf-8?B?ZjVucDlJTzBjNjdMM29XTWdkTmN1ekdxT0tXak14VThjYlc5UEptN0dqSlRl?= =?utf-8?B?bWE4dDB5TXFMdERvZVJsRmJhemxRQklxRzIyVWd3aWNCLzd0b0s3YUltdkRB?= =?utf-8?B?Wk91Tm9MaGdQMktwcSt2NXYySkJ2QzdlZ29obWh3c0cxRkxJaU5RVk1nMEFX?= =?utf-8?B?Y1FtdlB6czZ1T2IzNG9lNmdWNGFXK0NLZ1JZQk9LT0VMZGhzZDhReUx2aXJi?= =?utf-8?B?WE11Uyt4dXhhWGxVNElSNzdjUjBqV01PbDlYMlhVV1V0a05COWk5SVR4Tm16?= =?utf-8?B?NjBFc21qbzEwVEVqN1U3WXNtZTh6MXZSaGhnRjRac3lEeURvcEZXV3pKYnBH?= =?utf-8?B?eVEzVjlLQytaS2NyVDBSUXRMb0t6ZXFSalhMSnFVN25GTVpzM3hFNXduRWxL?= =?utf-8?B?SGo3SmtudHRmYjA4bUtqdllFSXNKVVFHZ3VxOXNqSjhHSi8zNTFNVkdmMk5w?= =?utf-8?B?L1g4Uk0xb3Fsc2VoNDN0U3llaFd2c1ZjejhIaEsvWjc0TGl0OUdtMDkrV2I1?= =?utf-8?B?V3lnd0UzM29JVUt5N2NGZGRYVGZpL3NPUzkyMHVJRjBaWjk2NjFoT1lGM2lZ?= =?utf-8?B?cVVsM2wrZWZaQTY0YjNiZThJaVFKZW1MY0JxcDVITmY3N1JCYUFxZVUzRnk1?= =?utf-8?B?ZDc4VHpiZ1lLM0NEOUkyaGFOcDlITE5pS0lVaVZkakljR0JTMDY4QVlwT01p?= =?utf-8?B?UmpiT0IyZFJFYXVjOC9VUHoxUk9xbnpqeEJKWVI5NHFoeEh4ODZiQ05pRlQz?= =?utf-8?B?UjZlOFkrWkRMaDgxZW5Ob1lWR2xLVGFuK2k3cEd4aXVtNVQ4SVRHbWh0YytJ?= =?utf-8?B?cDR1anNQaEZlYUlwL0dKOUVpM2Z0T0VMUVpBWHc1dWZ6TEpZeTExc3V0T2xR?= =?utf-8?B?WDNoc0NHbmJvN2xnZHVleVFhZUFHb2ZzT3k2R1NzZHlSbDhhYUhIemNIcEdB?= =?utf-8?B?RE5IemlEdTNmU0hiOU00M3pqaGwwZzRodS9IcDMxMW9qbE1yeFljZ3BGVS8y?= =?utf-8?B?MFQ2d01rQ0JHY3o1ZkhvNFdHYzBpZWRpSm5HMmFENGd3c1E1blh0TkNiK2ZW?= =?utf-8?B?V2NpWVp1MU5PL2FBYlV0b1BXSStWNGptL1JEcVNqN2xnUUxuOFZFVWxMa3p2?= =?utf-8?B?QVRhZnIvOTNRTGR1TnRObGVXZHZ2bDlBUFljc1M4dUs2eDNoUEV1Y2tkeVNy?= =?utf-8?B?YkZNcU1QNFYvdjFqQzU1SXVwenZJVGFhdDBRcS90MUxST1B6NFJjQmcyb3VL?= =?utf-8?B?dU1IREI3ZXNqVU9JOTdSdU82elI2L0lNOXgvV2QrR1lsZ2dXOUxWL2NzWmR3?= =?utf-8?B?cHJzak0vTnJ3RnFNUnF5OTcxNTJLdmRIQTBtcWxLWUJ4NDJ1VFBLL1dFd2s0?= =?utf-8?B?L3c9PQ==?= X-OriginatorOrg: t2data.com X-MS-Exchange-CrossTenant-Network-Message-Id: f57a793c-d73a-4c56-b398-08dac182b2f7 X-MS-Exchange-CrossTenant-AuthSource: DB9P251MB0618.EURP251.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Nov 2022 12:13:51.9614 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 27928da5-aacd-4ba1-9566-c748a6863e6c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tZ3aaCRfs5+qL5bT2uiz6OLCQXfGGQPAKXpRqt8Yy947HCKTPo5Dr0RtuV1KY64wcNAaxAQ9uGFKWRLb2igmnQz0rLXiBjgTAZKmewHfGHI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB4P251MB0902 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on metis.ext.pengutronix.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=4.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2, SPF_HELO_PASS,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Subject: Re: [ptxdist] Speed up targetinstall of certain packages X-BeenThere: ptxdist@pengutronix.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: PTXdist Development Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ptxdist@pengutronix.de Cc: ptxdist@pengutronix.de Sender: "ptxdist" X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: ptxdist-bounces@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false So I did an experiment with bash loadables (builtins). Turns out bash has a bunch of builtin examples. I had ptxdist build me a bash-5.2 for x86_64. With the "loadables" make target. $ diff -urN ../lib/ptxdist-2022.11.0/rules/bash.make rules/bash.make --- ../lib/ptxdist-2022.11.0/rules/bash.make 2022-10-21 16:54:50.000000000 +0200 +++ rules/bash.make 2022-11-08 12:51:38.609794625 +0100 @@ -77,6 +77,16 @@ --$(call ptx/wwo, PTXCONF_BASH_CURSES)-curses # ---------------------------------------------------------------------------- +# Compile +# ---------------------------------------------------------------------------- + +$(STATEDIR)/bash.compile: + @$(call targetinfo) + @$(call world/compile, BASH) + @$(call compile, BASH, loadables) + @$(call touch) + +# ---------------------------------------------------------------------------- Since mkdir is slightly less than half of the external binary calls I had it replaced. I copied the scripts/lib/ptxd_make_xpkg_pkg.sh to my build path and created a shell-loadables directory there. $ cp platform-secplatform-x86_64/build-target/bash-5.2/examples/loadables/mkdir scripts/lib/shell-loadables/ $ ls -1 scripts/lib/shell-loadables/ mkdir $ diff -urN ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh scripts/lib/ptxd_make_xpkg_pkg.sh --- ../lib/ptxdist-2022.11.0/scripts/lib/ptxd_make_xpkg_pkg.sh 2022-10-21 16:54:50.000000000 +0200 +++ scripts/lib/ptxd_make_xpkg_pkg.sh 2022-11-08 13:08:18.074550829 +0100 @@ -682,6 +682,8 @@ ptxd_install_file() { local cmd="file" + #export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x; + enable -f scripts/lib/shell-loadables/mkdir mkdir ptxd_install_file_impl "$@" || ptxd_install_error "install_file failed!" } without mkdir builtin: finished target timezone.targetinstall.post real 0m17,043s user 0m13,522s sys 0m4,138s with mkdir builtin: finished target timezone.targetinstall.post real 0m12,345s user 0m9,899s sys 0m2,916s On 11/8/22 11:13 AM, Christian Melki wrote: > > > On 11/4/22 8:12 PM, Alexander Dahl wrote: >> Hello Christian, >> >> Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian: >>> Hello, >>> >>> I have question regarding the speedup of daily work. >>> >>> We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with >>> >>> @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no) >>> >>> Compile and install stages are empty. The just call targetinfo and touch to skip the stages. >>> >>> The frontend depends on the backend and the backend obviously depends on our application which is called by the backend. >>> So our web-frontend.in file looks like this: >>> ## SECTION=project_specific >>> >>> config WEB_FRONTEND >>>     bool >>>     select APP_LAYER >>>     select WEB_BACKEND >>>     prompt "e-mode Web Frontend" >>>     help >>> >>> As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and Frontend are executed again. However this is only a Run-Time only dependency. It is a third-party archive and install_tree takes quite some time even on fast build hosts. Even it if is just a minute it is annoying to spend the time waiting during image creation. Trying to solve that i found "if RUNTIME"  für Run-Time only Dependencys in the documentation here: >>> >>> https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail >>> >>> So my expectation would be that if i change the webfrontend.in file like this: >>> >>> config WEB_FRONTEND >>>     bool >>>     select APP_LAYER    if RUNTIME >>>     select WEB_BACKEND   if RUNTIME >>>     prompt "e-mode Web Frontend" >>>     help >> >> That sounds reasonable and I would have done it the same. >> >>> The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect that the web-frontend is deployed earlier if no build dependency is given. >>> >>> Am i missing something, oder is the "if RUNTIME" Switch not working properly in my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages? >> >> Not sure how that should behave.  However if you want to speed up the >> build: make sure you call ptxdist with -q or --quiet parameter.  The >> output on screen takes suprisingly much time, even with modern >> terminals, and especially when doing targetinstall of many many files >> (as usually the case with web frontends. been there, done that.) >> >> Greets >> Alex >> > > I have a slight disagreement here. I don't think the console is slow. > So I did some investigation, mostly since the slowness bugs me too. > I did a: > export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x; > as time measurement and trace in the shellscript in question (mostly scripts/lib/ptxd_make_xpkg_pkg.sh). > This pretty linear progressive cpu consumption, albeit the big chunks were due to calling of external binaries. > > Then I did a timezone package clean and targetinstall. > When targetinstalling a zone there was calls to several binaries. > > For one zone I had external calls to (in order): > echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir, install, install, chmod, chmod, chown, echo. > > Some of these could be internal builtins, but the consumed time suggested otherwise. > Either way. Each install took about 26 ms and I could account the majority of that time in forking external programs and waiting for them to return. > > So my conclusion is: The whole thing is a bit slow and bash doesn't help. > > Regards, > Christian > >>> >>> Thanks for any feedback. >>> BR, >>> Christian >> >