From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Tue, 08 Nov 2022 11:17:32 +0100 Received: from metis.ext.pengutronix.de ([2001:67c:670:201:290:27ff:fe1d:cc33]) by lore.white.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1osLff-001HGs-QH for lore@lore.pengutronix.de; Tue, 08 Nov 2022 11:17:32 +0100 Received: from localhost ([127.0.0.1] helo=metis.ext.pengutronix.de) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1osLff-0005km-45; Tue, 08 Nov 2022 11:17:31 +0100 Received: from mail-am6eur05on2077.outbound.protection.outlook.com ([40.107.22.77] helo=EUR05-AM6-obe.outbound.protection.outlook.com) by metis.ext.pengutronix.de with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1osLfF-0005kZ-O7; Tue, 08 Nov 2022 11:17:06 +0100 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gcWmEdqbVB9KgDGbSjctQX+MRRfG2ZsNjhL9JK+A5OBhQ7CMq2+1c7Hu4ky+nHyr71lm4E4nWdKopMAeCaI/YeLzXn7VS9vJ/WFlb6xuw2BZSf9nmwuNdA1FYcU4xc5adjQKI7Ifz3ch+w/Q2B49VI/DGYlEkX4E9ybzX1xM97xKLX7+qPPtBzRLVqtPd4QCL7sKDY1CqSxy1HcwrmUWi/UhloSNzdde7USza+xjBtnIhORlql1wAUBBAkJc41se/e+mK2xuYvR8rbg8rV5J+gPjzVyWi0A4jzgZwPd4q6SDSRwin99D+aICXae3MNA1IGx9R0CssiIU1fHYxwT4WQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=79gcKhe/q9yxlUgA+dHzBZRP536kfXbxc4H4Ykkg4OU=; b=YKWFJLzqS5BtsVw8enkucjNMdRyHXV2m7WKfmVaPRWVPag7pJKhT4wsHaQWg1JmDnC6szEjD/lcnV/D5UNZYhuQOemyzijyDZf58qclkp9uC4z6Zw+9pu5bo8wdMivH7UzRxCp+7YwbFAXLmqwwnI6OHK9OoGv+ixUMwZOXnEYnqNIOwO9QvtYA51Btx6OZb8mMnuEdhRG+ZrfDXZaDSwAcJmdPrjhIvduVvFUQK4Ealc4UAP7bgcB7mZKZlG9+MJiCId58yW0RETq3UCdGFLCrr80YO0r7Qser0y30NXe8Upic4EEUo78SRIH0kjti6eyxYZD3Idw/KMKqvLCP6BQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=t2data.com; dmarc=pass action=none header.from=t2data.com; dkim=pass header.d=t2data.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=t2datacom.onmicrosoft.com; s=selector1-t2datacom-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=79gcKhe/q9yxlUgA+dHzBZRP536kfXbxc4H4Ykkg4OU=; b=Bi94EhqhBLDhMbiBkC/OXXqgu9WLNfPxr/jI5QoCez1hTUXgGWpLbcMgUUDpTjVx18e1SWcQH1lQSaZpYdJrdtQc6TzaCTnwXBk2FL7yPBsV1bTr1PBwyYRn0qiDO4GJn+6wwwo5l8RQlH9fsqYvyJ/ep4VF5BM8Dcm6JnqWKZM= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=t2data.com; Received: from DB9P251MB0618.EURP251.PROD.OUTLOOK.COM (2603:10a6:10:334::22) by AS1P251MB0557.EURP251.PROD.OUTLOOK.COM (2603:10a6:20b:4af::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5791.26; Tue, 8 Nov 2022 10:17:03 +0000 Received: from DB9P251MB0618.EURP251.PROD.OUTLOOK.COM ([fe80::fafd:ebc3:4f2d:f330]) by DB9P251MB0618.EURP251.PROD.OUTLOOK.COM ([fe80::fafd:ebc3:4f2d:f330%7]) with mapi id 15.20.5791.027; Tue, 8 Nov 2022 10:17:03 +0000 Message-ID: <2aa3b38a-9025-d88e-8364-fea4ada10f3a@t2data.com> Date: Tue, 8 Nov 2022 11:13:44 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.1 Content-Language: en-US To: post@lespocky.de, CG@eks-engel.de, Michael Olbrich References: <75f0b7f568884d33af8f5c6bc23bd5dd@eks-engel.de> From: Christian Melki In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: GV3P280CA0056.SWEP280.PROD.OUTLOOK.COM (2603:10a6:150:9::10) To DB9P251MB0618.EURP251.PROD.OUTLOOK.COM (2603:10a6:10:334::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB9P251MB0618:EE_|AS1P251MB0557:EE_ X-MS-Office365-Filtering-Correlation-Id: dea4200e-90d5-49d9-6f19-08dac17261bb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rrvBduXvxCBZpvK5iH/bZ/jZi904vleg7vXfkPJjpe63JJwOkPqX3T55lfnZx601EjN0M5SuWJYD0y6HJ2kAiEXmSTReybfRLtTf5hZalcqJT6xjHP0MQER2JcbidO/Ckh/778ZNDwV/pUbrH63CtEnpZlJxBL5OuUrc5ybijIMPS+PoJvKKBfCNWNnOJiF3M6bxjE8dsNP4xfBBNZY0JVpVqkBoZ6MoyO9Q1AZI1ZT4cMluIWl8f9ldaLwOmQ8p3F/uRJpPWErrxBUC8qh7FhoTrkDUFDjPE3QEuyDK5C0sdX+8vpGoqsJmoIdTMwTbYJenZ1CjbjWMAjBCYIEVw5gOddgJzTZgldVGRTsyUUiZVsdaq+OaRLq3HP6nhWulFGRnvRFxVp9eOrq+93uD3x6NrhX87NKVpxHAv/fMQHJKEnvimuz8eGVXFp3pw+AkPEUt5aJ/gWB4CAzM2bZ0P270q0yWteMw2IiBQ3sW16HHcMLghKEZMsUz0t/04D559HFAk5KHYYlwk1qjA90MQOhDlc0dQSvwH4sd/7mYM06DsduiIHyQMSosQZ/8uWTevP9cHRMRBHQuPSqTMGFI4cj28PappPI9W2C9et9MvCi3FZpHzvBYhMunajZgkYL7VgS9Dy7fw8wnOU1kTkmvR0cxCTOSpMCArUetVfevfeMY0W3wJSrY9vImp+yNDMYa1e5OfcwmoMEJthBYr8MShM3IwS69+BAM43dcd+8hdmfc/mEWp/4lMLr58aGcpyC1ap8aRVsHITMlZ/Az5jYCCZ4ucMSBHNaDwyZ+2ggfDoT+RvwPBAs1AOavxJJtTO+u X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DB9P251MB0618.EURP251.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(13230022)(346002)(366004)(396003)(136003)(376002)(39830400003)(451199015)(36756003)(86362001)(31686004)(31696002)(6666004)(2906002)(44832011)(2616005)(66574015)(26005)(186003)(53546011)(6512007)(83380400001)(6506007)(66556008)(4326008)(66476007)(8676002)(66946007)(8936002)(316002)(478600001)(966005)(6486002)(41300700001)(38100700002)(5660300002)(6916009)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YjhIZjZueDNDVnBrUUY5ZUgrQ25qYzdvK2F1QjZaejNRNTVpcFA0QlpvU0Y0?= =?utf-8?B?aURmU2JDeFdYNTNtUUNDZzdIR1RuWVkyOWtGNndpcjJmTUVIZm92S2NFc2Vn?= =?utf-8?B?NUpoWDAvbFJFRnFaZXhLSDMxUDFBZDNuazlRbG12Q1dvRXhUeXlXcEhkMnBa?= =?utf-8?B?U2dqMUVqbFIyRk1BSVFKL3FIS0N1SW1taWlGOGcxbWFkenQ2ckVDTHdpeHZS?= =?utf-8?B?TUh5WTBoUG5kWEdhNHlQdW5zeFdPajlyVExyeTBkaXR1UGg0eTZ5WHBUNEto?= =?utf-8?B?dlI1cENUWFFLWDdOY0RqeE5wOHlmV3orWXlhYk9NanhUOXErZTgwR05ldFpu?= =?utf-8?B?UXVZczJOaHk3amVNY2tkSjZJVW9ObDlaaFcwejlCcjk5Z0owTW1uQ3VHTW83?= =?utf-8?B?TnhFb09mdFM0UGNpQlNpcGhjMC9GMWZEb1BGZzhXdXh2cUh1cFBNUGUySU82?= =?utf-8?B?cktkRXVmbzlCc3l4RFZIdXBGRGxFbVZCbzVzbkJQZTlDcUZGS3dFWTl5UVk5?= =?utf-8?B?MlZRbUJ5R2pQWGJJVTdIL3I0dURGc0Yzc296bG90dzZXeU5rYzZJWTA4RDRK?= =?utf-8?B?SWh4c3FrUlNabUhRanpWc3YvOGtha2NqQzhPaWtPL2xDSnJHMmN4cTgvbkxF?= =?utf-8?B?UnMwRk41ak9PR3RHazh3T1ZzUlNTL0ZoSnRobDV1SjBXbzJod3NVZjl3K2dG?= =?utf-8?B?SmlhMkNkcW55QnA5YlMzTFZ1V0VSYnJvRFYzcWtCWGUzNzR3N1Y5L2s5d1hv?= =?utf-8?B?NHhhaFN4UnRPNXVPR21HNGU2QS9DSFFnOEIzcUdsRFNUenpSQ1poa3F6c2Vx?= =?utf-8?B?a0tQUzZnZFUyZFVmL2UyeTMzT25KWnFDOUxrNndpWWNtS2JDMWRnQ2xuWk1Z?= =?utf-8?B?NGkzbFpEdXErbmNidzlNQlFGZlNLTGNHbjJEdGNUWnlXcExlU3NoZzFiSlo3?= =?utf-8?B?cW8xVnBWTUVDbFo3M0pvYzQ0YmNMUkNmVU9zOEQwcjQvU3FEU3I5RnpNbWFn?= =?utf-8?B?T0Q1eGNMOTF5OXZlcGhRYUhrTFFoakZBc1FHbVZRTEZxLzF3Mml6VHQ1SXho?= =?utf-8?B?WTRoVHUxeCszenpwTDQ4aklCODdkWk9XdHhhTGwrRkhWMmRRaEdTOFhrc3VF?= =?utf-8?B?VFhublEvZTlHeEFxbjIyRzhjVCszd296WGl0NmhkcXF0WHVJTytFd3dSb29m?= =?utf-8?B?djFZYjVoNk93TFJpZVVyVkFlZ3loeDZDYlZVK2VBWTRmWVJobGpWcWh6Yk9Q?= =?utf-8?B?dWN2R1ZOMUp6Z3JNWVRKYWcrSHJFK1dNU1RSeUlTa0E1bE54QndKL0dKR3pJ?= =?utf-8?B?L3JGVWQ0WmxlemtQdnlDVDdQdGY5WG5HYWxNU1hod1BoVTlXTlA4RDNIdkcz?= =?utf-8?B?RlpHZWIxc3BIdXdBcXdRY2w5Z0FQMHVhTG1uSkkwNE42T1NER2pBWkozMTFy?= =?utf-8?B?emhYalhvSS9DajNFWnhTK0swWXJSQnJMWVROZktqeXVONC8xcFBNUm5kTmF6?= =?utf-8?B?Vmk5VE12V2h1OFhSUVpOZUNjK3JvQkJuU00ybVRCK0ljdko3QmRGU2wvUUE0?= =?utf-8?B?OFNhZERWbDAvVXN4QTdPRm8vQnpQcExHNElnVWVqWUFwdHZXUjFVTkFxRWxX?= =?utf-8?B?MWdoTE9DaFRqbGxPNVFxVEwvYmIwSHI1NGVuVkkvY25Bd1drNFowcWpQZXln?= =?utf-8?B?OW9uL015Ym0zS29rSVlvS3UzNjVNN3g4cU84N3YzWkxXWVc1bXhsSWRBVmVv?= =?utf-8?B?TVdUWnVoSUVNMGFLSDhnT2swYUNBWDFxL1dzSWRrcnlnVzFDcXkzZG9YUER6?= =?utf-8?B?dHlqeVRqRGhuTCtpTDQrQ2U5VnAzYU5VT0ZoVXVPLzJjQW9HQjdRU0tjWWU2?= =?utf-8?B?ZTZxMkIrTkwvWkpWa3c2SUx2bGFTMEhWZmRPZGlzMlRyVTUrUWREV0FIdnpR?= =?utf-8?B?cEVpblA0aWV3SjhBeUlWYUxNMUxGR2toa1RjdGlxT1Y2aTZrSE1BMnE2Zkth?= =?utf-8?B?Y09oZi9iclUwa25ZUWtoeEl3YUFYNU1WbEE3VzJ4cGhGalpIdUx6c1krWlJz?= =?utf-8?B?Zlc0bGs2WSt5OVBaeE5LTHRvczlkWUZZVHpFZS91UWxsSThZOS9iZWF0VWdO?= =?utf-8?B?bG9BOSsyUytoeWJ6dFpLalNnVUQrOUp5QUFPZXc5eUpuMlh5aks0WEJMaUho?= =?utf-8?B?K0E9PQ==?= X-OriginatorOrg: t2data.com X-MS-Exchange-CrossTenant-Network-Message-Id: dea4200e-90d5-49d9-6f19-08dac17261bb X-MS-Exchange-CrossTenant-AuthSource: DB9P251MB0618.EURP251.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Nov 2022 10:17:03.7701 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 27928da5-aacd-4ba1-9566-c748a6863e6c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JbSyG9ORln1Qbl1ew6ECT/49NeNE5/fAiON6s2fhn2h50UWgt2myDIQMixuLXOJutvrcU8BWUpIwoMQeLfoIWm5RgWbkFP+J2df33913ArA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS1P251MB0557 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on metis.ext.pengutronix.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=4.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2, SPF_HELO_PASS,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Subject: Re: [ptxdist] Speed up targetinstall of certain packages X-BeenThere: ptxdist@pengutronix.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: PTXdist Development Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ptxdist@pengutronix.de Cc: ptxdist@pengutronix.de Sender: "ptxdist" X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: ptxdist-bounces@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false On 11/4/22 8:12 PM, Alexander Dahl wrote: > Hello Christian, > > Am Fri, Nov 04, 2022 at 03:37:02PM +0000 schrieb Gieseler, Christian: >> Hello, >> >> I have question regarding the speedup of daily work. >> >> We have frontend and backend of our webgui deployed with separate packages. Only task of these package is to deploy the files with >> >> @$(call install_tree, web-frontend, -, -, $(WEB_FRONTEND_DIR)/var-www/, /var/www/,no) >> >> Compile and install stages are empty. The just call targetinfo and touch to skip the stages. >> >> The frontend depends on the backend and the backend obviously depends on our application which is called by the backend. >> So our web-frontend.in file looks like this: >> ## SECTION=project_specific >> >> config WEB_FRONTEND >> bool >> select APP_LAYER >> select WEB_BACKEND >> prompt "e-mode Web Frontend" >> help >> >> As expected if i clean and compile APP_LAYER the targetinstallstage of Backend and Frontend are executed again. However this is only a Run-Time only dependency. It is a third-party archive and install_tree takes quite some time even on fast build hosts. Even it if is just a minute it is annoying to spend the time waiting during image creation. Trying to solve that i found "if RUNTIME" für Run-Time only Dependencys in the documentation here: >> >> https://www.ptxdist.org/doc/daily_work_section.html#controlling-package-dependencies-in-more-detail >> >> So my expectation would be that if i change the webfrontend.in file like this: >> >> config WEB_FRONTEND >> bool >> select APP_LAYER if RUNTIME >> select WEB_BACKEND if RUNTIME >> prompt "e-mode Web Frontend" >> help > > That sounds reasonable and I would have done it the same. > >> The "if RUNTIME" would make sure that the targetinstall stage is not executed again if i just execute a "ptxdist clean app-layer" followed by a "ptxdist images". Same with ptxdist clean root; ptxdist images. It is clear that all targetinstall stages are executed again, but i would expect that the web-frontend is deployed earlier if no build dependency is given. >> >> Am i missing something, oder is the "if RUNTIME" Switch not working properly in my ptxdist-2018.12 version? Or does it have no effect on targetinstall stages? > > Not sure how that should behave. However if you want to speed up the > build: make sure you call ptxdist with -q or --quiet parameter. The > output on screen takes suprisingly much time, even with modern > terminals, and especially when doing targetinstall of many many files > (as usually the case with web frontends. been there, done that.) > > Greets > Alex > I have a slight disagreement here. I don't think the console is slow. So I did some investigation, mostly since the slowness bugs me too. I did a: export PS4='+[${EPOCHREALTIME}][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'; set -x; as time measurement and trace in the shellscript in question (mostly scripts/lib/ptxd_make_xpkg_pkg.sh). This pretty linear progressive cpu consumption, albeit the big chunks were due to calling of external binaries. Then I did a timezone package clean and targetinstall. When targetinstalling a zone there was calls to several binaries. For one zone I had external calls to (in order): echo, mkdir, printf(?), flock, ls, rm, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, flock, mkdir, mkdir, mkdir, mkdir, mkdir, install, install, chmod, chmod, chown, echo. Some of these could be internal builtins, but the consumed time suggested otherwise. Either way. Each install took about 26 ms and I could account the majority of that time in forking external programs and waiting for them to return. So my conclusion is: The whole thing is a bit slow and bash doesn't help. Regards, Christian >> >> Thanks for any feedback. >> BR, >> Christian >