From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Fri, 13 May 2022 15:46:50 +0200 Received: from metis.ext.pengutronix.de ([2001:67c:670:201:290:27ff:fe1d:cc33]) by lore.white.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1npVd4-00BzzC-MD for lore@lore.pengutronix.de; Fri, 13 May 2022 15:46:50 +0200 Received: from localhost ([127.0.0.1] helo=metis.ext.pengutronix.de) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1npVd4-0007qF-19; Fri, 13 May 2022 15:46:50 +0200 Received: from mail-eopbgr70053.outbound.protection.outlook.com ([40.107.7.53] helo=EUR04-HE1-obe.outbound.protection.outlook.com) by metis.ext.pengutronix.de with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1npVck-0007q6-Ur for ptxdist@pengutronix.de; Fri, 13 May 2022 15:46:36 +0200 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jbynThkih/4wuqdtPJP5XDywZ3attWOp0uDRg933UWGQzABk4aqsKGaI2vIH4c/fEbXEGmeCTw+kU073ySCTXWyVtDY9JP5kqQTmrVphEgKoQw24Rsa0pja9v7kqZ6OjsOtrBwmv8KSMy+a5LTHbM8DfCKPZrYoQspEFbLO79HTO/6LCzbz6EqyN0yipMV38d6fYVcA14kYYng9g26mkUpqHVCxh48ZbkOtzYFL+lBnXG8UfQ7mfy2PBgQb/i3kOuTKvFvkdu5J6cbvzKmX77ZZ1wNiMsS/iDNTTP9bOXb7rkZ1VRBa6g6k620ppo95vw7iCcd3X0fAOAl9dsFHVOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UbaKAdsqKON6S5gngVc7RjBgo1B17gQhLXbXZ6xmdjc=; b=c8SYumuFK4Af8z1BL4o3LEKKvvndxc9XlkR3jMT+RQYZbKuKxZKXieGh8qlL7c4PdbMVq3L2TcTJwIqU92WdCOYLfVUI2WLzZ0IIshWEzpalbOczrORkzaOWPBik5hunQhj98eQjQgHWmQZG2uFzAkYrk11ibD2hGC2C8hFeRt/GUDXtUEaedzbVVWUMu35Eg5wl+NftN8f0p/KJr9c+2nXxuaZQ5uCXOIt2mZCakxWId/2lpbYXBNv3gU8lSW63mfxIagc8neBEXcl0DRPQc1pJdUID7ArYOsxPvjyBivICXiWzfPpC4euPEuTGZ1Pi2lP3UjKx6vzv6kNWhxNWtg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=t2data.com; dmarc=pass action=none header.from=t2data.com; dkim=pass header.d=t2data.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=t2datacom.onmicrosoft.com; s=selector1-t2datacom-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UbaKAdsqKON6S5gngVc7RjBgo1B17gQhLXbXZ6xmdjc=; b=ft057zsuzlRL8E8pG/GdKIZNtpyZkNfQSP9ruL/qbc5EnQoQ1TbR+M1qit9+TO3hS4EIHSIiSQn2WYYw7HuSHU3w9E0C6+4eoM1MNQuIkAYE1E/sDK/soFXCxkwH5Xt8MqvIZqH/Wd+TqAZasgwoH4NC63phCraQvm5ouHHESHk= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=t2data.com; Received: from DB9P251MB0618.EURP251.PROD.OUTLOOK.COM (2603:10a6:10:334::22) by DB4P251MB0952.EURP251.PROD.OUTLOOK.COM (2603:10a6:10:38c::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5250.14; Fri, 13 May 2022 13:46:28 +0000 Received: from DB9P251MB0618.EURP251.PROD.OUTLOOK.COM ([fe80::1d7f:19a9:18f9:57af]) by DB9P251MB0618.EURP251.PROD.OUTLOOK.COM ([fe80::1d7f:19a9:18f9:57af%8]) with mapi id 15.20.5250.014; Fri, 13 May 2022 13:46:28 +0000 From: Christian Melki To: ptxdist@pengutronix.de Date: Fri, 13 May 2022 15:46:21 +0200 Message-Id: <20220513134621.1963504-1-christian.melki@t2data.com> X-Mailer: git-send-email 2.34.1 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: GV3P280CA0001.SWEP280.PROD.OUTLOOK.COM (2603:10a6:150:b::31) To DB9P251MB0618.EURP251.PROD.OUTLOOK.COM (2603:10a6:10:334::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3c46c67e-4291-40f3-019f-08da34e6faa0 X-MS-TrafficTypeDiagnostic: DB4P251MB0952:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IZxXcKFIVSsWCNwEUTd2Mvum2WPIXLj3JRIJxxc8qjOlGiAoBID9ujTTuhEHDPQnEvrdK9hoCcObjf/q4MwNpMzcmA/qmblwyPzMdNbgyd6rZ05OyS2O1aNOYlpgBDAKZtYVP6IjJrLqyvHZXCKfGGpU7IPO5thzwrz9oFv0sxximUYHFjzeWzKe90/CWyZIauYkTi0egRag5Ga0RgeN0QzYEbfgOYdbdIRhEnVV9TjOfh/zkAuaS5qC5XOYj+UrXMQCqEUchSd/UEppId+A8Agd53mGtSs8ocISFNpiwjsZchzaf9GiVXm/2xve7FD7mAPzwmsXUcvdRkSv5pHkuNmov4BOi/74shgqL2JqPPqVfJiIJfyyRKX/MFCZyttJmGhCydeXBYCgMogVL58EYAPH3I+AuOBu9wgVrnhYl9b+N1CLGa90By2DaGpcFd59DeoKWq1N06tORYIgG1Wzyd3Z2D7ax0cr0Hek8qHd3rL+Rx+0etQGfdggrlsGC1sBuQ0Rs6eBF0vv2xFwOaQJpVZZIxgj/kFvt6EZhAd8BCk6wBHEutfYKB7tAaYPdKJT3Qqy5XZsfziR1tcdkHfNS/kaz1fK9NF9rjsgln7le7525kQnvxsr/VCkcVO74YD1I0ywJqohhL+LzQ71pMaKICAfy6ep06gz/+VKt4mNdraMhQJRu9ic2Dr+ZPAIQJfwWQl6CLPIwwQeX2rUdGy20FD6tFiyM/J3x23VXtQkfneaG8eQHFpzUPIX5nFnW0ZahNm5eu88be+vfR8Zg+l0iA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DB9P251MB0618.EURP251.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(13230001)(346002)(396003)(366004)(376002)(39830400003)(136003)(44832011)(36756003)(30864003)(8676002)(6666004)(6916009)(66946007)(66476007)(66556008)(6486002)(52116002)(316002)(83380400001)(8936002)(508600001)(966005)(2906002)(38350700002)(38100700002)(19627235002)(186003)(5660300002)(26005)(1076003)(2616005)(6512007)(86362001)(6506007)(559001)(579004); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?9fDSndNqLrcPB1OxI5NhvtRDJvDJWkGyYHriREP2z7dE0YciJex6Hgx/3MpX?= =?us-ascii?Q?OjppFGcmMpRyTjFaCdPT6ura9pviiOQ+qWUy1IeEX/7IZvh0KjwGPZnE5uuY?= =?us-ascii?Q?mJXfUHQyNkiiXKsmPOgQhZcfV0XiK+CGZKGJwRyxhMru7rKXT9GCan6PFuCz?= =?us-ascii?Q?YU2ORPgzSgd3sVR5u9rpg0rEWIzxZq8ciLB8bu5tapQnQRw1EFNSrgTFmzzD?= =?us-ascii?Q?d7h/vQHnD1NPpDAsQWDKS++62SnoNnSvsbnjDDLsYIBLXCv5uTUYNVFntC50?= =?us-ascii?Q?426H75Xi/szCWN4pyiGvpStDb7kYq75FgeRPyoKdkG7k+RHzGWCoTmOE4/eU?= =?us-ascii?Q?5iqQ8ddoJLTdaTEn6xShFrqf9W7Q/NmokO+bTUEiZ4gasUmvUlfdyRJaXbjI?= =?us-ascii?Q?n/KvMJ1fn6/S4t1xM7Wz+He2q26kaE3tZa4yDmZ4GBzvbN5GOjZ4jz/ymtjA?= =?us-ascii?Q?e6ZUyu1iy1GHH4BN9xWYR2EiiS15FuxZ2HTQDQXMvzuasgo2F2NNPn9Zu0l3?= =?us-ascii?Q?bjTGfLrNcyaFsaSirUz/7vs4u3y5JZqavrBiyl9bGmnFBvE1LSl86bw5TkAm?= =?us-ascii?Q?tLk9/q3VCRcOAvpWJ/+YU04Hy29FfUfIcrdG+q0FQL/kP9INQBMVR2HATF89?= =?us-ascii?Q?ociiaecNU5V/cI/gpoReijpMiwFVb7xS84GE9wqtZStG06kvGvs/k8qVRPZh?= =?us-ascii?Q?qFNA9HbHcReb8gkQqdgcebbyluvNSBee/JtNkU8GxuRo11bpjlg3HXEIELkB?= =?us-ascii?Q?rKGpG2hqtIz7ErDLLq9QMxXQlkrT7JZtR7oaXieZS9r4nBEWaKuW/TBKVTK2?= =?us-ascii?Q?xuBP8M0/X4l2uRJgJK0OD7bqZMeyoeiBvhhqgBH7g89V7b3p6AsMfRRih+Mt?= =?us-ascii?Q?wEiLBN2Ir3bOkWyTkf4kv7z8gTEJ/utzNliJ9Q5RoI9IsOTPmjMllsizo5A5?= =?us-ascii?Q?0yJEV/gbV1UonsJILvv0LWVS6uVH4n4O5tKQGR/lc2yoZUtQdUB77Raixw2V?= =?us-ascii?Q?VhlTnANSFUTMCL8/59+lIZG8Bl7WTBlCYFrHsnM3UL5lm+Wdu2mmhxwEPqXm?= =?us-ascii?Q?V7ltUSmSfswXsk4WQhFNu6DOWOB/UbKY9lX44lgbEuie+Z03d/MQpwyWdfNs?= =?us-ascii?Q?iZI5daUPrU2zJmaWTGrZvwaxUFqLunZpRn4oz78fBHVdyoy7OB+mfFTMQSUi?= =?us-ascii?Q?PENDO9fBWPjJIi42xtf2RlsdGiHJzjfH9NMVT+v7qEQOHmAXWicuVx93nJ2u?= =?us-ascii?Q?hsFICy038UqLjM2Y8kzMejlM3nTg/UUspxsyow2yem/cg7X6tfS+40Liaixe?= =?us-ascii?Q?ggP05/62ojSzeq+GhKtecfX/qf4vX4hy/r8S0AeHNal7Z2bCpdK6OMkK8bhy?= =?us-ascii?Q?6gyyc9F/jbYEiuLSO/O+0Y9pyf4dPS2flMbm7bjPnwi3+stjtQdh6zHnd7Q6?= =?us-ascii?Q?3f00Sk0V5Tr1s2LIZ59CbpDpsR2ZlpM3Sdg5gFZRKTTRw25zikrMEjJr/d1G?= =?us-ascii?Q?znTZTWDGZuJMJ4qsmFBDzxltIUD2ePD2T3ASDaQyXOeK6nty5Zzi4NpCR08t?= =?us-ascii?Q?ttVJRWqW8SwcADgE7IGLKkB0+ws3DC0YXBAwtI19j0ooWfRK+y86wBsxZZPp?= =?us-ascii?Q?4JQhNqLxGOa/8g4m3TCD/FPCxd8I+oAANvc8BOeZhT96VG732FeSzVp50KiE?= =?us-ascii?Q?KhyYBCKUUQBqn5SqNhuHYPIkIzQ9ICj97b5ApzotBYx/L+u7tztqdcl3hCj6?= =?us-ascii?Q?HZYUgQwn7QTMx3xkz2s0bINPUwPDyqBeacp5GDsZNf3vWStAx2cq?= X-OriginatorOrg: t2data.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3c46c67e-4291-40f3-019f-08da34e6faa0 X-MS-Exchange-CrossTenant-AuthSource: DB9P251MB0618.EURP251.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 May 2022 13:46:28.0288 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 27928da5-aacd-4ba1-9566-c748a6863e6c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ymb0SpQdVEXTFH0qNrEPoHIg1wSnzN8sluDa5RwRehB15hHiGrmGqj0jAGgnn6rLv14kg5u9guXDAm0e64lQhPtLPDoNucmoWl0u5G5m36Q= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB4P251MB0952 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on metis.ext.pengutronix.de X-Spam-Level: X-Spam-Status: No, score=-2.1 required=4.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.2 Subject: [ptxdist] [PATCH] host-ninja: Version bump and fork change. 0.8.2 -> 1.10.2.g51db2.kitware.jobserver-1 X-BeenThere: ptxdist@pengutronix.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: PTXdist Development Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ptxdist@pengutronix.de Sender: "ptxdist" X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: ptxdist-bounces@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false Almost 5 years in the making. Lots of bugs and behavior has changed. Upstream release notes from v1.8.2 (mostly referrals to git changes). v1.9.0: https://groups.google.com/forum/#!topic/ninja-build/nY5Kb7zUvcg v1.10.0: https://groups.google.com/d/msg/ninja-build/piOltAhywFA/zPfkrTtRCwAJ v1.10.1: https://groups.google.com/d/msg/ninja-build/QQM54eAhrjU/q5zn_zTlAQAJ v1.10.2: https://groups.google.com/d/msg/ninja-build/oobwq_F0PpA/FeJC5LoRBgAJ Now follow in the path of others. Due to the popular but stalled jobserver feature addition (6 years), people has started forking ninja just for this feature. https://github.com/buildroot/buildroot/commit/227d7e0cbaaf093f509f5728f06fad5f53caed7b Do the same for ptxdist. * Point ninja URL to the kitware fork. * Upstream ninja now uses CMAKE. Drop python configuration. * Reflect the CMAKE / PYTHON changes in the in file. * Ninja is now default OOT. But lets keep it as it was for now. Disable OOT. * Drop the old patchset which was the jobserver beside the fix for ptxdist file descriptor handling. * Forward file descriptor handling to a new patchset. Signed-off-by: Christian Melki --- .../0001-don-t-close-open-fds.patch} | 6 +- .../series | 4 + ...dd-GNU-make-jobserver-client-support.patch | 478 --------------- ...l-monitoring-to-SubprocessSet-DoWork.patch | 554 ------------------ ...er-when-jN-is-forced-on-command-line.patch | 192 ------ .../0004-Honor-lN-from-MAKEFLAGS.patch | 128 ---- ...e-LinePrinter-for-TokenPool-messages.patch | 122 ---- .../0006-Prepare-PR-for-merging.patch | 151 ----- patches/ninja-1.8.2/series | 10 - rules/host-ninja.in | 2 +- rules/host-ninja.make | 22 +- 11 files changed, 19 insertions(+), 1650 deletions(-) rename patches/{ninja-1.8.2/0007-don-t-close-open-fds.patch => ninja-1.10.2.g51db2.kitware.jobserver-1/0001-don-t-close-open-fds.patch} (88%) create mode 100644 patches/ninja-1.10.2.g51db2.kitware.jobserver-1/series delete mode 100644 patches/ninja-1.8.2/0001-Add-GNU-make-jobserver-client-support.patch delete mode 100644 patches/ninja-1.8.2/0002-Add-TokenPool-monitoring-to-SubprocessSet-DoWork.patch delete mode 100644 patches/ninja-1.8.2/0003-Ignore-jobserver-when-jN-is-forced-on-command-line.patch delete mode 100644 patches/ninja-1.8.2/0004-Honor-lN-from-MAKEFLAGS.patch delete mode 100644 patches/ninja-1.8.2/0005-Use-LinePrinter-for-TokenPool-messages.patch delete mode 100644 patches/ninja-1.8.2/0006-Prepare-PR-for-merging.patch delete mode 100644 patches/ninja-1.8.2/series diff --git a/patches/ninja-1.8.2/0007-don-t-close-open-fds.patch b/patches/ninja-1.10.2.g51db2.kitware.jobserver-1/0001-don-t-close-open-fds.patch similarity index 88% rename from patches/ninja-1.8.2/0007-don-t-close-open-fds.patch rename to patches/ninja-1.10.2.g51db2.kitware.jobserver-1/0001-don-t-close-open-fds.patch index 8da54d4bd..a2dfc5b8e 100644 --- a/patches/ninja-1.8.2/0007-don-t-close-open-fds.patch +++ b/patches/ninja-1.10.2.g51db2.kitware.jobserver-1/0001-don-t-close-open-fds.patch @@ -10,10 +10,10 @@ Signed-off-by: Michael Olbrich 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/configure.py b/configure.py -index 41d95469c00d..d5e34a0eeb3a 100755 +index 57e1ca3ab369..e49802f4c75b 100755 --- a/configure.py +++ b/configure.py -@@ -186,7 +186,7 @@ class Bootstrap: +@@ -191,7 +191,7 @@ class Bootstrap: try: if self.verbose: print(cmdline) @@ -22,7 +22,7 @@ index 41d95469c00d..d5e34a0eeb3a 100755 except subprocess.CalledProcessError: print('when running: ', cmdline) raise -@@ -683,4 +683,4 @@ if options.bootstrap: +@@ -716,4 +716,4 @@ if options.bootstrap: if options.verbose: rebuild_args.append('-v') diff --git a/patches/ninja-1.10.2.g51db2.kitware.jobserver-1/series b/patches/ninja-1.10.2.g51db2.kitware.jobserver-1/series new file mode 100644 index 000000000..05bd05b30 --- /dev/null +++ b/patches/ninja-1.10.2.g51db2.kitware.jobserver-1/series @@ -0,0 +1,4 @@ +# generated by git-ptx-patches +#tag:base --start-number 1 +0001-don-t-close-open-fds.patch +# 9f9c2c8544d68ab2c7db95cac6cf30bc - git-ptx-patches magic diff --git a/patches/ninja-1.8.2/0001-Add-GNU-make-jobserver-client-support.patch b/patches/ninja-1.8.2/0001-Add-GNU-make-jobserver-client-support.patch deleted file mode 100644 index 5c23e036e..000000000 --- a/patches/ninja-1.8.2/0001-Add-GNU-make-jobserver-client-support.patch +++ /dev/null @@ -1,478 +0,0 @@ -From: Stefan Becker -Date: Tue, 22 Mar 2016 13:48:07 +0200 -Subject: [PATCH] Add GNU make jobserver client support - -- add new TokenPool interface -- GNU make implementation for TokenPool parses and verifies the magic - information from the MAKEFLAGS environment variable -- RealCommandRunner tries to acquire TokenPool - * if no token pool is available then there is no change in behaviour -- When a token pool is available then RealCommandRunner behaviour - changes as follows - * CanRunMore() only returns true if TokenPool::Acquire() returns true - * StartCommand() calls TokenPool::Reserve() - * WaitForCommand() calls TokenPool::Release() - -Documentation for GNU make jobserver - - http://make.mad-scientist.net/papers/jobserver-implementation/ - -Fixes https://github.com/ninja-build/ninja/issues/1139 ---- - configure.py | 2 + - src/build.cc | 59 ++++++++----- - src/build.h | 3 + - src/tokenpool-gnu-make.cc | 211 ++++++++++++++++++++++++++++++++++++++++++++++ - src/tokenpool-none.cc | 27 ++++++ - src/tokenpool.h | 26 ++++++ - 6 files changed, 308 insertions(+), 20 deletions(-) - create mode 100644 src/tokenpool-gnu-make.cc - create mode 100644 src/tokenpool-none.cc - create mode 100644 src/tokenpool.h - -diff --git a/configure.py b/configure.py -index a4437489426e..41d95469c00d 100755 ---- a/configure.py -+++ b/configure.py -@@ -499,6 +499,7 @@ for name in ['build', - objs += cxx(name) - if platform.is_windows(): - for name in ['subprocess-win32', -+ 'tokenpool-none', - 'includes_normalize-win32', - 'msvc_helper-win32', - 'msvc_helper_main-win32']: -@@ -508,6 +509,7 @@ if platform.is_windows(): - objs += cc('getopt') - else: - objs += cxx('subprocess-posix') -+ objs += cxx('tokenpool-gnu-make') - if platform.is_aix(): - objs += cc('getopt') - if platform.is_msvc(): -diff --git a/src/build.cc b/src/build.cc -index 61ef0e849add..cc796ff838fa 100644 ---- a/src/build.cc -+++ b/src/build.cc -@@ -38,6 +38,7 @@ - #include "graph.h" - #include "state.h" - #include "subprocess.h" -+#include "tokenpool.h" - #include "util.h" - - namespace { -@@ -347,7 +348,7 @@ bool Plan::AddSubTarget(Node* node, Node* dependent, string* err) { - } - - Edge* Plan::FindWork() { -- if (ready_.empty()) -+ if (!more_ready()) - return NULL; - set::iterator e = ready_.begin(); - Edge* edge = *e; -@@ -485,8 +486,8 @@ void Plan::Dump() { - } - - struct RealCommandRunner : public CommandRunner { -- explicit RealCommandRunner(const BuildConfig& config) : config_(config) {} -- virtual ~RealCommandRunner() {} -+ explicit RealCommandRunner(const BuildConfig& config); -+ virtual ~RealCommandRunner(); - virtual bool CanRunMore(); - virtual bool StartCommand(Edge* edge); - virtual bool WaitForCommand(Result* result); -@@ -495,9 +496,18 @@ struct RealCommandRunner : public CommandRunner { - - const BuildConfig& config_; - SubprocessSet subprocs_; -+ TokenPool *tokens_; - map subproc_to_edge_; - }; - -+RealCommandRunner::RealCommandRunner(const BuildConfig& config) : config_(config) { -+ tokens_ = TokenPool::Get(); -+} -+ -+RealCommandRunner::~RealCommandRunner() { -+ delete tokens_; -+} -+ - vector RealCommandRunner::GetActiveEdges() { - vector edges; - for (map::iterator e = subproc_to_edge_.begin(); -@@ -508,14 +518,18 @@ vector RealCommandRunner::GetActiveEdges() { - - void RealCommandRunner::Abort() { - subprocs_.Clear(); -+ if (tokens_) -+ tokens_->Clear(); - } - - bool RealCommandRunner::CanRunMore() { - size_t subproc_number = - subprocs_.running_.size() + subprocs_.finished_.size(); - return (int)subproc_number < config_.parallelism -- && ((subprocs_.running_.empty() || config_.max_load_average <= 0.0f) -- || GetLoadAverage() < config_.max_load_average); -+ && (subprocs_.running_.empty() || -+ ((config_.max_load_average <= 0.0f || -+ GetLoadAverage() < config_.max_load_average) -+ && (!tokens_ || tokens_->Acquire()))); - } - - bool RealCommandRunner::StartCommand(Edge* edge) { -@@ -523,6 +537,8 @@ bool RealCommandRunner::StartCommand(Edge* edge) { - Subprocess* subproc = subprocs_.Add(command, edge->use_console()); - if (!subproc) - return false; -+ if (tokens_) -+ tokens_->Reserve(); - subproc_to_edge_.insert(make_pair(subproc, edge)); - - return true; -@@ -536,6 +552,9 @@ bool RealCommandRunner::WaitForCommand(Result* result) { - return false; - } - -+ if (tokens_) -+ tokens_->Release(); -+ - result->status = subproc->Finish(); - result->output = subproc->GetOutput(); - -@@ -644,23 +663,23 @@ bool Builder::Build(string* err) { - // Second, we attempt to wait for / reap the next finished command. - while (plan_.more_to_do()) { - // See if we can start any more commands. -- if (failures_allowed && command_runner_->CanRunMore()) { -- if (Edge* edge = plan_.FindWork()) { -- if (!StartEdge(edge, err)) { -- Cleanup(); -- status_->BuildFinished(); -- return false; -- } -- -- if (edge->is_phony()) { -- plan_.EdgeFinished(edge, Plan::kEdgeSucceeded); -- } else { -- ++pending_commands; -- } -+ if (failures_allowed && plan_.more_ready() && -+ command_runner_->CanRunMore()) { -+ Edge* edge = plan_.FindWork(); -+ if (!StartEdge(edge, err)) { -+ Cleanup(); -+ status_->BuildFinished(); -+ return false; -+ } - -- // We made some progress; go back to the main loop. -- continue; -+ if (edge->is_phony()) { -+ plan_.EdgeFinished(edge, Plan::kEdgeSucceeded); -+ } else { -+ ++pending_commands; - } -+ -+ // We made some progress; go back to the main loop. -+ continue; - } - - // See if we can reap any finished commands. -diff --git a/src/build.h b/src/build.h -index 43786f1c928f..cca7e8d8181d 100644 ---- a/src/build.h -+++ b/src/build.h -@@ -53,6 +53,9 @@ struct Plan { - /// Returns true if there's more work to be done. - bool more_to_do() const { return wanted_edges_ > 0 && command_edges_ > 0; } - -+ /// Returns true if there's more edges ready to start -+ bool more_ready() const { return !ready_.empty(); } -+ - /// Dumps the current state of the plan. - void Dump(); - -diff --git a/src/tokenpool-gnu-make.cc b/src/tokenpool-gnu-make.cc -new file mode 100644 -index 000000000000..a8f9b7139d23 ---- /dev/null -+++ b/src/tokenpool-gnu-make.cc -@@ -0,0 +1,211 @@ -+// Copyright 2016 Google Inc. All Rights Reserved. -+// -+// Licensed under the Apache License, Version 2.0 (the "License"); -+// you may not use this file except in compliance with the License. -+// You may obtain a copy of the License at -+// -+// http://www.apache.org/licenses/LICENSE-2.0 -+// -+// Unless required by applicable law or agreed to in writing, software -+// distributed under the License is distributed on an "AS IS" BASIS, -+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -+// See the License for the specific language governing permissions and -+// limitations under the License. -+ -+#include "tokenpool.h" -+ -+#include -+#include -+#include -+#include -+#include -+#include -+#include -+#include -+ -+// TokenPool implementation for GNU make jobserver -+// (http://make.mad-scientist.net/papers/jobserver-implementation/) -+struct GNUmakeTokenPool : public TokenPool { -+ GNUmakeTokenPool(); -+ virtual ~GNUmakeTokenPool(); -+ -+ virtual bool Acquire(); -+ virtual void Reserve(); -+ virtual void Release(); -+ virtual void Clear(); -+ -+ bool Setup(); -+ -+ private: -+ int available_; -+ int used_; -+ -+#ifdef _WIN32 -+ // @TODO -+#else -+ int rfd_; -+ int wfd_; -+ -+ struct sigaction old_act_; -+ bool restore_; -+ -+ static int dup_rfd_; -+ static void CloseDupRfd(int signum); -+ -+ bool CheckFd(int fd); -+ bool SetAlarmHandler(); -+#endif -+ -+ void Return(); -+}; -+ -+// every instance owns an implicit token -> available_ == 1 -+GNUmakeTokenPool::GNUmakeTokenPool() : available_(1), used_(0), -+ rfd_(-1), wfd_(-1), restore_(false) { -+} -+ -+GNUmakeTokenPool::~GNUmakeTokenPool() { -+ Clear(); -+ if (restore_) -+ sigaction(SIGALRM, &old_act_, NULL); -+} -+ -+bool GNUmakeTokenPool::CheckFd(int fd) { -+ if (fd < 0) -+ return false; -+ int ret = fcntl(fd, F_GETFD); -+ if (ret < 0) -+ return false; -+ return true; -+} -+ -+int GNUmakeTokenPool::dup_rfd_ = -1; -+ -+void GNUmakeTokenPool::CloseDupRfd(int signum) { -+ close(dup_rfd_); -+ dup_rfd_ = -1; -+} -+ -+bool GNUmakeTokenPool::SetAlarmHandler() { -+ struct sigaction act; -+ memset(&act, 0, sizeof(act)); -+ act.sa_handler = CloseDupRfd; -+ if (sigaction(SIGALRM, &act, &old_act_) < 0) { -+ perror("sigaction:"); -+ return(false); -+ } else { -+ restore_ = true; -+ return(true); -+ } -+} -+ -+bool GNUmakeTokenPool::Setup() { -+ const char *value = getenv("MAKEFLAGS"); -+ if (value) { -+ // GNU make <= 4.1 -+ const char *jobserver = strstr(value, "--jobserver-fds="); -+ // GNU make => 4.2 -+ if (!jobserver) -+ jobserver = strstr(value, "--jobserver-auth="); -+ if (jobserver) { -+ int rfd = -1; -+ int wfd = -1; -+ if ((sscanf(jobserver, "%*[^=]=%d,%d", &rfd, &wfd) == 2) && -+ CheckFd(rfd) && -+ CheckFd(wfd) && -+ SetAlarmHandler()) { -+ printf("ninja: using GNU make jobserver.\n"); -+ rfd_ = rfd; -+ wfd_ = wfd; -+ return true; -+ } -+ } -+ } -+ -+ return false; -+} -+ -+bool GNUmakeTokenPool::Acquire() { -+ if (available_ > 0) -+ return true; -+ -+#ifdef USE_PPOLL -+ pollfd pollfds[] = {{rfd_, POLLIN, 0}}; -+ int ret = poll(pollfds, 1, 0); -+#else -+ fd_set set; -+ struct timeval timeout = { 0, 0 }; -+ FD_ZERO(&set); -+ FD_SET(rfd_, &set); -+ int ret = select(rfd_ + 1, &set, NULL, NULL, &timeout); -+#endif -+ if (ret > 0) { -+ dup_rfd_ = dup(rfd_); -+ -+ if (dup_rfd_ != -1) { -+ struct sigaction act, old_act; -+ int ret = 0; -+ -+ memset(&act, 0, sizeof(act)); -+ act.sa_handler = CloseDupRfd; -+ if (sigaction(SIGCHLD, &act, &old_act) == 0) { -+ char buf; -+ -+ // block until token read, child exits or timeout -+ alarm(1); -+ ret = read(dup_rfd_, &buf, 1); -+ alarm(0); -+ -+ sigaction(SIGCHLD, &old_act, NULL); -+ } -+ -+ CloseDupRfd(0); -+ -+ if (ret > 0) { -+ available_++; -+ return true; -+ } -+ } -+ } -+ return false; -+} -+ -+void GNUmakeTokenPool::Reserve() { -+ available_--; -+ used_++; -+} -+ -+void GNUmakeTokenPool::Return() { -+ const char buf = '+'; -+ while (1) { -+ int ret = write(wfd_, &buf, 1); -+ if (ret > 0) -+ available_--; -+ if ((ret != -1) || (errno != EINTR)) -+ return; -+ // write got interrupted - retry -+ } -+} -+ -+void GNUmakeTokenPool::Release() { -+ available_++; -+ used_--; -+ if (available_ > 1) -+ Return(); -+} -+ -+void GNUmakeTokenPool::Clear() { -+ while (used_ > 0) -+ Release(); -+ while (available_ > 1) -+ Return(); -+} -+ -+struct TokenPool *TokenPool::Get(void) { -+ GNUmakeTokenPool *tokenpool = new GNUmakeTokenPool; -+ if (tokenpool->Setup()) -+ return tokenpool; -+ else -+ delete tokenpool; -+ return NULL; -+} -diff --git a/src/tokenpool-none.cc b/src/tokenpool-none.cc -new file mode 100644 -index 000000000000..602b3316f54d ---- /dev/null -+++ b/src/tokenpool-none.cc -@@ -0,0 +1,27 @@ -+// Copyright 2016 Google Inc. All Rights Reserved. -+// -+// Licensed under the Apache License, Version 2.0 (the "License"); -+// you may not use this file except in compliance with the License. -+// You may obtain a copy of the License at -+// -+// http://www.apache.org/licenses/LICENSE-2.0 -+// -+// Unless required by applicable law or agreed to in writing, software -+// distributed under the License is distributed on an "AS IS" BASIS, -+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -+// See the License for the specific language governing permissions and -+// limitations under the License. -+ -+#include "tokenpool.h" -+ -+#include -+#include -+#include -+#include -+#include -+#include -+ -+// No-op TokenPool implementation -+struct TokenPool *TokenPool::Get(void) { -+ return NULL; -+} -diff --git a/src/tokenpool.h b/src/tokenpool.h -new file mode 100644 -index 000000000000..f560b1083b65 ---- /dev/null -+++ b/src/tokenpool.h -@@ -0,0 +1,26 @@ -+// Copyright 2016 Google Inc. All Rights Reserved. -+// -+// Licensed under the Apache License, Version 2.0 (the "License"); -+// you may not use this file except in compliance with the License. -+// You may obtain a copy of the License at -+// -+// http://www.apache.org/licenses/LICENSE-2.0 -+// -+// Unless required by applicable law or agreed to in writing, software -+// distributed under the License is distributed on an "AS IS" BASIS, -+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -+// See the License for the specific language governing permissions and -+// limitations under the License. -+ -+// interface to token pool -+struct TokenPool { -+ virtual ~TokenPool() {} -+ -+ virtual bool Acquire() = 0; -+ virtual void Reserve() = 0; -+ virtual void Release() = 0; -+ virtual void Clear() = 0; -+ -+ // returns NULL if token pool is not available -+ static struct TokenPool *Get(void); -+}; diff --git a/patches/ninja-1.8.2/0002-Add-TokenPool-monitoring-to-SubprocessSet-DoWork.patch b/patches/ninja-1.8.2/0002-Add-TokenPool-monitoring-to-SubprocessSet-DoWork.patch deleted file mode 100644 index f8ee646d7..000000000 --- a/patches/ninja-1.8.2/0002-Add-TokenPool-monitoring-to-SubprocessSet-DoWork.patch +++ /dev/null @@ -1,554 +0,0 @@ -From: Stefan Becker -Date: Fri, 27 May 2016 16:47:10 +0300 -Subject: [PATCH] Add TokenPool monitoring to SubprocessSet::DoWork() - -Improve on the original jobserver client implementation. This makes -ninja a more aggressive GNU make jobserver client. - -- add monitor interface to TokenPool -- TokenPool is passed down when main loop indicates that more work is - ready and would be allowed to start if a token becomes available -- posix: update DoWork() to monitor TokenPool read file descriptor -- WaitForCommand() exits when DoWork() sets token flag -- Main loop starts over when WaitForCommand() sets token exit status ---- - src/build.cc | 53 +++++++++++++++++++++++++++++++++++------------ - src/build.h | 3 ++- - src/build_test.cc | 9 ++++++-- - src/exit_status.h | 3 ++- - src/subprocess-posix.cc | 33 +++++++++++++++++++++++++++-- - src/subprocess-win32.cc | 2 +- - src/subprocess.h | 8 ++++++- - src/subprocess_test.cc | 47 ++++++++++++++++++++++++++++------------- - src/tokenpool-gnu-make.cc | 5 +++++ - src/tokenpool.h | 6 ++++++ - 10 files changed, 134 insertions(+), 35 deletions(-) - -diff --git a/src/build.cc b/src/build.cc -index cc796ff838fa..219bb9f1ff48 100644 ---- a/src/build.cc -+++ b/src/build.cc -@@ -49,8 +49,9 @@ struct DryRunCommandRunner : public CommandRunner { - - // Overridden from CommandRunner: - virtual bool CanRunMore(); -+ virtual bool AcquireToken(); - virtual bool StartCommand(Edge* edge); -- virtual bool WaitForCommand(Result* result); -+ virtual bool WaitForCommand(Result* result, bool more_ready); - - private: - queue finished_; -@@ -60,12 +61,16 @@ bool DryRunCommandRunner::CanRunMore() { - return true; - } - -+bool DryRunCommandRunner::AcquireToken() { -+ return true; -+} -+ - bool DryRunCommandRunner::StartCommand(Edge* edge) { - finished_.push(edge); - return true; - } - --bool DryRunCommandRunner::WaitForCommand(Result* result) { -+bool DryRunCommandRunner::WaitForCommand(Result* result, bool more_ready) { - if (finished_.empty()) - return false; - -@@ -489,8 +494,9 @@ struct RealCommandRunner : public CommandRunner { - explicit RealCommandRunner(const BuildConfig& config); - virtual ~RealCommandRunner(); - virtual bool CanRunMore(); -+ virtual bool AcquireToken(); - virtual bool StartCommand(Edge* edge); -- virtual bool WaitForCommand(Result* result); -+ virtual bool WaitForCommand(Result* result, bool more_ready); - virtual vector GetActiveEdges(); - virtual void Abort(); - -@@ -527,9 +533,12 @@ bool RealCommandRunner::CanRunMore() { - subprocs_.running_.size() + subprocs_.finished_.size(); - return (int)subproc_number < config_.parallelism - && (subprocs_.running_.empty() || -- ((config_.max_load_average <= 0.0f || -- GetLoadAverage() < config_.max_load_average) -- && (!tokens_ || tokens_->Acquire()))); -+ (config_.max_load_average <= 0.0f || -+ GetLoadAverage() < config_.max_load_average)); -+} -+ -+bool RealCommandRunner::AcquireToken() { -+ return (!tokens_ || tokens_->Acquire()); - } - - bool RealCommandRunner::StartCommand(Edge* edge) { -@@ -544,14 +553,23 @@ bool RealCommandRunner::StartCommand(Edge* edge) { - return true; - } - --bool RealCommandRunner::WaitForCommand(Result* result) { -+bool RealCommandRunner::WaitForCommand(Result* result, bool more_ready) { - Subprocess* subproc; -- while ((subproc = subprocs_.NextFinished()) == NULL) { -- bool interrupted = subprocs_.DoWork(); -+ subprocs_.ResetTokenAvailable(); -+ while (((subproc = subprocs_.NextFinished()) == NULL) && -+ !subprocs_.IsTokenAvailable()) { -+ bool interrupted = subprocs_.DoWork(more_ready ? tokens_ : NULL); - if (interrupted) - return false; - } - -+ // token became available -+ if (subproc == NULL) { -+ result->status = ExitTokenAvailable; -+ return true; -+ } -+ -+ // command completed - if (tokens_) - tokens_->Release(); - -@@ -662,9 +680,14 @@ bool Builder::Build(string* err) { - // command runner. - // Second, we attempt to wait for / reap the next finished command. - while (plan_.more_to_do()) { -- // See if we can start any more commands. -- if (failures_allowed && plan_.more_ready() && -- command_runner_->CanRunMore()) { -+ // See if we can start any more commands... -+ bool can_run_more = -+ failures_allowed && -+ plan_.more_ready() && -+ command_runner_->CanRunMore(); -+ -+ // ... but we also need a token to do that. -+ if (can_run_more && command_runner_->AcquireToken()) { - Edge* edge = plan_.FindWork(); - if (!StartEdge(edge, err)) { - Cleanup(); -@@ -685,7 +708,7 @@ bool Builder::Build(string* err) { - // See if we can reap any finished commands. - if (pending_commands) { - CommandRunner::Result result; -- if (!command_runner_->WaitForCommand(&result) || -+ if (!command_runner_->WaitForCommand(&result, can_run_more) || - result.status == ExitInterrupted) { - Cleanup(); - status_->BuildFinished(); -@@ -693,6 +716,10 @@ bool Builder::Build(string* err) { - return false; - } - -+ // We might be able to start another command; start the main loop over. -+ if (result.status == ExitTokenAvailable) -+ continue; -+ - --pending_commands; - if (!FinishCommand(&result, err)) { - Cleanup(); -diff --git a/src/build.h b/src/build.h -index cca7e8d8181d..ca605e62e0e3 100644 ---- a/src/build.h -+++ b/src/build.h -@@ -108,6 +108,7 @@ private: - struct CommandRunner { - virtual ~CommandRunner() {} - virtual bool CanRunMore() = 0; -+ virtual bool AcquireToken() = 0; - virtual bool StartCommand(Edge* edge) = 0; - - /// The result of waiting for a command. -@@ -119,7 +120,7 @@ struct CommandRunner { - bool success() const { return status == ExitSuccess; } - }; - /// Wait for a command to complete, or return false if interrupted. -- virtual bool WaitForCommand(Result* result) = 0; -+ virtual bool WaitForCommand(Result* result, bool more_ready) = 0; - - virtual vector GetActiveEdges() { return vector(); } - virtual void Abort() {} -diff --git a/src/build_test.cc b/src/build_test.cc -index 46ab33ef86c8..a1022edcf546 100644 ---- a/src/build_test.cc -+++ b/src/build_test.cc -@@ -445,8 +445,9 @@ struct FakeCommandRunner : public CommandRunner { - - // CommandRunner impl - virtual bool CanRunMore(); -+ virtual bool AcquireToken(); - virtual bool StartCommand(Edge* edge); -- virtual bool WaitForCommand(Result* result); -+ virtual bool WaitForCommand(Result* result, bool more_ready); - virtual vector GetActiveEdges(); - virtual void Abort(); - -@@ -547,6 +548,10 @@ bool FakeCommandRunner::CanRunMore() { - return last_command_ == NULL; - } - -+bool FakeCommandRunner::AcquireToken() { -+ return true; -+} -+ - bool FakeCommandRunner::StartCommand(Edge* edge) { - assert(!last_command_); - commands_ran_.push_back(edge->EvaluateCommand()); -@@ -575,7 +580,7 @@ bool FakeCommandRunner::StartCommand(Edge* edge) { - return true; - } - --bool FakeCommandRunner::WaitForCommand(Result* result) { -+bool FakeCommandRunner::WaitForCommand(Result* result, bool more_ready) { - if (!last_command_) - return false; - -diff --git a/src/exit_status.h b/src/exit_status.h -index a714ece791f7..75ebf6a7a0ce 100644 ---- a/src/exit_status.h -+++ b/src/exit_status.h -@@ -18,7 +18,8 @@ - enum ExitStatus { - ExitSuccess, - ExitFailure, -- ExitInterrupted -+ ExitTokenAvailable, -+ ExitInterrupted, - }; - - #endif // NINJA_EXIT_STATUS_H_ -diff --git a/src/subprocess-posix.cc b/src/subprocess-posix.cc -index 1de22c38f7fa..980fadf78e0d 100644 ---- a/src/subprocess-posix.cc -+++ b/src/subprocess-posix.cc -@@ -13,6 +13,7 @@ - // limitations under the License. - - #include "subprocess.h" -+#include "tokenpool.h" - - #include - #include -@@ -219,7 +220,7 @@ Subprocess *SubprocessSet::Add(const string& command, bool use_console) { - } - - #ifdef USE_PPOLL --bool SubprocessSet::DoWork() { -+bool SubprocessSet::DoWork(struct TokenPool* tokens) { - vector fds; - nfds_t nfds = 0; - -@@ -233,6 +234,12 @@ bool SubprocessSet::DoWork() { - ++nfds; - } - -+ if (tokens) { -+ pollfd pfd = { tokens->GetMonitorFd(), POLLIN | POLLPRI, 0 }; -+ fds.push_back(pfd); -+ ++nfds; -+ } -+ - interrupted_ = 0; - int ret = ppoll(&fds.front(), nfds, NULL, &old_mask_); - if (ret == -1) { -@@ -265,11 +272,20 @@ bool SubprocessSet::DoWork() { - ++i; - } - -+ if (tokens) { -+ pollfd *pfd = &fds[nfds - 1]; -+ if (pfd->fd >= 0) { -+ assert(pfd->fd == tokens->GetMonitorFd()); -+ if (pfd->revents != 0) -+ token_available_ = true; -+ } -+ } -+ - return IsInterrupted(); - } - - #else // !defined(USE_PPOLL) --bool SubprocessSet::DoWork() { -+bool SubprocessSet::DoWork(struct TokenPool* tokens) { - fd_set set; - int nfds = 0; - FD_ZERO(&set); -@@ -284,6 +300,13 @@ bool SubprocessSet::DoWork() { - } - } - -+ if (tokens) { -+ int fd = tokens->GetMonitorFd(); -+ FD_SET(fd, &set); -+ if (nfds < fd+1) -+ nfds = fd+1; -+ } -+ - interrupted_ = 0; - int ret = pselect(nfds, &set, 0, 0, 0, &old_mask_); - if (ret == -1) { -@@ -312,6 +335,12 @@ bool SubprocessSet::DoWork() { - ++i; - } - -+ if (tokens) { -+ int fd = tokens->GetMonitorFd(); -+ if ((fd >= 0) && FD_ISSET(fd, &set)) -+ token_available_ = true; -+ } -+ - return IsInterrupted(); - } - #endif // !defined(USE_PPOLL) -diff --git a/src/subprocess-win32.cc b/src/subprocess-win32.cc -index 4bab71939d6d..9b415b0b7bc3 100644 ---- a/src/subprocess-win32.cc -+++ b/src/subprocess-win32.cc -@@ -236,7 +236,7 @@ Subprocess *SubprocessSet::Add(const string& command, bool use_console) { - return subprocess; - } - --bool SubprocessSet::DoWork() { -+bool SubprocessSet::DoWork(struct TokenPool* tokens) { - DWORD bytes_read; - Subprocess* subproc; - OVERLAPPED* overlapped; -diff --git a/src/subprocess.h b/src/subprocess.h -index b2d486ca400c..bf1a46090bc1 100644 ---- a/src/subprocess.h -+++ b/src/subprocess.h -@@ -77,6 +77,8 @@ struct Subprocess { - friend struct SubprocessSet; - }; - -+struct TokenPool; -+ - /// SubprocessSet runs a ppoll/pselect() loop around a set of Subprocesses. - /// DoWork() waits for any state change in subprocesses; finished_ - /// is a queue of subprocesses as they finish. -@@ -85,13 +87,17 @@ struct SubprocessSet { - ~SubprocessSet(); - - Subprocess* Add(const string& command, bool use_console = false); -- bool DoWork(); -+ bool DoWork(struct TokenPool* tokens); - Subprocess* NextFinished(); - void Clear(); - - vector running_; - queue finished_; - -+ bool token_available_; -+ bool IsTokenAvailable() { return token_available_; } -+ void ResetTokenAvailable() { token_available_ = false; } -+ - #ifdef _WIN32 - static BOOL WINAPI NotifyInterrupted(DWORD dwCtrlType); - static HANDLE ioport_; -diff --git a/src/subprocess_test.cc b/src/subprocess_test.cc -index 0a8c2061b7f2..e759ea4574bc 100644 ---- a/src/subprocess_test.cc -+++ b/src/subprocess_test.cc -@@ -43,10 +43,12 @@ TEST_F(SubprocessTest, BadCommandStderr) { - Subprocess* subproc = subprocs_.Add("cmd /c ninja_no_such_command"); - ASSERT_NE((Subprocess *) 0, subproc); - -+ subprocs_.ResetTokenAvailable(); - while (!subproc->Done()) { - // Pretend we discovered that stderr was ready for writing. -- subprocs_.DoWork(); -+ subprocs_.DoWork(NULL); - } -+ ASSERT_EQ(false, subprocs_.IsTokenAvailable()); - - EXPECT_EQ(ExitFailure, subproc->Finish()); - EXPECT_NE("", subproc->GetOutput()); -@@ -57,10 +59,12 @@ TEST_F(SubprocessTest, NoSuchCommand) { - Subprocess* subproc = subprocs_.Add("ninja_no_such_command"); - ASSERT_NE((Subprocess *) 0, subproc); - -+ subprocs_.ResetTokenAvailable(); - while (!subproc->Done()) { - // Pretend we discovered that stderr was ready for writing. -- subprocs_.DoWork(); -+ subprocs_.DoWork(NULL); - } -+ ASSERT_EQ(false, subprocs_.IsTokenAvailable()); - - EXPECT_EQ(ExitFailure, subproc->Finish()); - EXPECT_NE("", subproc->GetOutput()); -@@ -76,9 +80,11 @@ TEST_F(SubprocessTest, InterruptChild) { - Subprocess* subproc = subprocs_.Add("kill -INT $$"); - ASSERT_NE((Subprocess *) 0, subproc); - -+ subprocs_.ResetTokenAvailable(); - while (!subproc->Done()) { -- subprocs_.DoWork(); -+ subprocs_.DoWork(NULL); - } -+ ASSERT_EQ(false, subprocs_.IsTokenAvailable()); - - EXPECT_EQ(ExitInterrupted, subproc->Finish()); - } -@@ -88,7 +94,7 @@ TEST_F(SubprocessTest, InterruptParent) { - ASSERT_NE((Subprocess *) 0, subproc); - - while (!subproc->Done()) { -- bool interrupted = subprocs_.DoWork(); -+ bool interrupted = subprocs_.DoWork(NULL); - if (interrupted) - return; - } -@@ -100,9 +106,11 @@ TEST_F(SubprocessTest, InterruptChildWithSigTerm) { - Subprocess* subproc = subprocs_.Add("kill -TERM $$"); - ASSERT_NE((Subprocess *) 0, subproc); - -+ subprocs_.ResetTokenAvailable(); - while (!subproc->Done()) { -- subprocs_.DoWork(); -+ subprocs_.DoWork(NULL); - } -+ ASSERT_EQ(false, subprocs_.IsTokenAvailable()); - - EXPECT_EQ(ExitInterrupted, subproc->Finish()); - } -@@ -112,7 +120,7 @@ TEST_F(SubprocessTest, InterruptParentWithSigTerm) { - ASSERT_NE((Subprocess *) 0, subproc); - - while (!subproc->Done()) { -- bool interrupted = subprocs_.DoWork(); -+ bool interrupted = subprocs_.DoWork(NULL); - if (interrupted) - return; - } -@@ -124,9 +132,11 @@ TEST_F(SubprocessTest, InterruptChildWithSigHup) { - Subprocess* subproc = subprocs_.Add("kill -HUP $$"); - ASSERT_NE((Subprocess *) 0, subproc); - -+ subprocs_.ResetTokenAvailable(); - while (!subproc->Done()) { -- subprocs_.DoWork(); -+ subprocs_.DoWork(NULL); - } -+ ASSERT_EQ(false, subprocs_.IsTokenAvailable()); - - EXPECT_EQ(ExitInterrupted, subproc->Finish()); - } -@@ -136,7 +146,7 @@ TEST_F(SubprocessTest, InterruptParentWithSigHup) { - ASSERT_NE((Subprocess *) 0, subproc); - - while (!subproc->Done()) { -- bool interrupted = subprocs_.DoWork(); -+ bool interrupted = subprocs_.DoWork(NULL); - if (interrupted) - return; - } -@@ -151,9 +161,11 @@ TEST_F(SubprocessTest, Console) { - subprocs_.Add("test -t 0 -a -t 1 -a -t 2", /*use_console=*/true); - ASSERT_NE((Subprocess*)0, subproc); - -+ subprocs_.ResetTokenAvailable(); - while (!subproc->Done()) { -- subprocs_.DoWork(); -+ subprocs_.DoWork(NULL); - } -+ ASSERT_EQ(false, subprocs_.IsTokenAvailable()); - - EXPECT_EQ(ExitSuccess, subproc->Finish()); - } -@@ -165,9 +177,11 @@ TEST_F(SubprocessTest, SetWithSingle) { - Subprocess* subproc = subprocs_.Add(kSimpleCommand); - ASSERT_NE((Subprocess *) 0, subproc); - -+ subprocs_.ResetTokenAvailable(); - while (!subproc->Done()) { -- subprocs_.DoWork(); -+ subprocs_.DoWork(NULL); - } -+ ASSERT_EQ(false, subprocs_.IsTokenAvailable()); - ASSERT_EQ(ExitSuccess, subproc->Finish()); - ASSERT_NE("", subproc->GetOutput()); - -@@ -198,12 +212,13 @@ TEST_F(SubprocessTest, SetWithMulti) { - ASSERT_EQ("", processes[i]->GetOutput()); - } - -+ subprocs_.ResetTokenAvailable(); - while (!processes[0]->Done() || !processes[1]->Done() || - !processes[2]->Done()) { - ASSERT_GT(subprocs_.running_.size(), 0u); -- subprocs_.DoWork(); -+ subprocs_.DoWork(NULL); - } -- -+ ASSERT_EQ(false, subprocs_.IsTokenAvailable()); - ASSERT_EQ(0u, subprocs_.running_.size()); - ASSERT_EQ(3u, subprocs_.finished_.size()); - -@@ -235,8 +250,10 @@ TEST_F(SubprocessTest, SetWithLots) { - ASSERT_NE((Subprocess *) 0, subproc); - procs.push_back(subproc); - } -+ subprocs_.ResetTokenAvailable(); - while (!subprocs_.running_.empty()) -- subprocs_.DoWork(); -+ subprocs_.DoWork(NULL); -+ ASSERT_EQ(false, subprocs_.IsTokenAvailable()); - for (size_t i = 0; i < procs.size(); ++i) { - ASSERT_EQ(ExitSuccess, procs[i]->Finish()); - ASSERT_NE("", procs[i]->GetOutput()); -@@ -252,9 +269,11 @@ TEST_F(SubprocessTest, SetWithLots) { - // that stdin is closed. - TEST_F(SubprocessTest, ReadStdin) { - Subprocess* subproc = subprocs_.Add("cat -"); -+ subprocs_.ResetTokenAvailable(); - while (!subproc->Done()) { -- subprocs_.DoWork(); -+ subprocs_.DoWork(NULL); - } -+ ASSERT_EQ(false, subprocs_.IsTokenAvailable()); - ASSERT_EQ(ExitSuccess, subproc->Finish()); - ASSERT_EQ(1u, subprocs_.finished_.size()); - } -diff --git a/src/tokenpool-gnu-make.cc b/src/tokenpool-gnu-make.cc -index a8f9b7139d23..396bb7d87443 100644 ---- a/src/tokenpool-gnu-make.cc -+++ b/src/tokenpool-gnu-make.cc -@@ -33,6 +33,7 @@ struct GNUmakeTokenPool : public TokenPool { - virtual void Reserve(); - virtual void Release(); - virtual void Clear(); -+ virtual int GetMonitorFd(); - - bool Setup(); - -@@ -201,6 +202,10 @@ void GNUmakeTokenPool::Clear() { - Return(); - } - -+int GNUmakeTokenPool::GetMonitorFd() { -+ return(rfd_); -+} -+ - struct TokenPool *TokenPool::Get(void) { - GNUmakeTokenPool *tokenpool = new GNUmakeTokenPool; - if (tokenpool->Setup()) -diff --git a/src/tokenpool.h b/src/tokenpool.h -index f560b1083b65..301e1998ee8e 100644 ---- a/src/tokenpool.h -+++ b/src/tokenpool.h -@@ -21,6 +21,12 @@ struct TokenPool { - virtual void Release() = 0; - virtual void Clear() = 0; - -+#ifdef _WIN32 -+ // @TODO -+#else -+ virtual int GetMonitorFd() = 0; -+#endif -+ - // returns NULL if token pool is not available - static struct TokenPool *Get(void); - }; diff --git a/patches/ninja-1.8.2/0003-Ignore-jobserver-when-jN-is-forced-on-command-line.patch b/patches/ninja-1.8.2/0003-Ignore-jobserver-when-jN-is-forced-on-command-line.patch deleted file mode 100644 index cd4b78f69..000000000 --- a/patches/ninja-1.8.2/0003-Ignore-jobserver-when-jN-is-forced-on-command-line.patch +++ /dev/null @@ -1,192 +0,0 @@ -From: Stefan Becker -Date: Sun, 12 Nov 2017 16:58:55 +0200 -Subject: [PATCH] Ignore jobserver when -jN is forced on command line - -This emulates the behaviour of GNU make. - -- add parallelism_from_cmdline flag to build configuration -- set the flag when -jN is given on command line -- pass the flag to TokenPool::Get() -- GNUmakeTokenPool::Setup() - * prints a warning when the flag is true and jobserver was detected - * returns false, i.e. jobserver will be ignored -- ignore config.parallelism in CanRunMore() when we have a valid - TokenPool, because it gets always initialized to a default when not - given on the command line ---- - src/build.cc | 10 ++++++---- - src/build.h | 4 +++- - src/ninja.cc | 1 + - src/tokenpool-gnu-make.cc | 34 +++++++++++++++++++--------------- - src/tokenpool-none.cc | 4 ++-- - src/tokenpool.h | 4 ++-- - 6 files changed, 33 insertions(+), 24 deletions(-) - -diff --git a/src/build.cc b/src/build.cc -index 219bb9f1ff48..bc26bdade61b 100644 ---- a/src/build.cc -+++ b/src/build.cc -@@ -507,7 +507,7 @@ struct RealCommandRunner : public CommandRunner { - }; - - RealCommandRunner::RealCommandRunner(const BuildConfig& config) : config_(config) { -- tokens_ = TokenPool::Get(); -+ tokens_ = TokenPool::Get(config_.parallelism_from_cmdline); - } - - RealCommandRunner::~RealCommandRunner() { -@@ -529,9 +529,11 @@ void RealCommandRunner::Abort() { - } - - bool RealCommandRunner::CanRunMore() { -- size_t subproc_number = -- subprocs_.running_.size() + subprocs_.finished_.size(); -- return (int)subproc_number < config_.parallelism -+ bool parallelism_limit_not_reached = -+ tokens_ || // ignore config_.parallelism -+ ((int) (subprocs_.running_.size() + -+ subprocs_.finished_.size()) < config_.parallelism); -+ return parallelism_limit_not_reached - && (subprocs_.running_.empty() || - (config_.max_load_average <= 0.0f || - GetLoadAverage() < config_.max_load_average)); -diff --git a/src/build.h b/src/build.h -index ca605e62e0e3..6bc6fea26e94 100644 ---- a/src/build.h -+++ b/src/build.h -@@ -128,7 +128,8 @@ struct CommandRunner { - - /// Options (e.g. verbosity, parallelism) passed to a build. - struct BuildConfig { -- BuildConfig() : verbosity(NORMAL), dry_run(false), parallelism(1), -+ BuildConfig() : verbosity(NORMAL), dry_run(false), -+ parallelism(1), parallelism_from_cmdline(false), - failures_allowed(1), max_load_average(-0.0f) {} - - enum Verbosity { -@@ -139,6 +140,7 @@ struct BuildConfig { - Verbosity verbosity; - bool dry_run; - int parallelism; -+ bool parallelism_from_cmdline; - int failures_allowed; - /// The maximum load average we must not exceed. A negative value - /// means that we do not have any limit. -diff --git a/src/ninja.cc b/src/ninja.cc -index ed004ac8f1fe..4332636c1b64 100644 ---- a/src/ninja.cc -+++ b/src/ninja.cc -@@ -1063,6 +1063,7 @@ int ReadFlags(int* argc, char*** argv, - if (*end != 0 || value <= 0) - Fatal("invalid -j parameter"); - config->parallelism = value; -+ config->parallelism_from_cmdline = true; - break; - } - case 'k': { -diff --git a/src/tokenpool-gnu-make.cc b/src/tokenpool-gnu-make.cc -index 396bb7d87443..af4be05a31cf 100644 ---- a/src/tokenpool-gnu-make.cc -+++ b/src/tokenpool-gnu-make.cc -@@ -1,4 +1,4 @@ --// Copyright 2016 Google Inc. All Rights Reserved. -+// Copyright 2016-2017 Google Inc. All Rights Reserved. - // - // Licensed under the Apache License, Version 2.0 (the "License"); - // you may not use this file except in compliance with the License. -@@ -35,7 +35,7 @@ struct GNUmakeTokenPool : public TokenPool { - virtual void Clear(); - virtual int GetMonitorFd(); - -- bool Setup(); -+ bool Setup(bool ignore); - - private: - int available_; -@@ -100,7 +100,7 @@ bool GNUmakeTokenPool::SetAlarmHandler() { - } - } - --bool GNUmakeTokenPool::Setup() { -+bool GNUmakeTokenPool::Setup(bool ignore) { - const char *value = getenv("MAKEFLAGS"); - if (value) { - // GNU make <= 4.1 -@@ -109,16 +109,20 @@ bool GNUmakeTokenPool::Setup() { - if (!jobserver) - jobserver = strstr(value, "--jobserver-auth="); - if (jobserver) { -- int rfd = -1; -- int wfd = -1; -- if ((sscanf(jobserver, "%*[^=]=%d,%d", &rfd, &wfd) == 2) && -- CheckFd(rfd) && -- CheckFd(wfd) && -- SetAlarmHandler()) { -- printf("ninja: using GNU make jobserver.\n"); -- rfd_ = rfd; -- wfd_ = wfd; -- return true; -+ if (ignore) { -+ printf("ninja: warning: -jN forced on command line; ignoring GNU make jobserver.\n"); -+ } else { -+ int rfd = -1; -+ int wfd = -1; -+ if ((sscanf(jobserver, "%*[^=]=%d,%d", &rfd, &wfd) == 2) && -+ CheckFd(rfd) && -+ CheckFd(wfd) && -+ SetAlarmHandler()) { -+ printf("ninja: using GNU make jobserver.\n"); -+ rfd_ = rfd; -+ wfd_ = wfd; -+ return true; -+ } - } - } - } -@@ -206,9 +210,9 @@ int GNUmakeTokenPool::GetMonitorFd() { - return(rfd_); - } - --struct TokenPool *TokenPool::Get(void) { -+struct TokenPool *TokenPool::Get(bool ignore) { - GNUmakeTokenPool *tokenpool = new GNUmakeTokenPool; -- if (tokenpool->Setup()) -+ if (tokenpool->Setup(ignore)) - return tokenpool; - else - delete tokenpool; -diff --git a/src/tokenpool-none.cc b/src/tokenpool-none.cc -index 602b3316f54d..199b22264bc6 100644 ---- a/src/tokenpool-none.cc -+++ b/src/tokenpool-none.cc -@@ -1,4 +1,4 @@ --// Copyright 2016 Google Inc. All Rights Reserved. -+// Copyright 2016-2017 Google Inc. All Rights Reserved. - // - // Licensed under the Apache License, Version 2.0 (the "License"); - // you may not use this file except in compliance with the License. -@@ -22,6 +22,6 @@ - #include - - // No-op TokenPool implementation --struct TokenPool *TokenPool::Get(void) { -+struct TokenPool *TokenPool::Get(bool ignore) { - return NULL; - } -diff --git a/src/tokenpool.h b/src/tokenpool.h -index 301e1998ee8e..878a0933c2f0 100644 ---- a/src/tokenpool.h -+++ b/src/tokenpool.h -@@ -1,4 +1,4 @@ --// Copyright 2016 Google Inc. All Rights Reserved. -+// Copyright 2016-2017 Google Inc. All Rights Reserved. - // - // Licensed under the Apache License, Version 2.0 (the "License"); - // you may not use this file except in compliance with the License. -@@ -28,5 +28,5 @@ struct TokenPool { - #endif - - // returns NULL if token pool is not available -- static struct TokenPool *Get(void); -+ static struct TokenPool *Get(bool ignore); - }; diff --git a/patches/ninja-1.8.2/0004-Honor-lN-from-MAKEFLAGS.patch b/patches/ninja-1.8.2/0004-Honor-lN-from-MAKEFLAGS.patch deleted file mode 100644 index 2a2505001..000000000 --- a/patches/ninja-1.8.2/0004-Honor-lN-from-MAKEFLAGS.patch +++ /dev/null @@ -1,128 +0,0 @@ -From: Stefan Becker -Date: Sun, 12 Nov 2017 18:04:12 +0200 -Subject: [PATCH] Honor -lN from MAKEFLAGS - -This emulates the behaviour of GNU make. - -- build: make a copy of max_load_average and pass it to TokenPool. -- GNUmakeTokenPool: if we detect a jobserver and a valid -lN argument in - MAKEFLAGS then set max_load_average to N. ---- - src/build.cc | 10 +++++++--- - src/tokenpool-gnu-make.cc | 19 +++++++++++++++---- - src/tokenpool-none.cc | 2 +- - src/tokenpool.h | 2 +- - 4 files changed, 24 insertions(+), 9 deletions(-) - -diff --git a/src/build.cc b/src/build.cc -index bc26bdade61b..6eaf299caeec 100644 ---- a/src/build.cc -+++ b/src/build.cc -@@ -501,13 +501,17 @@ struct RealCommandRunner : public CommandRunner { - virtual void Abort(); - - const BuildConfig& config_; -+ // copy of config_.max_load_average; can be modified by TokenPool setup -+ double max_load_average_; - SubprocessSet subprocs_; - TokenPool *tokens_; - map subproc_to_edge_; - }; - - RealCommandRunner::RealCommandRunner(const BuildConfig& config) : config_(config) { -- tokens_ = TokenPool::Get(config_.parallelism_from_cmdline); -+ max_load_average_ = config.max_load_average; -+ tokens_ = TokenPool::Get(config_.parallelism_from_cmdline, -+ max_load_average_); - } - - RealCommandRunner::~RealCommandRunner() { -@@ -535,8 +539,8 @@ bool RealCommandRunner::CanRunMore() { - subprocs_.finished_.size()) < config_.parallelism); - return parallelism_limit_not_reached - && (subprocs_.running_.empty() || -- (config_.max_load_average <= 0.0f || -- GetLoadAverage() < config_.max_load_average)); -+ (max_load_average_ <= 0.0f || -+ GetLoadAverage() < max_load_average_)); - } - - bool RealCommandRunner::AcquireToken() { -diff --git a/src/tokenpool-gnu-make.cc b/src/tokenpool-gnu-make.cc -index af4be05a31cf..fb654c4d88ba 100644 ---- a/src/tokenpool-gnu-make.cc -+++ b/src/tokenpool-gnu-make.cc -@@ -35,7 +35,7 @@ struct GNUmakeTokenPool : public TokenPool { - virtual void Clear(); - virtual int GetMonitorFd(); - -- bool Setup(bool ignore); -+ bool Setup(bool ignore, double& max_load_average); - - private: - int available_; -@@ -100,7 +100,7 @@ bool GNUmakeTokenPool::SetAlarmHandler() { - } - } - --bool GNUmakeTokenPool::Setup(bool ignore) { -+bool GNUmakeTokenPool::Setup(bool ignore, double& max_load_average) { - const char *value = getenv("MAKEFLAGS"); - if (value) { - // GNU make <= 4.1 -@@ -118,9 +118,20 @@ bool GNUmakeTokenPool::Setup(bool ignore) { - CheckFd(rfd) && - CheckFd(wfd) && - SetAlarmHandler()) { -+ const char *l_arg = strstr(value, " -l"); -+ int load_limit = -1; -+ - printf("ninja: using GNU make jobserver.\n"); - rfd_ = rfd; - wfd_ = wfd; -+ -+ // translate GNU make -lN to ninja -lN -+ if (l_arg && -+ (sscanf(l_arg + 3, "%d ", &load_limit) == 1) && -+ (load_limit > 0)) { -+ max_load_average = load_limit; -+ } -+ - return true; - } - } -@@ -210,9 +221,9 @@ int GNUmakeTokenPool::GetMonitorFd() { - return(rfd_); - } - --struct TokenPool *TokenPool::Get(bool ignore) { -+struct TokenPool *TokenPool::Get(bool ignore, double& max_load_average) { - GNUmakeTokenPool *tokenpool = new GNUmakeTokenPool; -- if (tokenpool->Setup(ignore)) -+ if (tokenpool->Setup(ignore, max_load_average)) - return tokenpool; - else - delete tokenpool; -diff --git a/src/tokenpool-none.cc b/src/tokenpool-none.cc -index 199b22264bc6..e8e25426c39f 100644 ---- a/src/tokenpool-none.cc -+++ b/src/tokenpool-none.cc -@@ -22,6 +22,6 @@ - #include - - // No-op TokenPool implementation --struct TokenPool *TokenPool::Get(bool ignore) { -+struct TokenPool *TokenPool::Get(bool ignore, double& max_load_average) { - return NULL; - } -diff --git a/src/tokenpool.h b/src/tokenpool.h -index 878a0933c2f0..f9e8cc2ee081 100644 ---- a/src/tokenpool.h -+++ b/src/tokenpool.h -@@ -28,5 +28,5 @@ struct TokenPool { - #endif - - // returns NULL if token pool is not available -- static struct TokenPool *Get(bool ignore); -+ static struct TokenPool *Get(bool ignore, double& max_load_average); - }; diff --git a/patches/ninja-1.8.2/0005-Use-LinePrinter-for-TokenPool-messages.patch b/patches/ninja-1.8.2/0005-Use-LinePrinter-for-TokenPool-messages.patch deleted file mode 100644 index 1894be5a1..000000000 --- a/patches/ninja-1.8.2/0005-Use-LinePrinter-for-TokenPool-messages.patch +++ /dev/null @@ -1,122 +0,0 @@ -From: Stefan Becker -Date: Wed, 6 Dec 2017 22:14:21 +0200 -Subject: [PATCH] Use LinePrinter for TokenPool messages - -- replace printf() with calls to LinePrinter -- print GNU make jobserver message only when verbose build is requested ---- - src/build.cc | 1 + - src/tokenpool-gnu-make.cc | 22 ++++++++++++++++------ - src/tokenpool-none.cc | 4 +++- - src/tokenpool.h | 4 +++- - 4 files changed, 23 insertions(+), 8 deletions(-) - -diff --git a/src/build.cc b/src/build.cc -index 6eaf299caeec..754d362c7162 100644 ---- a/src/build.cc -+++ b/src/build.cc -@@ -511,6 +511,7 @@ struct RealCommandRunner : public CommandRunner { - RealCommandRunner::RealCommandRunner(const BuildConfig& config) : config_(config) { - max_load_average_ = config.max_load_average; - tokens_ = TokenPool::Get(config_.parallelism_from_cmdline, -+ config_.verbosity == BuildConfig::VERBOSE, - max_load_average_); - } - -diff --git a/src/tokenpool-gnu-make.cc b/src/tokenpool-gnu-make.cc -index fb654c4d88ba..b0d3e6ebc463 100644 ---- a/src/tokenpool-gnu-make.cc -+++ b/src/tokenpool-gnu-make.cc -@@ -23,6 +23,8 @@ - #include - #include - -+#include "line_printer.h" -+ - // TokenPool implementation for GNU make jobserver - // (http://make.mad-scientist.net/papers/jobserver-implementation/) - struct GNUmakeTokenPool : public TokenPool { -@@ -35,7 +37,7 @@ struct GNUmakeTokenPool : public TokenPool { - virtual void Clear(); - virtual int GetMonitorFd(); - -- bool Setup(bool ignore, double& max_load_average); -+ bool Setup(bool ignore, bool verbose, double& max_load_average); - - private: - int available_; -@@ -100,7 +102,9 @@ bool GNUmakeTokenPool::SetAlarmHandler() { - } - } - --bool GNUmakeTokenPool::Setup(bool ignore, double& max_load_average) { -+bool GNUmakeTokenPool::Setup(bool ignore, -+ bool verbose, -+ double& max_load_average) { - const char *value = getenv("MAKEFLAGS"); - if (value) { - // GNU make <= 4.1 -@@ -109,8 +113,10 @@ bool GNUmakeTokenPool::Setup(bool ignore, double& max_load_average) { - if (!jobserver) - jobserver = strstr(value, "--jobserver-auth="); - if (jobserver) { -+ LinePrinter printer; -+ - if (ignore) { -- printf("ninja: warning: -jN forced on command line; ignoring GNU make jobserver.\n"); -+ printer.PrintOnNewLine("ninja: warning: -jN forced on command line; ignoring GNU make jobserver.\n"); - } else { - int rfd = -1; - int wfd = -1; -@@ -121,7 +127,9 @@ bool GNUmakeTokenPool::Setup(bool ignore, double& max_load_average) { - const char *l_arg = strstr(value, " -l"); - int load_limit = -1; - -- printf("ninja: using GNU make jobserver.\n"); -+ if (verbose) { -+ printer.PrintOnNewLine("ninja: using GNU make jobserver.\n"); -+ } - rfd_ = rfd; - wfd_ = wfd; - -@@ -221,9 +229,11 @@ int GNUmakeTokenPool::GetMonitorFd() { - return(rfd_); - } - --struct TokenPool *TokenPool::Get(bool ignore, double& max_load_average) { -+struct TokenPool *TokenPool::Get(bool ignore, -+ bool verbose, -+ double& max_load_average) { - GNUmakeTokenPool *tokenpool = new GNUmakeTokenPool; -- if (tokenpool->Setup(ignore, max_load_average)) -+ if (tokenpool->Setup(ignore, verbose, max_load_average)) - return tokenpool; - else - delete tokenpool; -diff --git a/src/tokenpool-none.cc b/src/tokenpool-none.cc -index e8e25426c39f..1c1c499c8d9c 100644 ---- a/src/tokenpool-none.cc -+++ b/src/tokenpool-none.cc -@@ -22,6 +22,8 @@ - #include - - // No-op TokenPool implementation --struct TokenPool *TokenPool::Get(bool ignore, double& max_load_average) { -+struct TokenPool *TokenPool::Get(bool ignore, -+ bool verbose, -+ double& max_load_average) { - return NULL; - } -diff --git a/src/tokenpool.h b/src/tokenpool.h -index f9e8cc2ee081..4bf477f20c8a 100644 ---- a/src/tokenpool.h -+++ b/src/tokenpool.h -@@ -28,5 +28,7 @@ struct TokenPool { - #endif - - // returns NULL if token pool is not available -- static struct TokenPool *Get(bool ignore, double& max_load_average); -+ static struct TokenPool *Get(bool ignore, -+ bool verbose, -+ double& max_load_average); - }; diff --git a/patches/ninja-1.8.2/0006-Prepare-PR-for-merging.patch b/patches/ninja-1.8.2/0006-Prepare-PR-for-merging.patch deleted file mode 100644 index bf44d5f4b..000000000 --- a/patches/ninja-1.8.2/0006-Prepare-PR-for-merging.patch +++ /dev/null @@ -1,151 +0,0 @@ -From: Stefan Becker -Date: Sat, 7 Apr 2018 17:11:21 +0300 -Subject: [PATCH] Prepare PR for merging - -- fix Windows build error in no-op TokenPool implementation -- improve Acquire() to block for a maximum of 100ms -- address review comments ---- - src/build.h | 2 ++ - src/tokenpool-gnu-make.cc | 53 ++++++++++++++++++++++++++++++++++++++++------- - src/tokenpool-none.cc | 7 +------ - 3 files changed, 49 insertions(+), 13 deletions(-) - -diff --git a/src/build.h b/src/build.h -index 6bc6fea26e94..530812bb9a2a 100644 ---- a/src/build.h -+++ b/src/build.h -@@ -120,6 +120,8 @@ struct CommandRunner { - bool success() const { return status == ExitSuccess; } - }; - /// Wait for a command to complete, or return false if interrupted. -+ /// If more_ready is true then the optional TokenPool is monitored too -+ /// and we return when a token becomes available. - virtual bool WaitForCommand(Result* result, bool more_ready) = 0; - - virtual vector GetActiveEdges() { return vector(); } -diff --git a/src/tokenpool-gnu-make.cc b/src/tokenpool-gnu-make.cc -index b0d3e6ebc463..4132bb06d9dd 100644 ---- a/src/tokenpool-gnu-make.cc -+++ b/src/tokenpool-gnu-make.cc -@@ -1,4 +1,4 @@ --// Copyright 2016-2017 Google Inc. All Rights Reserved. -+// Copyright 2016-2018 Google Inc. All Rights Reserved. - // - // Licensed under the Apache License, Version 2.0 (the "License"); - // you may not use this file except in compliance with the License. -@@ -19,6 +19,7 @@ - #include - #include - #include -+#include - #include - #include - #include -@@ -153,6 +154,15 @@ bool GNUmakeTokenPool::Acquire() { - if (available_ > 0) - return true; - -+ // Please read -+ // -+ // http://make.mad-scientist.net/papers/jobserver-implementation/ -+ // -+ // for the reasoning behind the following code. -+ // -+ // Try to read one character from the pipe. Returns true on success. -+ // -+ // First check if read() would succeed without blocking. - #ifdef USE_PPOLL - pollfd pollfds[] = {{rfd_, POLLIN, 0}}; - int ret = poll(pollfds, 1, 0); -@@ -164,33 +174,62 @@ bool GNUmakeTokenPool::Acquire() { - int ret = select(rfd_ + 1, &set, NULL, NULL, &timeout); - #endif - if (ret > 0) { -+ // Handle potential race condition: -+ // - the above check succeeded, i.e. read() should not block -+ // - the character disappears before we call read() -+ // -+ // Create a duplicate of rfd_. The duplicate file descriptor dup_rfd_ -+ // can safely be closed by signal handlers without affecting rfd_. - dup_rfd_ = dup(rfd_); - - if (dup_rfd_ != -1) { - struct sigaction act, old_act; - int ret = 0; - -+ // Temporarily replace SIGCHLD handler with our own - memset(&act, 0, sizeof(act)); - act.sa_handler = CloseDupRfd; - if (sigaction(SIGCHLD, &act, &old_act) == 0) { -- char buf; -- -- // block until token read, child exits or timeout -- alarm(1); -- ret = read(dup_rfd_, &buf, 1); -- alarm(0); -+ struct itimerval timeout; -+ -+ // install a 100ms timeout that generates SIGALARM on expiration -+ memset(&timeout, 0, sizeof(timeout)); -+ timeout.it_value.tv_usec = 100 * 1000; // [ms] -> [usec] -+ if (setitimer(ITIMER_REAL, &timeout, NULL) == 0) { -+ char buf; -+ -+ // Now try to read() from dup_rfd_. Return values from read(): -+ // -+ // 1. token read -> 1 -+ // 2. pipe closed -> 0 -+ // 3. alarm expires -> -1 (EINTR) -+ // 4. child exits -> -1 (EINTR) -+ // 5. alarm expired before entering read() -> -1 (EBADF) -+ // 6. child exited before entering read() -> -1 (EBADF) -+ // 7. child exited before handler is installed -> go to 1 - 3 -+ ret = read(dup_rfd_, &buf, 1); -+ -+ // disarm timer -+ memset(&timeout, 0, sizeof(timeout)); -+ setitimer(ITIMER_REAL, &timeout, NULL); -+ } - - sigaction(SIGCHLD, &old_act, NULL); - } - - CloseDupRfd(0); - -+ // Case 1 from above list - if (ret > 0) { - available_++; - return true; - } - } - } -+ -+ // read() would block, i.e. no token available, -+ // cases 2-6 from above list or -+ // select() / poll() / dup() / sigaction() / setitimer() failed - return false; - } - -diff --git a/src/tokenpool-none.cc b/src/tokenpool-none.cc -index 1c1c499c8d9c..4c592875b4ad 100644 ---- a/src/tokenpool-none.cc -+++ b/src/tokenpool-none.cc -@@ -1,4 +1,4 @@ --// Copyright 2016-2017 Google Inc. All Rights Reserved. -+// Copyright 2016-2018 Google Inc. All Rights Reserved. - // - // Licensed under the Apache License, Version 2.0 (the "License"); - // you may not use this file except in compliance with the License. -@@ -14,11 +14,6 @@ - - #include "tokenpool.h" - --#include --#include --#include --#include --#include - #include - - // No-op TokenPool implementation diff --git a/patches/ninja-1.8.2/series b/patches/ninja-1.8.2/series deleted file mode 100644 index ef13f1510..000000000 --- a/patches/ninja-1.8.2/series +++ /dev/null @@ -1,10 +0,0 @@ -# generated by git-ptx-patches -#tag:base --start-number 1 -0001-Add-GNU-make-jobserver-client-support.patch -0002-Add-TokenPool-monitoring-to-SubprocessSet-DoWork.patch -0003-Ignore-jobserver-when-jN-is-forced-on-command-line.patch -0004-Honor-lN-from-MAKEFLAGS.patch -0005-Use-LinePrinter-for-TokenPool-messages.patch -0006-Prepare-PR-for-merging.patch -0007-don-t-close-open-fds.patch -# e4c44c2e231b79ea6c0263de5d69c6d3 - git-ptx-patches magic diff --git a/rules/host-ninja.in b/rules/host-ninja.in index ef0e14fe8..fa568e714 100644 --- a/rules/host-ninja.in +++ b/rules/host-ninja.in @@ -2,7 +2,7 @@ config HOST_NINJA tristate - select HOST_SYSTEM_PYTHON3 + select HOST_CMAKE default y if ALLYES help Ninja build tool diff --git a/rules/host-ninja.make b/rules/host-ninja.make index d0622e2df..8b8b51a68 100644 --- a/rules/host-ninja.make +++ b/rules/host-ninja.make @@ -14,28 +14,28 @@ HOST_PACKAGES-$(PTXCONF_HOST_NINJA) += host-ninja # # Paths and names # -HOST_NINJA_VERSION := 1.8.2 -HOST_NINJA_MD5 := 5fdb04461cc7f5d02536b3bfc0300166 +HOST_NINJA_VERSION := 1.10.2.g51db2.kitware.jobserver-1 +HOST_NINJA_MD5 := 0bdd69f1013deb74465fe5d03e8fa2ea HOST_NINJA := ninja-$(HOST_NINJA_VERSION) HOST_NINJA_SUFFIX := tar.gz -HOST_NINJA_URL := https://github.com/ninja-build/ninja/archive/v$(HOST_NINJA_VERSION).$(HOST_NINJA_SUFFIX) +HOST_NINJA_URL := https://github.com/Kitware/ninja/archive/refs/tags/v$(HOST_NINJA_VERSION).$(HOST_NINJA_SUFFIX) HOST_NINJA_SOURCE := $(SRCDIR)/$(HOST_NINJA).$(HOST_NINJA_SUFFIX) HOST_NINJA_DIR := $(HOST_BUILDDIR)/$(HOST_NINJA) HOST_NINJA_LICENSE := Apache-2.0 +# ---------------------------------------------------------------------------- +# Prepare +# ---------------------------------------------------------------------------- + +HOST_NINJA_BUILD_OOT := NO + # ---------------------------------------------------------------------------- # Compile # ---------------------------------------------------------------------------- +HOST_NINJA_CONF_TOOL := cmake HOST_NINJA_CONF_OPT := \ - --bootstrap \ - $(if $(filter 1,$(PTXDIST_VERBOSE)),--verbose) - -$(STATEDIR)/host-ninja.compile: - @$(call targetinfo) - @$(call world/execute, HOST_NINJA, \ - $(SYSTEMPYTHON3) ./configure.py $(HOST_NINJA_CONF_OPT)) - @$(call touch) + $(HOST_CMAKE_OPT) # ---------------------------------------------------------------------------- # Install -- 2.34.1