From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Fri, 19 Aug 2022 12:40:03 +0200 Received: from metis.ext.pengutronix.de ([2001:67c:670:201:290:27ff:fe1d:cc33]) by lore.white.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1oOzQ4-0040RC-ET for lore@lore.pengutronix.de; Fri, 19 Aug 2022 12:40:03 +0200 Received: from localhost ([127.0.0.1] helo=metis.ext.pengutronix.de) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1oOzQ2-0006x7-Uo; Fri, 19 Aug 2022 12:40:02 +0200 Received: from mail-qt1-x830.google.com ([2607:f8b0:4864:20::830]) by metis.ext.pengutronix.de with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1oOzPM-0006wy-Jy for ptxdist@pengutronix.com; Fri, 19 Aug 2022 12:39:21 +0200 Received: by mail-qt1-x830.google.com with SMTP id s11so3004271qtx.6 for ; Fri, 19 Aug 2022 03:39:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:sender:from:to:cc; bh=CYg2N0jnLiCBpOZwai37yagUClxvk6wRLMlo19Mowlw=; b=cZU3YwzhLx+phaOkI2EHSGWFATbllodcWql4odvsSouWen0MkXOnrGZtlCpMBlGGNw lvWrJmjORHYUrsGHvvPwXr5JI/I0W0IJB24ZVLQDajcW0U67y2r5V+pEJvzuxAWL4BqC kHZ4uPA5TIUrozJ6Xm7IVGKw8sH0QYzeZlUvndK9slSjQKcc3Wsr5Ommxdbx7Rn8UHBV b5vA+eOnItvLq8ThMPqfyUjvTaUYMpEp6Z8mKyPKytJtBu+6uvgyUJzHEaVGF1cZLpP/ zm4DxRhLrVRHDyI0a4HD60g6Lr3h8AfSLzKovZkkOVUyKTRmtrxiQHe6WH46F9b0fr21 z8Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:sender:x-gm-message-state:from:to:cc; bh=CYg2N0jnLiCBpOZwai37yagUClxvk6wRLMlo19Mowlw=; b=ADIyiJJ2N8717zHYUtZZkhYc34li+ZQijw7zcygUgXrWqZRLVT6+kWNKgR+/+TpDX6 OWb2NdNtvS8bElhbjjh1ICplgNC6aL/IEPDQEiNsmRQ5ujVGNc7PL4m/4riTCO7iD271 Op04OJF4zHcbH/bO2WkT2/mm0a3kV7cjKaTLpvPjNcd5prbPEDyNXOeYeh5YWpPHJfiL dbwnbMv3qRgodLZvp7onm6LNKTPIIbrhd9ne77f1dBmKV7gtObG6EEKZoN6hSZpavgGn FU6sDu99A+6ghq5mmj29GU2pfYPUsfReu9uLb3Hg3kbiSPYgrt2UCVes1VkM/TQrNlJR G4vQ== X-Gm-Message-State: ACgBeo3XsLHZsanXnYY/TrHBbXGW5RUD7hmbfIWF8PY377TRFzULxhez ESMyBIN+qX1Hj/+OtsbqPCUZnxwORf8= X-Google-Smtp-Source: AA6agR4VD30lO62d1Lbl7PQtLt1GyxFEujBTq1SKXYZXaEp3macWNPEoqOWhM0U7y4dLEENawxnQyA== X-Received: by 2002:ac8:7d90:0:b0:326:b431:6cd3 with SMTP id c16-20020ac87d90000000b00326b4316cd3mr5887519qtd.511.1660905558289; Fri, 19 Aug 2022 03:39:18 -0700 (PDT) Received: from jring-t580-gns3.gridpoint.com.com (pool-108-18-228-177.washdc.fios.verizon.net. [108.18.228.177]) by smtp.gmail.com with ESMTPSA id q30-20020a37f71e000000b006baef6daa45sm3214056qkj.119.2022.08.19.03.39.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Aug 2022 03:39:17 -0700 (PDT) From: jon@ringle.org To: ptxdist@pengutronix.com Date: Fri, 19 Aug 2022 06:39:13 -0400 Message-Id: <20220819103913.2869281-1-jon@ringle.org> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on metis.ext.pengutronix.de X-Spam-Level: X-Spam-Status: No, score=-1.5 required=4.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.2 Subject: [ptxdist] [PATCH 1/1] ptxd_make_get: Add support for downloading s3:// urls via aws s3 cp X-BeenThere: ptxdist@pengutronix.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: PTXdist Development Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ptxdist@pengutronix.de Cc: Jon Ringle Sender: "ptxdist" X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: ptxdist-bounces@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false From: Jon Ringle This allows downloading files stored in an AWS S3 bucket via an s3:// url This allows you to point PTXCONF_SETUP_PTXMIRROR to an s3:// bucket Signed-off-by: Jon Ringle --- scripts/lib/ptxd_make_get.sh | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/scripts/lib/ptxd_make_get.sh b/scripts/lib/ptxd_make_get.sh index ee6b01346..d575ac13e 100644 --- a/scripts/lib/ptxd_make_get.sh +++ b/scripts/lib/ptxd_make_get.sh @@ -270,6 +270,35 @@ ptxd_make_get_svn() { } export -f ptxd_make_get_svn +# +# in env: +# +# ${path} : local file name +# ${url} : the url to download +# ${opts[]} : an array of options +# +ptxd_make_get_s3() { + local temp_file + set -- "${opts[@]}" + unset opts + + local file="${url##*/}" + + # remove any pending or half downloaded files + rm -f -- "${path}."* + + temp_file="$(mktemp "${path}.XXXXXXXXXX")" || ptxd_bailout "failed to create tempfile" + + aws s3 cp "${url}" "${temp_file}" && { + chmod 644 -- "${temp_file}" && + touch -- "${temp_file}" && + mv -- "${temp_file}" "${path}" + } + + ptxd_make_serialize_put +} +export -f ptxd_make_get_s3 + # # check if download is disabled @@ -412,6 +441,10 @@ ptxd_make_get() { ptxd_make_get_download_permitted && ptxd_make_get_http && return ;; + s3://*) + ptxd_make_get_download_permitted && + ptxd_make_get_s3 && return + ;; file*) local thing="${url/file:\/\///}" -- 2.25.1