From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Tue, 30 Aug 2022 09:02:59 +0200 Received: from metis.ext.pengutronix.de ([2001:67c:670:201:290:27ff:fe1d:cc33]) by lore.white.stw.pengutronix.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1oSvH1-001Sba-TG for lore@lore.pengutronix.de; Tue, 30 Aug 2022 09:02:59 +0200 Received: from localhost ([127.0.0.1] helo=metis.ext.pengutronix.de) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1oSvH1-0004q8-5t; Tue, 30 Aug 2022 09:02:59 +0200 Received: from drehscheibe.grey.stw.pengutronix.de ([2a0a:edc0:0:c01:1d::a2]) by metis.ext.pengutronix.de with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oSvGM-0004pg-0R; Tue, 30 Aug 2022 09:02:18 +0200 Received: from [2a0a:edc0:0:1101:1d::39] (helo=dude03.red.stw.pengutronix.de) by drehscheibe.grey.stw.pengutronix.de with esmtp (Exim 4.94.2) (envelope-from ) id 1oSvGK-002ope-Mk; Tue, 30 Aug 2022 09:02:16 +0200 Received: from mol by dude03.red.stw.pengutronix.de with local (Exim 4.94.2) (envelope-from ) id 1oSvGJ-001nDl-VG; Tue, 30 Aug 2022 09:02:16 +0200 Date: Tue, 30 Aug 2022 09:02:15 +0200 From: Michael Olbrich To: jon@ringle.org Message-ID: References: <20220819103913.2869281-1-jon@ringle.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220819103913.2869281-1-jon@ringle.org> X-Sent-From: Pengutronix Hildesheim X-URL: http://www.pengutronix.de/ X-IRC: #ptxdist @freenode X-Accept-Language: de,en X-Accept-Content-Type: text/plain Subject: Re: [ptxdist] [PATCH 1/1] ptxd_make_get: Add support for downloading s3:// urls via aws s3 cp X-BeenThere: ptxdist@pengutronix.de X-Mailman-Version: 2.1.29 Precedence: list List-Id: PTXdist Development Mailing List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: ptxdist@pengutronix.de Cc: ptxdist@pengutronix.com, Jon Ringle Sender: "ptxdist" X-SA-Exim-Connect-IP: 127.0.0.1 X-SA-Exim-Mail-From: ptxdist-bounces@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false On Fri, Aug 19, 2022 at 06:39:13AM -0400, jon@ringle.org wrote: > From: Jon Ringle > > This allows downloading files stored in an AWS S3 bucket via an s3:// url > This allows you to point PTXCONF_SETUP_PTXMIRROR to an s3:// bucket > > Signed-off-by: Jon Ringle > --- > scripts/lib/ptxd_make_get.sh | 33 +++++++++++++++++++++++++++++++++ > 1 file changed, 33 insertions(+) > > diff --git a/scripts/lib/ptxd_make_get.sh b/scripts/lib/ptxd_make_get.sh > index ee6b01346..d575ac13e 100644 > --- a/scripts/lib/ptxd_make_get.sh > +++ b/scripts/lib/ptxd_make_get.sh > @@ -270,6 +270,35 @@ ptxd_make_get_svn() { > } > export -f ptxd_make_get_svn > > +# > +# in env: > +# > +# ${path} : local file name > +# ${url} : the url to download > +# ${opts[]} : an array of options > +# > +ptxd_make_get_s3() { > + local temp_file > + set -- "${opts[@]}" > + unset opts > + > + local file="${url##*/}" unused? > + > + # remove any pending or half downloaded files > + rm -f -- "${path}."* > + > + temp_file="$(mktemp "${path}.XXXXXXXXXX")" || ptxd_bailout "failed to create tempfile" > + > + aws s3 cp "${url}" "${temp_file}" && { aws s3 ... || return ... to avoid the { } and indention. Michael > + chmod 644 -- "${temp_file}" && > + touch -- "${temp_file}" && > + mv -- "${temp_file}" "${path}" > + } > + > + ptxd_make_serialize_put > +} > +export -f ptxd_make_get_s3 > + > > # > # check if download is disabled > @@ -412,6 +441,10 @@ ptxd_make_get() { > ptxd_make_get_download_permitted && > ptxd_make_get_http && return > ;; > + s3://*) > + ptxd_make_get_download_permitted && > + ptxd_make_get_s3 && return > + ;; > file*) > local thing="${url/file:\/\///}" > > -- > 2.25.1 > > > -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |