public inbox for [email protected]
help / color / mirror / Atom feedDownload failures on apt archive
7+ messages / 3 participants
[nested] [flat]
* Download failures on apt archive
@ 2023-05-18 14:35 James Coleman <[email protected]>
2023-05-18 16:48 ` Re: Download failures on apt archive Bradford Boyle <[email protected]>
2023-05-18 19:51 ` Re: Download failures on apt archive Christoph Berg <[email protected]>
0 siblings, 2 replies; 7+ messages in thread
From: James Coleman @ 2023-05-18 14:35 UTC (permalink / raw)
To: [email protected]
Hello,
We've been trying to do direct downloads from the apt archive (e.g.
[1]) and many of the files fail to download with an AccessDenied error
with XML from S3 like:
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>FRN2GJREB814S5FD</RequestId>
<HostId>
47zx5Bji8uWyhDlF+59x0dJhK/8+8TInBCVeQqwhIZecGE2pZfPlFZEIO2/aVtam+My7h/WY9ls=
</HostId>
</Error>
For example [2] fails but for example [3] succeeds.
This seems to be repeatable by file across browsers and with my
co-workers, so I don't think it's just a client issue.
Thanks,
James Coleman
1: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html
2: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/libpq-dev_14.7-1.pgdg110%...
3. https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/postgresql-14_14.1-1.pgdg...
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Download failures on apt archive
2023-05-18 14:35 Download failures on apt archive James Coleman <[email protected]>
@ 2023-05-18 16:48 ` Bradford Boyle <[email protected]>
1 sibling, 0 replies; 7+ messages in thread
From: Bradford Boyle @ 2023-05-18 16:48 UTC (permalink / raw)
To: James Coleman <[email protected]>; +Cc: [email protected]
Hi James,
apt-archive.p.o is hosted via an AWS S3 bucket which doesn't include/generate
index lists (e.g, index.html) which is why [1] doesn't work. If you are
familiar with the structure of Debian repos, you can view the repo's index by
browsing the appropriate files directly. For example,
https://apt-archive.postgresql.org/pub/repos/apt/dists/${DISTRIBUTION}-pgdg-archive/InRelease
where distribution is bullseye, focal, etc. The "InRelease" file will contain
paths for various "Packages" files. You can view the list of packages by
replacing "InRelease" with the appropraite package path. For example
https://apt-archive.postgresql.org/pub/repos/apt/dists/${DISTRIBUTION}-pgdg-archive/main/binary-amd64/Packages
Using this, we can see why [2] isn't working for you -- there is no libpq-dev
in bullseye-pgdg-archive, the package is libpq5-dev.
Hope this helps somewhat.
--Bradford
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Download failures on apt archive
2023-05-18 14:35 Download failures on apt archive James Coleman <[email protected]>
@ 2023-05-18 19:51 ` Christoph Berg <[email protected]>
2023-05-19 03:48 ` Re: Download failures on apt archive Bradford Boyle <[email protected]>
1 sibling, 1 reply; 7+ messages in thread
From: Christoph Berg @ 2023-05-18 19:51 UTC (permalink / raw)
To: James Coleman <[email protected]>; Bradford Boyle <[email protected]>; +Cc: [email protected]
Re: James Coleman
> Hello,
>
> We've been trying to do direct downloads from the apt archive (e.g.
> [1]) and many of the files fail to download with an AccessDenied error
> with XML from S3 like:
>
> <Error>
> <Code>AccessDenied</Code>
Hmm, admittedly I've never tried to recheck if all the package links
worked after the initial setup, so it's well possible something went
wrong in the years since then.
I'll do some QA there tomorrow. Thanks for reporting!
Re: Bradford Boyle
> Hi James,
>
> apt-archive.p.o is hosted via an AWS S3 bucket which doesn't include/generate
> index lists (e.g, index.html) which is why [1] doesn't work.
There are manually uploaded index.html files generated from the
database mentioned earlier.
> Using this, we can see why [2] isn't working for you -- there is no libpq-dev
> in bullseye-pgdg-archive, the package is libpq5-dev.
libpq-dev is correct, the 5 is only in libpq5.deb.
Christoph
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Download failures on apt archive
2023-05-18 14:35 Download failures on apt archive James Coleman <[email protected]>
2023-05-18 19:51 ` Re: Download failures on apt archive Christoph Berg <[email protected]>
@ 2023-05-19 03:48 ` Bradford Boyle <[email protected]>
2023-05-19 13:06 ` Re: Download failures on apt archive Christoph Berg <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Bradford Boyle @ 2023-05-19 03:48 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; +Cc: James Coleman <[email protected]>; [email protected]
> There are manually uploaded index.html files generated from the
> database mentioned earlier.
Yep, I see that now. I should have read the code for
generate-archive-lists more closely before replying.
> libpq-dev is correct, the 5 is only in libpq5.deb.
Right again -- I don't know what I was doing wrong with my grep commands
this morning but I'll blame it on a lack of sleep and coffee.
It looks like the links on [1] are incorrect -- it gives [2] as the link
for libpq-dev_14.7-1.pgdg110+1_amd64.deb but looking at the Package
file, I think the correct link is [3] (i.e., pool/main vs pool/14).
I think the issue has to do with set_package_component in
generate-pgdg-source moving lib packages to a different component when
the source postgresql version is older than PG_MAIN_VERSION; it looks
generate-archive-lists assumes component is always main though.
--Bradford
[1]: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html
[2]: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/libpq-dev_14.7-1.pgdg110%...
[3]: https://apt-archive.postgresql.org/pub/repos/apt/pool/14/p/postgresql-14/libpq-dev_14.7-1.pgdg110%2B...
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Download failures on apt archive
2023-05-18 14:35 Download failures on apt archive James Coleman <[email protected]>
2023-05-18 19:51 ` Re: Download failures on apt archive Christoph Berg <[email protected]>
2023-05-19 03:48 ` Re: Download failures on apt archive Bradford Boyle <[email protected]>
@ 2023-05-19 13:06 ` Christoph Berg <[email protected]>
2023-05-19 15:31 ` Re: Download failures on apt archive Christoph Berg <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Christoph Berg @ 2023-05-19 13:06 UTC (permalink / raw)
To: Bradford Boyle <[email protected]>; +Cc: James Coleman <[email protected]>; [email protected]
Re: Bradford Boyle
> It looks like the links on [1] are incorrect -- it gives [2] as the link
> for libpq-dev_14.7-1.pgdg110+1_amd64.deb but looking at the Package
> file, I think the correct link is [3] (i.e., pool/main vs pool/14).
Ah right, thanks for spotting that, I had missed that when trying to
debugging the issue earlier today. I remember now that I had omitted
the extra pool locations, but I missed that they are still used.
So in summary, the files are all there, but the path in the
referencing web page is wrong. Will fix.
> I think the issue has to do with set_package_component in
> generate-pgdg-source moving lib packages to a different component when
> the source postgresql version is older than PG_MAIN_VERSION; it looks
> generate-archive-lists assumes component is always main though.
Yeah I had the idea that for the archive, I could just ignore the
changed component and shovel everything into "main", but of course in
that case the files would have to be uploaded to that location.
Christoph
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Download failures on apt archive
2023-05-18 14:35 Download failures on apt archive James Coleman <[email protected]>
2023-05-18 19:51 ` Re: Download failures on apt archive Christoph Berg <[email protected]>
2023-05-19 03:48 ` Re: Download failures on apt archive Bradford Boyle <[email protected]>
2023-05-19 13:06 ` Re: Download failures on apt archive Christoph Berg <[email protected]>
@ 2023-05-19 15:31 ` Christoph Berg <[email protected]>
2023-05-20 02:00 ` Re: Download failures on apt archive James Coleman <[email protected]>
0 siblings, 1 reply; 7+ messages in thread
From: Christoph Berg @ 2023-05-19 15:31 UTC (permalink / raw)
To: Bradford Boyle <[email protected]>; +Cc: James Coleman <[email protected]>; [email protected]
Re: To Bradford Boyle
> So in summary, the files are all there, but the path in the
> referencing web page is wrong. Will fix.
I managed to fix the problems, the index listings in package
directories like
https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html
will now also reference files located in other directories like pool/14/p/...
James, Bradford: Thanks for spotting and debugging!
Christoph
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Download failures on apt archive
2023-05-18 14:35 Download failures on apt archive James Coleman <[email protected]>
2023-05-18 19:51 ` Re: Download failures on apt archive Christoph Berg <[email protected]>
2023-05-19 03:48 ` Re: Download failures on apt archive Bradford Boyle <[email protected]>
2023-05-19 13:06 ` Re: Download failures on apt archive Christoph Berg <[email protected]>
2023-05-19 15:31 ` Re: Download failures on apt archive Christoph Berg <[email protected]>
@ 2023-05-20 02:00 ` James Coleman <[email protected]>
0 siblings, 0 replies; 7+ messages in thread
From: James Coleman @ 2023-05-20 02:00 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; +Cc: Bradford Boyle <[email protected]>; [email protected]
On Fri, May 19, 2023 at 11:31 AM Christoph Berg <[email protected]> wrote:
>
> Re: To Bradford Boyle
> > So in summary, the files are all there, but the path in the
> > referencing web page is wrong. Will fix.
>
> I managed to fix the problems, the index listings in package
> directories like
> https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html
> will now also reference files located in other directories like pool/14/p/...
>
> James, Bradford: Thanks for spotting and debugging!
Thanks for fixing this!
James
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2023-05-20 02:00 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-05-18 14:35 Download failures on apt archive James Coleman <[email protected]>
2023-05-18 16:48 ` Bradford Boyle <[email protected]>
2023-05-18 19:51 ` Christoph Berg <[email protected]>
2023-05-19 03:48 ` Bradford Boyle <[email protected]>
2023-05-19 13:06 ` Christoph Berg <[email protected]>
2023-05-19 15:31 ` Christoph Berg <[email protected]>
2023-05-20 02:00 ` James Coleman <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox