Subject: Download failures on apt archive

From: James Coleman <[email-masked]>
Date: Thu, 18 May 2023 10:35:42 -0400

Hello,

We've been trying to do direct downloads from the apt archive (e.g.
[1]) and many of the files fail to download with an AccessDenied error
with XML from S3 like:

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>FRN2GJREB814S5FD</RequestId>
<HostId>
47zx5Bji8uWyhDlF+59x0dJhK/8+8TInBCVeQqwhIZecGE2pZfPlFZEIO2/aVtam+My7h/WY9ls=
</HostId>
</Error>

For example [2] fails but for example [3] succeeds.

This seems to be repeatable by file across browsers and with my
co-workers, so I don't think it's just a client issue.

Thanks,
James Coleman

1: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html
2: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/libpq-dev_14.7-1.pgdg110%2B1_amd64.deb
3. https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/postgresql-14_14.1-1.pgdg20.04%2B1_arm64.deb

---

From: Bradford Boyle <[email-masked]>
Date: Thu, 18 May 2023 09:48:21 -0700

Hi James,

apt-archive.p.o is hosted via an AWS S3 bucket which doesn't include/generate
index lists (e.g, index.html) which is why [1] doesn't work. If you are
familiar with the structure of Debian repos, you can view the repo's index by
browsing the appropriate files directly. For example,

https://apt-archive.postgresql.org/pub/repos/apt/dists/${DISTRIBUTION}-pgdg-archive/InRelease

where distribution is bullseye, focal, etc. The "InRelease" file will contain
paths for various "Packages" files. You can view the list of packages by
replacing "InRelease" with the appropraite package path. For example

https://apt-archive.postgresql.org/pub/repos/apt/dists/${DISTRIBUTION}-pgdg-archive/main/binary-amd64/Packages

Using this, we can see why [2] isn't working for you -- there is no libpq-dev
in bullseye-pgdg-archive, the package is libpq5-dev.

Hope this helps somewhat.

--Bradford

---

From: Christoph Berg <[email-masked]>
Date: Thu, 18 May 2023 21:51:31 +0200

Re: James Coleman
> <Code>AccessDenied</Code>

Hmm, admittedly I've never tried to recheck if all the package links
worked after the initial setup, so it's well possible something went
wrong in the years since then.

I'll do some QA there tomorrow. Thanks for reporting!

Re: Bradford Boyle
> index lists (e.g, index.html) which is why [1] doesn't work.

There are manually uploaded index.html files generated from the
database mentioned earlier.

> in bullseye-pgdg-archive, the package is libpq5-dev.

libpq-dev is correct, the 5 is only in libpq5.deb.

Christoph

---

From: Bradford Boyle <[email-masked]>
Date: Thu, 18 May 2023 20:48:51 -0700

> database mentioned earlier.

Yep, I see that now. I should have read the code for
generate-archive-lists more closely before replying.

> libpq-dev is correct, the 5 is only in libpq5.deb.

Right again -- I don't know what I was doing wrong with my grep commands
this morning but I'll blame it on a lack of sleep and coffee.

It looks like the links on [1] are incorrect -- it gives [2] as the link
for libpq-dev_14.7-1.pgdg110+1_amd64.deb but looking at the Package
file, I think the correct link is [3] (i.e., pool/main vs pool/14).

I think the issue has to do with set_package_component in
generate-pgdg-source moving lib packages to a different component when
the source postgresql version is older than PG_MAIN_VERSION; it looks
generate-archive-lists assumes component is always main though.

--Bradford

[1]: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html
[2]: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/libpq-dev_14.7-1.pgdg110%2B1_amd64.deb
[3]: https://apt-archive.postgresql.org/pub/repos/apt/pool/14/p/postgresql-14/libpq-dev_14.7-1.pgdg110%2B1_amd64.deb

---

From: Christoph Berg <[email-masked]>
Date: Fri, 19 May 2023 15:06:30 +0200

Re: Bradford Boyle
> file, I think the correct link is [3] (i.e., pool/main vs pool/14).

Ah right, thanks for spotting that, I had missed that when trying to
debugging the issue earlier today. I remember now that I had omitted
the extra pool locations, but I missed that they are still used.

So in summary, the files are all there, but the path in the
referencing web page is wrong. Will fix.

> generate-archive-lists assumes component is always main though.

Yeah I had the idea that for the archive, I could just ignore the
changed component and shovel everything into "main", but of course in
that case the files would have to be uploaded to that location.

Christoph

---

From: Christoph Berg <[email-masked]>
Date: Fri, 19 May 2023 17:31:27 +0200

Re: To Bradford Boyle
> referencing web page is wrong. Will fix.

I managed to fix the problems, the index listings in package
directories like
https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html
will now also reference files located in other directories like pool/14/p/...

James, Bradford: Thanks for spotting and debugging!

Christoph

---

From: James Coleman <[email-masked]>
Date: Fri, 19 May 2023 22:00:25 -0400

> James, Bradford: Thanks for spotting and debugging!

Thanks for fixing this!

James