public inbox for [email protected]
help / color / mirror / Atom feedapt.postgresql.org django app for www.postgresql.org
12+ messages / 4 participants
[nested] [flat]
* apt.postgresql.org django app for www.postgresql.org
@ 2019-01-22 14:00 Christoph Berg <[email protected]>
2019-01-22 14:14 ` Re: apt.postgresql.org django app for www.postgresql.org Ray O'Donnell <[email protected]>
2019-01-30 15:48 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
0 siblings, 2 replies; 12+ messages in thread
From: Christoph Berg @ 2019-01-22 14:00 UTC (permalink / raw)
To: pgsql-www; +Cc: Adrian Vondendriesch <[email protected]>
Hi,
We have implemented a repository browser for apt.postgresql.org to be
run on www.postgresql.org/repos/apt/. This has been in the works for
years, but has somehow never reached this list.
It is mostly feature-complete, i.e. browsing the list of supported
distributions, source and binary packages works. It supports searching
for packages and filenames in packages.
Dependencies:
python-apt, postgresql-*-debversion, pg_trgm (from postgresql-*-contrib)
To test, create a mirror of a (subset of) apt.postgresql.org:
* debmirror -v -h apt.postgresql.org -r pub/repos/apt --method=http -d sid-pgdg --omit-suite-symlinks -s main,12 -a source,amd64 --exclude='\.deb$' --getcontents --no-check-gpg /srv/repo
* set APT_DIR to that directory in settings_local.py
* run import-packagelists.py
TODO:
* The contained homepage at /repos/apt/ is an outdated mockup, the
list of supported dists should be rendered from the database
* There are no links to /repos/apt/ from anywhere else in the pg.o
page yet
* import-packagelists.py uses django to open the PG connection, but
this should be rewritten to use plain psycopg2.
* The idea is that import-packagelists.py writes directly to the
database, so appropriate GRANTs are needed to allow writing to the
apt_* tables only
* The "qa" part contains queries that have not yet been ported from an
older datamodel yet and is hence disabled
* There is no documentation yet
Still, it already works nicely if you point the browser directly at
/repos/apt, so please review :)
Christoph
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: apt.postgresql.org django app for www.postgresql.org
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
@ 2019-01-22 14:14 ` Ray O'Donnell <[email protected]>
2019-01-22 14:16 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
1 sibling, 1 reply; 12+ messages in thread
From: Ray O'Donnell @ 2019-01-22 14:14 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; pgsql-www; Adrian Vondendriesch <[email protected]>
On 22/01/2019 14:00, Christoph Berg wrote:
> Hi,
>
> We have implemented a repository browser for apt.postgresql.org to be
> run on www.postgresql.org/repos/apt/. This has been in the works for
> years, but has somehow never reached this list.
[snip]
> Still, it already works nicely if you point the browser directly at
> /repos/apt, so please review :)
Maybe I was too quick off the mark, but I'm getting a 404 at the above URL.
Ray.
--
Raymond O'Donnell // Galway // Ireland
[email protected]
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: apt.postgresql.org django app for www.postgresql.org
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-01-22 14:14 ` Re: apt.postgresql.org django app for www.postgresql.org Ray O'Donnell <[email protected]>
@ 2019-01-22 14:16 ` Christoph Berg <[email protected]>
2019-01-22 14:17 ` Re: apt.postgresql.org django app for www.postgresql.org Ray O'Donnell <[email protected]>
0 siblings, 1 reply; 12+ messages in thread
From: Christoph Berg @ 2019-01-22 14:16 UTC (permalink / raw)
To: Ray O'Donnell <[email protected]>; +Cc: pgsql-www; Adrian Vondendriesch <[email protected]>
Re: Ray O'Donnell 2019-01-22 <[email protected]>
> Maybe I was too quick off the mark, but I'm getting a 404 at the above URL.
That's why there's a patch attached :)
Christoph
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: apt.postgresql.org django app for www.postgresql.org
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-01-22 14:14 ` Re: apt.postgresql.org django app for www.postgresql.org Ray O'Donnell <[email protected]>
2019-01-22 14:16 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
@ 2019-01-22 14:17 ` Ray O'Donnell <[email protected]>
0 siblings, 0 replies; 12+ messages in thread
From: Ray O'Donnell @ 2019-01-22 14:17 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; pgsql-www; Adrian Vondendriesch <[email protected]>
On 22/01/2019 14:16, Christoph Berg wrote:
> Re: Ray O'Donnell 2019-01-22 <[email protected]>
>> Maybe I was too quick off the mark, but I'm getting a 404 at the above URL.
>
> That's why there's a patch attached :)
>
Ah, OK, I misunderstood - I thought it was up and running. :-)
Ray.
--
Raymond O'Donnell // Galway // Ireland
[email protected]
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: apt.postgresql.org django app for www.postgresql.org
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
@ 2019-01-30 15:48 ` Christoph Berg <[email protected]>
2019-02-27 15:09 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
1 sibling, 1 reply; 12+ messages in thread
From: Christoph Berg @ 2019-01-30 15:48 UTC (permalink / raw)
To: pgsql-www; Adrian Vondendriesch <[email protected]>
Updated for python3. I had to fix several places where {{package}} was
rendered as "Package Object". The new syntax is {{package.package}}.
Christoph
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: apt.postgresql.org django app for www.postgresql.org
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-01-30 15:48 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
@ 2019-02-27 15:09 ` Christoph Berg <[email protected]>
2019-02-27 15:29 ` Re: apt.postgresql.org django app for www.postgresql.org Jonathan S. Katz <[email protected]>
2020-07-08 13:08 ` Re: apt.postgresql.org django app for www.postgresql.org Dave Page <[email protected]>
0 siblings, 2 replies; 12+ messages in thread
From: Christoph Berg @ 2019-02-27 15:09 UTC (permalink / raw)
To: pgsql-www; Adrian Vondendriesch <[email protected]>
Re: To PostgreSQL WWW 2019-01-30 <[email protected]>
> Updated for python3. I had to fix several places where {{package}} was
> rendered as "Package Object". The new syntax is {{package.package}}.
Ping?
Christoph
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: apt.postgresql.org django app for www.postgresql.org
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-01-30 15:48 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-02-27 15:09 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
@ 2019-02-27 15:29 ` Jonathan S. Katz <[email protected]>
2019-02-27 15:42 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
1 sibling, 1 reply; 12+ messages in thread
From: Jonathan S. Katz @ 2019-02-27 15:29 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; pgsql-www; Adrian Vondendriesch <[email protected]>
On 2/27/19 10:09 AM, Christoph Berg wrote:
> Re: To PostgreSQL WWW 2019-01-30 <[email protected]>
>> Updated for python3. I had to fix several places where {{package}} was
>> rendered as "Package Object". The new syntax is {{package.package}}.
>
> Ping?
tl;dr I do think having a repo browser on pgweb would be nice, but we
need to solve a few fundamental questions first:
1. Will we make the same thing available for the yums?
2. What will be the manual overhead on the pgweb, pginfra, + release team?
And the rest of it:
This will take some involved review. I made a quick pass, and saw a few
things.
First, an off-list comment that Magnus had raised (amongst a few
others): is this be set up to handle RPM/YUM, and if not, what will it
take to get it there? (Will somewhat answer this below).
There are a few things that are immediate nonstarters for me:
1. There is a bunch of DDL hanging out in SQL files that should be in
the Django migration file (and in fact, I don't see a migration file)
2. There appear to be some hardcoded things specific to deb/apt (e.g.
"all" => "amd64") which would not work if we wanted to extend this to
rpm/yum. The question becomes can we / do we want to build a
one-size-fits-all system?
Also:
1. This appears to introduce an extension from PGXN, "debversion" -- I
don't believe we've added any pgxn extensions directly into pgweb (I may
be wrong) but we would have to see what that means from an
administration standpoint.
2. There seem to be some easy improvements to make in some of the logic:
def search(request):
if 'package' in request.GET and request.GET['package']:
package = request.GET['package']
becomes:
def search(request):
if request.GET.get('package'):
package = request.GET['package']
3. For now, the "active" booleans scare me until I understand how much
of this is automated and how much of this is manual. Adding more manual
steps to each release terrifies me, mostly because we have more than our
fair share at the moment :)
Thanks,
Jonathan
Attachments:
[application/pgp-signature] signature.asc (833B, 2-signature.asc)
download
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: apt.postgresql.org django app for www.postgresql.org
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-01-30 15:48 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-02-27 15:09 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-02-27 15:29 ` Re: apt.postgresql.org django app for www.postgresql.org Jonathan S. Katz <[email protected]>
@ 2019-02-27 15:42 ` Christoph Berg <[email protected]>
0 siblings, 0 replies; 12+ messages in thread
From: Christoph Berg @ 2019-02-27 15:42 UTC (permalink / raw)
To: Jonathan S. Katz <[email protected]>; +Cc: pgsql-www; Adrian Vondendriesch <[email protected]>
Re: Jonathan S. Katz 2019-02-27 <[email protected]>
> tl;dr I do think having a repo browser on pgweb would be nice, but we
> need to solve a few fundamental questions first:
>
> 1. Will we make the same thing available for the yums?
I don't think it is fair to burden the extra work of supporting yum on
the initial implementation of this. I don't think it makes much sense
to extend the system to support both via the same because the package
structures are really different (both because of deb vs. rpm and the
actual packaging structure). Of course it makes sense to think about
if we could share some infrastructure with yum, but I don't think that
would fly.
> 2. What will be the manual overhead on the pgweb, pginfra, + release team?
Basically none. I'd handle all the necessary database updates from the
atalia.postgresql.org host by integrating it into the "add new
package" workflow.
> First, an off-list comment that Magnus had raised (amongst a few
> others): is this be set up to handle RPM/YUM, and if not, what will it
> take to get it there? (Will somewhat answer this below).
(See above)
> There are a few things that are immediate nonstarters for me:
>
> 1. There is a bunch of DDL hanging out in SQL files that should be in
> the Django migration file (and in fact, I don't see a migration file)
That's a leftover from when it was first written 5 years ago. Fixable.
> 2. There appear to be some hardcoded things specific to deb/apt (e.g.
> "all" => "amd64") which would not work if we wanted to extend this to
> rpm/yum. The question becomes can we / do we want to build a
> one-size-fits-all system?
"all" is a special snowflake in the Debian architecture list. The
repository solves that by including "all" in each Packages files for
all architectures. If information about an Arch:all package is
requested, we need to pull it from any of the "real" architectures.
> Also:
>
> 1. This appears to introduce an extension from PGXN, "debversion" -- I
> don't believe we've added any pgxn extensions directly into pgweb (I may
> be wrong) but we would have to see what that means from an
> administration standpoint.
It would allow proper comparison of Debian version numbers, but I
think we aren't really using this anywhere, so it could just be
removed.
> def search(request):
> if 'package' in request.GET and request.GET['package']:
> package = request.GET['package']
>
> becomes:
>
> def search(request):
> if request.GET.get('package'):
> package = request.GET['package']
Will fix.
> 3. For now, the "active" booleans scare me until I understand how much
> of this is automated and how much of this is manual. Adding more manual
> steps to each release terrifies me, mostly because we have more than our
> fair share at the moment :)
There are the markers for "is this Debian release still active". This
would be up to me to handle from the apt.pg.o side. (We can't really
automate this.)
Thanks,
Christoph
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: apt.postgresql.org django app for www.postgresql.org
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-01-30 15:48 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-02-27 15:09 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
@ 2020-07-08 13:08 ` Dave Page <[email protected]>
2020-07-08 13:20 ` Re: apt.postgresql.org django app for www.postgresql.org Dave Page <[email protected]>
2020-07-09 09:31 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
1 sibling, 2 replies; 12+ messages in thread
From: Dave Page @ 2020-07-08 13:08 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; pgsql-www; Adrian Vondendriesch <[email protected]>
Hi
On Wed, Feb 27, 2019 at 3:09 PM Christoph Berg <[email protected]> wrote:
> Re: To PostgreSQL WWW 2019-01-30 <[email protected]>
> > Updated for python3. I had to fix several places where {{package}} was
> > rendered as "Package Object". The new syntax is {{package.package}}.
>
> Ping?
>
I spent a bunch of time playing with this, as I intend to get repo
browsing for both Yum and Apt onto the website.
There was quite a bit of work to do to get it working with modern versions
of Django and Python 3. Once I got through enough of that to start looking
at the actual functionality what I found was *really* comprehensive.
Unfortunately I think there's actually far more there than we should put on
the main website.
- I think the QA section is clearly something that's aimed at you as
maintainers of the apt repos. This definitely doesn't belong on the main
website in my opinion.
- The madison interface is also interesting (academically), but I think is
of little use to the vast majority of our users; I'm not even sure that the
majority of Debian/Ubuntu users would know about rmadison.
- Similarly, I think the binary and source package pages are far more
comprehensive than most of our users need or would care about.
One of the biggest barriers of adoption to PostgreSQL is the perceived
complexity, including that of getting it up and running. That's why I'm
spending a lot of time at the moment trying to simplify and clarify the
download and installation processes. I think what we have in this patch
will simply be information overload for most of our users.
My suggestion is that we incorporate a relatively simple browser into the
main website, which allows users to easily browse the available packages
and see the details of them. I already have the repo scanning part of that
done for both apt and yum, generating JSON output in a way that can be
integrated with our download server sync process, which can load that into
the website database.
I would support a separate site (probably under apt.enterprisedb.com) that
supports the level of functionality you have in your patch; and I think
much, if not all of the code you currently have could be used for that.
This could of course be linked from the main website.
Thanks.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: apt.postgresql.org django app for www.postgresql.org
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-01-30 15:48 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-02-27 15:09 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2020-07-08 13:08 ` Re: apt.postgresql.org django app for www.postgresql.org Dave Page <[email protected]>
@ 2020-07-08 13:20 ` Dave Page <[email protected]>
1 sibling, 0 replies; 12+ messages in thread
From: Dave Page @ 2020-07-08 13:20 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; pgsql-www; Adrian Vondendriesch <[email protected]>
On Wed, Jul 8, 2020 at 2:08 PM Dave Page <[email protected]> wrote:
> Hi
>
> On Wed, Feb 27, 2019 at 3:09 PM Christoph Berg <[email protected]> wrote:
>
>> Re: To PostgreSQL WWW 2019-01-30 <[email protected]>
>> > Updated for python3. I had to fix several places where {{package}} was
>> > rendered as "Package Object". The new syntax is {{package.package}}.
>>
>> Ping?
>>
>
> I spent a bunch of time playing with this, as I intend to get repo
> browsing for both Yum and Apt onto the website.
>
> There was quite a bit of work to do to get it working with modern versions
> of Django and Python 3. Once I got through enough of that to start looking
> at the actual functionality what I found was *really* comprehensive.
> Unfortunately I think there's actually far more there than we should put on
> the main website.
>
> - I think the QA section is clearly something that's aimed at you as
> maintainers of the apt repos. This definitely doesn't belong on the main
> website in my opinion.
>
> - The madison interface is also interesting (academically), but I think is
> of little use to the vast majority of our users; I'm not even sure that the
> majority of Debian/Ubuntu users would know about rmadison.
>
> - Similarly, I think the binary and source package pages are far more
> comprehensive than most of our users need or would care about.
>
> One of the biggest barriers of adoption to PostgreSQL is the perceived
> complexity, including that of getting it up and running. That's why I'm
> spending a lot of time at the moment trying to simplify and clarify the
> download and installation processes. I think what we have in this patch
> will simply be information overload for most of our users.
>
> My suggestion is that we incorporate a relatively simple browser into the
> main website, which allows users to easily browse the available packages
> and see the details of them. I already have the repo scanning part of that
> done for both apt and yum, generating JSON output in a way that can be
> integrated with our download server sync process, which can load that into
> the website database.
>
> I would support a separate site (probably under apt.enterprisedb.com)
> that supports the level of functionality you have in your patch; and I
> think much, if not all of the code you currently have could be used for
> that. This could of course be linked from the main website.
>
Obviously that should be apt.postgresql.org :-)
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: apt.postgresql.org django app for www.postgresql.org
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-01-30 15:48 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-02-27 15:09 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2020-07-08 13:08 ` Re: apt.postgresql.org django app for www.postgresql.org Dave Page <[email protected]>
@ 2020-07-09 09:31 ` Christoph Berg <[email protected]>
2020-07-10 12:50 ` Re: apt.postgresql.org django app for www.postgresql.org Dave Page <[email protected]>
1 sibling, 1 reply; 12+ messages in thread
From: Christoph Berg @ 2020-07-09 09:31 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: pgsql-www; Adrian Vondendriesch <[email protected]>
Re: Dave Page
> I spent a bunch of time playing with this, as I intend to get repo
> browsing for both Yum and Apt onto the website.
>
> There was quite a bit of work to do to get it working with modern versions
> of Django and Python 3. Once I got through enough of that to start looking
> at the actual functionality what I found was *really* comprehensive.
> Unfortunately I think there's actually far more there than we should put on
> the main website.
Hi Dave,
thanks for picking this up.
It's well possible that I overshot the goal when I picked this up from
Adrian and put more and more info into it.
> - I think the QA section is clearly something that's aimed at you as
> maintainers of the apt repos. This definitely doesn't belong on the main
> website in my opinion.
Yeah that could go.
> - The madison interface is also interesting (academically), but I think is
> of little use to the vast majority of our users; I'm not even sure that the
> majority of Debian/Ubuntu users would know about rmadison.
I'm using that daily on the Debian archive, and it would help me a lot
if it were there. But we don't have to link it from every page, it's
just some API endpoint, we don't have to confuse users by linking to
it.
> - Similarly, I think the binary and source package pages are far more
> comprehensive than most of our users need or would care about.
>
> One of the biggest barriers of adoption to PostgreSQL is the perceived
> complexity, including that of getting it up and running. That's why I'm
> spending a lot of time at the moment trying to simplify and clarify the
> download and installation processes. I think what we have in this patch
> will simply be information overload for most of our users.
Ack. We can probably merge the source and binary views into a single
(bigger) page with less clutter that users would reach by default. We
can still have the detailed pages linked from that for users that need
to know the details. I'm sure we can find a way to do that that
doesn't spoil the complexity reduction idea.
> My suggestion is that we incorporate a relatively simple browser into the
> main website, which allows users to easily browse the available packages
> and see the details of them.
That'd be about what I said above, I think.
> I already have the repo scanning part of that
> done for both apt and yum, generating JSON output in a way that can be
> integrated with our download server sync process, which can load that into
> the website database.
Is that online somewhere?
> I would support a separate site (probably under apt.enterprisedb.com) that
> supports the level of functionality you have in your patch; and I think
> much, if not all of the code you currently have could be used for that.
> This could of course be linked from the main website.
Maintaining two sets of interfaces is probably too much. I think we
can get the "main" one to work, we just need to remove lots of
clutter.
> Obviously that should be apt.postgresql.org :-)
That got me for a second. ;)
Christoph
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: apt.postgresql.org django app for www.postgresql.org
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-01-30 15:48 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-02-27 15:09 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2020-07-08 13:08 ` Re: apt.postgresql.org django app for www.postgresql.org Dave Page <[email protected]>
2020-07-09 09:31 ` Re: apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
@ 2020-07-10 12:50 ` Dave Page <[email protected]>
0 siblings, 0 replies; 12+ messages in thread
From: Dave Page @ 2020-07-10 12:50 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; Dave Page <[email protected]>; pgsql-www; Adrian Vondendriesch <[email protected]>
On Thu, Jul 9, 2020 at 10:31 AM Christoph Berg <[email protected]> wrote:
> Re: Dave Page
> > I spent a bunch of time playing with this, as I intend to get repo
> > browsing for both Yum and Apt onto the website.
> >
> > There was quite a bit of work to do to get it working with modern
> versions
> > of Django and Python 3. Once I got through enough of that to start
> looking
> > at the actual functionality what I found was *really* comprehensive.
> > Unfortunately I think there's actually far more there than we should put
> on
> > the main website.
>
> Hi Dave,
>
> thanks for picking this up.
>
> It's well possible that I overshot the goal when I picked this up from
> Adrian and put more and more info into it.
>
:-)
>
> > - I think the QA section is clearly something that's aimed at you as
> > maintainers of the apt repos. This definitely doesn't belong on the main
> > website in my opinion.
>
> Yeah that could go.
>
> > - The madison interface is also interesting (academically), but I think
> is
> > of little use to the vast majority of our users; I'm not even sure that
> the
> > majority of Debian/Ubuntu users would know about rmadison.
>
> I'm using that daily on the Debian archive, and it would help me a lot
> if it were there. But we don't have to link it from every page, it's
> just some API endpoint, we don't have to confuse users by linking to
> it.
>
> > - Similarly, I think the binary and source package pages are far more
> > comprehensive than most of our users need or would care about.
> >
> > One of the biggest barriers of adoption to PostgreSQL is the perceived
> > complexity, including that of getting it up and running. That's why I'm
> > spending a lot of time at the moment trying to simplify and clarify the
> > download and installation processes. I think what we have in this patch
> > will simply be information overload for most of our users.
>
> Ack. We can probably merge the source and binary views into a single
> (bigger) page with less clutter that users would reach by default. We
> can still have the detailed pages linked from that for users that need
> to know the details. I'm sure we can find a way to do that that
> doesn't spoil the complexity reduction idea.
>
Yeah. As a side note, the other thing I'm trying to do is be fairly
consistent design-wise between apt, yum and zypp.
>
> > My suggestion is that we incorporate a relatively simple browser into the
> > main website, which allows users to easily browse the available packages
> > and see the details of them.
>
> That'd be about what I said above, I think.
>
> > I already have the repo scanning part of that
> > done for both apt and yum, generating JSON output in a way that can be
> > integrated with our download server sync process, which can load that
> into
> > the website database.
>
> Is that online somewhere?
>
No, but I've attached the WIP scripts. Comments welcome. Note that I've
studiously avoided using any modules or external utilities that are only
available on Debian/Ubuntu. Sample output for Apt looks like:
[
{
"Architecture": "arm64",
"Build": "3.pgdg+1",
"Description": "debug symbols for pg-rage-terminator-9.6",
"Distribution": "sid",
"Filename":
"pool/main/p/pg-rage-terminator/pg-rage-terminator-9.6-dbgsym_0.1.7-3.pgdg+1_arm64.deb",
"Licence": "",
"Maintainer": "Adrian Vondendriesch <[email protected]>",
"Package": "pg-rage-terminator-9.6-dbgsym",
"Repo": "sid-pgdg-testing",
"Version": "0.1.7"
},
{
"Architecture": "arm64",
"Build": "1.pgdg+1",
"Description": "PostgreSQL management tool - GUI application\npgAdmin
is an open source administration and management tool for the\nPostgreSQL
database. It includes a graphical administration interface, an SQL\nquery
tool, a procedural code debugger and much more. The tool is designed
to\nanswer the needs of developers, DBAs and system administrators
alike.\n\nThis package installs the GUI application.",
"Distribution": "sid",
"Filename": "pool/main/p/pgadmin4/pgadmin4_4.21-1.pgdg+1_arm64.deb",
"Licence": "",
"Maintainer": "Debian PostgreSQL Maintainers <
[email protected]>",
"Package": "pgadmin4",
"Repo": "sid-pgdg-testing",
"Url": "https://www.pgadmin.org/";,
"Version": "4.21"
}
]
And for Yum:
[
{
"Architecture": "ppc64le",
"Build": "9.rhel7",
"Description": "pgAgent is a job scheduler for PostgreSQL which may be
managed\nusing pgAdmin.",
"Distribution": "rhel-7",
"Filename":
"9.5/redhat/rhel-7-ppc64le/pgagent_95-3.4.0-9.rhel7.ppc64le.rpm",
"Licence": "PostgreSQL",
"Maintainer": "",
"Package": "pgagent_95",
"Repo": "9.5",
"Url": "http://www.pgadmin.org/";,
"Version": "3.4.0"
},
{
"Architecture": "ppc64le",
"Build": "1.f25",
"Description": "The PostgreSQL Audit extension (pgaudit) provides
detailed session\nand/or object audit logging via the standard PostgreSQL
logging\nfacility.\n\nThe goal of the PostgreSQL Audit extension (pgaudit)
is to provide\nPostgreSQL users with capability to produce audit logs often
required to\ncomply with government, financial, or ISO
certifications.\n\nAn audit is an official inspection of an individual's or
organization's\naccounts, typically by an independent body. The information
gathered by\nthe PostgreSQL Audit extension (pgaudit) is properly called an
audit\ntrail or audit log. The term audit log is used in this
documentation.",
"Distribution": "rhel-7",
"Filename":
"9.5/redhat/rhel-7-ppc64le/pgaudit10_95-1.0.5-1.f25.ppc64le.rpm",
"Licence": "BSD",
"Maintainer": "",
"Package": "pgaudit10_95",
"Repo": "9.5",
"Url": "https://www.pgaudit.org";,
"Version": "1.0.5"
}
]
>
> > I would support a separate site (probably under apt.enterprisedb.com)
> that
> > supports the level of functionality you have in your patch; and I think
> > much, if not all of the code you currently have could be used for that.
> > This could of course be linked from the main website.
>
> Maintaining two sets of interfaces is probably too much. I think we
> can get the "main" one to work, we just need to remove lots of
> clutter.
>
I have a suspicion that what I think would be appropriate for most of our
users wouldn't be enough for you. Definitely open to discussion though!
>
> > Obviously that should be apt.postgresql.org :-)
>
> That got me for a second. ;)
Yeah, muscle memory :-p
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EDB: http://www.enterprisedb.com
Attachments:
[text/x-python-script] scan-apt.py (4.4K, 3-scan-apt.py)
download | inline:
import argparse
import json
import os
from pathlib import Path
def get_package_files(repo_dir):
# Get a list of all the Packages files
package_files = []
for path in Path(repo_dir).rglob('Packages'):
package_files.append(path.absolute())
return package_files
def get_distribution(package_file):
# Find the Release file corresponding to the Packages file, and get the
# distribution name from it.
with open(os.path.dirname(package_file) + '/Release', "r") as release_data:
for release_line in release_data:
if release_line.startswith("Archive: "):
return release_line[9:]
return None
def get_licence(deb_file):
# TODO: Figure out a sane way to get the license
license = ''
return license
def get_packages(package_file):
# Extract all the packages from a package file
packages = []
with open(package_file, "r") as package_data:
package = ""
first = True
for line in package_data:
if line.startswith('Package: ') and not first:
# Get the distribution
distribution = get_distribution(package_file)
if distribution is not None:
package = package + '\nDistribution: ' + distribution
packages.append(package)
package = ""
else:
first = False
package = package + line
return packages
def get_package(package_data):
# Decode a Package entry into a dictionary
package = {}
in_description = False
for line in package_data.splitlines():
# Package
if line.startswith("Package: "):
package['Package'] = line[9:]
# Version
if line.startswith("Version: "):
# The build is normally prefixed with a -, but sometimes
# just .pgdg
package['Version'] = line[9:].split('-')[0]
package['Build'] = ''.join(line[9:].split('-')[1:])
if '.pgdg' in package['Version']:
version = line[9:].split('.pgdg')
package['Version'] = version[0]
package['Build'] = 'pgdg' + version[1]
# Architecture
if line.startswith("Architecture: "):
package['Architecture'] = line[14:]
# Filename
if line.startswith("Filename: "):
package['Filename'] = line[10:]
# Licence
licence = get_licence(line[10:])
if licence is not None:
package['Licence'] = licence
# Description. This can be multi-line. Treat the first line
# as normal, then scan the rest until we hit the end
if in_description:
if line.strip() == ".":
package['Description'] = package['Description'] + "\n"
# The description ends when we encounter a line that doesn't start
# with a space.
elif not line.startswith(" "):
in_description = False
else:
package['Description'] = \
package['Description'] + '\n' + line.strip()
if line.startswith("Description: "):
package['Description'] = line[13:]
in_description = True
# Distribution/Repo
if line.startswith("Distribution: "):
package['Distribution'] = line[14:].split('-')[0]
package['Repo'] = line[14:]
# URL
if line.startswith("Homepage: "):
package['Url'] = line[10:]
# Packager
if line.startswith("Maintainer: "):
package['Maintainer'] = line[12:]
return package
def main():
# Command line arguments
parser = argparse.ArgumentParser(description='Scan a set of APT repos and '
'generate a JSON catalog of '
'the contents.')
parser.add_argument("repo", help="the repo directory, or directory "
"containing multiple repos")
args = parser.parse_args()
package_info = []
package_files = get_package_files(args.repo)
for package_file in package_files:
packages = get_packages(package_file)
for package in packages:
package_info.append(get_package(package))
with open('apt.json', 'w') as output_file:
json.dump(package_info, output_file, indent=2, sort_keys=True)
if __name__ == "__main__":
main()
[text/x-sh] sync-repos.sh (483B, 4-sync-repos.sh)
download | inline:
#!/bin/sh
echo "Syncing APT repos..."
rsync -avz --delete --include "*/" --include="Release" --include="Packages" --exclude="*" rsync://ftp.postgresql.org/pgsql-ftp/repos/apt .
echo "Syncing Yum repos..."
rsync -avz --delete --include "*/" --include="*/repodata/*" --exclude="*" rsync://ftp.postgresql.org/pgsql-ftp/repos/yum .
echo "Syncing Zypp repos..."
rsync -avz --delete --include "*/" --include="*/repodata/*" --exclude="*" rsync://ftp.postgresql.org/pgsql-ftp/repos/zypp .
[text/x-python-script] scan-yum.py (2.1K, 5-scan-yum.py)
download | inline:
import argparse
import json
import os
from pathlib import Path
import repomd
def get_repos(repo_dir):
# Get a list of all repo dirs
repos = []
for path in Path(repo_dir).rglob('repomd.xml'):
repos.append(path.parent.parent.absolute())
return repos
def get_distribution(repo):
distribution_dir = repo.name
parts = distribution_dir.split('-')
distribution = '-'.join(parts[:2])
return distribution
def get_package_info(repo, base_path):
packages = []
repo_data = repomd.load('file://' + str(repo))
for package_data in repo_data:
package = {'Package': package_data.name,
'Version': package_data.version,
'Build': package_data.release,
'Architecture': package_data.arch,
'Filename': os.path.relpath(str(repo) + '/' +
package_data.location,
base_path),
'Description': package_data.description,
'Distribution': get_distribution(repo),
'Url': package_data.url,
'Maintainer': package_data.vendor,
'Repo': os.path.relpath(str(repo), base_path).split('/')[0],
'Licence': package_data.license
}
packages.append(package)
return packages
def main():
# Command line arguments
parser = argparse.ArgumentParser(description='Scan a set of APT repos and '
'generate a JSON catalog of '
'the contents.')
parser.add_argument("repo", help="the repo directory, or directory "
"containing multiple repos")
args = parser.parse_args()
package_info = []
repos = get_repos(args.repo)
for repo in repos:
package_info.extend(get_package_info(repo, args.repo))
with open('yum.json', 'w') as output_file:
json.dump(package_info, output_file, indent=2, sort_keys=True)
if __name__ == "__main__":
main()
^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2020-07-10 12:50 UTC | newest]
Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-01-22 14:00 apt.postgresql.org django app for www.postgresql.org Christoph Berg <[email protected]>
2019-01-22 14:14 ` Ray O'Donnell <[email protected]>
2019-01-22 14:16 ` Christoph Berg <[email protected]>
2019-01-22 14:17 ` Ray O'Donnell <[email protected]>
2019-01-30 15:48 ` Christoph Berg <[email protected]>
2019-02-27 15:09 ` Christoph Berg <[email protected]>
2019-02-27 15:29 ` Jonathan S. Katz <[email protected]>
2019-02-27 15:42 ` Christoph Berg <[email protected]>
2020-07-08 13:08 ` Dave Page <[email protected]>
2020-07-08 13:20 ` Dave Page <[email protected]>
2020-07-09 09:31 ` Christoph Berg <[email protected]>
2020-07-10 12:50 ` Dave Page <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox