pg.ddx.io pgsql-www@postgresql.org mailing list archive
help / color / mirror / Atom feedPython Tooling
9+ messages / 3 participants
[nested] [flat]
* Python Tooling
@ 2026-08-17 16:13 Tristan Partin <tristan@partin.io>
0 siblings, 1 reply; 9+ messages in thread
From: Tristan Partin @ 2026-08-17 16:13 UTC (permalink / raw)
To: pgsql-www <pgsql-www@lists.postgresql.org>
If I spent some time to improve the Python tooling for pgweb, would it
be accepted? Thinks we could do:
- Add Python typing and type checking
- Add linter checks
- Use pyproject.toml
- Update dependencies
Additionally, I would like to understand how people actually develop
this repo. `pip install -r requirements.txt` doesn't work for me because
my Python 3.11 toolchain cannot compile the version of psycopg that we
pin. Python 3.11 is what is recommended in the docs.
--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Python Tooling
@ 2026-08-17 16:24 Magnus Hagander <magnus@hagander.net>
parent: Tristan Partin <tristan@partin.io>
0 siblings, 1 reply; 9+ messages in thread
From: Magnus Hagander @ 2026-08-17 16:24 UTC (permalink / raw)
To: Tristan Partin <tristan@partin.io>; +Cc: pgsql-www <pgsql-www@lists.postgresql.org>
On Mon, 17 Aug 2026 at 18:13, Tristan Partin <tristan@partin.io> wrote:
> If I spent some time to improve the Python tooling for pgweb, would it
> be accepted? Thinks we could do:
>
> - Add Python typing and type checking
>
That would be very useful I think.
> - Add linter checks
>
That may or may not be useful depending on exactly what it is :) But
probably.
> - Use pyproject.toml
> - Update dependencies
>
> Additionally, I would like to understand how people actually develop
> this repo. `pip install -r requirements.txt` doesn't work for me because
> my Python 3.11 toolchain cannot compile the version of psycopg that we
> pin. Python 3.11 is what is recommended in the docs.
>
>
The basic answer is, "use whatever is in debian" because that's what it's
deployed on, and at this point that's trixie.
That also means we do *not* want to change the dependencies to anything
other than whatr's there.
But unfortunately, we (which mostly means me) are very bad at keeping them
up to date. Because for example Debian Trixie is on python 3.13, not 3.11.
And since our currently existing tests are literally just "passes
pycodestyle", it wouldn't catch that (and those tests pass on both 3.11 and
3.13).
//Magnus
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Python Tooling
@ 2026-08-31 20:23 Tristan Partin <tristan@partin.io>
parent: Magnus Hagander <magnus@hagander.net>
0 siblings, 2 replies; 9+ messages in thread
From: Tristan Partin @ 2026-08-31 20:23 UTC (permalink / raw)
To: Magnus Hagander <magnus@hagander.net>; +Cc: pgsql-www <pgsql-www@lists.postgresql.org>
On Mon Aug 17, 2026 at 4:24 PM UTC, Magnus Hagander wrote:
> On Mon, 17 Aug 2026 at 18:13, Tristan Partin <tristan@partin.io> wrote:
>
>> If I spent some time to improve the Python tooling for pgweb, would it
>> be accepted? Thinks we could do:
>>
>> - Add Python typing and type checking
>>
>
> That would be very useful I think.
Noted.
>> - Add linter checks
>>
>
> That may or may not be useful depending on exactly what it is :) But
> probably.
I'll see what this could look like.
>> - Use pyproject.toml
>> - Update dependencies
>>
>> Additionally, I would like to understand how people actually develop
>> this repo. `pip install -r requirements.txt` doesn't work for me because
>> my Python 3.11 toolchain cannot compile the version of psycopg that we
>> pin. Python 3.11 is what is recommended in the docs.
>>
>>
> The basic answer is, "use whatever is in debian" because that's what it's
> deployed on, and at this point that's trixie.
>
> That also means we do *not* want to change the dependencies to anything
> other than whatr's there.
Does this mean that we rely on the versions of Python packages within
Trixie and the requirements.txt is basically ignored?
> But unfortunately, we (which mostly means me) are very bad at keeping them
> up to date. Because for example Debian Trixie is on python 3.13, not 3.11.
> And since our currently existing tests are literally just "passes
> pycodestyle", it wouldn't catch that (and those tests pass on both 3.11 and
> 3.13).
Maybe we can change that.
--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Python Tooling
@ 2026-08-31 20:26 Jonathan S. Katz <jkatz@postgresql.org>
parent: Tristan Partin <tristan@partin.io>
1 sibling, 1 reply; 9+ messages in thread
From: Jonathan S. Katz @ 2026-08-31 20:26 UTC (permalink / raw)
To: Tristan Partin <tristan@partin.io>; Magnus Hagander <magnus@hagander.net>; +Cc: pgsql-www <pgsql-www@lists.postgresql.org>
On 8/31/26 4:23 PM, Tristan Partin wrote:
>> The basic answer is, "use whatever is in debian" because that's what it's
>> deployed on, and at this point that's trixie.
>>
>> That also means we do *not* want to change the dependencies to anything
>> other than whatr's there.
>
> Does this mean that we rely on the versions of Python packages within
> Trixie and the requirements.txt is basically ignored?
FWIW I will use requirements.txt in my local environment, but that only
happens when it gets nuked, so it's a once-every-few-years ordeal.
(I could probably get the pgweb env to load from a container; perhaps
this is a suggestion towards a Dockerfile?)
Jonathan
Attachments:
[application/pgp-signature] OpenPGP_signature.asc (839B, ../../a323d506-699b-4dc5-ad38-9486c4913f9f@postgresql.org/2-OpenPGP_signature.asc)
download
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Python Tooling
@ 2026-08-31 20:30 Magnus Hagander <magnus@hagander.net>
parent: Tristan Partin <tristan@partin.io>
1 sibling, 0 replies; 9+ messages in thread
From: Magnus Hagander @ 2026-08-31 20:30 UTC (permalink / raw)
To: Tristan Partin <tristan@partin.io>; +Cc: pgsql-www <pgsql-www@lists.postgresql.org>
On Mon, 31 Aug 2026 at 22:23, Tristan Partin <tristan@partin.io> wrote:
> On Mon Aug 17, 2026 at 4:24 PM UTC, Magnus Hagander wrote:
> > On Mon, 17 Aug 2026 at 18:13, Tristan Partin <tristan@partin.io> wrote:
> >> - Use pyproject.toml
> >> - Update dependencies
> >>
> >> Additionally, I would like to understand how people actually develop
> >> this repo. `pip install -r requirements.txt` doesn't work for me because
> >> my Python 3.11 toolchain cannot compile the version of psycopg that we
> >> pin. Python 3.11 is what is recommended in the docs.
> >>
> >>
> > The basic answer is, "use whatever is in debian" because that's what it's
> > deployed on, and at this point that's trixie.
> >
> > That also means we do *not* want to change the dependencies to anything
> > other than whatr's there.
>
> Does this mean that we rely on the versions of Python packages within
> Trixie and the requirements.txt is basically ignored?
>
Correct. The only things we're currently pulling from python reqiurements
is django itself (and the associated asgiref) and the cvss package (which
is not available on Debian).
> But unfortunately, we (which mostly means me) are very bad at keeping them
> > up to date. Because for example Debian Trixie is on python 3.13, not
> 3.11.
> > And since our currently existing tests are literally just "passes
> > pycodestyle", it wouldn't catch that (and those tests pass on both 3.11
> and
> > 3.13).
>
> Maybe we can change that.
>
Yeah, that would definitely be an improvement :)
--
Magnus Hagander
Me: https://www.hagander.net/ <http://www.hagander.net/;
Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/;
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Python Tooling
@ 2026-08-31 20:32 Tristan Partin <tristan@partin.io>
parent: Jonathan S. Katz <jkatz@postgresql.org>
0 siblings, 1 reply; 9+ messages in thread
From: Tristan Partin @ 2026-08-31 20:32 UTC (permalink / raw)
To: Jonathan S. Katz <jkatz@postgresql.org>; Magnus Hagander <magnus@hagander.net>; +Cc: pgsql-www <pgsql-www@lists.postgresql.org>
On Mon Aug 31, 2026 at 8:26 PM UTC, Jonathan S. Katz wrote:
> On 8/31/26 4:23 PM, Tristan Partin wrote:
>
>>> The basic answer is, "use whatever is in debian" because that's what it's
>>> deployed on, and at this point that's trixie.
>>>
>>> That also means we do *not* want to change the dependencies to anything
>>> other than whatr's there.
>>
>> Does this mean that we rely on the versions of Python packages within
>> Trixie and the requirements.txt is basically ignored?
>
> FWIW I will use requirements.txt in my local environment, but that only
> happens when it gets nuked, so it's a once-every-few-years ordeal.
>
> (I could probably get the pgweb env to load from a container; perhaps
> this is a suggestion towards a Dockerfile?)
Are you referring to deploying as a container or a devcontainer? To
expand on what I am thinking that we should use native Python tooling to
manage dependencies. For instance, we would encode all our dependencies
in the pyproject.toml, and then actually run the server with `uv run
...`.
Python tooling moves so much faster than Debian that I don't think it
makes sense to depend on Debian for anything other than the base OS and
maybe the Python interpreter.
--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Python Tooling
@ 2026-08-31 20:38 Magnus Hagander <magnus@hagander.net>
parent: Tristan Partin <tristan@partin.io>
0 siblings, 1 reply; 9+ messages in thread
From: Magnus Hagander @ 2026-08-31 20:38 UTC (permalink / raw)
To: Tristan Partin <tristan@partin.io>; +Cc: Jonathan S. Katz <jkatz@postgresql.org>; pgsql-www <pgsql-www@lists.postgresql.org>
On Mon, 31 Aug 2026 at 22:32, Tristan Partin <tristan@partin.io> wrote:
> On Mon Aug 31, 2026 at 8:26 PM UTC, Jonathan S. Katz wrote:
> > On 8/31/26 4:23 PM, Tristan Partin wrote:
> >
> >>> The basic answer is, "use whatever is in debian" because that's what
> it's
> >>> deployed on, and at this point that's trixie.
> >>>
> >>> That also means we do *not* want to change the dependencies to anything
> >>> other than whatr's there.
> >>
> >> Does this mean that we rely on the versions of Python packages within
> >> Trixie and the requirements.txt is basically ignored?
> >
> > FWIW I will use requirements.txt in my local environment, but that only
> > happens when it gets nuked, so it's a once-every-few-years ordeal.
> >
> > (I could probably get the pgweb env to load from a container; perhaps
> > this is a suggestion towards a Dockerfile?)
>
> Are you referring to deploying as a container or a devcontainer? To
> expand on what I am thinking that we should use native Python tooling to
> manage dependencies. For instance, we would encode all our dependencies
> in the pyproject.toml, and then actually run the server with `uv run
> ...`.
>
> Python tooling moves so much faster than Debian that I don't think it
> makes sense to depend on Debian for anything other than the base OS and
> maybe the Python interpreter.
>
I think you're going to have a very steep uphill battle to fight to get
that done. Mainly because the vast majority of dependencies we have lack
the basic idea of a stable release/branch, LTS or whatever you want to call
it. Django itself does, which makes it work. But we don't have the
resources to go around and manually figure out which changes are needed to
which of our many apps whenever we're forced to upgrade something because
of a security update. For very little actual gain, since just sticking to
the supported versions has served us very well for many years.
--
Magnus Hagander
Me: https://www.hagander.net/ <http://www.hagander.net/;
Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/;
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Python Tooling
@ 2026-08-31 21:06 Tristan Partin <tristan@partin.io>
parent: Magnus Hagander <magnus@hagander.net>
0 siblings, 1 reply; 9+ messages in thread
From: Tristan Partin @ 2026-08-31 21:06 UTC (permalink / raw)
To: Magnus Hagander <magnus@hagander.net>; +Cc: Jonathan S. Katz <jkatz@postgresql.org>; pgsql-www <pgsql-www@lists.postgresql.org>
On Mon Aug 31, 2026 at 8:38 PM UTC, Magnus Hagander wrote:
> On Mon, 31 Aug 2026 at 22:32, Tristan Partin <tristan@partin.io> wrote:
>
>> On Mon Aug 31, 2026 at 8:26 PM UTC, Jonathan S. Katz wrote:
>> > On 8/31/26 4:23 PM, Tristan Partin wrote:
>> >
>> >>> The basic answer is, "use whatever is in debian" because that's what
>> it's
>> >>> deployed on, and at this point that's trixie.
>> >>>
>> >>> That also means we do *not* want to change the dependencies to anything
>> >>> other than whatr's there.
>> >>
>> >> Does this mean that we rely on the versions of Python packages within
>> >> Trixie and the requirements.txt is basically ignored?
>> >
>> > FWIW I will use requirements.txt in my local environment, but that only
>> > happens when it gets nuked, so it's a once-every-few-years ordeal.
>> >
>> > (I could probably get the pgweb env to load from a container; perhaps
>> > this is a suggestion towards a Dockerfile?)
>>
>> Are you referring to deploying as a container or a devcontainer? To
>> expand on what I am thinking that we should use native Python tooling to
>> manage dependencies. For instance, we would encode all our dependencies
>> in the pyproject.toml, and then actually run the server with `uv run
>> ...`.
>>
>> Python tooling moves so much faster than Debian that I don't think it
>> makes sense to depend on Debian for anything other than the base OS and
>> maybe the Python interpreter.
>>
>
> I think you're going to have a very steep uphill battle to fight to get
> that done. Mainly because the vast majority of dependencies we have lack
> the basic idea of a stable release/branch, LTS or whatever you want to call
> it. Django itself does, which makes it work. But we don't have the
> resources to go around and manually figure out which changes are needed to
> which of our many apps whenever we're forced to upgrade something because
> of a security update. For very little actual gain, since just sticking to
> the supported versions has served us very well for many years.
I hear your concerns. Two thoughts:
1) The fact that Debian is an LTS distro does not necessarily mean that
security updates will get backported. For instance, if software xyz has
a vulnerability in version 1.X that isn't fixed until version 2.3, it is
seriously unlikely that Debian has the resources to maintain a forked
version of 1.X with the security fix. As you said, most software doesn't
maintain an LTS branch. Debian tracks CVEs affecting its current package
set[0]. I counted 48 CVEs affecting Python and other Python packages.
I decided to look at one of these for an example: CVE-2026-47180[1]. It
affects the python3-zeroconf package[2]. Trixie has version 0.147.0
packaged, which was released just about a year ago. GitHub reports[3]
that the CVE was fixed in 0.149.5, and affected versions are everything
less than that. This CVE was published on 2026-05-21, so Debian has been
knowingly vulnerable for over 3 months. If we managed our own
dependencies, we could have updated the day of the disclosure to 0.149.5
which was released 4 days before the CVE was published. I understand
being able to do this requires resources and time...
2. But I think that this problem can largely be automated away. On other
software forges (GitHub, etc.), you can have automations for bumping
dependencies upon new releases, and the forges can notify you of
security vulnerabilities in your dependencies. If we didn't want to move
Git hosting to other forges, we could also do our own automations using
systemd timers that automagically sends patches to the list when a new
release of one of our dependencies is published.
Out of curiosity, how often is the Debian server restarted to pull in
new updates?
[0]: https://security-tracker.debian.org/tracker/status/release/stable
[1]: https://security-tracker.debian.org/tracker/CVE-2026-47180
[2]: https://packages.debian.org/trixie/python3-zeroconf
[3]: https://github.com/python-zeroconf/python-zeroconf/security/advisories/GHSA-9pgc-3ccv-5297
--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Python Tooling
@ 2026-09-01 21:39 Magnus Hagander <magnus@hagander.net>
parent: Tristan Partin <tristan@partin.io>
0 siblings, 0 replies; 9+ messages in thread
From: Magnus Hagander @ 2026-09-01 21:39 UTC (permalink / raw)
To: Tristan Partin <tristan@partin.io>; +Cc: Jonathan S. Katz <jkatz@postgresql.org>; pgsql-www <pgsql-www@lists.postgresql.org>
On Mon, 31 Aug 2026 at 23:06, Tristan Partin <tristan@partin.io> wrote:
> On Mon Aug 31, 2026 at 8:38 PM UTC, Magnus Hagander wrote:
> > On Mon, 31 Aug 2026 at 22:32, Tristan Partin <tristan@partin.io> wrote:
> >
> >> On Mon Aug 31, 2026 at 8:26 PM UTC, Jonathan S. Katz wrote:
> >> > On 8/31/26 4:23 PM, Tristan Partin wrote:
> >> >
> >> >>> The basic answer is, "use whatever is in debian" because that's what
> >> it's
> >> >>> deployed on, and at this point that's trixie.
> >> >>>
> >> >>> That also means we do *not* want to change the dependencies to
> anything
> >> >>> other than whatr's there.
> >> >>
> >> >> Does this mean that we rely on the versions of Python packages within
> >> >> Trixie and the requirements.txt is basically ignored?
> >> >
> >> > FWIW I will use requirements.txt in my local environment, but that
> only
> >> > happens when it gets nuked, so it's a once-every-few-years ordeal.
> >> >
> >> > (I could probably get the pgweb env to load from a container; perhaps
> >> > this is a suggestion towards a Dockerfile?)
> >>
> >> Are you referring to deploying as a container or a devcontainer? To
> >> expand on what I am thinking that we should use native Python tooling to
> >> manage dependencies. For instance, we would encode all our dependencies
> >> in the pyproject.toml, and then actually run the server with `uv run
> >> ...`.
> >>
> >> Python tooling moves so much faster than Debian that I don't think it
> >> makes sense to depend on Debian for anything other than the base OS and
> >> maybe the Python interpreter.
> >>
> >
> > I think you're going to have a very steep uphill battle to fight to get
> > that done. Mainly because the vast majority of dependencies we have lack
> > the basic idea of a stable release/branch, LTS or whatever you want to
> call
> > it. Django itself does, which makes it work. But we don't have the
> > resources to go around and manually figure out which changes are needed
> to
> > which of our many apps whenever we're forced to upgrade something because
> > of a security update. For very little actual gain, since just sticking to
> > the supported versions has served us very well for many years.
>
> I hear your concerns. Two thoughts:
>
> 1) The fact that Debian is an LTS distro does not necessarily mean that
> security updates will get backported. For instance, if software xyz has
> a vulnerability in version 1.X that isn't fixed until version 2.3, it is
> seriously unlikely that Debian has the resources to maintain a forked
> version of 1.X with the security fix. As you said, most software doesn't
> maintain an LTS branch. Debian tracks CVEs affecting its current package
> set[0]. I counted 48 CVEs affecting Python and other Python packages.
I decided to look at one of these for an example: CVE-2026-47180[1]. It
> affects the python3-zeroconf package[2]. Trixie has version 0.147.0
> packaged, which was released just about a year ago. GitHub reports[3]
> that the CVE was fixed in 0.149.5, and affected versions are everything
> less than that. This CVE was published on 2026-05-21, so Debian has been
> knowingly vulnerable for over 3 months. If we managed our own
> dependencies, we could have updated the day of the disclosure to 0.149.5
> which was released 4 days before the CVE was published. I understand
> being able to do this requires resources and time...
>
Oh absolutely, it's far from perfect. The most important thing remains, of
course, to minimize dependencies against packages that don't have a stable
branch handling as much as at all possible.
> 2. But I think that this problem can largely be automated away. On other
> software forges (GitHub, etc.), you can have automations for bumping
> dependencies upon new releases, and the forges can notify you of
> security vulnerabilities in your dependencies. If we didn't want to move
> Git hosting to other forges, we could also do our own automations using
> systemd timers that automagically sends patches to the list when a new
> release of one of our dependencies is published.
>
Bumping a dependency is not the hard part. The problem arises when the
update is not compatible with what you're using. Then you need to also
figure out exactly how and rewrite your code to work with the new version,
before you can update it.
Having complete testing coverage helps with that insofar that at least you
will *know* that things will break. You still have to fix them, but you
won't accidentally push an update to production. But without that, there's
a huge risk of just suddenly breaking things, and then you're stuck.
(And yes, we've seen that happen numerous times with some of our other apps
we host, and it's an absolute PITA).
Out of curiosity, how often is the Debian server restarted to pull in
> new updates?
>
Generally depending on need. We do check for updates every couple of hours
normally, sooner if it's one of those "omg the world is ending"-security
alerts that hits the press...
//Magnus
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2026-09-01 21:39 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-08-17 16:13 Python Tooling Tristan Partin <tristan@partin.io>
2026-08-17 16:24 ` Magnus Hagander <magnus@hagander.net>
2026-08-31 20:23 ` Tristan Partin <tristan@partin.io>
2026-08-31 20:26 ` Jonathan S. Katz <jkatz@postgresql.org>
2026-08-31 20:32 ` Tristan Partin <tristan@partin.io>
2026-08-31 20:38 ` Magnus Hagander <magnus@hagander.net>
2026-08-31 21:06 ` Tristan Partin <tristan@partin.io>
2026-09-01 21:39 ` Magnus Hagander <magnus@hagander.net>
2026-08-31 20:30 ` Magnus Hagander <magnus@hagander.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox