Subject: Time to drop plpython2?

From: Andres Freund <[email-masked]>
Date: Sun, 31 Oct 2021 11:45:48 -0700

Hi,

The last release of python2 was 2.7.18 released 2020-04-20, with support
already having ended before that 2020-01-01.

To me it seems time to drop plpython2 support. Supporting plpython2
until ~7 years after python2 is EOL is already quite long... It'd be one
thing if it were completely painless, but imo it's not.

I was about to list better plpython2/3 support (including the regex
replacements for the regress tests) as a TODO for the meson proposal,
but it doesn't really seem worth investing in that...

Greetings,

Andres Freund

---

From: Tom Lane <[email-masked]>
Date: Sun, 31 Oct 2021 14:49:22 -0400

> thing if it were completely painless, but imo it's not.

7 years?  Oh, you're envisioning dropping plpython2 in HEAD but keeping
it alive in the back branches.  Yeah, it's a bit hard to justify
continuing support in HEAD.

			regards, tom lane

---

From: Andres Freund <[email-masked]>
Date: Sun, 31 Oct 2021 12:05:42 -0700

Hi,

On 2021-10-31 14:49:22 -0400, Tom Lane wrote:
> it alive in the back branches.

Yea. It's annoying to support python2, but I don't think it's severe
enough to consider dropping support in released branches. We might get
to that point for the newer released versions, but for now...

> Yeah, it's a bit hard to justify continuing support in HEAD.

Cool.

Greetings,

Andres Freund

---

From: Thomas Munro <[email-masked]>
Date: Mon, 1 Nov 2021 09:00:49 +1300

> > Yeah, it's a bit hard to justify continuing support in HEAD.

+1, it's dropping out of distros, it'd be unsupportable without
unreasonable effort.

---

From: Peter Eisentraut <[email-masked]>
Date: Mon, 1 Nov 2021 06:53:01 +0100

> but it doesn't really seem worth investing in that...

Usually, we have dropped support for older Python versions when 
continued support would be a burden for some ongoing development 
project.  This would certainly count as one of those, I think.

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 4 Nov 2021 19:58:54 +0100

I see you have posted a patch for this in the meson thread 
(https://www.postgresql.org/message-id/attachment/127770/v5-0003-plpython-Drop-support-python2.patch). 
  Here is my review of that.

I would change the search order in configure from

PGAC_PATH_PROGS(PYTHON, [python python3 python2])

to

PGAC_PATH_PROGS(PYTHON, [python3 python])

This makes sure you don't accidentally pick up a "python" binary that 
points to Python 2.  This would otherwise immediately generate lots of 
build failures on older platforms that still have Python 2 around.

You left two FIXME sections in plpython.h.  AFAICT, we can make the 
substitutions corresponding to those #define's in the source code and 
remove those blocks.

src/pl/plpython/expected/README should be updated for the removed files.

Some documentation updates are needed.  I see possibly relevant text in 
the following files:

hstore.sgml
install-windows.sgml
installation.sgml
json.sgml
libpq.sgml
plpython.sgml
ref/comment.sgml (examples)
ref/create_transform.sgml (examples)
ref/drop_transform.sgml (examples)
standalone-profile.xsl

src/tools/msvc/ has lots of Python-related content.

More stuff to clean up:

contrib/hstore_plpython/.gitignore
contrib/jsonb_plpython/.gitignore
contrib/ltree_plpython/.gitignore
src/pl/plpython/.gitignore

Finally, morally related, there is some Python 2/3 compat code in 
contrib/unaccent/generate_unaccent_rules.py that could be removed. 
Also, arguably, change the shebang line in that script.

---

From: Andres Freund <[email-masked]>
Date: Thu, 4 Nov 2021 12:54:43 -0700

Hi,

On 2021-11-04 19:58:54 +0100, Peter Eisentraut wrote:
> I see you have posted a patch for this in the meson thread (https://www.postgresql.org/message-id/attachment/127770/v5-0003-plpython-Drop-support-python2.patch).

Yea, I was planning to post that here after a bit more polish. I mostly wanted
to get rid of the gross gross hack I did for the transmutation of the
regression test files.

> This makes sure you don't accidentally pick up a "python" binary that points
> to Python 2.  This would otherwise immediately generate lots of build
> failures on older platforms that still have Python 2 around.

Yes, we should do that, at least for now. I did see build failures that
required me to specify the python version to avoid issues around it.

> substitutions corresponding to those #define's in the source code and remove
> those blocks.

Yea, it shouldn't be hard. Just required more time than I had to send it out
before Nov 1st ;)

With meson I'd do a version: '>= 3' or such, to filter out a bare 'python'
being python2, but I don't think there's an equally trivial way to do that
with autoconf.

> contrib/unaccent/generate_unaccent_rules.py that could be removed. Also,
> arguably, change the shebang line in that script.

Hm. So far the python used for plpython and python for code generation etc is
independent. I don't know if plpython actually can be cross-compiled, but if
so, they'd have to be independent.  Otherwise I'd say we should just invoke
contrib/unaccent/generate_unaccent_rules.py with a python chosen by
configure/meson, rather than relying on a shebang that can't be adjusted
without modifying source code.

Another thing I wondered about is what we want to do with the extension
names. Do we want to leave it named plpython3u? Do we want to have a plpython
that depends on plpython3u?

I'd be inclined to just keep it at plpython3u for now, but there's an argument
that going for plpython would be better long term: Presumably there will be
python 4 at some point - but I'd expect that to not be a breaking release,
given the disaster that python 3 is. Making a non-versioned name better?

Greetings,

Andres Freund

---

From: Tom Lane <[email-masked]>
Date: Thu, 04 Nov 2021 16:26:39 -0400

> that depends on plpython3u?

I think we want to keep plpython3u.  Maybe we can point plpythonu
at that, but I'm concerned about the potential for user confusion.
In particular, I think there's a nonzero probability that someone
will choose to maintain plpython2u/plpythonu outside of core,
just because they still don't want to migrate their Python code.

> given the disaster that python 3 is. Making a non-versioned name better?

Meh.  If there is a python 4, I'd expect it to be named that way precisely
because it *is* a breaking release.  Why would we set ourselves up for
a repeat of this mess?

			regards, tom lane

---

From: Peter Eisentraut <[email-masked]>
Date: Wed, 10 Nov 2021 11:56:10 +0100

> without modifying source code.

We don't rely on the shebang for running this (see Makefile).  I just 
see some trend of people changing shebang lines as a sort of signal, 
"this script has abandoned Python 2".  It's not very important.

> that depends on plpython3u?

I would tend to mirror what the Python community does with 
/usr/bin/python.  Right now, most people are removing /usr/bin/python 
and just provide /usr/bin/python3.  This would be analogous to what your 
patch does.  Some time in the future, they may add /usr/bin/python back, 
and then we could do the same, meaning add a plpythonu that depends on 
plpython3u.

---

From: Tom Lane <[email-masked]>
Date: Sun, 14 Nov 2021 21:24:31 -0500

... btw, there's a fairly critical gating factor for any plan to drop
python2 support: the buildfarm.  I just counted, and there are exactly
as many members running python 2.x as 3.x (49 apiece), not counting
Windows machines that aren't running configure.  We can't commit
something that's going to make half the buildfarm go red.

(It's likely that some fraction of them do already have python3 installed,
in which case the search order change Peter recommended would be enough to
fix it.  But I'm sure not all do.)

This ties into the business about converting the build system to meson,
as that also requires python 3 --- with, IIUC, a higher minimum version
than we might otherwise need.  I'm disinclined to cause two separate
flag days for buildfarm owners, so what I now think is we ought to put
this idea on the shelf until we've finished that conversion or decided
we're not gonna do it.  We need to identify exactly what needs to be
installed before we start pestering the owners.

			regards, tom lane

---

From: Andres Freund <[email-masked]>
Date: Mon, 15 Nov 2021 09:06:32 -0800

Hi,

> fix it.  But I'm sure not all do.)

How about committing the order change alone? That seems like something
warranted completely in isolation? Afterwards we can see how many run what and
go from there?

> installed before we start pestering the owners.

Yea, that's true.

Greetings,

Andres Freund

---

From: Tom Lane <[email-masked]>
Date: Mon, 15 Nov 2021 12:19:51 -0500

> warranted completely in isolation? Afterwards we can see how many run what and

I don't think that's warranted.  The existing design is that we let
the user say which python is "python", and I do not think we should
change that in advance of actually dropping python2 support.

I was wondering about simply probing to see if python3 exists (and if
so, what version it is exactly), as an additional configure test that
doesn't hook into anything.  That would give us some information without
suddenly changing what is being tested.

			regards, tom lane

---

From: Andres Freund <[email-masked]>
Date: Mon, 15 Nov 2021 10:12:46 -0800

Hi,

> change that in advance of actually dropping python2 support.

Hm. I think it'd be ok, given that python 2 is well past EOL. But I also see
your point.

> suddenly changing what is being tested.

But this is probably a good compromise. Were you thinking of doing a proper
autoconf test or just putting something like python3 --version || true in
configure?

I guess it'd be easiest to interpret if we output the current PYTHON version
and, iff < 3.0, also output the 'python3' version?

Greetings,

Andres Freund

---

From: Tom Lane <[email-masked]>
Date: Mon, 15 Nov 2021 13:18:22 -0500

> autoconf test or just putting something like python3 --version || true in

Hm, I had in mind an actual configure test; but since it's just a
temporary exploratory measure, shortcuts are fine.  However, I'm
not sure that what you suggest would result in capturing anything
in config.log.

> and, iff < 3.0, also output the 'python3' version?

We'll already know if PYTHON is set, at least so far as the buildfarm
is concerned.

			regards, tom lane

---

From: Peter Eisentraut <[email-masked]>
Date: Mon, 15 Nov 2021 19:52:19 +0100

On 15.11.21 19:18, Tom Lane wrote:
> Andres Freund<[email-masked]>  writes:

I'm not sure this is really going to end up moving things forward.

I think we should just write to the build farm owners, we plan to drop 
python2 support in, say, 60 days, please update your setup to use 
python3 or disable python support.

If we add this test first, then all we're going to learn is probably 
that 60% of those who are currently using python2 don't have python3 
installed, and then we're still going to have to send that above email.

---

From: Tom Lane <[email-masked]>
Date: Mon, 15 Nov 2021 14:26:55 -0500

> installed, and then we're still going to have to send that above email.

I don't know what fraction don't have python3 installed, and it doesn't
matter much, since it's unlikely that it's either 0% or 100%.  What
I hoped to learn was, of those who *do* have some python3 installed,
which version it is.  That might inform our thoughts about where to
set the minimum python3 version.

Relevant data points:

* Our docs claim the minimum 3.x version for pl/python itself is 3.1,
but this is unbacked by any testing; the oldest 3.x in the buildfarm
is 3.4.3 (three such animals).

* Meson only promises support back to python 3.6, but if that's
accurate it's going to be a problem for us, because there are lots
of live LTS distributions with older python3 (RHEL7, Solaris 11.3,
AIX 7.2 for starters).  I've been planning to do some testing and
see if meson will run under python 3.4 or 3.5.

But this probably belongs on the meson thread, since that looks like
it will be a much stronger constraint than pl/python is.

			regards, tom lane

---

From: Andres Freund <[email-masked]>
Date: Mon, 15 Nov 2021 12:00:19 -0800

Hi,

Continuing the discussion from https://www.postgresql.org/message-id/2146739.1637004415%40sss.pgh.pa.us

> see if meson will run under python 3.4 or 3.5.

Slightly older versions, which do work to build postgres with the proposed
patchset, run on python3 3.5. I don't think it's likely we could make the
versions that only required 3.4 work reasonably.

Is RHEL7 really an issue? I only have Centos 7 around, but that has python
3.6.

I don't know much about AIX, but according to https://repology.org/project/python/versions
the AIX toolbox has 3.7.11. I don't know enough about AIX to know whether
there's other sources of python3, bison, etc that are common. On the AIX
system I have access to they all seem to be symlinked to /opt/freeware, which
I understand is that toolbox stuff?

Greetings,

Andres Freund

---

From: Tom Lane <[email-masked]>
Date: Mon, 15 Nov 2021 15:30:02 -0500

> versions that only required 3.4 work reasonably.

OK, thanks for the datapoint.

> Is RHEL7 really an issue? I only have Centos 7 around, but that has python

The info I checked said that RHEL7 originally shipped with 3.3.
I'm not sure that Red Hat would've outright replaced that, but they
do have a notion of add-on "software collections", and I'm certain
that they would have provided newer pythons via that mechanism.
So it should only be a big issue for someone who didn't want to install
an add-on collection.  I suppose the overlap of that group with the
people who will want to put PG 15+ on that platform is probably nil.

> I understand is that toolbox stuff?

Hmm, I was basing that on (a) what I can see installed on gcc119,
which is 3.5.1, and (b) AIX 7.2's 2015 release date, which matches up
pretty well with python 3.5.  As with RHEL, it's entirely likely that
IBM has made newer releases available as add-ons, but there's no
guarantee that any given installation would have that.

Solaris is a bit worse, since they shipped 11.3 a shade earlier,
with python 3.4.3.  No idea about add-on conventions on that
platform.

There's room to argue that all three of these will be out of
primary support before PG15 ships, so maybe we don't need to
worry about whether we can build with their default toolsets.
Still, it's a tradeoff I'd rather not make.

			regards, tom lane

---

From: Robert Haas <[email-masked]>
Date: Mon, 15 Nov 2021 15:36:11 -0500

> people who will want to put PG 15+ on that platform is probably nil.

It'd only be an issue if they want to compile from source, right?
We're not speaking of changing the runtime prerequisites, IIUC.

I think it's really important that we continue to run on all of the
supported Linux distributions and even some recently-out-of-support
ones if they are popular. My experience is that many people stay on
whatever version of RHEL they've got for quite a long time. I think
we'll be doing ourselves a disservice if such people find that they
can't easily upgrade to newer versions of PostgreSQL without
installing a bunch of new software and/or upgrading software they've
already got on the machine.

On the other hand, the class of users that I'm thinking about does not
typically install anything from source, so they are not directly
impacted by build prerequisites.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

---

From: Andres Freund <[email-masked]>
Date: Mon, 15 Nov 2021 13:04:05 -0800

Hi,

> people who will want to put PG 15+ on that platform is probably nil.

3.6 appears to be in the 'update' repository, which I think is separate from
the add-on collection? It appears to be enabled by default in centos7, not
sure how much that means.

Name        : python3
Version     : 3.6.8
Release     : 18.el7
Architecture: x86_64
Size        : 39896
Packager    : CentOS BuildSystem <http://bugs.centos.org>
Group       : Unspecified
URL         : https://www.python.org/
Repository  : updates
Summary     : Interpreter of the Python programming language
Source      : python3-3.6.8-18.el7.src.rpm

> guarantee that any given installation would have that.

I think AIX itself only ships with python 2.7, and all python3 versions are
from toolbox. But I don't actually know.

> with python 3.4.3.  No idea about add-on conventions on that

Based solely on some person in meson's irc channel, it appears that they added
a newer python3 sometime in 11.4's lifetime:

alanc:
Solaris 11.4 currently ships Python 3.7 & 3.9
as the maintainer of the meson package for Solaris, I deal with the
lack of newer python versions on older Solaris releases by only packaging
meson for Solaris 11.4, and telling anyone who wants it for older releases to
upgrade their OS, or build all the dependencies themselves
...
or it's 11.4.0 from 2018, and not updated with the latest updates (which are
only available for support customers right now unfortunately) - 3.7 was added
in 11.4.12
https://docs.oracle.com/cd/E37838_01/html/E99082/bundledfreeware.html
...

> Still, it's a tradeoff I'd rather not make.

There's two other paths worth mentioning:
- muon is a WIP alternative implementation of meson in plain C99. Doesn't yet
  have enough feature coverage.
- pyinstaller generates an executable from the meson python code, and supports
  solaris and aix

Greetings,

Andres Freund

---

From: Andres Freund <[email-masked]>
Date: Mon, 15 Nov 2021 13:07:37 -0800

Hi, 

>We're not speaking of changing the runtime prerequisites, IIUC.

Correct.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Tom Lane <[email-masked]>
Date: Mon, 15 Nov 2021 16:12:12 -0500

> We're not speaking of changing the runtime prerequisites, IIUC.

I'm not sure.  Does it make sense to document that pl/python has
a different Python version requirement than the build system does?
If we do, who exactly is going to be testing that such a combination
works?  Will it even be possible to compile pl/python against Python
headers/libs of a different Python generation than meson is running
under?

ISTM we'd be a lot better off saying "the minimum Python version is
3.something", full stop, and then making sure that that minimum is
represented in the buildfarm.  But it's not quite clear yet what
"something" needs to be.

> ones if they are popular.

I agree completely, which is why I'm raising the point.

			regards, tom lane

---

From: Peter Eisentraut <[email-masked]>
Date: Mon, 15 Nov 2021 23:45:52 +0100

> is 3.4.3 (three such animals).

I confirmed locally that 3.2.6 still works with PL/Python.  I expect 
that Python 3.1 also still works, except that I can't build it at the 
moment.  Python 3.0.* was sort of a dud, so it's not really interesting 
to consider.

Partially because of the need to keep supporting Python 2, there hasn't 
been any work done in PL/Python to make use of any newer APIs, so the 
Python 3 minimum version is frozen in time at the moment.

---

From: Robert Haas <[email-masked]>
Date: Tue, 16 Nov 2021 10:16:04 -0500

> headers/libs of a different Python generation than meson is running

Hmm, that's true. I hadn't considered the fact that anyone who is
packaging PostgreSQL probably also wants to build plpython. However,
it's possible that a side-by-side install of a newer python version
could be used for the build system while building against the system
python for plpython. That might or might not be too exotic a
configuration for someone to consider.

---

From: Andrew Dunstan <[email-masked]>
Date: Tue, 16 Nov 2021 11:26:41 -0500

> installed before we start pestering the owners.

crake has been using 2.7, but has 3.9.7 installed. I tried switching to
that but ran into an issue with upgrading things from 9.5 on. It would
have been all the way back to 9.2 but the plpython tests drop the
extension even though the transform tests in contrib don't, and neither
do the plperl tests drop plperlu. I'm rather inclined to say we should
stop doing that, or at least be consistent about it.

Having rebuilt all the branches from 9.5 and up the cross version
upgrade tests are now passing on crake.

My other machine with an old python instance is bowerbird. It has python
3.4 installed but not used, alongside 2.7 which is udsed. I will install
the latest and see if that can be made to work.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

---

From: Andrew Dunstan <[email-masked]>
Date: Thu, 18 Nov 2021 16:33:13 -0500

> the latest and see if that can be made to work.

bowerbird is now building with python 3.10

cheers

andrew