Subject: Re: Replace uses of deprecated Python module distutils.sysconfig

From: Peter Eisentraut <[email-masked]>
Date: Tue, 18 Jan 2022 17:47:37 +0100

On 18.01.22 16:24, Tom Lane wrote:
> Peter Eisentraut <[email-masked]> writes:
>> Also, considering the failure on prairiedog, I do see now on
>> <https://docs.python.org/3/library/sysconfig.html> that the sysconfig
>> module is "New in version 3.2".  I had interpreted the fact that it
>> exists in version 2.7 that that includes all higher versions, but
>> obviously there were multiple branches involved, so that was a mistaken
>> assumption.
> 
> Hm.  I installed 3.1 because we claim support for that.  I don't mind
> updating to 3.2 (as long as we adjust the docs to match), but it seems
> kinda moot unless you figure out a solution for the include-path
> issue.  I see that platforms as recent as Debian 10 are failing,
> so I don't think we can dismiss that as not needing fixing.

I have reverted this for now.

I don't have a clear idea how to fix this in the long run.  We would 
perhaps need to determine at which points the various platforms had 
fixed this issue in their Python installations and select between the 
old and the new approach based on that.  Seems messy.

---

From: Tom Lane <[email-masked]>
Date: Tue, 18 Jan 2022 12:04:21 -0500

> old and the new approach based on that.  Seems messy.

Are we sure it's an issue within Python, rather than something we
could dodge by invoking sysconfig differently?  It's hard to believe
that sysconfig could be totally unfit for the purpose of finding out
the include path and would remain so for multiple years.

			regards, tom lane

---

From: Tom Lane <[email-masked]>
Date: Tue, 18 Jan 2022 19:18:11 -0500

> the include path and would remain so for multiple years.

I dug up a Debian 9 image and found that I could reproduce the problem
against its python2 (2.7.13) installation, but not its python3 (3.5.3):

$ python2 -m sysconfig | grep include
        include = "/usr/local/include/python2.7"
        platinclude = "/usr/local/include/python2.7"
...
$ python3 -m sysconfig | grep include
        include = "/usr/include/python3.5m"
        platinclude = "/usr/include/python3.5m"
...

Looking at the buildfarm animals that failed this way, 10 out of 11
are using python 2.x.  The lone exception is Andrew's prion.  I wonder
if there is something unusual about its python3 installation.

Anyway, based on these results, we might have better luck switching to
sysconfig after we start forcing python3.  I'm tempted to resurrect the
idea of changing configure's probe order to "python3 python python2"
in the meantime, just so we can see how much of the buildfarm is ready
for that.

			regards, tom lane

---

From: Tom Lane <[email-masked]>
Date: Wed, 19 Jan 2022 11:21:15 -0500

I wrote:
> sysconfig after we start forcing python3.

On the other hand, that answer is not back-patchable, and we surely
need a back-patchable fix, because people will try to build the
back branches against newer pythons.

Based on the buildfarm results so far, the problem can be described
as "some installations say /usr/local when they should have said /usr".
I experimented with the attached delta patch and it fixes the problem
on my Debian 9 image.  (I don't know Python, so there may be a better
way to do this.)  We'd have to also bump the minimum 3.x version to
3.2, but that seems very unlikely to bother anyone.

			regards, tom lane

diff --git a/config/python.m4 b/config/python.m4
index 8ca1eaa64b..c65356c6ac 100644
--- a/config/python.m4
+++ b/config/python.m4
@@ -56,13 +56,20 @@ AC_MSG_RESULT([$python_configdir])
 
 AC_MSG_CHECKING([Python include directories])
 python_includespec=`${PYTHON} -c "
-import sysconfig
-a = '-I' + sysconfig.get_path('include')
-b = '-I' + sysconfig.get_path('platinclude')
+import sysconfig, os.path
+a = sysconfig.get_path('include')
+b = sysconfig.get_path('platinclude')
+# Some versions of sysconfig report '/usr/local/include'
+# when they should have said '/usr/include'
+if not os.path.exists(a + '/Python.h'):
+    aalt = a.replace('/usr/local/', '/usr/', 1)
+    if os.path.exists(aalt + '/Python.h'):
+        a = aalt
+        b = b.replace('/usr/local/', '/usr/', 1)
 if a == b:
-    print(a)
+    print('-I' + a)
 else:
-    print(a + ' ' + b)"`
+    print('-I' + a + ' -I' + b)"`
 if test "$PORTNAME" = win32 ; then
     python_includespec=`echo $python_includespec | sed 's,[[\]],/,g'`
 fi
diff --git a/configure b/configure
index 9c856cb1d5..f88db9467d 100755
--- a/configure
+++ b/configure
@@ -10370,13 +10370,20 @@ $as_echo "$python_configdir" >&6; }
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking Python include directories" >&5
 $as_echo_n "checking Python include directories... " >&6; }
 python_includespec=`${PYTHON} -c "
-import sysconfig
-a = '-I' + sysconfig.get_path('include')
-b = '-I' + sysconfig.get_path('platinclude')
+import sysconfig, os.path
+a = sysconfig.get_path('include')
+b = sysconfig.get_path('platinclude')
+# Some versions of sysconfig report '/usr/local/include'
+# when they should have said '/usr/include'
+if not os.path.exists(a + '/Python.h'):
+    aalt = a.replace('/usr/local/', '/usr/', 1)
+    if os.path.exists(aalt + '/Python.h'):
+        a = aalt
+        b = b.replace('/usr/local/', '/usr/', 1)
 if a == b:
-    print(a)
+    print('-I' + a)
 else:
-    print(a + ' ' + b)"`
+    print('-I' + a + ' -I' + b)"`
 if test "$PORTNAME" = win32 ; then
     python_includespec=`echo $python_includespec | sed 's,[\],/,g'`
 fi

---

From: Peter Eisentraut <[email-masked]>
Date: Wed, 19 Jan 2022 17:49:26 +0100

> in the meantime, just so we can see how much of the buildfarm is ready

This seems sensible in any case, given that we have quasi-committed to 
enforcing Python 3 soon.

---

From: Tom Lane <[email-masked]>
Date: Wed, 19 Jan 2022 15:40:12 -0500

Peter Eisentraut <[email-masked]> writes:
>> ... I'm tempted to resurrect the

> This seems sensible in any case, given that we have quasi-committed to 

Done.  (I couldn't find any equivalent logic in the MSVC build scripts
though; is there something I missed?)

			regards, tom lane

---

From: Juan José Santamaría Flecha <[email-masked]>
Date: Wed, 19 Jan 2022 22:20:37 +0100

On Wed, Jan 19, 2022 at 9:40 PM Tom Lane <[email-masked]> wrote:

> MSVC will use the path configured in src\tools\msvc\config.pl $config->{"python"},
there is no ambiguity.

Regards,

Juan José Santamaría Flecha

---

From: Tom Lane <[email-masked]>
Date: Wed, 19 Jan 2022 20:27:59 -0500

> MSVC will use the path configured in src\tools\msvc\config.pl $config->{"python"},

Ah, right.  We have only three active Windows animals that are building
with python, and all three are using 3.something, so that side of the
house seems to be ready to go.

			regards, tom lane

---

From: Andrew Dunstan <[email-masked]>
Date: Thu, 20 Jan 2022 10:04:56 -0500

> if there is something unusual about its python3 installation.

It's an Amazon Linux instance, and using their packages, which seem a
bit odd (there's nothing in /usr/local/include). Maybe we should be
looking at INCLUEPY?

[ec2-user@ip-172-31-22-42 bf]$ python3 -m sysconfig | grep include
    include = "/usr/local/include/python3.6m"
    platinclude = "/usr/local/include/python3.6m"
    CONFIG_ARGS = "'--build=x86_64-redhat-linux-gnu'
'--host=x86_64-redhat-linux-gnu' '--target=x86_64-amazon-linux-gnu'
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64'
'--libexecdir=/usr/libexec' '--localstatedir=/var'
'--sharedstatedir=/var/lib' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--enable-ipv6' '--enable-shared'
'--with-computed-gotos=yes' '--with-dbmliborder=gdbm:ndbm:bdb'
'--with-system-expat' '--with-system-ffi'
'--enable-loadable-sqlite-extensions' '--with-dtrace' '--with-valgrind'
'--without-ensurepip' '--enable-optimizations'
'build_alias=x86_64-redhat-linux-gnu'
'host_alias=x86_64-redhat-linux-gnu'
'target_alias=x86_64-amazon-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC
-fwrapv  ' 'LDFLAGS= -g  ' 'CPPFLAGS= '
'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'"
    CONFINCLUDEDIR = "/usr/include"
    CONFINCLUDEPY = "/usr/include/python3.6m"
    INCLDIRSTOMAKE = "/usr/include  /usr/include/python3.6m"
    INCLUDEDIR = "/usr/include"
    INCLUDEPY = "/usr/include/python3.6m"

I have upgraded it to python 3.8, but got similar results.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com