Subject: Re: plpython vs _POSIX_C_SOURCE

From: Andres Freund <[email-masked]>
Date: Tue, 24 Jan 2023 17:48:56 -0800

Hi,

On 2023-01-24 16:16:06 -0500, Tom Lane wrote:
> Andres Freund <[email-masked]> writes:
> > Python's _POSIX_C_SOURCE value is set to a specific value in their configure
> > script:
>
> > if test $define_xopen_source = yes
> > then
> >   ...
> >   AC_DEFINE(_POSIX_C_SOURCE, 200809L, Define to activate features from IEEE Stds 1003.1-2008)
> > fi
>
> Hm.  I looked into Python 3.2 (the oldest release we still support)
> and it has similar code but
>
>   AC_DEFINE(_POSIX_C_SOURCE, 200112L, Define to activate features from IEEE Stds 1003.1-2001)
>
> So yeah it's fixed (or else not defined) for any particular Python
> release, but could vary across releases.

Looks like it changed in 3.3:

$ git grep -E 'AC_DEFINE.*_POSIX_C_SOURCE' v3.2 v3.3.0
v3.2:configure.in:  AC_DEFINE(_POSIX_C_SOURCE, 200112L, Define to activate features from IEEE Stds 1003.1-2001)
v3.3.0:configure.ac:  AC_DEFINE(_POSIX_C_SOURCE, 200809L, Define to activate features from IEEE Stds 1003.1-2008)

I'm not sure we need to care a lot about a build with python 3.3 triggering a
bunch of warnings.

Personally I'd just bump the python requirements to well above it - the last
3.2 release was Oct. 12, 2014.

Official EOL date:
Ver	Last Release	EOL Date
3.2	2014-10-11	2016-02-20
3.3	2017-09-19	2017-09-29
>From 3.4 on there's just an official last release:
3.4	2019-03-18
3.5	2020-09-05
3.6	2021-09-04
3.7	2023-06-??

> > Solaris and AIX are the ones missing.
> > I guess I'll test them manually. It seems promising not to need this stuff
> > anymore?
>
> Given that hoverfly is AIX, I'm betting there's an issue there.

Doesn't look that way.

I found plenty problems on AIX, but all independent of _POSIX_C_SOURCE.

Both autoconf and meson builds seem to need externally specified
-D_LARGE_FILES=1 to build successfully when using plpython, otherwise we end
up with conflicting signatures with lseek. I see that Noah has that in his
buildfarm config.  ISTM that we should just move that into our build specs.

To get 64 bit autoconf to link plpython3.so correctly, I needed to add to
manually add -lpthread:
ld: 0711-317 ERROR: Undefined symbol: .pthread_init
...

I suspect Noah might not hit this, because one of the dependencies he has
enabled already adds it to the backend LDFLAGS.

Also for autoconf, I needed to link
$prefix/lib/python3.11/config-3.11/libpython3.11.a
to
$prefix/lib/libpython3.11.a
That might be a python version difference or be related to building python
with --enable-shared - but I see saw other problems without --enable-shared.

I ran out of energy to test on aix with xlc, I spent way more time on this
than I have already. I'll pick it up later.

I also tested 64bit solaris. No relevant warnings (lots of other warnings
though), tests pass, with both acc and gcc.

> >> Anyway, I'm still of the opinion that what a11cf433413 tried to do
> >> was the best available fix, and we need to do whatever we have to do
> >> to plpython's headers to reinstate that coding rule.
>
> > You think it's not a viable path to just remove the _POSIX_C_SOURCE,
> > _XOPEN_SOURCE undefines?
>
> I think at the least that will result in warnings on some platforms,
> and at the worst in actual build problems.  Maybe there are no more
> of the latter a dozen years after the fact, but ...

I think it might be ok. I tested nearly all OSs that we support, with the
exception of DragonFlyBSD and Illumos, which both are very similar to tested
operating systems.

> Nice idea.  We did not have that option while we were using HAVE_SNPRINTF
> ourselves, but now that we are not I concur that this should work.

Cool.

> (I confirmed that their code looks the same in Python 3.2.)
> Note that you'd better make it
>
> #define HAVE_SNPRINTF 1
>
> or you risk macro-redefinition warnings.

Good point.

I guess I'll push that part first, given that we have agreement how it should
look like.

Greetings,

Andres Freund

---

From: Andres Freund <[email-masked]>
Date: Tue, 24 Jan 2023 18:32:46 -0800

Hi,

> with --enable-shared - but I see saw other problems without --enable-shared.

That actually doesn't quite work right. One needs to either link to the file
by name (i.e. just $prefix/lib/libpython3.11.so instead of -lpython3.11), or
create a wrapper .a "manually". I.e.

ar crs $prefix/lib/libpython3.11.a $prefix/lib/libpython3.11.so

I tried quite a few things and got confused between the attempts.

> than I have already. I'll pick it up later.

Building in the background now.

Greetings,

Andres Freund

---

From: Andres Freund <[email-masked]>
Date: Tue, 24 Jan 2023 20:28:35 -0800

Hi,

> Building in the background now.

Also passes.

Thus I think getting rid of the #undefines is the best plan going
forward. Fewer complicated rules to follow => fewer rule violations.

Patches attached.

Greetings,

Andres Freund
>From eb3fb959b3db609db44437f783fc58cdb7f28e9f Mon Sep 17 00:00:00 2001
From: Andres Freund <[email-masked]>
Date: Tue, 24 Jan 2023 11:42:35 -0800
Subject: [PATCH v1 1/3] plpython: Avoid the need to redefine *printf macros

Until now we undefined and then redefined a lot of *printf macros due to
worries about conflicts with Python.h macro definitions. Current Python.h
doesn't define any *printf macros, and older versions just defined snprintf,
vsnprintf, guarded by #if defined(MS_WIN32) && !defined(HAVE_SNPRINTF).

Thus we can replace the undefine/define section with a single
 #define HAVE_SNPRINTF 1

Discussion: https://postgr.es/m/[email-masked]
---
 src/pl/plpython/plpython.h | 48 +++-----------------------------------
 1 file changed, 3 insertions(+), 45 deletions(-)

diff --git a/src/pl/plpython/plpython.h b/src/pl/plpython/plpython.h
index 2af0d04d1f8..2d18fc2dc1b 100644
--- a/src/pl/plpython/plpython.h
+++ b/src/pl/plpython/plpython.h
@@ -30,17 +30,10 @@
 #undef _XOPEN_SOURCE
 
 /*
- * Sometimes python carefully scribbles on our *printf macros.
- * So we undefine them here and redefine them after it's done its dirty deed.
+ * Python versions <= 3.8 otherwise define a replacement, causing macro
+ * redefinition warnings.
  */
-#undef vsnprintf
-#undef snprintf
-#undef vsprintf
-#undef sprintf
-#undef vfprintf
-#undef fprintf
-#undef vprintf
-#undef printf
+#define HAVE_SNPRINTF 1
 
 #if defined(_MSC_VER) && defined(_DEBUG)
 /* Python uses #pragma to bring in a non-default libpython on VC++ if
@@ -63,41 +56,6 @@
 #undef TEXTDOMAIN
 #define TEXTDOMAIN PG_TEXTDOMAIN("plpython")
 
-/* put back our *printf macros ... this must match src/include/port.h */
-#ifdef vsnprintf
-#undef vsnprintf
-#endif
-#ifdef snprintf
-#undef snprintf
-#endif
-#ifdef vsprintf
-#undef vsprintf
-#endif
-#ifdef sprintf
-#undef sprintf
-#endif
-#ifdef vfprintf
-#undef vfprintf
-#endif
-#ifdef fprintf
-#undef fprintf
-#endif
-#ifdef vprintf
-#undef vprintf
-#endif
-#ifdef printf
-#undef printf
-#endif
-
-#define vsnprintf		pg_vsnprintf
-#define snprintf		pg_snprintf
-#define vsprintf		pg_vsprintf
-#define sprintf			pg_sprintf
-#define vfprintf		pg_vfprintf
-#define fprintf			pg_fprintf
-#define vprintf			pg_vprintf
-#define printf(...)		pg_printf(__VA_ARGS__)
-
 /*
  * Used throughout, so it's easier to just include it everywhere.
  */
-- 
2.38.0

>From 54f95177b0f32522862cc3589c587cbe3595304b Mon Sep 17 00:00:00 2001
From: Andres Freund <[email-masked]>
Date: Tue, 24 Jan 2023 19:03:36 -0800
Subject: [PATCH v1 2/3] plpython: Stop undefining _POSIX_C_SOURCE,
 _XOPEN_SOURCE

We undefined them to avoid warnings about macro redefinitions. But we haven't
fully followed the necessary include order, since at least 147c2482542, in
2011.  Recently the combination of the include order rules not being followed
and undefining _POSIX_C_SOURCE started to cause a compile failure, starting
with 03023a2664f. Undefining _POSIX_C_SOURCE hides clock_gettime(), which is
referenced in an inline function as of 03023a2664f, whereas it was a macro
before.

After seeing some evidence that undefining _POSIX_C_SOURCE et al isn't
required, I tried to build postgres with plpython on most of our supported
platforms (except DragonFlyBSD and Illumos, but similar systems were tested),
with/without the #undefines. No compiler warning / behavioral difference.

The oldest supported python version, 3.2, defines _POSIX_C_SOURCE to 200112L
ad _XOPEN_SOURCE to 600, whereas newer versions of python use 200809L/700
respectively. As _POSIX_C_SOURCE/_XOPEN_SOURCE will default to the newer
operating system on most platforms, it's possible that when using python 3.2
new warnings would be emitted - but that seems acceptable.

Discussion: https://postgr.es/m/[email-masked]
---
 src/pl/plpython/plpython.h | 18 ++++--------------
 1 file changed, 4 insertions(+), 14 deletions(-)

diff --git a/src/pl/plpython/plpython.h b/src/pl/plpython/plpython.h
index 2d18fc2dc1b..91f6f8d8607 100644
--- a/src/pl/plpython/plpython.h
+++ b/src/pl/plpython/plpython.h
@@ -12,23 +12,13 @@
 #ifndef PLPYTHON_H
 #define PLPYTHON_H
 
-/*
- * Include order should be: postgres.h, other postgres headers, plpython.h,
- * other plpython headers.  (In practice, other plpython headers will also
- * include this file, so that they can compile standalone.)
- */
-#ifndef POSTGRES_H
+/* postgres.h needs to be included before Python.h, as usual */
+#if !defined(POSTGRES_H)
 #error postgres.h must be included before plpython.h
+#elif defined(Py_PYTHON_H)
+#error Python.h must be included via plpython.h
 #endif
 
-/*
- * Undefine some things that get (re)defined in the Python headers. They aren't
- * used by the PL/Python code, and all PostgreSQL headers should be included
- * earlier, so this should be pretty safe.
- */
-#undef _POSIX_C_SOURCE
-#undef _XOPEN_SOURCE
-
 /*
  * Python versions <= 3.8 otherwise define a replacement, causing macro
  * redefinition warnings.
-- 
2.38.0

---

From: Tom Lane <[email-masked]>
Date: Tue, 24 Jan 2023 23:37:44 -0500

> Patches attached.

+1 for 0001.  I'm still nervous about 0002.  However, maybe the
cases that we had trouble with are legacy issues that nobody cares
about anymore in 2023.  We can always look for another answer if
we get complaints, I guess.

			regards, tom lane

---

From: Andres Freund <[email-masked]>
Date: Wed, 25 Jan 2023 00:52:23 -0800

Hi,

On 2023-01-24 23:37:44 -0500, Tom Lane wrote:
> +1 for 0001.

Cool, will push tomorrow.

> I'm still nervous about 0002.  However, maybe the cases that we had trouble
> with are legacy issues that nobody cares about anymore in 2023.  We can
> always look for another answer if we get complaints, I guess.

Yea, it's a patch that should be easily revertable, if it comes to that. I'll
add a note to the commit message about potentially needing to do that if
there's not easily addressed fallout.

Greetings,

Andres Freund

---

From: Robert Haas <[email-masked]>
Date: Wed, 25 Jan 2023 08:31:23 -0500

> we get complaints, I guess.

It feels like things are changing so fast these days that whatever was
happening 12 years ago is not likely to be relevant. Compilers change
enough to cause warnings and even errors in just a few years. A decade
is long enough for an entire platform to become irrelevant.

Plus, the cost of experimentation here seems very low. Sure, something
might break, but if it does, we can just change it back, or change it
again. That's not really a big deal. The thing that would be a big
deal, maybe, is if we released and only found out afterward that this
caused some subtle and horrible problem for which we had no
back-patchable fix, but that seems pretty unlikely.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

---

From: Andres Freund <[email-masked]>
Date: Wed, 25 Jan 2023 13:26:49 -0800

Hi,

Pushed the patches. So far no fallout, and hoverfly recovered.

I just checked a few of the more odd animals (Illumos, Solaris, old OpenBSD,
AIX) that already ran without finding new warnings.

There's a few more animals to run before I'll fully relax though.

> back-patchable fix, but that seems pretty unlikely.

Agreed.

Greetings,

Andres Freund