($INBOX_DIR/description missing)
help / color / mirror / Atom feed[RFC] building postgres with meson
130+ messages / 13 participants
[nested] [flat]
* [RFC] building postgres with meson
@ 2021-10-12 08:37 Andres Freund <[email protected]>
0 siblings, 11 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-12 08:37 UTC (permalink / raw)
To: pgsql-hackers
Hi,
For the last year or so I've on and off tinkered with $subject. I think
it's in a state worth sharing now. First, let's look at a little
comparison.
My workstation:
non-cached configure:
current: 11.80s
meson: 6.67s
non-cached build (world-bin):
current: 40.46s
ninja: 7.31s
no-change build:
current: 1.17s
ninja: 0.06s
test world:
current: 105s
meson: 63s
What actually started to motivate me however were the long times windows
builds took to come back with testsresults. On CI, with the same machine
config:
build:
current: 202s (doesn't include genbki etc)
meson+ninja: 140s
meson+msbuild: 206s
test:
current: 1323s (many commands)
meson: 903s (single command)
(note that the test comparison isn't quite fair - there's a few tests
missing, but it's just small contrib ones afaik)
The biggest difference to me however is not the speed, but how readable
the output is.
Running the tests with meson in a terminal, shows the number of tests
that completed out of how many total, how much time has passed, how long
the currently running tests already have been running.
At the end of a testrun a count of tests is shown:
188/189 postgresql:tap+pg_basebackup / pg_basebackup/t/010_pg_basebackup.pl OK 39.51s 110 subtests passed
189/189 postgresql:isolation+snapshot_too_old / snapshot_too_old/isolation OK 62.93s
Ok: 188
Expected Fail: 0
Fail: 1
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Full log written to /tmp/meson/meson-logs/testlog.txt
The log has the output of the tests and ends with:
Summary of Failures:
120/189 postgresql:tap+recovery / recovery/t/007_sync_rep.pl ERROR 7.16s (exit status 255 or signal 127 SIGinvalid)
Quite the difference to make check-world -jnn output.
So, now that the teasing is done, let me explain a bit what lead me down
this path:
Autoconf + make is not being actively developed. Especially autoconf is
*barely* in maintenance mode - despite many shortcomings and bugs. It's
also technology that very few want to use - autoconf m4 is scary, and
it's scarier for people that started more recently than a lot of us
committers for example.
Recursive make as we use it is hard to get right. One reason the clean
make build is so slow compared to meson is that we had to resort to
.NOTPARALLEL to handle dependencies in a bunch of places. And despite
that, I quite regularly see incremental build failures that can be
resolved by retrying the build.
While we have incremental build via --enable-depend, they don't work
that reliable (i.e. misses necessary rebuilds) and yet is often too
aggressive. More modern build system can keep track of the precise
command used to build a target and rebuild it when that command changes.
We also don't just have the autoconf / make buildsystem, there's also
the msvc project generator - something most of us unix-y folks do not
like to touch. I think that, combined with there being no easy way to
run all tests, and it being just different, really hurt our windows
developer appeal (and subsequently the quality of postgres on
windows). I'm not saying this to ding the project generator - that was
well before there were decent "meta" buildsystems out there (and in some
ways it is a small one itself).
The last big issue I have with the current situation is that there's no
good test integration. make check-world output is essentially unreadable
/ not automatically parseable. Which led to the buildfarm having a
separate list of things it needs to test, so that failures can be
pinpointed and paired with appropriate logs. That approach unfortunately
doesn't scale well to multi-core CPUs, slowing down the buildfarm by a
fair bit.
This all led to me to experiment with improvements. I tried a few
somewhat crazy but incremental things like converting our buildsystem to
non-recursive make (I got it to build the backend, but it's too hard to
do manually I think), or to not run tests during the recursive make
check-world, but to append commands to a list of tests, that then is run
by a helper (can kinda be made to work). In the end I concluded that
the amount of time we'd need to invest to maintain our more-and-more
custom buildsystem going forward doesn't make sense.
Which lead me to look around and analyze which other buildsystems there
are that could make some sense for us. The halfway decent list includes,
I think:
1) cmake
2) bazel
3) meson
cmake would be a decent choice, I think. However, I just can't fully
warm up to it. Something about it just doesn't quite sit right with
me. That's not a good enough reason to prevent others from suggesting to
use it, but it's good enough to justify not investing a lot of time in
it myself.
Bazel has some nice architectural properties. But it requires a JVM to
run - I think that basically makes it insuitable for us. And the build
information seems quite arduous to maintain too.
Which left me with meson. It is a meta-buildsystem that can do the
actual work of building via ninja (the most common one, also targeted by
cmake), msbuild (visual studio project files, important for GUI work)
and xcode projects (I assume that's for a macos IDE, but I haven't tried
to use it). Meson roughly does what autoconf+automake did, in a
python-esque DSL, and outputs build-instructions for ninja / msbuild /
xcode. One interesting bit is that meson itself is written in python (
and fairly easy to contribute too - I got a few changes in now).
I don't think meson is perfect architecturally - e.g. its insistence on
not having functions ends up making it a bit harder to not end up
duplicating code. There's some user-interface oddities that are now hard
to fix fully, due to the faily wide usage. But all-in-all it's pretty
nice to use.
Its worth calling out that a lot of large open source projects have been
/ are migrating to meson. qemu/kvm, mesa (core part of graphics stack on
linux and also widely used in other platforms), a good chunk of GNOME,
and quite a few more. Due to that it seems unlikely to be abandoned
soon.
As far as I can tell the only OS that postgres currently supports that
meson doesn't support is HPUX. It'd likely be fairly easy to add
gcc-on-hpux support, a chunk more to add support for the proprietary
ones.
The attached patch (meson support is 0016, the rest is prerequisites
that aren't that interesting at this stage) converts most of postgres to
meson. There's a few missing contrib modules, only about half the
optional library dependencies are implemented, and I've only built on
x64. It builds on freebsd, linux, macos and windows (both ninja and
msbuild) and cross builds from linux to windows. Thomas helped make the
freebsd / macos pieces a reality, thanks!
I took a number of shortcuts (although there used to be a *lot*
more). So this shouldn't be reviewed to the normal standard of the
community - it's a prototype. But I think it's in a complete enough
shape that it allows to do a well-informed evaluation.
What doesn't yet work/ build:
- plenty optional libraries, contrib, NLS, docs build
- PGXS - and I don't yet know what to best do about it. One
backward-compatible way would be to continue use makefiles for pgxs,
but do the necessary replacement of Makefile.global.in via meson (and
not use that for postgres' own build). But that doesn't really
provide a nicer path for building postgres extensions on windows, so
it'd definitely not be a long-term path.
- JIT bitcode generation for anything but src/backend.
- anything but modern-ish x86. That's proably a small amount of work,
but something that needs to be done.
- exporting all symbols for extension modules on windows (the stuff for
postgres is implemented). Instead I marked the relevant symbols als
declspec(dllexport). I think we should do that regardless of the
buildsystem change. Restricting symbol visibility via gcc's
-fvisibility=hidden for extensions results in a substantially reduced
number of exported symbols, and even reduces object size (and I think
improves the code too). I'll send an email about that separately.
There's a lot more stuff to talk about, but I'll stop with a small bit
of instructions below:
Demo / instructions:
# Get code
git remote add andres [email protected]:anarazel/postgres.git
git fetch andres
git checkout --track andres/meson
# setup build directory
meson setup build --buildtype debug
cd build
# build (uses automatically as many cores as available)
ninja
# change configuration, build again
meson configure -Dssl=openssl
ninja
# run all tests
meson test
# run just recovery tests
meson test --suite setup --suite recovery
# list tests
meson test --list
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 09:08 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
10 siblings, 0 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-12 09:08 UTC (permalink / raw)
To: pgsql-hackers
Hi,
On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
> non-cached build (world-bin):
> current: 40.46s
> ninja: 7.31s
Interestingly this is pretty close to the minimum achievable on my
machine from the buildsystem perspective.
A build with -fuse-ld=lld, which the above didn't use, takes 6.979s. The
critical path is
bison gram.y -> gram.c 4.13s
gcc gram.c -> gram.o 2.05s
gcc postgres .... 0.317
A very helpful visualization is to transform ninja's build logs into a
tracefile with https://github.com/nico/ninjatracing
I attached an example - the trace.json.gz can be uploaded as-is to
https://ui.perfetto.dev/
It's quite a bit of of fun to look at imo.
There's a few other things quickly apparent:
- genbki prevents build progress due to dependencies on the generated
headers.
- the absolutely stupid way I implemented the python2->python3
regression test output conversion uses up a fair bit of resources
- tablecmds.c, pg_dump.c, xlog.c and a few other files are starting to
big enough to be problematic compile-time wise
Greetings,
Andres Freund
Attachments:
[application/gzip] trace.json.gz (23.6K, ../../[email protected]/2-trace.json.gz)
download
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 13:30 Peter Eisentraut <[email protected]>
parent: Andres Freund <[email protected]>
10 siblings, 2 replies; 130+ messages in thread
From: Peter Eisentraut @ 2021-10-12 13:30 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers
On 12.10.21 10:37, Andres Freund wrote:
> For the last year or so I've on and off tinkered with $subject. I think
> it's in a state worth sharing now. First, let's look at a little
> comparison.
I played with $subject a few years ago and liked it. I think, like you
said, meson is the best way forward. I support this project.
One problem I noticed back then was that some choices that we currently
determine ourselves in configure or the makefiles are hardcoded in
meson. For example, at the time, gcc on macOS was not supported. Meson
thought, if you are on macOS, you are surely using the Apple compiler,
and it supports these options. Fixing that required patches deep in the
bowels of the meson source code (and, in practice, waiting for a new
release etc.). I strongly suspect this isn't the only such problem.
For example, the shared library build behavior has been carefully tuned
in opinionated ways. With the autotools chain, one can override
anything with enough violence; so we have always felt free to do that.
I haven't followed it in a while, so I don't know what the situation is
now; but it is a concern, because we have always felt free to try new
and unusual build tools (Sun compiler, Intel compiler,
clang-when-it-was-new) early without waiting for anyone else.
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 15:00 Robert Haas <[email protected]>
parent: Peter Eisentraut <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Robert Haas @ 2021-10-12 15:00 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
On Tue, Oct 12, 2021 at 9:31 AM Peter Eisentraut
<[email protected]> wrote:
> One problem I noticed back then was that some choices that we currently
> determine ourselves in configure or the makefiles are hardcoded in
> meson. For example, at the time, gcc on macOS was not supported. Meson
> thought, if you are on macOS, you are surely using the Apple compiler,
> and it supports these options. Fixing that required patches deep in the
> bowels of the meson source code (and, in practice, waiting for a new
> release etc.). I strongly suspect this isn't the only such problem.
> For example, the shared library build behavior has been carefully tuned
> in opinionated ways. With the autotools chain, one can override
> anything with enough violence; so we have always felt free to do that.
> I haven't followed it in a while, so I don't know what the situation is
> now; but it is a concern, because we have always felt free to try new
> and unusual build tools (Sun compiler, Intel compiler,
> clang-when-it-was-new) early without waiting for anyone else.
I think we're going to need some solution to this problem. We have too
many people here with strong opinions about questions like this for me
to feel good about the idea that we're going to collectively be OK
with leaving these sorts of decisions up to some other project.
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 15:08 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
10 siblings, 0 replies; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-12 15:08 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers
On 10/12/21 4:37 AM, Andres Freund wrote:
> git remote add andres [email protected]:anarazel/postgres.git
ITYM:
git remote add andres git://github.com/anarazel/postgres.git
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 15:21 Josef Šimánek <[email protected]>
parent: Andres Freund <[email protected]>
10 siblings, 1 reply; 130+ messages in thread
From: Josef Šimánek @ 2021-10-12 15:21 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
.
út 12. 10. 2021 v 10:37 odesílatel Andres Freund <[email protected]> napsal:
>
> Hi,
>
> For the last year or so I've on and off tinkered with $subject. I think
> it's in a state worth sharing now. First, let's look at a little
> comparison.
>
> My workstation:
>
> non-cached configure:
> current: 11.80s
> meson: 6.67s
>
> non-cached build (world-bin):
> current: 40.46s
> ninja: 7.31s
>
> no-change build:
> current: 1.17s
> ninja: 0.06s
>
> test world:
> current: 105s
> meson: 63s
>
>
> What actually started to motivate me however were the long times windows
> builds took to come back with testsresults. On CI, with the same machine
> config:
>
> build:
> current: 202s (doesn't include genbki etc)
> meson+ninja: 140s
> meson+msbuild: 206s
>
>
> test:
> current: 1323s (many commands)
> meson: 903s (single command)
>
> (note that the test comparison isn't quite fair - there's a few tests
> missing, but it's just small contrib ones afaik)
>
>
> The biggest difference to me however is not the speed, but how readable
> the output is.
>
> Running the tests with meson in a terminal, shows the number of tests
> that completed out of how many total, how much time has passed, how long
> the currently running tests already have been running.
>
> At the end of a testrun a count of tests is shown:
>
> 188/189 postgresql:tap+pg_basebackup / pg_basebackup/t/010_pg_basebackup.pl OK 39.51s 110 subtests passed
> 189/189 postgresql:isolation+snapshot_too_old / snapshot_too_old/isolation OK 62.93s
>
>
> Ok: 188
> Expected Fail: 0
> Fail: 1
> Unexpected Pass: 0
> Skipped: 0
> Timeout: 0
>
> Full log written to /tmp/meson/meson-logs/testlog.txt
>
>
> The log has the output of the tests and ends with:
>
> Summary of Failures:
> 120/189 postgresql:tap+recovery / recovery/t/007_sync_rep.pl ERROR 7.16s (exit status 255 or signal 127 SIGinvalid)
>
>
> Quite the difference to make check-world -jnn output.
>
>
> So, now that the teasing is done, let me explain a bit what lead me down
> this path:
>
> Autoconf + make is not being actively developed. Especially autoconf is
> *barely* in maintenance mode - despite many shortcomings and bugs. It's
> also technology that very few want to use - autoconf m4 is scary, and
> it's scarier for people that started more recently than a lot of us
> committers for example.
>
> Recursive make as we use it is hard to get right. One reason the clean
> make build is so slow compared to meson is that we had to resort to
> .NOTPARALLEL to handle dependencies in a bunch of places. And despite
> that, I quite regularly see incremental build failures that can be
> resolved by retrying the build.
>
> While we have incremental build via --enable-depend, they don't work
> that reliable (i.e. misses necessary rebuilds) and yet is often too
> aggressive. More modern build system can keep track of the precise
> command used to build a target and rebuild it when that command changes.
>
>
> We also don't just have the autoconf / make buildsystem, there's also
> the msvc project generator - something most of us unix-y folks do not
> like to touch. I think that, combined with there being no easy way to
> run all tests, and it being just different, really hurt our windows
> developer appeal (and subsequently the quality of postgres on
> windows). I'm not saying this to ding the project generator - that was
> well before there were decent "meta" buildsystems out there (and in some
> ways it is a small one itself).
>
>
> The last big issue I have with the current situation is that there's no
> good test integration. make check-world output is essentially unreadable
> / not automatically parseable. Which led to the buildfarm having a
> separate list of things it needs to test, so that failures can be
> pinpointed and paired with appropriate logs. That approach unfortunately
> doesn't scale well to multi-core CPUs, slowing down the buildfarm by a
> fair bit.
>
>
> This all led to me to experiment with improvements. I tried a few
> somewhat crazy but incremental things like converting our buildsystem to
> non-recursive make (I got it to build the backend, but it's too hard to
> do manually I think), or to not run tests during the recursive make
> check-world, but to append commands to a list of tests, that then is run
> by a helper (can kinda be made to work). In the end I concluded that
> the amount of time we'd need to invest to maintain our more-and-more
> custom buildsystem going forward doesn't make sense.
>
>
> Which lead me to look around and analyze which other buildsystems there
> are that could make some sense for us. The halfway decent list includes,
> I think:
> 1) cmake
> 2) bazel
> 3) meson
>
>
> cmake would be a decent choice, I think. However, I just can't fully
> warm up to it. Something about it just doesn't quite sit right with
> me. That's not a good enough reason to prevent others from suggesting to
> use it, but it's good enough to justify not investing a lot of time in
> it myself.
>
> Bazel has some nice architectural properties. But it requires a JVM to
> run - I think that basically makes it insuitable for us. And the build
> information seems quite arduous to maintain too.
>
> Which left me with meson. It is a meta-buildsystem that can do the
> actual work of building via ninja (the most common one, also targeted by
> cmake), msbuild (visual studio project files, important for GUI work)
> and xcode projects (I assume that's for a macos IDE, but I haven't tried
> to use it). Meson roughly does what autoconf+automake did, in a
> python-esque DSL, and outputs build-instructions for ninja / msbuild /
> xcode. One interesting bit is that meson itself is written in python (
> and fairly easy to contribute too - I got a few changes in now).
>
>
> I don't think meson is perfect architecturally - e.g. its insistence on
> not having functions ends up making it a bit harder to not end up
> duplicating code. There's some user-interface oddities that are now hard
> to fix fully, due to the faily wide usage. But all-in-all it's pretty
> nice to use.
>
>
> Its worth calling out that a lot of large open source projects have been
> / are migrating to meson. qemu/kvm, mesa (core part of graphics stack on
> linux and also widely used in other platforms), a good chunk of GNOME,
> and quite a few more. Due to that it seems unlikely to be abandoned
> soon.
>
>
> As far as I can tell the only OS that postgres currently supports that
> meson doesn't support is HPUX. It'd likely be fairly easy to add
> gcc-on-hpux support, a chunk more to add support for the proprietary
> ones.
>
>
> The attached patch (meson support is 0016, the rest is prerequisites
> that aren't that interesting at this stage) converts most of postgres to
> meson. There's a few missing contrib modules, only about half the
> optional library dependencies are implemented, and I've only built on
> x64. It builds on freebsd, linux, macos and windows (both ninja and
> msbuild) and cross builds from linux to windows. Thomas helped make the
> freebsd / macos pieces a reality, thanks!
>
> I took a number of shortcuts (although there used to be a *lot*
> more). So this shouldn't be reviewed to the normal standard of the
> community - it's a prototype. But I think it's in a complete enough
> shape that it allows to do a well-informed evaluation.
>
> What doesn't yet work/ build:
>
> - plenty optional libraries, contrib, NLS, docs build
>
> - PGXS - and I don't yet know what to best do about it. One
> backward-compatible way would be to continue use makefiles for pgxs,
> but do the necessary replacement of Makefile.global.in via meson (and
> not use that for postgres' own build). But that doesn't really
> provide a nicer path for building postgres extensions on windows, so
> it'd definitely not be a long-term path.
>
> - JIT bitcode generation for anything but src/backend.
>
> - anything but modern-ish x86. That's proably a small amount of work,
> but something that needs to be done.
>
> - exporting all symbols for extension modules on windows (the stuff for
> postgres is implemented). Instead I marked the relevant symbols als
> declspec(dllexport). I think we should do that regardless of the
> buildsystem change. Restricting symbol visibility via gcc's
> -fvisibility=hidden for extensions results in a substantially reduced
> number of exported symbols, and even reduces object size (and I think
> improves the code too). I'll send an email about that separately.
>
>
>
>
> There's a lot more stuff to talk about, but I'll stop with a small bit
> of instructions below:
>
>
> Demo / instructions:
> # Get code
> git remote add andres [email protected]:anarazel/postgres.git
> git fetch andres
> git checkout --track andres/meson
>
> # setup build directory
> meson setup build --buildtype debug
> cd build
>
> # build (uses automatically as many cores as available)
> ninja
I'm getting errors at this step. You can find my output at
https://pastebin.com/Ar5VqfFG. Setup went well without errors. Is that
expected for now?
> # change configuration, build again
> meson configure -Dssl=openssl
> ninja
>
> # run all tests
> meson test
>
> # run just recovery tests
> meson test --suite setup --suite recovery
>
> # list tests
> meson test --list
>
>
> Greetings,
>
> Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 15:28 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
10 siblings, 1 reply; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-12 15:28 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers
On 10/12/21 4:37 AM, Andres Freund wrote:
> # setup build directory
> meson setup build --buildtype debug
I took this for an outing on msys2 and it just seems to hang. If it's not hanging it's unbelievably slow.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 15:47 Tom Lane <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Tom Lane @ 2021-10-12 15:47 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
Robert Haas <[email protected]> writes:
> I think we're going to need some solution to this problem. We have too
> many people here with strong opinions about questions like this for me
> to feel good about the idea that we're going to collectively be OK
> with leaving these sorts of decisions up to some other project.
Agreed. I'm willing to put up with the costs of moving to some
other build system, but not if it dictates choices we don't want to
make about the end products.
> From my point of view, the time it takes to run configure is annoying,
> but the build time is pretty fine. On my system, configure takes about
> 33 seconds, and a full rebuild with 'make -j8' takes 14.5 seconds (I
> am using ccache). Moreover, most of the time when I run make, I'm only
> doing a partial rebuild, so it's near-instantaneous.
Read about Autoconf's --cache-file option. That and ccache are
absolutely essential tools IMO.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 15:50 Andrew Dunstan <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-12 15:50 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers
On 10/12/21 11:28 AM, Andrew Dunstan wrote:
> On 10/12/21 4:37 AM, Andres Freund wrote:
>> # setup build directory
>> meson setup build --buildtype debug
> I took this for an outing on msys2 and it just seems to hang. If it's not hanging it's unbelievably slow.
>
>
It hung because it expected the compiler to be 'ccache cc'. Hanging in
such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
but it then errored out looking for perl libs. I think msys2 is going to
be a bit difficult here :-(
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 16:15 Andres Freund <[email protected]>
parent: Peter Eisentraut <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-12 16:15 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 15:30:57 +0200, Peter Eisentraut wrote:
> I played with $subject a few years ago and liked it. I think, like you
> said, meson is the best way forward. I support this project.
Cool.
> One problem I noticed back then was that some choices that we currently
> determine ourselves in configure or the makefiles are hardcoded in meson.
Yea, there's some of that. I think some degree of reduction in flexibility is
needed to realistically target multiple "backend" build-system like visual
studio project files etc. but I wish there were a bit less of that
nonetheless.
> For example, at the time, gcc on macOS was not supported. Meson thought, if
> you are on macOS, you are surely using the Apple compiler, and it supports
> these options.
I'm pretty sure this one now can just be overridden with CC=gcc. It can on
linux and windows, but I don't have ready interactive access with a mac
(leaving cirrus asside, which now has a "start a terminal" option...).
> For example, the shared library build behavior has been carefully tuned in
> opinionated ways. With the autotools chain, one can override anything with
> enough violence; so we have always felt free to do that. I haven't followed
> it in a while, so I don't know what the situation is now; but it is a
> concern, because we have always felt free to try new and unusual build tools
> (Sun compiler, Intel compiler, clang-when-it-was-new) early without waiting
> for anyone else.
It's possible to just take over building e.g. shared libraries ourselves with
custom targets. Although it'd be a bit annoying to do. The bigger problem is
that that e.g. wouldn't play that nicely with generating visual studio
projects, which require to generate link steps in a certain way. It'd build,
but the GUI might loose some of its options. Etc.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 16:59 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-12 16:59 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
> It hung because it expected the compiler to be 'ccache cc'. Hanging in
> such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
> but it then errored out looking for perl libs. I think msys2 is going to
> be a bit difficult here :-(
Hm. Yea, the perl thing is my fault - you should be able to get past it with
-Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
I can't reproduce the hanging though. I needed to install bison, flex and
ninja and disable perl as described above, but then it built just fine.
It does seems to crash somewhere in the main regression tests though, I think
I don't do the "set stack depth" dance correctly for msys.
If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?
(*) I've for now made most dependencies autodetected, unless you pass
--auto-features disabled to collectively disable all the auto-detected
features. Initially I had mirrored the autoconf behaviour, but I got sick of
forgetting to turn off readline or zlib on windows. And then it was useful to
test on multiple operating systems...
For working on windows meson's wraps are quite useful. I've not added that to
the git branch, but if you manually do
mkdir subprojects
meson wrap install lz4
meson wrap install zlib
building with -Dzlib=enabled -Dlz4=enabled will fall back to building lz4,
zlib as-needed.
I was wondering about adding a binary wrap for e.g. bison, flex on windows, so
that the process of getting a build going isn't as arduous.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 17:17 Andres Freund <[email protected]>
parent: Josef Šimánek <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-12 17:17 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 17:21:50 +0200, Josef Šimánek wrote:
> > # build (uses automatically as many cores as available)
> > ninja
>
> I'm getting errors at this step. You can find my output at
> https://pastebin.com/Ar5VqfFG. Setup went well without errors. Is that
> expected for now?
Thanks, that's helpful. And no, that's not expected (*), it should be fixed.
What OS / distribution / version is this?
Can you build postgres "normally" with --with-gss? Seems like we're ending up
with a version of gssapi that we're not compatible with.
You should be able to get past this by disabling gss using meson configure
-Dgssapi=disabled.
Greetings,
Andres Freund
* except kinda, in the sense that I'd expect it to be buggy, given that I've
run it only on a few machines and it's very, uh, bleeding edge
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 18:09 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-12 18:09 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 09:59:26 -0700, Andres Freund wrote:
> On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
> > It hung because it expected the compiler to be 'ccache cc'. Hanging in
> > such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
> > but it then errored out looking for perl libs. I think msys2 is going to
> > be a bit difficult here :-(
>
> Hm. Yea, the perl thing is my fault - you should be able to get past it with
> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
This is a weird one. I don't know much about msys, so it's probably related to
that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
shell commands that exists, but not according to msys's own python
$ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:', os.path.exists(p))"
does /usr/lib/perl5/core_perl/CORE exist: False
$ ls -ld /usr/lib/perl5/core_perl/CORE
drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE
So it's not too surprising that that doesn't work out. It's easy enough to
work around, but still pretty weird.
I pushed a workaround for the config-time error, but it doesn't yet recognize
msys perl correctly. But at least it's not alone in that - configure doesn't
seem to either, so I'm probably doing something wrong :)
> I can't reproduce the hanging though. I needed to install bison, flex and
> ninja and disable perl as described above, but then it built just fine.
>
> It does seems to crash somewhere in the main regression tests though, I think
> I don't do the "set stack depth" dance correctly for msys.
That was it - just hadn't ported setting -Wl,--stack=... for !msvc
windows. Pushed the fix for that out.
I guess I should figure out how to commandline install msys and add it to CI.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 18:11 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-12 18:11 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 12:59 PM, Andres Freund wrote:
>
>
> If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?
Here's the entire thing
# cat
C:/tools/msys64/home/Administrator/postgresql/build/meson-logs/meson-log.txt
Build started at 2021-10-12T18:08:34.387568
Main binary: C:/tools/msys64/mingw64/bin/python.exe
Build Options: -Dbuildtype=debug
Python system: Windows
The Meson build system
Version: 0.59.1
Source dir: C:/tools/msys64/home/Administrator/postgresql
Build dir: C:/tools/msys64/home/Administrator/postgresql/build
Build type: native build
Project name: postgresql
Project version: 15devel
Sanity testing C compiler: ccache cc
Is cross compiler: False.
Sanity check compiler command line: ccache cc sanitycheckc.c -o
sanitycheckc.exe -D_FILE_OFFSET_BITS=64
Sanity check compile stdout:
-----
Sanity check compile stderr:
-----
meson.build:1:0: ERROR: Compiler ccache cc can not compile programs.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 18:23 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-12 18:23 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 14:11:39 -0400, Andrew Dunstan wrote:
> On 10/12/21 12:59 PM, Andres Freund wrote:
> > If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?
> Here's the entire thing
> Sanity check compiler command line: ccache cc sanitycheckc.c -o
> sanitycheckc.exe -D_FILE_OFFSET_BITS=64
> Sanity check compile stdout:
>
> -----
> Sanity check compile stderr:
>
> -----
>
> meson.build:1:0: ERROR: Compiler ccache cc can not compile programs.
Huh, it's not a question of gcc vs cc, it's that meson automatically uses
ccache. And it looks like msys's ccache is broken at the moment (installed
yesterday):
$ ccache --version
ccache version 4.4.1
...
$ echo > test.c
$ ccache cc -c test.c
Segmentation fault (core dumped)
..
not sure how that leads to hanging, but it's not too surprising that things
don't work out after that...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 18:37 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-12 18:37 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 2:09 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-12 09:59:26 -0700, Andres Freund wrote:
>> On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
>>> It hung because it expected the compiler to be 'ccache cc'. Hanging in
>>> such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
>>> but it then errored out looking for perl libs. I think msys2 is going to
>>> be a bit difficult here :-(
>> Hm. Yea, the perl thing is my fault - you should be able to get past it with
>> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
> This is a weird one. I don't know much about msys, so it's probably related to
> that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
> shell commands that exists, but not according to msys's own python
>
> $ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:', os.path.exists(p))"
> does /usr/lib/perl5/core_perl/CORE exist: False
>
> $ ls -ld /usr/lib/perl5/core_perl/CORE
> drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE
Looks to me like a python issue:
# perl -e 'my $p = "/usr/lib/perl5/core_perl/CORE"; print qq(does $p
exist: ), -e $p, qq{\n};'
does /usr/lib/perl5/core_perl/CORE exist: 1
# python -c "import os; p = '/usr/lib/perl5/core_perl/CORE';
print(f'does {p} exist:', os.path.exists(p))"
does /usr/lib/perl5/core_perl/CORE exist: False
# cygpath -m /usr/lib/perl5/core_perl/CORE
C:/tools/msys64/usr/lib/perl5/core_perl/CORE
# python -c "import os; p =
'C:/tools/msys64/usr/lib/perl5/core_perl/CORE'; print(f'does {p}
exist:', os.path.exists(p))"
does C:/tools/msys64/usr/lib/perl5/core_perl/CORE exist: True
Clearly python is not understanding msys virtualized paths.
>
>
> I guess I should figure out how to commandline install msys and add it to CI.
>
here's what I do:
# msys2 outputs esc-[3J which clears the screen's scroll buffer. Nasty.
# so we redirect the output
# find the log in c:\Windows\System32 if needed
choco install -y --no-progress --limit-output msys2 > msys2inst.log
c:\tools\msys64\usr\bin\bash -l
'/c/vfiles/windows-uploads/msys2-packages.sh'
Here's what's in msys-packages.sh:
pacman -S --needed --noconfirm \
base-devel \
msys/git \
msys/ccache \
msys/vim \
msys/perl-Crypt-SSLeay \
mingw-w64-clang-x86_64-toolchain \
mingw-w64-x86_64-toolchain
# could do: pacman -S --needed --noconfirm development
# this is more economical. These should cover most of the things you
might
# want to configure with
pacman -S --needed --noconfirm \
msys/gettext-devel \
msys/icu-devel \
msys/libiconv-devel \
msys/libreadline-devel \
msys/libxml2-devel \
msys/libxslt-devel \
msys/openssl-devel \
msys/zlib-devel
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 18:42 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-12 18:42 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 2:23 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-12 14:11:39 -0400, Andrew Dunstan wrote:
>> On 10/12/21 12:59 PM, Andres Freund wrote:
>>> If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?
>> Here's the entire thing
>> Sanity check compiler command line: ccache cc sanitycheckc.c -o
>> sanitycheckc.exe -D_FILE_OFFSET_BITS=64
>> Sanity check compile stdout:
>>
>> -----
>> Sanity check compile stderr:
>>
>> -----
>>
>> meson.build:1:0: ERROR: Compiler ccache cc can not compile programs.
> Huh, it's not a question of gcc vs cc, it's that meson automatically uses
> ccache. And it looks like msys's ccache is broken at the moment (installed
> yesterday):
>
> $ ccache --version
> ccache version 4.4.1
> ...
>
> $ echo > test.c
> $ ccache cc -c test.c
> Segmentation fault (core dumped)
> ..
>
> not sure how that leads to hanging, but it's not too surprising that things
> don't work out after that...
>
Yes, I've had to disable ccache on fairywren.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 19:01 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-12 19:01 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 09:15:41 -0700, Andres Freund wrote:
> > For example, at the time, gcc on macOS was not supported. Meson thought, if
> > you are on macOS, you are surely using the Apple compiler, and it supports
> > these options.
>
> I'm pretty sure this one now can just be overridden with CC=gcc. It can on
> linux and windows, but I don't have ready interactive access with a mac
> (leaving cirrus asside, which now has a "start a terminal" option...).
It was a tad more complicated. But only because it took me a while to figure
out how to make gcc on macos actually work, independent of meson. Initially
gcc was always failing with errors about not finding the linker, and
installing binutils was a dead end.
Turns out just using a gcc at a specific path doesn't work, it ends up using
wrong internal binaries or something like that.
Once I got to that, the meson part was easy:
$ export PATH="/usr/local/opt/gcc/bin:$PATH"
$ CC=gcc-11 meson setup build-gcc
...
C compiler for the host machine: gcc-11 (gcc 11.2.0 "gcc-11 (Homebrew GCC 11.2.0) 11.2.0")
...
$ cd build-gcc
$ ninja test
...
181/181 postgresql:tap+subscription / subscription/t/100_bugs.pl OK 17.83s 5 subtests passed
Ok: 180
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 1
Timeout: 0
One thing that is nice with meson's testrunner is that it can parse the output
of tap tests and recognizes the number of completed / failed subtests. I
wonder whether we could make pg_regress' output tap compliant without the
output quality suffering too much.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 19:16 Andrew Dunstan <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-12 19:16 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 2:37 PM, Andrew Dunstan wrote:
> On 10/12/21 2:09 PM, Andres Freund wrote:
>> Hi,
>>
>> On 2021-10-12 09:59:26 -0700, Andres Freund wrote:
>>> On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
>>>> It hung because it expected the compiler to be 'ccache cc'. Hanging in
>>>> such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
>>>> but it then errored out looking for perl libs. I think msys2 is going to
>>>> be a bit difficult here :-(
>>> Hm. Yea, the perl thing is my fault - you should be able to get past it with
>>> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
>> This is a weird one. I don't know much about msys, so it's probably related to
>> that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
>> shell commands that exists, but not according to msys's own python
>>
>> $ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:', os.path.exists(p))"
>> does /usr/lib/perl5/core_perl/CORE exist: False
>>
>> $ ls -ld /usr/lib/perl5/core_perl/CORE
>> drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE
>
> Looks to me like a python issue:
>
>
> # perl -e 'my $p = "/usr/lib/perl5/core_perl/CORE"; print qq(does $p
> exist: ), -e $p, qq{\n};'
> does /usr/lib/perl5/core_perl/CORE exist: 1
>
> # python -c "import os; p = '/usr/lib/perl5/core_perl/CORE';
> print(f'does {p} exist:', os.path.exists(p))"
> does /usr/lib/perl5/core_perl/CORE exist: False
>
> # cygpath -m /usr/lib/perl5/core_perl/CORE
> C:/tools/msys64/usr/lib/perl5/core_perl/CORE
>
> # python -c "import os; p =
> 'C:/tools/msys64/usr/lib/perl5/core_perl/CORE'; print(f'does {p}
> exist:', os.path.exists(p))"
> does C:/tools/msys64/usr/lib/perl5/core_perl/CORE exist: True
>
>
> Clearly python is not understanding msys virtualized paths.
It's a matter of which python you use. The one that understands msys
paths is msys/python. The mingw64 packages are normally pure native
windows and so don't understand msys paths. I know it's confusing :-(
# /usr/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE';
print(f'does {p} exist:', os.path.exists(p))"
does /usr/lib/perl5/core_perl/CORE exist: True
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 19:29 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-12 19:29 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 14:37:04 -0400, Andrew Dunstan wrote:
> On 10/12/21 2:09 PM, Andres Freund wrote:
> >> Hm. Yea, the perl thing is my fault - you should be able to get past it with
> >> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
> > This is a weird one. I don't know much about msys, so it's probably related to
> > that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
> > shell commands that exists, but not according to msys's own python
> >
> > $ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:', os.path.exists(p))"
> > does /usr/lib/perl5/core_perl/CORE exist: False
> >
> > $ ls -ld /usr/lib/perl5/core_perl/CORE
> > drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE
> Looks to me like a python issue:
> Clearly python is not understanding msys virtualized paths.
Ah, it's a question of the *wrong* python being used :/. I somehow ended up
with both a mingw and an msys python, with the mingw python taking preference
over the msys one. The latter one does understand such paths.
> > I guess I should figure out how to commandline install msys and add it to CI.
> here's what I do:
Thanks!
Does that recipe get you to a build where ./configure --with-perl succeeds?
I see this here:
checking for Perl archlibexp... /usr/lib/perl5/core_perl
checking for Perl privlibexp... /usr/share/perl5/core_perl
checking for Perl useshrplib... true
checking for CFLAGS recommended by Perl... -DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -fno-strict-aliasing -fstack-protector-strong
checking for CFLAGS to compile embedded Perl... -DPERL_USE_SAFE_PUTENV
checking for flags to link embedded Perl... no
configure: error: could not determine flags for linking embedded Perl.
This probably means that ExtUtils::Embed or ExtUtils::MakeMaker is not
installed.
If I just include perl.h from a test file with gcc using the above flags it
fails to compile:
$ echo '#include <perl.h>' > test.c
$ gcc -DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -fno-strict-aliasing -fstack-protector-strong test.c -c -I /c/dev/msys64/usr/lib/perl5/core_perl/CORE
In file included from test.c:1:
C:/dev/msys64/usr/lib/perl5/core_perl/CORE/perl.h:1003:13: fatal error: sys/wait.h: No such file or directory
1003 | # include <sys/wait.h>
and ldopts bleats
$ perl -MExtUtils::Embed -e ldopts
Warning (mostly harmless): No library found for -lpthread
Warning (mostly harmless): No library found for -ldl
-Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--enable-auto-image-base -fstack-protector-strong -L/usr/lib/perl5/core_perl/CORE -lperl -lcrypt
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 19:55 John Naylor <[email protected]>
parent: Andres Freund <[email protected]>
10 siblings, 2 replies; 130+ messages in thread
From: John Naylor @ 2021-10-12 19:55 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Tue, Oct 12, 2021 at 4:37 AM Andres Freund <[email protected]> wrote:
[Meson prototype]
The build code looks pretty approachable for someone with no prior
exposure, and feels pretty nice when running it (I couldn't get a build
working but I'll leave that aside for now).
> As far as I can tell the only OS that postgres currently supports that
> meson doesn't support is HPUX. It'd likely be fairly easy to add
> gcc-on-hpux support, a chunk more to add support for the proprietary
> ones.
That would also have to work for all the dependencies, which were displayed
to me as:
ninja, gdbm, ca-certificates, [email protected], readline, sqlite and [email protected]
Also, could utility makefile targets be made to work? I'm thinking in
particular of update-unicode and reformat-dat-files, for example.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 20:02 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-12 20:02 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 3:29 PM, Andres Freund wrote:
>
> Does that recipe get you to a build where ./configure --with-perl succeeds?
>
> I see this here:
>
> checking for Perl archlibexp... /usr/lib/perl5/core_perl
> checking for Perl privlibexp... /usr/share/perl5/core_perl
> checking for Perl useshrplib... true
> checking for CFLAGS recommended by Perl... -DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -fno-strict-aliasing -fstack-protector-strong
> checking for CFLAGS to compile embedded Perl... -DPERL_USE_SAFE_PUTENV
> checking for flags to link embedded Perl... no
> configure: error: could not determine flags for linking embedded Perl.
> This probably means that ExtUtils::Embed or ExtUtils::MakeMaker is not
> installed.
>
> If I just include perl.h from a test file with gcc using the above flags it
> fails to compile:
You need to build against a native perl, like Strawberry or ActiveState.
(I have had mixed success with Strawberry) You do that by putting a path
to it at the start of the PATH. The wrinkle in this is that you need
prove to point to one that understands virtual paths. So you do
something like this:
PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 20:42 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-12 20:42 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 16:02:14 -0400, Andrew Dunstan wrote:
> You need to build against a native perl, like Strawberry or ActiveState.
> (I have had mixed success with Strawberry)
Do you understand why that is needed?
> You do that by putting a path to it at the start of the PATH. The wrinkle in
> this is that you need prove to point to one that understands virtual
> paths. So you do something like this:
>
>
> PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...
Oh my.
I'll try that later... I wonder if we could make this easier from our side?
This is a lot of magic to know.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 20:59 Andres Freund <[email protected]>
parent: John Naylor <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-12 20:59 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 15:55:22 -0400, John Naylor wrote:
> On Tue, Oct 12, 2021 at 4:37 AM Andres Freund <[email protected]> wrote:
> The build code looks pretty approachable for someone with no prior
> exposure, and feels pretty nice when running it
That's part of what attracted me...
> (I couldn't get a build working but I'll leave that aside for now).
If you want to do that separately, I'll try to fix it.
> > As far as I can tell the only OS that postgres currently supports that
> > meson doesn't support is HPUX. It'd likely be fairly easy to add
> > gcc-on-hpux support, a chunk more to add support for the proprietary
> > ones.
>
> That would also have to work for all the dependencies, which were displayed
> to me as:
>
> ninja, gdbm, ca-certificates, [email protected], readline, sqlite and [email protected]
meson does depend on ninja (to execute the build) and of course python. But
the rest should be optional dependencies. ninja builds without any
dependencies as long as you don't change its parser sources. python builds on
aix, hpux etc.
Not sure what way gdbm [email protected] and sqlite are pulled in? I assume readline
is for python...
> Also, could utility makefile targets be made to work? I'm thinking in
> particular of update-unicode and reformat-dat-files, for example.
Yes, that shouldn't be a problem. You can run arbitrary code in targets
(there's plenty need for that already in what I have so far).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 23:19 Josef Šimánek <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Josef Šimánek @ 2021-10-12 23:19 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
út 12. 10. 2021 v 19:17 odesílatel Andres Freund <[email protected]> napsal:
>
> Hi,
>
> On 2021-10-12 17:21:50 +0200, Josef Šimánek wrote:
> > > # build (uses automatically as many cores as available)
> > > ninja
> >
> > I'm getting errors at this step. You can find my output at
> > https://pastebin.com/Ar5VqfFG. Setup went well without errors. Is that
> > expected for now?
>
> Thanks, that's helpful. And no, that's not expected (*), it should be fixed.
>
> What OS / distribution / version is this?
Fedora 34 (64 bit)
> Can you build postgres "normally" with --with-gss? Seems like we're ending up
> with a version of gssapi that we're not compatible with.
Yes, I can.
> You should be able to get past this by disabling gss using meson configure
> -Dgssapi=disabled.
I tried to clean and start from scratch, but I'm getting different
error probably related to wrongly configured JIT (LLVM wasn't found
during meson setup). I'll debug on my side to provide more info.
Whole build error could be found at https://pastebin.com/hCFqcPvZ.
Setup log could be found at https://pastebin.com/wjbE1w56.
> Greetings,
>
> Andres Freund
>
> * except kinda, in the sense that I'd expect it to be buggy, given that I've
> run it only on a few machines and it's very, uh, bleeding edge
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-12 23:54 Andres Freund <[email protected]>
parent: Josef Šimánek <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-12 23:54 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 01:19:27 +0200, Josef Šimánek wrote:
> I tried to clean and start from scratch, but I'm getting different
> error probably related to wrongly configured JIT (LLVM wasn't found
> during meson setup). I'll debug on my side to provide more info.
../src/backend/jit/jit.c:91:73: error: ‘DLSUFFIX’ undeclared (first use in this function)
91 | snprintf(path, MAXPGPATH, "%s/%s%s", pkglib_path, jit_provider, DLSUFFIX);
| ^~~~~~~~
This *very* likely is related to building in a source tree that also contains
a "non-meson" build "in place". The problem is that the meson build picks up
the pg_config.h generated by ./configure in the "normal" build, rather than
the one meson generated itself.
You'd need to execute make distclean or such, or use a separate git checkout.
I forgot about this issue because I only ever build postgres from outside the
source-tree (by invoking configure from a separate directory), so there's
never build products in it. I think at least I need to make the build emit a
warning / error if there's a pg_config.h in the source tree...
This is the part of the jit code that's built regardless of llvm availability
- you'd get the same error in a few other places unrelated to jit.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 01:03 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-13 01:03 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 13:42:56 -0700, Andres Freund wrote:
> On 2021-10-12 16:02:14 -0400, Andrew Dunstan wrote:
> > You do that by putting a path to it at the start of the PATH. The wrinkle in
> > this is that you need prove to point to one that understands virtual
> > paths. So you do something like this:
> >
> >
> > PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...
>
> Oh my.
>
> I'll try that later... I wonder if we could make this easier from our side?
> This is a lot of magic to know.
I managed to get this working. At first it failed because I don't have
pexports - it's not available inside msys as far as I could tell. And seems to
be unmaintained. But replacing pexports with gendef fixed that.
There's this comment in src/pl/plperl/GNUmakefile
# Perl on win32 ships with import libraries only for Microsoft Visual C++,
# which are not compatible with mingw gcc. Therefore we need to build a
# new import library to link with.
but I seem to be able to link fine without going through that song-and-dance?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 11:54 Daniel Gustafsson <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Daniel Gustafsson @ 2021-10-13 11:54 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; pgsql-hackers
> On 12 Oct 2021, at 21:01, Andres Freund <[email protected]> wrote:
> One thing that is nice with meson's testrunner is that it can parse the output
> of tap tests and recognizes the number of completed / failed subtests. I
> wonder whether we could make pg_regress' output tap compliant without the
> output quality suffering too much.
I added a --tap option for TAP output to pg_regress together with Jinbao Chen
for giggles and killing some time a while back. It's not entirely done and
sort of PoC, but most of it works. Might not be of interest here, but in case
it is I've refreshed it slightly and rebased it. There might be better ways to
do it, but the aim was to make the diff against the guts of pg_regress small
and instead extract output functions for the different formats.
It omits the test timings, but that could be added either as a diagnostic line
following each status or as a YAML block in TAP 13 (the attached is standard
TAP, not version 13 but the change would be trivial).
If it's helpful and there's any interest for this I'm happy to finish it up now.
One thing that came out of this, is that we don't really handle the ignored
tests in the way the code thinks it does for normal output, the attached treats
ignored tests as SKIP tests.
--
Daniel Gustafsson https://vmware.com/
Attachments:
[application/octet-stream] pg_regress_tap.diff (14.1K, ../../[email protected]/2-pg_regress_tap.diff)
download | inline diff:
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index 05296f7ee1..dc60edf82e 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -95,6 +95,7 @@ static char *dlpath = PKGLIBDIR;
static char *user = NULL;
static _stringlist *extraroles = NULL;
static char *config_auth_datadir = NULL;
+static bool tap = false;
/* internal variables */
static const char *progname;
@@ -120,9 +121,71 @@ static int fail_ignore_count = 0;
static bool directory_exists(const char *dir);
static void make_directory(const char *dir);
-static void header(const char *fmt,...) pg_attribute_printf(1, 2);
+struct output_func
+{
+ void (*header)(const char *line);
+ void (*footer)(const char *difffilename, const char *logfilename);
+ void (*comment)(const char *comment);
+
+ void (*test_status_preamble)(const char *testname);
+
+ void (*test_status_ok)(const char *testname);
+ void (*test_status_failed)(const char *testname);
+ void (*test_status_ignored)(const char *testname);
+
+ void (*test_runtime)(const char *testname, double runtime);
+};
+
+
+void (*test_runtime)(const char *testname, double runtime);
+/* Text output format */
+static void header_text(const char *line);
+static void footer_text(const char *difffilename, const char *logfilename);
+static void comment_text(const char *comment);
+static void test_status_preamble_text(const char *testname);
+static void test_status_ok_text(const char *testname);
+static void test_status_failed_text(const char *testname);
+static void test_runtime_text(const char *testname, double runtime);
+
+struct output_func output_func_text =
+{
+ header_text,
+ footer_text,
+ comment_text,
+ test_status_preamble_text,
+ test_status_ok_text,
+ test_status_failed_text,
+ NULL,
+ test_runtime_text
+};
+
+/* TAP output format */
+static void header_tap(const char *line);
+static void footer_tap(const char *difffilename, const char *logfilename);
+static void comment_tap(const char *comment);
+static void test_status_ok_tap(const char *testname);
+static void test_status_failed_tap(const char *testname);
+static void test_status_ignored_tap(const char *testname);
+
+struct output_func output_func_tap =
+{
+ header_tap,
+ footer_tap,
+ comment_tap,
+ NULL,
+ test_status_ok_tap,
+ test_status_failed_tap,
+ test_status_ignored_tap,
+ NULL
+};
+
+struct output_func *output = &output_func_text;
+
+static void test_status_ok(const char *testname);
+
static void status(const char *fmt,...) pg_attribute_printf(1, 2);
static void psql_command(const char *database, const char *query,...) pg_attribute_printf(2, 3);
+static void status_end(void);
/*
* allow core files if possible.
@@ -206,18 +269,227 @@ split_to_stringlist(const char *s, const char *delim, _stringlist **listhead)
/*
* Print a progress banner on stdout.
*/
+static void
+header_text(const char *line)
+{
+ fprintf(stdout, "============== %-38s ==============\n", line);
+ fflush(stdout);
+}
+
+static void
+header_tap(const char *line)
+{
+ fprintf(stdout, "# %s\n", line);
+ fflush(stdout);
+}
+
static void
header(const char *fmt,...)
{
char tmp[64];
va_list ap;
+ if (!output->header)
+ return;
+
va_start(ap, fmt);
vsnprintf(tmp, sizeof(tmp), fmt, ap);
va_end(ap);
- fprintf(stdout, "============== %-38s ==============\n", tmp);
- fflush(stdout);
+ output->header(tmp);
+}
+
+static void
+footer_tap(const char *difffilename, const char *logfilename)
+{
+ status("1..%i\n", (fail_count + fail_ignore_count + success_count));
+ status_end();
+}
+
+static void
+footer(const char *difffilename, const char *logfilename)
+{
+ if (output->footer)
+ output->footer(difffilename, logfilename);
+}
+
+static void
+comment_text(const char *comment)
+{
+ status("%s", comment);
+}
+
+static void
+comment_tap(const char *comment)
+{
+ status("# %s", comment);
+}
+
+static void
+comment(const char *fmt,...)
+{
+ char tmp[256];
+ va_list ap;
+
+ if (!output->comment)
+ return;
+
+ va_start(ap, fmt);
+ vsnprintf(tmp, sizeof(tmp), fmt, ap);
+ va_end(ap);
+
+ output->comment(tmp);
+}
+
+static void
+test_status_preamble_text(const char *testname)
+{
+ status(_("test %-28s ... "), testname);
+}
+
+static void
+test_status_preamble(const char *testname)
+{
+ if (output->test_status_preamble)
+ output->test_status_preamble(testname);
+}
+
+static void
+test_status_ok_tap(const char *testname)
+{
+ /* There is no NLS translation here as "ok" is a protocol message */
+ status("ok %i - %s",
+ (fail_count + fail_ignore_count + success_count),
+ testname);
+}
+
+static void
+test_status_ok_text(const char *testname)
+{
+ (void) testname; /* unused */
+ status(_("ok ")); /* align with FAILED */
+}
+
+static void
+test_status_ok(const char *testname)
+{
+ success_count++;
+ if (output->test_status_ok)
+ output->test_status_ok(testname);
+}
+
+static void
+test_status_failed_tap(const char *testname)
+{
+ status("not ok %i - %s",
+ (fail_count + fail_ignore_count + success_count),
+ testname);
+}
+
+static void
+test_status_failed_text(const char *testname)
+{
+ status(_("FAILED"));
+}
+
+static void
+test_status_failed(const char *testname)
+{
+ fail_count++;
+ if (output->test_status_failed)
+ output->test_status_failed(testname);
+}
+
+static void
+test_status_ignored(const char *testname)
+{
+ fail_ignore_count++;
+ if (output->test_status_ignored)
+ output->test_status_ignored(testname);
+}
+
+static void
+test_status_ignored_tap(const char *testname)
+{
+ status("ok %i - %s # SKIP (ignored)",
+ (fail_count + fail_ignore_count + success_count),
+ testname);
+}
+
+static void
+test_runtime_text(const char *testname, double runtime)
+{
+ (void)testname;
+ status(_(" %8.0f ms"), runtime);
+}
+
+static void
+runtime(const char *testname, double runtime)
+{
+ if (output->test_runtime)
+ output->test_runtime(testname, runtime);
+}
+
+static void
+footer_text(const char *difffilename, const char *logfilename)
+{
+ char buf[256];
+
+ /*
+ * Emit nice-looking summary message
+ */
+ if (fail_count == 0 && fail_ignore_count == 0)
+ snprintf(buf, sizeof(buf),
+ _(" All %d tests passed. "),
+ success_count);
+ else if (fail_count == 0) /* fail_count=0, fail_ignore_count>0 */
+ snprintf(buf, sizeof(buf),
+ _(" %d of %d tests passed, %d failed test(s) ignored. "),
+ success_count,
+ success_count + fail_ignore_count,
+ fail_ignore_count);
+ else if (fail_ignore_count == 0) /* fail_count>0 && fail_ignore_count=0 */
+ snprintf(buf, sizeof(buf),
+ _(" %d of %d tests failed. "),
+ fail_count,
+ success_count + fail_count);
+ else
+ /* fail_count>0 && fail_ignore_count>0 */
+ snprintf(buf, sizeof(buf),
+ _(" %d of %d tests failed, %d of these failures ignored. "),
+ fail_count + fail_ignore_count,
+ success_count + fail_count + fail_ignore_count,
+ fail_ignore_count);
+
+ putchar('\n');
+ for (int i = strlen(buf); i > 0; i--)
+ putchar('=');
+ printf("\n%s\n", buf);
+ for (int i = strlen(buf); i > 0; i--)
+ putchar('=');
+ putchar('\n');
+ putchar('\n');
+
+ if (difffilename && logfilename)
+ {
+ printf(_("The differences that caused some tests to fail can be viewed in the\n"
+ "file \"%s\". A copy of the test summary that you see\n"
+ "above is saved in the file \"%s\".\n\n"),
+ difffilename, logfilename);
+ }
+}
+
+static void
+status_start(bool single, const char *testname)
+{
+ /* TAP only outputs after the test has finished */
+ if (tap)
+ return;
+
+ if (single)
+ status(_("test %-24s ... "), testname);
+ else
+ status(_(" %-24s ... "), testname);
}
/*
@@ -917,13 +1189,13 @@ initialize_environment(void)
#endif
if (pghost && pgport)
- printf(_("(using postmaster on %s, port %s)\n"), pghost, pgport);
+ comment(_("(using postmaster on %s, port %s)\n"), pghost, pgport);
if (pghost && !pgport)
- printf(_("(using postmaster on %s, default port)\n"), pghost);
+ comment(_("(using postmaster on %s, default port)\n"), pghost);
if (!pghost && pgport)
- printf(_("(using postmaster on Unix socket, port %s)\n"), pgport);
+ comment(_("(using postmaster on Unix socket, port %s)\n"), pgport);
if (!pghost && !pgport)
- printf(_("(using postmaster on Unix socket, default port)\n"));
+ comment(_("(using postmaster on Unix socket, default port)\n"));
}
convert_sourcefiles();
@@ -1167,9 +1439,10 @@ psql_command(const char *database, const char *query,...)
/* And now we can build and execute the shell command */
snprintf(psql_cmd, sizeof(psql_cmd),
- "\"%s%spsql\" -X -c \"%s\" \"%s\"",
+ "\"%s%spsql\" %s -X -c \"%s\" \"%s\"",
bindir ? bindir : "",
bindir ? "/" : "",
+ tap ? "-q" : "",
query_escaped,
database);
@@ -1704,6 +1977,9 @@ run_schedule(const char *schedule, test_start_function startfunc,
c++;
add_stringlist_item(&ignorelist, c);
+ test_status_ignored(c);
+ status_end();
+
/*
* Note: ignore: lines do not run the test, they just say that
* failure of this test when run later on is to be ignored. A bit
@@ -1762,7 +2038,7 @@ run_schedule(const char *schedule, test_start_function startfunc,
if (num_tests == 1)
{
- status(_("test %-28s ... "), tests[0]);
+ test_status_preamble(tests[0]);
pids[0] = (startfunc) (tests[0], &resultfiles[0], &expectfiles[0], &tags[0]);
INSTR_TIME_SET_CURRENT(starttimes[0]);
wait_for_tests(pids, statuses, stoptimes, NULL, 1);
@@ -1778,8 +2054,8 @@ run_schedule(const char *schedule, test_start_function startfunc,
{
int oldest = 0;
- status(_("parallel group (%d tests, in groups of %d): "),
- num_tests, max_connections);
+ comment(_("parallel group (%d tests, in groups of %d): "),
+ num_tests, max_connections);
for (i = 0; i < num_tests; i++)
{
if (i - oldest >= max_connections)
@@ -1799,7 +2075,7 @@ run_schedule(const char *schedule, test_start_function startfunc,
}
else
{
- status(_("parallel group (%d tests): "), num_tests);
+ comment(_("parallel group (%d tests): "), num_tests);
for (i = 0; i < num_tests; i++)
{
pids[i] = (startfunc) (tests[i], &resultfiles[i], &expectfiles[i], &tags[i]);
@@ -1818,7 +2094,7 @@ run_schedule(const char *schedule, test_start_function startfunc,
bool differ = false;
if (num_tests > 1)
- status(_(" %-28s ... "), tests[i]);
+ test_status_preamble(tests[i]);
/*
* Advance over all three lists simultaneously.
@@ -1858,27 +2134,18 @@ run_schedule(const char *schedule, test_start_function startfunc,
}
}
if (ignore)
- {
- status(_("failed (ignored)"));
- fail_ignore_count++;
- }
+ test_status_ignored(tests[i]);
else
- {
- status(_("FAILED"));
- fail_count++;
- }
+ test_status_failed(tests[i]);
}
else
- {
- status(_("ok ")); /* align with FAILED */
- success_count++;
- }
+ test_status_ok(tests[i]);
if (statuses[i] != 0)
log_child_failure(statuses[i]);
INSTR_TIME_SUBTRACT(stoptimes[i], starttimes[i]);
- status(_(" %8.0f ms"), INSTR_TIME_GET_MILLISEC(stoptimes[i]));
+ runtime(tests[i], INSTR_TIME_GET_MILLISEC(stoptimes[i]));
status_end();
}
@@ -1917,7 +2184,7 @@ run_single_test(const char *test, test_start_function startfunc,
*tl;
bool differ = false;
- status(_("test %-28s ... "), test);
+ test_status_preamble(test);
pid = (startfunc) (test, &resultfiles, &expectfiles, &tags);
INSTR_TIME_SET_CURRENT(starttime);
wait_for_tests(&pid, &exit_status, &stoptime, NULL, 1);
@@ -1947,15 +2214,9 @@ run_single_test(const char *test, test_start_function startfunc,
}
if (differ)
- {
- status(_("FAILED"));
- fail_count++;
- }
+ test_status_failed(test);
else
- {
- status(_("ok ")); /* align with FAILED */
- success_count++;
- }
+ test_status_ok(test);
if (exit_status != 0)
log_child_failure(exit_status);
@@ -2152,6 +2413,7 @@ regression_main(int argc, char *argv[],
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
{"make-testtablespace-dir", no_argument, NULL, 26},
+ {"tap", no_argument, NULL, 27},
{NULL, 0, NULL, 0}
};
@@ -2285,6 +2547,9 @@ regression_main(int argc, char *argv[],
case 26:
make_testtablespace_dir = true;
break;
+ case 27:
+ tap = true;
+ break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
@@ -2311,6 +2576,9 @@ regression_main(int argc, char *argv[],
exit(0);
}
+ if (tap)
+ output = &output_func_tap;
+
if (temp_instance && !port_specified_by_user)
/*
@@ -2636,54 +2904,20 @@ regression_main(int argc, char *argv[],
fclose(logfile);
- /*
- * Emit nice-looking summary message
- */
- if (fail_count == 0 && fail_ignore_count == 0)
- snprintf(buf, sizeof(buf),
- _(" All %d tests passed. "),
- success_count);
- else if (fail_count == 0) /* fail_count=0, fail_ignore_count>0 */
- snprintf(buf, sizeof(buf),
- _(" %d of %d tests passed, %d failed test(s) ignored. "),
- success_count,
- success_count + fail_ignore_count,
- fail_ignore_count);
- else if (fail_ignore_count == 0) /* fail_count>0 && fail_ignore_count=0 */
- snprintf(buf, sizeof(buf),
- _(" %d of %d tests failed. "),
- fail_count,
- success_count + fail_count);
- else
- /* fail_count>0 && fail_ignore_count>0 */
- snprintf(buf, sizeof(buf),
- _(" %d of %d tests failed, %d of these failures ignored. "),
- fail_count + fail_ignore_count,
- success_count + fail_count + fail_ignore_count,
- fail_ignore_count);
-
- putchar('\n');
- for (i = strlen(buf); i > 0; i--)
- putchar('=');
- printf("\n%s\n", buf);
- for (i = strlen(buf); i > 0; i--)
- putchar('=');
- putchar('\n');
- putchar('\n');
-
- if (file_size(difffilename) > 0)
- {
- printf(_("The differences that caused some tests to fail can be viewed in the\n"
- "file \"%s\". A copy of the test summary that you see\n"
- "above is saved in the file \"%s\".\n\n"),
- difffilename, logfilename);
- }
- else
+ if (file_size(difffilename) <= 0)
{
unlink(difffilename);
unlink(logfilename);
+
+ free(difffilename);
+ difffilename = NULL;
+ free(logfilename);
+ logfilename = NULL;
}
+ footer(difffilename, logfilename);
+ status_end();
+
if (fail_count != 0)
exit(1);
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 12:55 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-13 12:55 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 9:03 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-12 13:42:56 -0700, Andres Freund wrote:
>> On 2021-10-12 16:02:14 -0400, Andrew Dunstan wrote:
>>> You do that by putting a path to it at the start of the PATH. The wrinkle in
>>> this is that you need prove to point to one that understands virtual
>>> paths. So you do something like this:
>>>
>>>
>>> PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...
>> Oh my.
>>
>> I'll try that later... I wonder if we could make this easier from our side?
>> This is a lot of magic to know.
> I managed to get this working. At first it failed because I don't have
> pexports - it's not available inside msys as far as I could tell. And seems to
> be unmaintained. But replacing pexports with gendef fixed that.
>
> There's this comment in src/pl/plperl/GNUmakefile
>
> # Perl on win32 ships with import libraries only for Microsoft Visual C++,
> # which are not compatible with mingw gcc. Therefore we need to build a
> # new import library to link with.
>
> but I seem to be able to link fine without going through that song-and-dance?
>
It looks like you're not building a native postgres, but rather one
targeted at msys. To build one that's native (i.e. runs without any
presence of msys) you need to do these things before building:
MSYSTEM=MINGW64
MSYSTEM_CHOST=x86_64-w64-mingw32
PATH="/mingw64/bin:$PATH"
pexports will be in the resulting path, and the build will use the
native compiler.
You can use fairywren's config as a guide.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 15:51 John Naylor <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: John Naylor @ 2021-10-13 15:51 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Tue, Oct 12, 2021 at 4:59 PM Andres Freund <[email protected]> wrote:
> On 2021-10-12 15:55:22 -0400, John Naylor wrote:
> > (I couldn't get a build working but I'll leave that aside for now).
>
> If you want to do that separately, I'll try to fix it.
Okay, I pulled the latest commits and tried again:
[51/950] Compiling C object
src/interfaces/libpq/libpq.5.dylib.p/fe-connect.c.o
FAILED: src/interfaces/libpq/libpq.5.dylib.p/fe-connect.c.o
ccache cc -Isrc/interfaces/libpq/libpq.5.dylib.p -Isrc/interfaces/libpq
-I../src/interfaces/libpq -Isrc/port -I../src/port -Isrc/include
-I../src/include
-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers
-I/usr/local/opt/readline/include -I/usr/local/opt/gettext/include
-I/usr/local/opt/zlib/include -I/usr/local/opt/openssl/include
-fcolor-diagnostics -Wall -Winvalid-pch -Wextra -O0 -g -isysroot
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fno-strict-aliasing
-fwrapv -Wmissing-prototypes -Wpointer-arith -Werror=vla -Wendif-labels
-Wmissing-format-attribute -Wformat-security -Wdeclaration-after-statement
-Wno-unused-command-line-argument -Wno-missing-field-initializers
-Wno-sign-compare -Wno-unused-parameter -msse4.2
-F/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework
-DFRONTEND -MD -MQ src/interfaces/libpq/libpq.5.dylib.p/fe-connect.c.o -MF
src/interfaces/libpq/libpq.5.dylib.p/fe-connect.c.o.d -o
src/interfaces/libpq/libpq.5.dylib.p/fe-connect.c.o -c
../src/interfaces/libpq/fe-connect.c
In file included from ../src/interfaces/libpq/fe-connect.c:72:
In file included from
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers/ldap.h:1:
[the last line is repeated a bunch of times, then...]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers/ldap.h:1:10:
error: #include nested too deeply
#include <ldap.h>
^
Then the expected "undeclared identifier" errors that would arise from a
missing header. I tried compiling --with-ldap with the Make build, and only
got warnings about deprecated declarations -- that build completed.
I tried disabling ldap with the Meson build but I'll spare the details of
what went wrong there in case I did something wrong, so we can take things
one step at a time.
> > That would also have to work for all the dependencies, which were
displayed
> > to me as:
> >
> > ninja, gdbm, ca-certificates, [email protected], readline, sqlite and
[email protected]
>
> meson does depend on ninja (to execute the build) and of course python.
But
> the rest should be optional dependencies. ninja builds without any
> dependencies as long as you don't change its parser sources. python
builds on
> aix, hpux etc.
>
> Not sure what way gdbm [email protected] and sqlite are pulled in? I assume
readline
> is for python...
Hmm, weird.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 16:37 Andres Freund <[email protected]>
parent: John Naylor <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-13 16:37 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 11:51:03 -0400, John Naylor wrote:
> On Tue, Oct 12, 2021 at 4:59 PM Andres Freund <[email protected]> wrote:
>
> > On 2021-10-12 15:55:22 -0400, John Naylor wrote:
> > > (I couldn't get a build working but I'll leave that aside for now).
> >
> > If you want to do that separately, I'll try to fix it.
>
> Okay, I pulled the latest commits and tried again:
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers/ldap.h:1:
>
> [the last line is repeated a bunch of times, then...]
Oh. I actually saw that on CI at some point... That one is definitely
odd. Currently CI for OSX builds like
- brew install make coreutils ccache icu4c lz4 tcl-tk openldap
- brew install meson ninja [email protected]
..
PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig:$PKG_CONFIG_PATH"
PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH"
PKG_CONFIG_PATH="/usr/local/opt/openldap/lib/pkgconfig:$PKG_CONFIG_PATH"
export PKG_CONFIG_PATH
meson setup --buildtype debug -Dcassert=true -Dssl=openssl build
but I set that up knowing little about macos.
For the autoconf build CI currently does something similar via
LIBS="/usr/local/lib:$LIBS"
INCLUDES="/usr/local/include:$INCLUDES"
...
LIBS="/usr/local/opt/openldap/lib:$LIBS"
INCLUDES="/usr/local/opt/openldap/include:$INCLUDES"
...
--with-includes="$INCLUDES" \
--with-libs="$LIBS" \
are you doing something like that? Or does it work for you without? I vaguely
recall hitting a similar problem as you report when not passing
/usr/local/... to configure.
> i tried disabling ldap with the meson build but i'll spare the details of
> what went wrong there in case i did something wrong, so we can take things
> one step at a time.
you can change it for an existing builddir with
meson configure -dldap=disabled or when setting up a new builddir by passing
-dldap=disabled at that time.
> > > ninja, gdbm, ca-certificates, [email protected], readline, sqlite and
> [email protected]
> >
> > meson does depend on ninja (to execute the build) and of course python.
> but
> > the rest should be optional dependencies. ninja builds without any
> > dependencies as long as you don't change its parser sources. python
> builds on
> > aix, hpux etc.
> >
> > not sure what way gdbm [email protected] and sqlite are pulled in? i assume
> readline
> > is for python...
>
> Hmm, weird.
They're homebrew python deps: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/[email protected]#L28
which are optional things enabled explicitly:
https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/[email protected]#L123
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 17:19 John Naylor <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: John Naylor @ 2021-10-13 17:19 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Wed, Oct 13, 2021 at 12:37 PM Andres Freund <[email protected]> wrote:
> For the autoconf build CI currently does something similar via
> LIBS="/usr/local/lib:$LIBS"
> INCLUDES="/usr/local/include:$INCLUDES"
> ...
> LIBS="/usr/local/opt/openldap/lib:$LIBS"
> INCLUDES="/usr/local/opt/openldap/include:$INCLUDES"
> ...
> --with-includes="$INCLUDES" \
> --with-libs="$LIBS" \
>
> are you doing something like that? Or does it work for you without? I
vaguely
> recall hitting a similar problem as you report when not passing
> /usr/local/... to configure.
I didn't do anything like that for the autoconf build. I have in the past
done things retail, like
--with-icu ICU_CFLAGS='-I/usr/local/opt/icu4c/include/'
ICU_LIBS='-L/usr/local/opt/icu4c/lib/ -licui18n -licuuc -licudata'
> > i tried disabling ldap with the meson build but i'll spare the details
of
> > what went wrong there in case i did something wrong, so we can take
things
> > one step at a time.
>
> you can change it for an existing builddir with
> meson configure -dldap=disabled or when setting up a new builddir by
passing
> -dldap=disabled at that time.
Somehow our emails got lower-cased down here, but I tried it with capital D:
meson configure -Dldap=disabled
inside the build dir and got this:
../meson.build:278:2: ERROR: Tried to assign the invalid value "None" of
type NoneType to variable.
Line 278 is
ldap_r = ldap = dependency('', required : false)
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 17:26 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-13 17:26 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 08:55:38 -0400, Andrew Dunstan wrote:
> On 10/12/21 9:03 PM, Andres Freund wrote:
> > I managed to get this working. At first it failed because I don't have
> > pexports - it's not available inside msys as far as I could tell. And seems to
> > be unmaintained. But replacing pexports with gendef fixed that.
> >
> > There's this comment in src/pl/plperl/GNUmakefile
> >
> > # Perl on win32 ships with import libraries only for Microsoft Visual C++,
> > # which are not compatible with mingw gcc. Therefore we need to build a
> > # new import library to link with.
> >
> > but I seem to be able to link fine without going through that song-and-dance?
> >
>
>
> It looks like you're not building a native postgres, but rather one
> targeted at msys. To build one that's native (i.e. runs without any
> presence of msys) you need to do these things before building:
>
> MSYSTEM=MINGW64
> MSYSTEM_CHOST=x86_64-w64-mingw32
> PATH="/mingw64/bin:$PATH"
I had a config equivalent to this (slight difference in PATH, but the same gcc
being picked), and I just verified that it still works if I set up PATH like
that. I get a working plperl out of it. Without msys on PATH or such.
where perl526.dll
C:\perl\strawberry-5.26.3.1-64bit\perl\bin\perl526.dll
dumpbin /imports 'C:/Users/anfreund/src/pg-meson/build-mingw/tmp_install/lib/plperl.dll'|grep dll
Dump of file C:\Users\anfreund\src\pg-meson\build-mingw\tmp_install\lib\plperl.dll
KERNEL32.dll
msvcrt.dll
perl526.dll
dumpbin /imports .\build-mingw\tmp_install\bin\postgres.exe|grep dll
ADVAPI32.dll
KERNEL32.dll
msvcrt.dll
Secur32.dll
WLDAP32.dll
WS2_32.dll
do $$elog(NOTICE, "blob");$$ language plperl;
NOTICE: blob
DO
To me this looks like it's a plperl built without the import file recreation,
without being msys dependent?
> pexports will be in the resulting path, and the build will use the
> native compiler.
I don't see pexports anywhere in the msys installation. I can see it available
on sourceforge, and I see a few others asking where to get it from in the
context of msys, and being pointed to manually downloading it.
Seems like we should consider using gendef instead of pexports, given it's
available in msys?
$ pacman -Fy
$ pacman -F gendef.exe
...
mingw64/mingw-w64-x86_64-tools-git 9.0.0.6316.acdc7adc9-1 (mingw-w64-x86_64-toolchain) [installed]
mingw64/bin/gendef.exe
..
$ pacman -F pexports.exe
$ pacman -Fx pexports
<bunch of packages containing smtpexports.h>
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 17:42 Andres Freund <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-13 17:42 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 13:19:36 -0400, John Naylor wrote:
> On Wed, Oct 13, 2021 at 12:37 PM Andres Freund <[email protected]> wrote:
>
> > For the autoconf build CI currently does something similar via
> > LIBS="/usr/local/lib:$LIBS"
> > INCLUDES="/usr/local/include:$INCLUDES"
> > ...
> > LIBS="/usr/local/opt/openldap/lib:$LIBS"
> > INCLUDES="/usr/local/opt/openldap/include:$INCLUDES"
> > ...
> > --with-includes="$INCLUDES" \
> > --with-libs="$LIBS" \
> >
> > are you doing something like that? Or does it work for you without? I
> vaguely
> > recall hitting a similar problem as you report when not passing
> > /usr/local/... to configure.
>
> I didn't do anything like that for the autoconf build. I have in the past
> done things retail, like
I'll try to see how this works / what causes the breakage.
> Somehow our emails got lower-cased down here, but I tried it with capital D:
:)
> meson configure -Dldap=disabled
>
> inside the build dir and got this:
>
> ../meson.build:278:2: ERROR: Tried to assign the invalid value "None" of
> type NoneType to variable.
>
> Line 278 is
>
> ldap_r = ldap = dependency('', required : false)
Oops, I broke that when trying to clean things up. I guess I write too much C
;). It needs to be two lines.
I pushed the fix for that.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 18:40 John Naylor <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: John Naylor @ 2021-10-13 18:40 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Wed, Oct 13, 2021 at 1:42 PM Andres Freund <[email protected]> wrote:
> I pushed the fix for that.
Ok great, it builds now! :-) Now something's off with dynamic loading.
There are libraries in ./tmp_install/usr/local/lib/ but apparently initdb
doesn't know to look for them there:
$ cat /Users/john/pgdev/meson/build/testrun/main/pg_regress/log/initdb.log
dyld: Library not loaded: /usr/local/lib/libpq.5.dylib
Referenced from:
/Users/john/pgdev/meson/build/tmp_install/usr/local/bin/initdb
Reason: image not found
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 20:06 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-13 20:06 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/13/21 1:26 PM, Andres Freund wrote:
>
>> pexports will be in the resulting path, and the build will use the
>> native compiler.
> I don't see pexports anywhere in the msys installation. I can see it available
> on sourceforge, and I see a few others asking where to get it from in the
> context of msys, and being pointed to manually downloading it.
Weird. fairywren has it, which means that it must have been removed from
the packages at some stage, fairly recently as fairywren isn't that old.
I just confirmed the absence on a 100% fresh install.
It is in Strawberry's c/bin directory.
>
> Seems like we should consider using gendef instead of pexports, given it's
> available in msys?
Yeah. It's missing on my ancient msys animal (frogmouth), but it doesn't
build --with-perl.
jacana seems to have it.
If you prep a patch I'll test it.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 21:46 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-13 21:46 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 16:06:32 -0400, Andrew Dunstan wrote:
> If you prep a patch I'll test it.
Well, right now I'm wondering if the better fix is to just remove the whole
win32 block. I don't know how far back, but afaict it's not needed. Seems to
have been needed for narwhal at some point, according to 02b61dd08f99. But
narwhal is long dead.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 21:58 Josef Šimánek <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Josef Šimánek @ 2021-10-13 21:58 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
st 13. 10. 2021 v 1:54 odesílatel Andres Freund <[email protected]> napsal:
>
> Hi,
>
> On 2021-10-13 01:19:27 +0200, Josef Šimánek wrote:
> > I tried to clean and start from scratch, but I'm getting different
> > error probably related to wrongly configured JIT (LLVM wasn't found
> > during meson setup). I'll debug on my side to provide more info.
>
> ../src/backend/jit/jit.c:91:73: error: ‘DLSUFFIX’ undeclared (first use in this function)
> 91 | snprintf(path, MAXPGPATH, "%s/%s%s", pkglib_path, jit_provider, DLSUFFIX);
> | ^~~~~~~~
>
> This *very* likely is related to building in a source tree that also contains
> a "non-meson" build "in place". The problem is that the meson build picks up
> the pg_config.h generated by ./configure in the "normal" build, rather than
> the one meson generated itself.
>
> You'd need to execute make distclean or such, or use a separate git checkout.
>
> I forgot about this issue because I only ever build postgres from outside the
> source-tree (by invoking configure from a separate directory), so there's
> never build products in it. I think at least I need to make the build emit a
> warning / error if there's a pg_config.h in the source tree...
Hello, thanks for the hint. I can finally build using meson and run
regress tests.
The only problem I do have currently is auto-detection of perl. I'm
getting error related to missing "Opcode.pm". PERL is autodetected and
enabled (https://pastebin.com/xfRRrDcU).
I do get the same error when I enforce perl for current master build
(./configure --with-perl). Using ./configure with perl autodetection
skips plperl extension on my system.
Disabling perl manually for meson build (meson setup build
--reconfigure --buildtype debug -Dperl=disabled) works for me.
>
> This is the part of the jit code that's built regardless of llvm availability
> - you'd get the same error in a few other places unrelated to jit.
>
> Greetings,
>
> Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-13 23:11 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-13 23:11 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/13/21 5:46 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-13 16:06:32 -0400, Andrew Dunstan wrote:
>> If you prep a patch I'll test it.
> Well, right now I'm wondering if the better fix is to just remove the whole
> win32 block. I don't know how far back, but afaict it's not needed. Seems to
> have been needed for narwhal at some point, according to 02b61dd08f99. But
> narwhal is long dead.
>
Ok, I'll test it out.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 04:27 Thomas Munro <[email protected]>
parent: John Naylor <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Thomas Munro @ 2021-10-14 04:27 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
On Thu, Oct 14, 2021 at 4:51 AM John Naylor
<[email protected]> wrote:
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers/ldap.h:1:10: error: #include nested too deeply
> #include <ldap.h>
> ^
I vaguely recall that PostgreSQL should build OK against Apple's copy
of OpenLDAP. That recursive include loop is coming from a "framework"
header that contains just a couple of lines like #include <ldap.h> to
try to include the real header, which should also be in the include
path, somewhere like
/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include/ldap.h.
I think we'd need to figure out where that
-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers
directive is coming from and get rid of it, so we can include the real
header directly.
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 13:14 Dagfinn Ilmari Mannsåker <[email protected]>
parent: Josef Šimánek <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Dagfinn Ilmari Mannsåker @ 2021-10-14 13:14 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
Josef Šimánek <[email protected]> writes:
> The only problem I do have currently is auto-detection of perl. I'm
> getting error related to missing "Opcode.pm". PERL is autodetected and
> enabled (https://pastebin.com/xfRRrDcU).
Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
core module, and should be in /usr/lib64/perl5, judging by the paths in
the error message.
Which OS is this? Some Linux distributions have separate packages for
the interpreter itself and the included modules, and the packages can be
named confusingly. E.g. on older Redhat/Fedora versions you have to
install the 'perl-core' package to get all the modules, 'perl' is just
the interpreter and the bare minimum set of strictily necessary modules.
They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
minimal one.
- ilmari
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 13:19 Josef Šimánek <[email protected]>
parent: Dagfinn Ilmari Mannsåker <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Josef Šimánek @ 2021-10-14 13:19 UTC (permalink / raw)
To: Dagfinn Ilmari Mannsåker <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
čt 14. 10. 2021 v 15:14 odesílatel Dagfinn Ilmari Mannsåker
<[email protected]> napsal:
>
> Josef Šimánek <[email protected]> writes:
>
> > The only problem I do have currently is auto-detection of perl. I'm
> > getting error related to missing "Opcode.pm". PERL is autodetected and
> > enabled (https://pastebin.com/xfRRrDcU).
>
> Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
> core module, and should be in /usr/lib64/perl5, judging by the paths in
> the error message.
>
> Which OS is this? Some Linux distributions have separate packages for
> the interpreter itself and the included modules, and the packages can be
> named confusingly. E.g. on older Redhat/Fedora versions you have to
> install the 'perl-core' package to get all the modules, 'perl' is just
> the interpreter and the bare minimum set of strictily necessary modules.
>
> They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
> that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
> minimal one.
I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
package. Anyway it behaves differently with autoconf tools and the
meson build system. Is perl disabled by default in the current build
system?
>
> - ilmari
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 13:29 Alvaro Herrera <[email protected]>
parent: Josef Šimánek <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Alvaro Herrera @ 2021-10-14 13:29 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: Dagfinn Ilmari Mannsåker <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On 2021-Oct-14, Josef Šimánek wrote:
> I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> package. Anyway it behaves differently with autoconf tools and the
> meson build system. Is perl disabled by default in the current build
> system?
Yes, you have to use --with-perl in order to get it.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
"Puedes vivir sólo una vez, pero si lo haces bien, una vez es suficiente"
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 13:32 Dagfinn Ilmari Mannsåker <[email protected]>
parent: Josef Šimánek <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Dagfinn Ilmari Mannsåker @ 2021-10-14 13:32 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
Josef Šimánek <[email protected]> writes:
> čt 14. 10. 2021 v 15:14 odesílatel Dagfinn Ilmari Mannsåker
> <[email protected]> napsal:
>>
>> Josef Šimánek <[email protected]> writes:
>>
>> > The only problem I do have currently is auto-detection of perl. I'm
>> > getting error related to missing "Opcode.pm". PERL is autodetected and
>> > enabled (https://pastebin.com/xfRRrDcU).
>>
>> Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
>> core module, and should be in /usr/lib64/perl5, judging by the paths in
>> the error message.
>>
>> Which OS is this? Some Linux distributions have separate packages for
>> the interpreter itself and the included modules, and the packages can be
>> named confusingly. E.g. on older Redhat/Fedora versions you have to
>> install the 'perl-core' package to get all the modules, 'perl' is just
>> the interpreter and the bare minimum set of strictily necessary modules.
>>
>> They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
>> that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
>> minimal one.
>
> I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> package.`
Yes, it's a separate package, but the 'perl' package depends on all the
core module packages, so installing that should fix things. You appear
to only have 'perl-interpreter' installed.
> Anyway it behaves differently with autoconf tools and the meson build
> system. Is perl disabled by default in the current build system?
configure doesn't auto-detect any optional features, they have to be
explicitly enabled using --with-foo switches.
- ilmari
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 17:24 Andres Freund <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-14 17:24 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Josef Šimánek <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 10:29:42 -0300, Alvaro Herrera wrote:
> On 2021-Oct-14, Josef Šimánek wrote:
>
> > I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> > package. Anyway it behaves differently with autoconf tools and the
> > meson build system. Is perl disabled by default in the current build
> > system?
Hm, so it seems we should make the test separately verify that perl -M{Opcode,
ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
detection with a useful message?
> Yes, you have to use --with-perl in order to get it.
With the meson prototype I set most optional features to "auto", except for
LLVM, as that increases compile times noticeably.
For configure we didn't/don't want to do much auto-detection, because that
makes life harder for distributors. But meson has one switch controlling all
features set to 'auto' and not explicitly enabled/disabled:
--auto-features {enabled,disabled,auto} Override value of all 'auto' features (default: auto).
so the argument doesn't apply to the same degree there. We could default
auto-features to something else too.
There were two other reasons:
1) I got tired of needing to disable zlib, readline to be able to build on
windows.
2) Exercising all the dependency detection / checking seems important at this
stage
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 17:26 Andres Freund <[email protected]>
parent: Josef Šimánek <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-14 17:26 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 23:58:12 +0200, Josef Šimánek wrote:
> st 13. 10. 2021 v 1:54 odesílatel Andres Freund <[email protected]> napsal:
> > This *very* likely is related to building in a source tree that also contains
> > a "non-meson" build "in place". The problem is that the meson build picks up
> > the pg_config.h generated by ./configure in the "normal" build, rather than
> > the one meson generated itself.
> >
> > You'd need to execute make distclean or such, or use a separate git checkout.
> >
> > I forgot about this issue because I only ever build postgres from outside the
> > source-tree (by invoking configure from a separate directory), so there's
> > never build products in it. I think at least I need to make the build emit a
> > warning / error if there's a pg_config.h in the source tree...
>
> Hello, thanks for the hint. I can finally build using meson and run
> regress tests.
I yesterday pushed code that should detect this case (with an error). Should
now detect the situation both when you first run configure in tree, and then
meson, and the other way round (by the dirty hack of ./configure touch'ing
meson.build at the end for in-tree builds).
> The only problem I do have currently is auto-detection of perl. I'm
> getting error related to missing "Opcode.pm". PERL is autodetected and
> enabled (https://pastebin.com/xfRRrDcU).
>
> I do get the same error when I enforce perl for current master build
> (./configure --with-perl). Using ./configure with perl autodetection
> skips plperl extension on my system.
>
> Disabling perl manually for meson build (meson setup build
> --reconfigure --buildtype debug -Dperl=disabled) works for me.
Yay, thanks for testing!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 19:14 John Naylor <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: John Naylor @ 2021-10-14 19:14 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
I wrote:
> Ok great, it builds now! :-) Now something's off with dynamic loading.
There are libraries in ./tmp_install/usr/local/lib/ but apparently initdb
doesn't know to look for them there:
>
> $ cat /Users/john/pgdev/meson/build/testrun/main/pg_regress/log/initdb.log
> dyld: Library not loaded: /usr/local/lib/libpq.5.dylib
> Referenced from:
/Users/john/pgdev/meson/build/tmp_install/usr/local/bin/initdb
> Reason: image not found
After poking a bit more, this only happens when trying to run the tests. If
I specify a prefix, I can install, init, and start the server just fine, so
that much works.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 19:29 Josef Šimánek <[email protected]>
parent: Dagfinn Ilmari Mannsåker <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Josef Šimánek @ 2021-10-14 19:29 UTC (permalink / raw)
To: Dagfinn Ilmari Mannsåker <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
čt 14. 10. 2021 v 15:32 odesílatel Dagfinn Ilmari Mannsåker
<[email protected]> napsal:
>
> Josef Šimánek <[email protected]> writes:
>
> > čt 14. 10. 2021 v 15:14 odesílatel Dagfinn Ilmari Mannsåker
> > <[email protected]> napsal:
> >>
> >> Josef Šimánek <[email protected]> writes:
> >>
> >> > The only problem I do have currently is auto-detection of perl. I'm
> >> > getting error related to missing "Opcode.pm". PERL is autodetected and
> >> > enabled (https://pastebin.com/xfRRrDcU).
> >>
> >> Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
> >> core module, and should be in /usr/lib64/perl5, judging by the paths in
> >> the error message.
> >>
> >> Which OS is this? Some Linux distributions have separate packages for
> >> the interpreter itself and the included modules, and the packages can be
> >> named confusingly. E.g. on older Redhat/Fedora versions you have to
> >> install the 'perl-core' package to get all the modules, 'perl' is just
> >> the interpreter and the bare minimum set of strictily necessary modules.
> >>
> >> They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
> >> that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
> >> minimal one.
> >
> > I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> > package.`
>
> Yes, it's a separate package, but the 'perl' package depends on all the
> core module packages, so installing that should fix things. You appear
> to only have 'perl-interpreter' installed.
You're right. Installing "perl" or "perl-Opcode" manually fixes this
problem. Currently I only have "perl-interpreter" installed.
> > Anyway it behaves differently with autoconf tools and the meson build
> > system. Is perl disabled by default in the current build system?
>
> configure doesn't auto-detect any optional features, they have to be
> explicitly enabled using --with-foo switches.
>
> - ilmari
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 19:36 Josef Šimánek <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Josef Šimánek @ 2021-10-14 19:36 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; pgsql-hackers
čt 14. 10. 2021 v 19:24 odesílatel Andres Freund <[email protected]> napsal:
>
> Hi,
>
> On 2021-10-14 10:29:42 -0300, Alvaro Herrera wrote:
> > On 2021-Oct-14, Josef Šimánek wrote:
> >
> > > I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> > > package. Anyway it behaves differently with autoconf tools and the
> > > meson build system. Is perl disabled by default in the current build
> > > system?
>
> Hm, so it seems we should make the test separately verify that perl -M{Opcode,
> ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
> detection with a useful message?
I can confirm "perl -MOpcode" fails. ExtUtils::Embed and
ExtUtils::ParseXS are present. Looking at the local system history of
perl-interpreter package, it seems to be installed by default on
Fedora 34. Friendly error message would be welcomed.
>
> > Yes, you have to use --with-perl in order to get it.
>
> With the meson prototype I set most optional features to "auto", except for
> LLVM, as that increases compile times noticeably.
>
> For configure we didn't/don't want to do much auto-detection, because that
> makes life harder for distributors. But meson has one switch controlling all
> features set to 'auto' and not explicitly enabled/disabled:
> --auto-features {enabled,disabled,auto} Override value of all 'auto' features (default: auto).
> so the argument doesn't apply to the same degree there. We could default
> auto-features to something else too.
>
> There were two other reasons:
>
> 1) I got tired of needing to disable zlib, readline to be able to build on
> windows.
> 2) Exercising all the dependency detection / checking seems important at this
> stage
Clear, thanks for the info.
> Greetings,
>
> Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 20:34 Andres Freund <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-14 20:34 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: pgsql-hackers
Hi,
On October 14, 2021 12:14:16 PM PDT, John Naylor <[email protected]> wrote:
>I wrote:
>
>> Ok great, it builds now! :-) Now something's off with dynamic loading.
>There are libraries in ./tmp_install/usr/local/lib/ but apparently initdb
>doesn't know to look for them there:
>>
>> $ cat /Users/john/pgdev/meson/build/testrun/main/pg_regress/log/initdb.log
>> dyld: Library not loaded: /usr/local/lib/libpq.5.dylib
>> Referenced from:
>/Users/john/pgdev/meson/build/tmp_install/usr/local/bin/initdb
>> Reason: image not found
>
>After poking a bit more, this only happens when trying to run the tests. If
>I specify a prefix, I can install, init, and start the server just fine, so
>that much works.
Is this a Mac with SIP enabled? The Mac CI presumably has that disabled, which is why I didn't see this issue there. Probably need to implement whatever Tom figured out to do about that for the current way of running tests.
Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 20:54 John Naylor <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 2 replies; 130+ messages in thread
From: John Naylor @ 2021-10-14 20:54 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Thu, Oct 14, 2021 at 4:34 PM Andres Freund <[email protected]> wrote:
> Is this a Mac with SIP enabled? The Mac CI presumably has that disabled,
which is why I didn't see this issue there. Probably need to implement
whatever Tom figured out to do about that for the current way of running
tests.
System Information says it's disabled. Running "csrutil status" complains
of an unsupported configuration, which doesn't sound good, so I should
probably go fix that independent of anything else. :-/
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 21:16 John Naylor <[email protected]>
parent: John Naylor <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: John Naylor @ 2021-10-14 21:16 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
I wrote:
> > Is this a Mac with SIP enabled? The Mac CI presumably has that
disabled, which is why I didn't see this issue there. Probably need to
implement whatever Tom figured out to do about that for the current way of
running tests.
>
> System Information says it's disabled. Running "csrutil status" complains
of an unsupported configuration, which doesn't sound good, so I should
probably go fix that independent of anything else. :-/
Looking online, I wonder if the "unsupported" message might be overly
cautious. In any case, I do remember turning something off to allow a
debugger to run. Here are all the settings, in case it matters:
Apple Internal: disabled
Kext Signing: enabled
Filesystem Protections: enabled
Debugging Restrictions: disabled
DTrace Restrictions: enabled
NVRAM Protections: enabled
BaseSystem Verification: enabled
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 21:41 Sergey Shinderuk <[email protected]>
parent: John Naylor <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Sergey Shinderuk @ 2021-10-14 21:41 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 14.10.2021 23:54, John Naylor wrote:
> On Thu, Oct 14, 2021 at 4:34 PM Andres Freund <[email protected]
> <mailto:[email protected]>> wrote:
>
> > Is this a Mac with SIP enabled? The Mac CI presumably has that
> disabled, which is why I didn't see this issue there. Probably need to
> implement whatever Tom figured out to do about that for the current way
> of running tests.
>
> System Information says it's disabled. Running "csrutil status"
> complains of an unsupported configuration, which doesn't sound good, so
> I should probably go fix that independent of anything else. :-/
Maybe you could check that DYLD_LIBRARY_PATH is working for you?
% DYLD_FALLBACK_LIBRARY_PATH=
DYLD_LIBRARY_PATH=./tmp_install/usr/local/lib
./tmp_install/usr/local/bin/psql --version
psql (PostgreSQL) 15devel
Without DYLD_LIBRARY_PATH I get the error, as expected:
% DYLD_FALLBACK_LIBRARY_PATH= ./tmp_install/usr/local/bin/psql --version
dyld: Library not loaded: /usr/local/lib/libpq.5.dylib
Referenced from:
/Users/shinderuk/src/postgres-meson/build/./tmp_install/usr/local/bin/psql
Reason: image not found
I add "DYLD_FALLBACK_LIBRARY_PATH=" because otherwise dyld falls back to
/usr/lib/libpq.5.dylib provided by Apple (I am testing on Catalina).
% DYLD_PRINT_LIBRARIES=1 ./tmp_install/usr/local/bin/psql --version 2>&1
| grep libpq
dyld: loaded: <4EDF735E-2104-32AD-BE7B-B400ABFCF57C> /usr/lib/libpq.5.dylib
Regards,
--
Sergey Shinderuk https://postgrespro.com/
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 21:48 Tom Lane <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Tom Lane @ 2021-10-14 21:48 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
John Naylor <[email protected]> writes:
>> System Information says it's disabled. Running "csrutil status" complains
>> of an unsupported configuration, which doesn't sound good, so I should
>> probably go fix that independent of anything else. :-/
> Looking online, I wonder if the "unsupported" message might be overly
> cautious. In any case, I do remember turning something off to allow a
> debugger to run. Here are all the settings, in case it matters:
> Apple Internal: disabled
> Kext Signing: enabled
> Filesystem Protections: enabled
> Debugging Restrictions: disabled
> DTrace Restrictions: enabled
> NVRAM Protections: enabled
> BaseSystem Verification: enabled
I remember having seen that report too, after some previous software
upgrade that had started from a "SIP disabled" status. I'm mostly
guessing here, but my guess is that
(a) csrutil only considers the all-enabled and all-disabled states
of these individual flags to be "supported" cases.
(b) some one or more of these flags came along in a macOS update,
and if you did the update starting from a "disabled" state, you
nonetheless ended up with the new flags enabled, leading to the
mixed state that csrutil complains about.
I've lost count of the number of times I've seen macOS updates
be sloppy about preserving non-default settings, so I don't find
theory (b) to be even slightly surprising.
Whether the mixed state is actually problematic in any way,
I dunno. I don't recall having had any problems before noticing
that that was what I had.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 22:00 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 2 replies; 130+ messages in thread
From: Tom Lane @ 2021-10-14 22:00 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: John Naylor <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> Is this a Mac with SIP enabled? The Mac CI presumably has that disabled, which is why I didn't see this issue there. Probably need to implement whatever Tom figured out to do about that for the current way of running tests.
AFAIR the only cases we've made work are
(1) disable SIP
(2) avoid the need for (1) by always doing "make install" before
"make check".
Peter E. did some hacking towards another solution awhile ago,
but IIRC it involved changing the built binaries, and I think
we concluded that the benefits didn't justify that.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 22:08 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Tom Lane @ 2021-10-14 22:08 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Josef Šimánek <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> Hm, so it seems we should make the test separately verify that perl -M{Opcode,
> ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
> detection with a useful message?
Our existing policy is that we should check this at configure time,
not later. Since plperl won't work at all without Opcode, it seems
appropriate to add a check there if you say --with-perl. I wasn't
aware that Red Hat had unbundled that from the minimal perl
installation :-(.
OTOH, if they've not unbundled ExtUtils::Embed or ExtUtils::ParseXS,
I doubt it's worth the configure cycles to check for those separately.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 22:19 Andrew Dunstan <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Andrew Dunstan @ 2021-10-14 22:19 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/13/21 7:11 PM, Andrew Dunstan wrote:
> On 10/13/21 5:46 PM, Andres Freund wrote:
>> Hi,
>>
>> On 2021-10-13 16:06:32 -0400, Andrew Dunstan wrote:
>>> If you prep a patch I'll test it.
>> Well, right now I'm wondering if the better fix is to just remove the whole
>> win32 block. I don't know how far back, but afaict it's not needed. Seems to
>> have been needed for narwhal at some point, according to 02b61dd08f99. But
>> narwhal is long dead.
>>
> Ok, I'll test it out.
>
confirmed that jacana doesn't need this code to build or test plperl
(all I did was change the test from win32 to win32x). There would still
be work to do to fix the contrib bool_plperl, jsonb_plperl and
hstore_plperl modules.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 22:23 Thomas Munro <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 2 replies; 130+ messages in thread
From: Thomas Munro @ 2021-10-14 22:23 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
On Fri, Oct 15, 2021 at 11:00 AM Tom Lane <[email protected]> wrote:
> Peter E. did some hacking towards another solution awhile ago,
> but IIRC it involved changing the built binaries, and I think
> we concluded that the benefits didn't justify that.
Yeah, by now there are lots of useful blogs from various projects
figuring out that you can use the install_name_tool to adjust the
paths it uses to be absolute or relative to certain magic words, like
@executable_path/../lib/blah.dylib, which is tempting, but...
realistically, for serious hacking on a Mac, SIP is so annoying that
it isn't the only reason you'll want to turn it off: it stops
dtrace/dtruss/... from working, and somehow prevents debuggers from
working when you've ssh'd in from a remote machine with a proper
keyboard, and probably more things that I'm forgetting.
I wish I could find the Xnu source that shows exactly how and when the
environment is suppressed in this way to understand better, but it
doesn't jump out of Apple's github; maybe it's hiding in closed source
machinery...
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 22:40 Tom Lane <[email protected]>
parent: Thomas Munro <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Tom Lane @ 2021-10-14 22:40 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Andres Freund <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Thomas Munro <[email protected]> writes:
> I wish I could find the Xnu source that shows exactly how and when the
> environment is suppressed in this way to understand better, but it
> doesn't jump out of Apple's github; maybe it's hiding in closed source
> machinery...
I recall that we figured out awhile ago that the environment gets trimmed
when make (or whatever) executes some command via the shell; seemingly,
Apple has decided that /bin/sh is a security-critical program that mustn't
be run with a non-default DYLD_LIBRARY_PATH. Dunno if that helps you
find where the damage is done exactly.
(The silliness of this policy, when you pair it with the fact that they
don't reset PATH at the same time, seems blindingly obvious to me. But
apparently not to Apple.)
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 22:55 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-14 22:55 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 18:00:49 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > Is this a Mac with SIP enabled? The Mac CI presumably has that disabled, which is why I didn't see this issue there. Probably need to implement whatever Tom figured out to do about that for the current way of running tests.
>
> AFAIR the only cases we've made work are
>
> (1) disable SIP
>
> (2) avoid the need for (1) by always doing "make install" before
> "make check".
Ah, I thought it was more than that. In that case, John, does meson's test
succeed after you did the "proper" install? Assuming it's in a path that's
allowed to provide shared libraries?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 23:04 Tom Lane <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Tom Lane @ 2021-10-14 23:04 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Andres Freund <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
I wrote:
> I recall that we figured out awhile ago that the environment gets trimmed
> when make (or whatever) executes some command via the shell; seemingly,
> Apple has decided that /bin/sh is a security-critical program that mustn't
> be run with a non-default DYLD_LIBRARY_PATH. Dunno if that helps you
> find where the damage is done exactly.
BTW, here's the evidence for this theory:
[tgl@pro ~]$ cat checkenv.c
#include <stdio.h>
#include <stdlib.h>
int
main(int argc, char **argv)
{
char *pth = getenv("DYLD_LIBRARY_PATH");
if (pth)
printf("DYLD_LIBRARY_PATH = %s\n", pth);
else
printf("DYLD_LIBRARY_PATH is unset\n");
return 0;
}
[tgl@pro ~]$ gcc checkenv.c
[tgl@pro ~]$ ./a.out
DYLD_LIBRARY_PATH is unset
[tgl@pro ~]$ export DYLD_LIBRARY_PATH=/Users/tgl/pginstall/lib
[tgl@pro ~]$ ./a.out
DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
[tgl@pro ~]$ sh -c ./a.out
DYLD_LIBRARY_PATH is unset
[tgl@pro ~]$ ./a.out
DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
[tgl@pro ~]$ bash -c ./a.out
DYLD_LIBRARY_PATH is unset
You have to check the environment using an "unprivileged" program.
If you try to examine the environment using, say, "env", you will get
very misleading results. AFAICT, /usr/bin/env is *also* considered
security-critical, because I cannot get it to ever report that
DYLD_LIBRARY_PATH is set.
Hmm ... /usr/bin/perl seems to act the same way. It can see
ENV{'PATH'} but not ENV{'DYLD_LIBRARY_PATH'}.
This may indicate that they've applied this policy on a blanket
basis to everything in /bin and /usr/bin (and other system
directories, maybe), rather than singling out the shell.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 23:15 Andres Freund <[email protected]>
parent: Thomas Munro <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-14 23:15 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Tom Lane <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-15 11:23:00 +1300, Thomas Munro wrote:
> On Fri, Oct 15, 2021 at 11:00 AM Tom Lane <[email protected]> wrote:
> > Peter E. did some hacking towards another solution awhile ago,
> > but IIRC it involved changing the built binaries, and I think
> > we concluded that the benefits didn't justify that.
>
> Yeah, by now there are lots of useful blogs from various projects
> figuring out that you can use the install_name_tool to adjust the
> paths it uses to be absolute or relative to certain magic words, like
> @executable_path/../lib/blah.dylib, which is tempting, but...
> realistically, for serious hacking on a Mac, SIP is so annoying that
> it isn't the only reason you'll want to turn it off: it stops
> dtrace/dtruss/... from working, and somehow prevents debuggers from
> working when you've ssh'd in from a remote machine with a proper
> keyboard, and probably more things that I'm forgetting.
Meson has support for using install_name_tool to remove "build time" rpaths
and set "install time" rpaths during the installation process - which uses
install_name_tool on mac.
If, and perhaps that's too big an if, relative rpaths actually work despite
SIP, it might be worth setting a relative install_rpath, because afaict that
should then work both for a "real" installation and our temporary test one.
If absolute rpaths are required, it'd make the process a bit more expensive,
because we'd probably need to change a configure time option during the temporary
install. No actual rebuilds would be required, but still.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 23:23 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Tom Lane @ 2021-10-14 23:23 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Thomas Munro <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> If, and perhaps that's too big an if, relative rpaths actually work despite
> SIP, it might be worth setting a relative install_rpath, because afaict that
> should then work both for a "real" installation and our temporary test one.
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 23:27 John Naylor <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: John Naylor @ 2021-10-14 23:27 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Thu, Oct 14, 2021 at 6:55 PM Andres Freund <[email protected]> wrote:
> Ah, I thought it was more than that. In that case, John, does meson's test
> succeed after you did the "proper" install? Assuming it's in a path that's
> allowed to provide shared libraries?
Oh, it can act like installcheck? [checks] Yep, "meson test" ran fine (*).
It still ran the temp install first, but in any case it worked. The full
"configure step" was
meson setup build --buildtype debug -Dldap=disabled -Dcassert=true
-Dprefix=$(pwd)/inst
* (all passed but skipped subscription/t/012_collation.pl)
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 23:38 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-14 23:38 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Josef Šimánek <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 18:08:58 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > Hm, so it seems we should make the test separately verify that perl -M{Opcode,
> > ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
> > detection with a useful message?
>
> Our existing policy is that we should check this at configure time,
> not later.
Yea, I was thinking of configure (and meson's equivalent) as well.
> Since plperl won't work at all without Opcode, it seems
> appropriate to add a check there if you say --with-perl. I wasn't
> aware that Red Hat had unbundled that from the minimal perl
> installation :-(.
>
> OTOH, if they've not unbundled ExtUtils::Embed or ExtUtils::ParseXS,
> I doubt it's worth the configure cycles to check for those separately.
On debian the perl binary, with a sparse set of modules is in
perl-base. ExtUtils::Embed and ExtUtils::ParseXS are in
perl-modules-x.yy. Whereas Opcode is in libperlx.yy. But libperlx.yy depends
on perl-modules-x.yy so I guess an Opcode.pm check would suffice.
Seems we can just check all of them at once with with something like
perl -MOpcode -MExtUtils::Embed -MExtUtils::ParseXSNotAvailable -e ''
Can't locate ExtUtils/ParseXSNotAvailable.pm in @INC (you may need to install the ExtUtils::ParseXS3 module) (@INC contains: /home/andres/bin/perl5/lib/perl5/x86_64-linux-gnu-thread-multi /home/andres/bin/perl5/lib/perl5 /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1 /usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32 /usr/share/perl/5.32 /usr/local/lib/site_perl).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-14 23:51 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Tom Lane @ 2021-10-14 23:51 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Josef Šimánek <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2021-10-14 18:08:58 -0400, Tom Lane wrote:
>> Andres Freund <[email protected]> writes:
>>> Hm, so it seems we should make the test separately verify that perl -M{Opcode,
>>> ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
>>> detection with a useful message?
>> Our existing policy is that we should check this at configure time,
>> not later.
> Yea, I was thinking of configure (and meson's equivalent) as well.
Ah, sorry, I misunderstood what you meant by "test".
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-15 00:02 Andres Freund <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-15 00:02 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 19:27:17 -0400, John Naylor wrote:
> On Thu, Oct 14, 2021 at 6:55 PM Andres Freund <[email protected]> wrote:
> > Ah, I thought it was more than that. In that case, John, does meson's test
> > succeed after you did the "proper" install? Assuming it's in a path that's
> > allowed to provide shared libraries?
>
> Oh, it can act like installcheck? [checks] Yep, "meson test" ran fine (*).
> It still ran the temp install first, but in any case it worked.
As far as I understand Tom, our normal make check only works on OSX if
previously you ran make install. Which will have installed libpq into the
"proper" install location. Because all our binaries will, by default, have an
rpath to the library directory embedded, that then allows binaries in the
temporary install to work. But using the wrong libpq - which most of the time
turns out to be harmless, because libpq doesn't change that rapidly.
> * (all passed but skipped subscription/t/012_collation.pl)
That test requires ICU, so that's fine. I guess we could prevent the test from
being executed in the first place, but I don't think we've done that for cases
where it's one specific test in a t/ directory, where others in the same
directory do not have such dependencies.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-15 00:36 Thomas Munro <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Thomas Munro @ 2021-10-15 00:36 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
On Fri, Oct 15, 2021 at 12:04 PM Tom Lane <[email protected]> wrote:
> [tgl@pro ~]$ cat checkenv.c
> #include <stdio.h>
> #include <stdlib.h>
>
> int
> main(int argc, char **argv)
> {
> char *pth = getenv("DYLD_LIBRARY_PATH");
>
> if (pth)
> printf("DYLD_LIBRARY_PATH = %s\n", pth);
> else
> printf("DYLD_LIBRARY_PATH is unset\n");
>
> return 0;
> }
> [tgl@pro ~]$ gcc checkenv.c
> [tgl@pro ~]$ ./a.out
> DYLD_LIBRARY_PATH is unset
> [tgl@pro ~]$ export DYLD_LIBRARY_PATH=/Users/tgl/pginstall/lib
> [tgl@pro ~]$ ./a.out
> DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
> [tgl@pro ~]$ sh -c ./a.out
> DYLD_LIBRARY_PATH is unset
> [tgl@pro ~]$ ./a.out
> DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
> [tgl@pro ~]$ bash -c ./a.out
> DYLD_LIBRARY_PATH is unset
>
> You have to check the environment using an "unprivileged" program.
> If you try to examine the environment using, say, "env", you will get
> very misleading results. AFAICT, /usr/bin/env is *also* considered
> security-critical, because I cannot get it to ever report that
> DYLD_LIBRARY_PATH is set.
>
> Hmm ... /usr/bin/perl seems to act the same way. It can see
> ENV{'PATH'} but not ENV{'DYLD_LIBRARY_PATH'}.
>
> This may indicate that they've applied this policy on a blanket
> basis to everything in /bin and /usr/bin (and other system
> directories, maybe), rather than singling out the shell.
Looks like it. If I've found the right code here, it looks like where
any common-or-garden Unix runtime linker would ignore LD_LIBRARY_PATH
for a setuid binary, they've trained theirs to whack DYLD_*, and also
for code-signed and __RESTRICT-marked executables.
https://github.com/opensource-apple/dyld/blob/master/src/dyld.cpp#L1681
I suppose you could point SHELL at an unsigned copy of sh (codesign
--remove-signature, or something from brew/ports/x) so that GNU make
should respect, but I don't know how many other exec("/bin/sh") calls
might be hiding around the place (I guess perl calls system()?) and
might require some kind of LD_PRELOAD hackery... not much fun.
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-15 02:46 Tom Lane <[email protected]>
parent: Thomas Munro <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Tom Lane @ 2021-10-15 02:46 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Andres Freund <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Thomas Munro <[email protected]> writes:
> On Fri, Oct 15, 2021 at 12:04 PM Tom Lane <[email protected]> wrote:
>> This may indicate that they've applied this policy on a blanket
>> basis to everything in /bin and /usr/bin (and other system
>> directories, maybe), rather than singling out the shell.
> Looks like it. If I've found the right code here, it looks like where
> any common-or-garden Unix runtime linker would ignore LD_LIBRARY_PATH
> for a setuid binary, they've trained theirs to whack DYLD_*, and also
> for code-signed and __RESTRICT-marked executables.
> https://github.com/opensource-apple/dyld/blob/master/src/dyld.cpp#L1681
Ugh. That explains it, all right.
> I suppose you could point SHELL at an unsigned copy of sh (codesign
> --remove-signature, or something from brew/ports/x) so that GNU make
> should respect, but I don't know how many other exec("/bin/sh") calls
> might be hiding around the place (I guess perl calls system()?) and
> might require some kind of LD_PRELOAD hackery... not much fun.
Yeah. I thought about invoking everything via a small wrapper
that restores the correct setting of DYLD_LIBRARY_PATH. We could
perhaps make that work for the invocations of test postmasters
and psqls from "make" and TAP scripts, but hacking up our code's
sundry uses of system(3) like that seems like it'd be very messy,
if feasible at all.
BTW, the POSIX spec explicitly discourages letting SHELL affect the
behavior of system(3), so I bet that wouldn't help.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-15 03:20 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-15 03:20 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Thomas Munro <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 22:46:07 -0400, Tom Lane wrote:
> Thomas Munro <[email protected]> writes:
> > I suppose you could point SHELL at an unsigned copy of sh (codesign
> > --remove-signature, or something from brew/ports/x) so that GNU make
> > should respect, but I don't know how many other exec("/bin/sh") calls
> > might be hiding around the place (I guess perl calls system()?) and
> > might require some kind of LD_PRELOAD hackery... not much fun.
>
> Yeah. I thought about invoking everything via a small wrapper
> that restores the correct setting of DYLD_LIBRARY_PATH. We could
> perhaps make that work for the invocations of test postmasters
> and psqls from "make" and TAP scripts, but hacking up our code's
> sundry uses of system(3) like that seems like it'd be very messy,
> if feasible at all.
It does sound like using relative rpaths might be the thing we want - and like
they've been available since 10.5 or something.
Is there a reason we're using absolute rpaths on a bunch of platforms, rather
than relative ones, which'd allow relocation?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-15 18:50 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-15 18:50 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: Thomas Munro <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 19:23:58 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > If, and perhaps that's too big an if, relative rpaths actually work despite
> > SIP, it might be worth setting a relative install_rpath, because afaict that
> > should then work both for a "real" installation and our temporary test one.
>
> From what we know so far, it seems like SIP wouldn't interfere with
> that (if it works at all). I think what SIP desires to prevent is
> messing with a program's execution by setting DYLD_LIBRARY_PATH.
> As long as the program executable itself is saying where to find
> the library, I don't see why they should interfere with that.
Well, there's *some* danger with relative rpaths, because they might
accidentally be pointing somewhere non-existing and user-creatable. Not a huge
risk, but as you say:
> (Again, it seems blindingly stupid to forbid this while not blocking
> PATH or any of the other environment variables that have always affected
> execution. But what do I know.)
these aren't necessarily carefuly weighed considerations :/
But it seems to work well from what I gather.
> > If absolute rpaths are required, it'd make the process a bit more expensive,
>
> It'd also put the kibosh on relocatable install trees, though I dunno how
> much people really care about that.
We currently use absolute rpaths, or something equivalent.
The reason that running tests on macos works is that we set the "install_name"
of shared libraries to the intended installed location, using an absolute
path:
LINK.shared = $(COMPILER) -dynamiclib -install_name '$(libdir)/lib$(NAME).$(SO_MAJOR_VERSION)$(DLSUFFIX)' $(version_link) $(exported_symbols_list) -multiply_defined suppress
which on macos means that all libraries linking to that dylib reference it
under that absolute path.
On most other platforms we set an absolute rpath to the installation
directory, which has an equivalent effect:
rpathdir = $(libdir)
It seems to work quite well to change our own references to libpq in binaries
/ shared libs to be relative, but to leave the install_name of the libraries
intact. In combination with adding an rpath of @loader_path/../lib/ to
binaries and @loader_path/ to shlibs, the install will re relocatable.
It doesn't work as well to actually have a non-absolute install_name for
libraries (e.g. @rpath/libpq.dylib), because then external code linking to
libpq needs to add an rpath to the installation to make it work.
The advantage of this approach over Peter's is that it's not temp-install
specific - due to the relative paths, it makes installations relocatable
without relying [DY]LD_LIBRARY_PATH.
On other unixoid systems this whole mess is simpler, because we can just add
$ORIGIN to shared libraries and $ORIGIN/../lib/ to binaries. We don't need to
leave some absolute path in the libraries themself intact.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-15 22:36 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-15 22:36 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: Thomas Munro <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-15 11:50:30 -0700, Andres Freund wrote:
> It seems to work quite well to change our own references to libpq in binaries
> / shared libs to be relative, but to leave the install_name of the libraries
> intact. In combination with adding an rpath of @loader_path/../lib/ to
> binaries and @loader_path/ to shlibs, the install will re relocatable.
>
> It doesn't work as well to actually have a non-absolute install_name for
> libraries (e.g. @rpath/libpq.dylib), because then external code linking to
> libpq needs to add an rpath to the installation to make it work.
>
> The advantage of this approach over Peter's is that it's not temp-install
> specific - due to the relative paths, it makes installations relocatable
> without relying [DY]LD_LIBRARY_PATH.
>
> On other unixoid systems this whole mess is simpler, because we can just add
> $ORIGIN to shared libraries and $ORIGIN/../lib/ to binaries. We don't need to
> leave some absolute path in the libraries themself intact.
I implemented this for the meson build, and it seems to work nicely. The macos
part was harder than I hoped due to the install_name stuff, which meson
doesn't solve.
https://github.com/anarazel/postgres/commit/a35379c28989469cc4b701a8d7a22422e6302e09
After that the build directory is relocatale.
I don't immediately see a way to do this reasonably for the autoconf
build. We'd need a list of our own shared libraries from somewhere, and then
replace the references after building the objects?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-15 22:47 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-15 22:47 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: Thomas Munro <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-15 15:36:16 -0700, Andres Freund wrote:
> On 2021-10-15 11:50:30 -0700, Andres Freund wrote:
> > It seems to work quite well to change our own references to libpq in binaries
> > / shared libs to be relative, but to leave the install_name of the libraries
> > intact. In combination with adding an rpath of @loader_path/../lib/ to
> > binaries and @loader_path/ to shlibs, the install will re relocatable.
> >
> > It doesn't work as well to actually have a non-absolute install_name for
> > libraries (e.g. @rpath/libpq.dylib), because then external code linking to
> > libpq needs to add an rpath to the installation to make it work.
> >
> > The advantage of this approach over Peter's is that it's not temp-install
> > specific - due to the relative paths, it makes installations relocatable
> > without relying [DY]LD_LIBRARY_PATH.
> >
> > On other unixoid systems this whole mess is simpler, because we can just add
> > $ORIGIN to shared libraries and $ORIGIN/../lib/ to binaries. We don't need to
> > leave some absolute path in the libraries themself intact.
>
> I implemented this for the meson build, and it seems to work nicely. The macos
> part was harder than I hoped due to the install_name stuff, which meson
> doesn't solve.
>
> https://github.com/anarazel/postgres/commit/a35379c28989469cc4b701a8d7a22422e6302e09
>
> After that the build directory is relocatale.
Well, now that I think about it, it's still only relocatable in the sense that
postgres itself will continue to work. Outside code linking to e.g. libpq will
get the wrong path after relocation the source tree, due to the absolute
install_name.
But that doesn't seem solvable, unless we make the installed install_name to
be '@rpath/libpq...dylib' and require code linking to libpq to pass
-Wl,-rpath,/path/to/libpq when linking to libpq.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-19 18:34 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
10 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-19 18:34 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi Tom,
On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
> As far as I can tell the only OS that postgres currently supports that
> meson doesn't support is HPUX. It'd likely be fairly easy to add
> gcc-on-hpux support, a chunk more to add support for the proprietary
> ones.
Tom, wrt HPUX on pa-risc, what are your thoughts there? IIRC we gave up
supporting HP's compiler on pa-risc a while ago.
As I said it'd probably not be too hard to add meson support for hpux on hppa,
it's probably just a few branches. But that'd require access somewhere. The
gcc compile farm does not have a hppa member anymore...
I did notice that gcc will declare hppa-hpux obsolete in gcc 12 and will
remove at some point:
"The hppa[12]*-*-hpux10* and hppa[12]*-*-hpux11* configurations targeting 32-bit PA-RISC with HP-UX have been obsoleted and will be removed in a future release."
https://gcc.gnu.org/gcc-12/changes.html
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-19 19:22 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Tom Lane @ 2021-10-19 19:22 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
>> As far as I can tell the only OS that postgres currently supports that
>> meson doesn't support is HPUX. It'd likely be fairly easy to add
>> gcc-on-hpux support, a chunk more to add support for the proprietary
>> ones.
> Tom, wrt HPUX on pa-risc, what are your thoughts there? IIRC we gave up
> supporting HP's compiler on pa-risc a while ago.
Right. I am still testing with gcc on HP-PA. I'd kind of like to
keep it running just as an edge case for our spinlock support, but
I'm not sure that I want to do any huge amount of work on meson
to keep that going.
I do have a functioning OpenBSD installation on that machine, so
one alternative if the porting costs look too high is to replace
gaur with an OpenBSD animal. However, last I checked, OpenBSD
was about half the speed of HPUX on that hardware, so I'm not
real eager to go that way. gaur's already about the slowest
animal in the farm :-(
> As I said it'd probably not be too hard to add meson support for hpux on hppa,
> it's probably just a few branches. But that'd require access somewhere. The
> gcc compile farm does not have a hppa member anymore...
If you've got an idea where to look, I could add that to my
to-do queue.
In any case, I don't think we need to consider HPUX as a blocker
for the meson approach. The value-add from keeping gaur going
probably isn't terribly much. I'm more concerned about the
effort involved in getting meson going on some other old animals,
such as prairiedog.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-19 21:57 John Naylor <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: John Naylor @ 2021-10-19 21:57 UTC (permalink / raw)
To: Andres Freund <[email protected]>; Thomas Munro <[email protected]>; +Cc: pgsql-hackers
Hi,
I know this is still in the evaluation stage, but I did notice some
discrepencies in the Flex flags. With the attached patch, the read-only
data segment seems to match up pretty well now.
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] sync-flex-flags-with-autoconf-build.patch (5.1K, ../../CAFBsxsF3qasUjUrpsV5Ld1Q41A1YERJeUaUoc4zu0aGBfWOUrA@mail.gmail.com/3-sync-flex-flags-with-autoconf-build.patch)
download | inline diff:
diff --git a/contrib/cube/meson.build b/contrib/cube/meson.build
index 49276aed64..3cf7ebdd8e 100644
--- a/contrib/cube/meson.build
+++ b/contrib/cube/meson.build
@@ -6,7 +6,7 @@ cube_sources = files(
cubescan = custom_target('cubescan',
input: ['cubescan.l'],
output: ['cubescan.c'],
- command: [flex, '-CFe', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
+ command: [flex, '-o', '@OUTPUT@', '@INPUT@'])
cube_sources += custom_target('cubeparse',
input: 'cubeparse.y',
diff --git a/src/backend/parser/meson.build b/src/backend/parser/meson.build
index 491eacf20b..5ce4d09f31 100644
--- a/src/backend/parser/meson.build
+++ b/src/backend/parser/meson.build
@@ -28,7 +28,7 @@ parser_sources = [files('parser.c')]
backend_scanner = custom_target('scan',
input: ['scan.l'],
output: ['scan.c'],
- command: [flex, '-CF', '-p', '-p', '-o', '@OUTPUT@', '@INPUT0@'])
+ command: [flex, '-b', '-CF', '-p', '-p', '-o', '@OUTPUT@', '@INPUT0@'])
parser_sources += backend_scanner[0]
parser_sources += backend_parser_header[0]
diff --git a/src/backend/replication/meson.build b/src/backend/replication/meson.build
index 2573f166d7..ee12c6d49d 100644
--- a/src/backend/replication/meson.build
+++ b/src/backend/replication/meson.build
@@ -17,7 +17,7 @@ backend_sources += files(
repl_scanner = custom_target('repl_scanner',
input : files('repl_scanner.l'),
output : ['repl_scanner.c'],
- command : [flex, '-CF', '-p', '-p', '-o', '@OUTPUT0@', '@INPUT@']
+ command: [flex, '-o', '@OUTPUT@', '@INPUT@']
)
generated_backend_sources += custom_target('repl_gram',
@@ -30,7 +30,7 @@ generated_backend_sources += custom_target('repl_gram',
syncrep_scanner = custom_target('syncrep_scanner',
input: 'syncrep_scanner.l',
output: 'syncrep_scanner.c',
- command: [flex, '-CF', '-p', '-p', '-o', '@OUTPUT0@', '@INPUT@'])
+ command: [flex, '-o', '@OUTPUT@', '@INPUT@'])
generated_backend_sources += custom_target('syncrep_gram',
input: 'syncrep_gram.y',
diff --git a/src/backend/utils/adt/meson.build b/src/backend/utils/adt/meson.build
index 086fde8ff0..e1cea1eb4e 100644
--- a/src/backend/utils/adt/meson.build
+++ b/src/backend/utils/adt/meson.build
@@ -109,7 +109,7 @@ backend_sources += files(
jsonpath_scan = custom_target('jsonpath_scan',
input: ['jsonpath_scan.l'],
output: ['jsonpath_scan.c'],
- command: [flex, '-CF', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
+ command: [flex, '-b', '-CF', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
# jsonpath_scan is compiled as part of jsonpath_gram
generated_backend_sources += custom_target('jsonpath_parse',
diff --git a/src/backend/utils/misc/meson.build b/src/backend/utils/misc/meson.build
index 5274c8aa1a..2c0090ad33 100644
--- a/src/backend/utils/misc/meson.build
+++ b/src/backend/utils/misc/meson.build
@@ -18,7 +18,7 @@ backend_sources += files(
guc_scan = custom_target('guc_scan',
input: ['guc-file.l'],
output: ['guc-file.c.h'],
- command: [flex, '-CF', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
+ command: [flex, '-o', '@OUTPUT@', '@INPUT@'])
generated_backend_sources += guc_scan
diff --git a/src/bin/pgbench/meson.build b/src/bin/pgbench/meson.build
index 5c4a778ff3..bc135abebf 100644
--- a/src/bin/pgbench/meson.build
+++ b/src/bin/pgbench/meson.build
@@ -9,7 +9,7 @@ pgbench_sources = files(
exprscan = custom_target('exprscan',
input : files('exprscan.l'),
output : ['exprscan.c'],
- command : [flex, '-CF', '-p', '-p', '-o', '@OUTPUT0@', '@INPUT@']
+ command : [flex, '-o', '@OUTPUT0@', '@INPUT@']
)
exprparse = custom_target('exprparse',
diff --git a/src/bin/psql/meson.build b/src/bin/psql/meson.build
index 98921f801d..e56beb28e1 100644
--- a/src/bin/psql/meson.build
+++ b/src/bin/psql/meson.build
@@ -18,7 +18,7 @@ psql_sources = files(
psql_sources += custom_target('psqlscanslash',
input: ['psqlscanslash.l'],
output: ['psqlscanslash.c'],
- command: [flex, '-CFe', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
+ command: [flex, '-b', '-Cfe', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
psql_sources += custom_target('psql_help',
input: ['create_help.pl'],
diff --git a/src/fe_utils/meson.build b/src/fe_utils/meson.build
index b305727d96..e3f0b34cf1 100644
--- a/src/fe_utils/meson.build
+++ b/src/fe_utils/meson.build
@@ -16,7 +16,7 @@ fe_utils_sources = files(
fe_utils_sources += custom_target('psqlscan',
input: ['psqlscan.l'],
output: ['psqlscan.c'],
- command: [flex, '-Cfe', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
+ command: [flex, '-b', '-Cfe', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
fe_utils = static_library('fe_utils',
fe_utils_sources + generated_headers,
diff --git a/src/test/isolation/meson.build b/src/test/isolation/meson.build
index 637b480755..ea8baa2063 100644
--- a/src/test/isolation/meson.build
+++ b/src/test/isolation/meson.build
@@ -8,7 +8,7 @@ isolation_sources = pg_regress_c + files(
spec_scanner = custom_target('specscanner',
input : files('specscanner.l'),
output : ['specscanner.c'],
- command : [flex, '-CF', '-p', '-p', '-o', '@OUTPUT0@', '@INPUT@']
+ command : [flex, '-o', '@OUTPUT0@', '@INPUT@']
)
isolationtester_sources = files('isolationtester.c')
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-20 00:31 Andres Freund <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-20 00:31 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
Hi,
On 2021-10-19 17:57:31 -0400, John Naylor wrote:
> I know this is still in the evaluation stage, but I did notice some
> discrepencies in the Flex flags. With the attached patch, the read-only
> data segment seems to match up pretty well now.
Good catch. I think I just copied them around...
I wish we had a bit more consistency in the flags, so we could centralize
them. Seems there's no reason to not use -p -p and -b everywhere?
I also need to make meson use our flex wrapper for the relevant versions... I
can see the warning that'd be fixed by it on macos CI. Will do that and push
it out to my github repo together with your changes.
Thanks!
Andres
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-20 01:08 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-20 01:08 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-19 15:22:15 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
> >> As far as I can tell the only OS that postgres currently supports that
> >> meson doesn't support is HPUX. It'd likely be fairly easy to add
> >> gcc-on-hpux support, a chunk more to add support for the proprietary
> >> ones.
>
> > Tom, wrt HPUX on pa-risc, what are your thoughts there? IIRC we gave up
> > supporting HP's compiler on pa-risc a while ago.
>
> Right. I am still testing with gcc on HP-PA. I'd kind of like to
> keep it running just as an edge case for our spinlock support, but
> I'm not sure that I want to do any huge amount of work on meson
> to keep that going.
Makes sense. While that does test an odd special case for our spinlock
implementation, it's also the only supported platform with that edge case, and
it seems extremely unlikely that there ever will be a new platform with such
odd/limited atomic operations.
> I do have a functioning OpenBSD installation on that machine, so
> one alternative if the porting costs look too high is to replace
> gaur with an OpenBSD animal. However, last I checked, OpenBSD
> was about half the speed of HPUX on that hardware, so I'm not
> real eager to go that way. gaur's already about the slowest
> animal in the farm :-(
Yea, that doesn't sound enticing. Seems like we either should keep it running
on hp-ux or just drop parisc support?
> > As I said it'd probably not be too hard to add meson support for hpux on hppa,
> > it's probably just a few branches. But that'd require access somewhere. The
> > gcc compile farm does not have a hppa member anymore...
>
> If you've got an idea where to look, I could add that to my to-do queue.
It might even just work. Looks like meson does have pa-risc detection. While
it doesn't have any specifically for hpux, it just falls back to python's
sys.platform in that case. python3 -c 'import sys;print(sys.platform)'
meson generates output for ninja to execute (basically a faster make that's
partially faster by being much less flexible. Intended to be output by more
user-friendly buildsystems ). Ninja can be built by a minimal python script,
or with cmake. The former doesn't seem to have hpux support, the latter does I
think.
https://github.com/ninja-build/ninja
So it could be interesting to see if ninja builds.
I've not taught the PG meson the necessary stuff for a 32 bit build. So
there's no point is trying whether meson works that much. I'll try to do that,
and let you know.
> I'm more concerned about the effort involved in getting meson going on some
> other old animals, such as prairiedog.
Yea, that's an *old* OS version. One version too old to have support for
@rpath, added in 10.5 :(. Is there a reason to run 10.4 specifically?
According to wikipedia 10.5 is the last version to support ppc.
Looks like python still supports building back to 10.4.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-20 01:10 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Tom Lane @ 2021-10-20 01:10 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: John Naylor <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> I wish we had a bit more consistency in the flags, so we could centralize
> them. Seems there's no reason to not use -p -p and -b everywhere?
I don't think we care enough about performance of most of the scanners
to make them all backup-free, so -1 to that idea.
We could possibly replace the command line switches with %option
entries in the files themselves. But I think the reason we haven't
done so for -b is that the Makefile still needs to know about it
so as to know what to do with the lex.backup output file.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-20 01:26 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Tom Lane @ 2021-10-20 01:26 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2021-10-19 15:22:15 -0400, Tom Lane wrote:
>> I'm more concerned about the effort involved in getting meson going on some
>> other old animals, such as prairiedog.
> Yea, that's an *old* OS version. One version too old to have support for
> @rpath, added in 10.5 :(. Is there a reason to run 10.4 specifically?
> According to wikipedia 10.5 is the last version to support ppc.
My notes say
Currently running OSX 10.4.11 (last release of Tiger); although 10.5 Leopard
supports PPCs, it refuses to install if CPU speed < 867MHz, well beyond the
Cube's ability. Wikipedia does suggest it's possible to run Leopard, but...
https://en.wikipedia.org/wiki/Mac_OS_X_Leopard#Usage_on_unsupported_hardware
I'm not sure that I have install media for 10.5 anymore, either --- ISTR
some machine's CD drive failing and not letting me get the CD back out.
If I did have it, I don't think there'd be a way to update past 10.5.0
(surely Apple no longer has those updaters on-line?), so on the whole
I think that path is a nonstarter.
I do have 10.5 running on an old G4 PowerMac, but that machine is (a)
noisy (b) power-hungry and (c) getting flaky, so I'm uneager to spin up
a buildfarm animal on it.
As with the HPPA, a potential compromise is to spin up some newer
BSD-ish system on it. I agree that OSX 10.4 is uninteresting as a
software platform, but I'd like to keep 32-bit PPC represented in
the farm.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-20 01:35 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-20 01:35 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
Hi,
On 2021-10-19 17:31:22 -0700, Andres Freund wrote:
> I also need to make meson use our flex wrapper for the relevant versions... I
> can see the warning that'd be fixed by it on macos CI. Will do that and push
> it out to my github repo together with your changes.
That turned out to be more work than I anticipated, so I pushed your changes
out separately.
There's this bit in plflex.pl that talks about adjusting yywrap() for msvc. I
didn't implement that and didn't see any compilation problems. Looks like that
originally hails from 2011, in 08a0c2dabc3b9d59d72d7a79ed867b8e37d275a7
Hm. Seems not worth carrying forward unless it actually causes trouble?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-20 01:49 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-20 01:49 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-19 21:26:53 -0400, Tom Lane wrote:
> My notes say
>
> Currently running OSX 10.4.11 (last release of Tiger); although 10.5 Leopard
> supports PPCs, it refuses to install if CPU speed < 867MHz, well beyond the
> Cube's ability. Wikipedia does suggest it's possible to run Leopard, but...
> https://en.wikipedia.org/wiki/Mac_OS_X_Leopard#Usage_on_unsupported_hardware
>
> I'm not sure that I have install media for 10.5 anymore, either --- ISTR
> some machine's CD drive failing and not letting me get the CD back out.
> If I did have it, I don't think there'd be a way to update past 10.5.0
> (surely Apple no longer has those updaters on-line?), so on the whole
> I think that path is a nonstarter.
That does indeed sound like a nonstarter.
> I do have 10.5 running on an old G4 PowerMac, but that machine is (a)
> noisy (b) power-hungry and (c) getting flaky, so I'm uneager to spin up
> a buildfarm animal on it.
Understandable.
> As with the HPPA, a potential compromise is to spin up some newer
> BSD-ish system on it. I agree that OSX 10.4 is uninteresting as a
> software platform, but I'd like to keep 32-bit PPC represented in
> the farm.
I assume the reason 32-bit PPC is interesting is that it's commonly run big
endian?
I wonder when it'll be faster to run 32bit ppc via qemu than natively :)
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-20 02:04 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Tom Lane @ 2021-10-20 02:04 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2021-10-19 21:26:53 -0400, Tom Lane wrote:
>> As with the HPPA, a potential compromise is to spin up some newer
>> BSD-ish system on it. I agree that OSX 10.4 is uninteresting as a
>> software platform, but I'd like to keep 32-bit PPC represented in
>> the farm.
> I assume the reason 32-bit PPC is interesting is that it's commonly run big
> endian?
Aside from bit width and endianness, I believe it's a somewhat smaller
instruction set than the newer CPUs.
> I wonder when it'll be faster to run 32bit ppc via qemu than natively :)
I think qemu would have a ways to go for that. More to the point,
I've found that its emulation is not as precise as one might wish...
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-20 02:41 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-20 02:41 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-19 18:49:43 -0700, Andres Freund wrote:
> I wonder when it'll be faster to run 32bit ppc via qemu than natively :)
Freebsd didn't seem to want to boot, but surprisingly a debian buster image
started at least the installer without problems... Will probably take a while
to see if it actually works.
I assume to make it acceptable from a build-speed perspective one would have
to use distcc with the compiler running outside.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-20 16:01 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-20 16:01 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-19 19:41:56 -0700, Andres Freund wrote:
> On 2021-10-19 18:49:43 -0700, Andres Freund wrote:
> > I wonder when it'll be faster to run 32bit ppc via qemu than natively :)
>
> Freebsd didn't seem to want to boot, but surprisingly a debian buster image
> started at least the installer without problems... Will probably take a while
> to see if it actually works.
The build was quite slow (cold ccache cache, only 1 cpu):
real 106m33.418s
user 86m36.363s
sys 17m33.830s
But the actual test time wasn't *too* bad, compared to the 32bit ppc animals
real 12m14.944s
user 0m51.622s
sys 0m44.743s
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-21 21:48 Andres Freund <[email protected]>
parent: John Naylor <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-10-21 21:48 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 15:55:22 -0400, John Naylor wrote:
> Also, could utility makefile targets be made to work? I'm thinking in
> particular of update-unicode and reformat-dat-files, for example.
Implementing reformat-dat-files was trivial:
https://github.com/anarazel/postgres/commit/29c1ce1ad4731290714978da5ce81e99ef051bec
However, update-unicode is a bit harder. Partially not directly because of
meson, but because update-unicode as-is afaict doesn't support VPATH builds,
and meson enforces those.
make update-unicode
...
make -C src/common/unicode update-unicode
'/usr/bin/perl' generate-unicode_norm_table.pl
Can't open perl script "generate-unicode_norm_table.pl": No such file or directory
It's not too hard to fix. See attached for the minimal stuff that I
immediately found to be needed. There's likely more,
e.g. src/backend/utils/mb/Unicode - but I didn't immediately see where that's
invoked from.
The slightly bigger issue making update-unicode work with meson is that meson
doesn't provide support for invoking build targets in specific directories
(because it doesn't map nicely to e.g. msbuild). But scripts like
src/common/unicode/generate-unicode_norm_table.pl rely on CWD. It's not hard
to work around that, but IMO it's better for such scripts to not rely on CWD.
Greetings,
Andres Freund
Attachments:
[text/x-diff] update-unicode.diff (2.2K, ../../[email protected]/2-update-unicode.diff)
download | inline diff:
diff --git i/src/common/unicode/Makefile w/src/common/unicode/Makefile
index a3683dd86b9..e69054d4671 100644
--- i/src/common/unicode/Makefile
+++ w/src/common/unicode/Makefile
@@ -12,14 +12,14 @@ subdir = src/common/unicode
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
+override CPPFLAGS := -DFRONTEND -I$(abs_top_builddir)/src/common/unicode $(CPPFLAGS)
LIBS += $(PTHREAD_LIBS)
# By default, do nothing.
all:
update-unicode: unicode_norm_table.h unicode_combining_table.h unicode_east_asian_fw_table.h unicode_normprops_table.h unicode_norm_hashfunc.h
- mv $^ ../../../src/include/common/
+ mv $^ $(top_srcdir)/src/include/common/
$(MAKE) normalization-check
# These files are part of the Unicode Character Database. Download
@@ -33,7 +33,7 @@ UnicodeData.txt EastAsianWidth.txt DerivedNormalizationProps.txt CompositionExcl
unicode_norm_hashfunc.h: unicode_norm_table.h
unicode_norm_table.h: generate-unicode_norm_table.pl UnicodeData.txt CompositionExclusions.txt
- $(PERL) generate-unicode_norm_table.pl
+ $(PERL) $^
unicode_combining_table.h: generate-unicode_combining_table.pl UnicodeData.txt
$(PERL) $^ >$@
@@ -58,7 +58,7 @@ submake-common:
$(MAKE) -C .. all
norm_test_table.h: generate-norm_test_table.pl NormalizationTest.txt
- perl generate-norm_test_table.pl NormalizationTest.txt $@
+ perl $^ $@
.PHONY: normalization-check
diff --git i/contrib/unaccent/Makefile w/contrib/unaccent/Makefile
index b8307d1601e..d6c466e07ad 100644
--- i/contrib/unaccent/Makefile
+++ w/contrib/unaccent/Makefile
@@ -27,12 +27,12 @@ include $(top_builddir)/src/Makefile.global
include $(top_srcdir)/contrib/contrib-global.mk
endif
-update-unicode: unaccent.rules
+update-unicode: $(srcdir)/unaccent.rules
# Allow running this even without --with-python
PYTHON ?= python
-unaccent.rules: generate_unaccent_rules.py ../../src/common/unicode/UnicodeData.txt Latin-ASCII.xml
+$(srcdir)/unaccent.rules: generate_unaccent_rules.py ../../src/common/unicode/UnicodeData.txt Latin-ASCII.xml
$(PYTHON) $< --unicode-data-file $(word 2,$^) --latin-ascii-file $(word 3,$^) >$@
# Only download it once; dependencies must match src/common/unicode/
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-10-22 15:55 John Naylor <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: John Naylor @ 2021-10-22 15:55 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Thu, Oct 21, 2021 at 5:48 PM Andres Freund <[email protected]> wrote:
> However, update-unicode is a bit harder. Partially not directly because
of
> meson, but because update-unicode as-is afaict doesn't support VPATH
builds,
> and meson enforces those.
> make update-unicode
> ...
> make -C src/common/unicode update-unicode
> '/usr/bin/perl' generate-unicode_norm_table.pl
> Can't open perl script "generate-unicode_norm_table.pl": No such file or
directory
>
> It's not too hard to fix. See attached for the minimal stuff that I
> immediately found to be needed.
Thanks for doing that, it works well enough for demonstration. With your
patch, and using an autoconf VPATH build, the unicode tables work fine, but
it complains of a permission error in generate_unaccent_rules.py. That
seems to be because the script is invoked directly rather than as an
argument to the python interpreter.
> The slightly bigger issue making update-unicode work with meson is that
meson
> doesn't provide support for invoking build targets in specific directories
> (because it doesn't map nicely to e.g. msbuild). But scripts like
> src/common/unicode/generate-unicode_norm_table.pl rely on CWD. It's not
hard
> to work around that, but IMO it's better for such scripts to not rely on
CWD.
Yeah. I encountered a further issue: With autoconf on HEAD, with a source
tree build executed in contrib/unaccent:
$ touch generate_unaccent_rules.py
$ make update-unicode
generate_unaccent_rules.py --unicode-data-file
../../src/common/unicode/UnicodeData.txt --latin-ascii-file Latin-ASCII.xml
>unaccent.rules
/bin/sh: generate_unaccent_rules.py: command not found
make: *** [unaccent.rules] Error 127
make: *** Deleting file `unaccent.rules'
...so in this case it seems not to know to use CWD here.
Anyway, this can be put off until the very end, since it's not run often.
You've demonstrated how these targets would work, and that's good enough
for now.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson -v
@ 2021-10-31 23:24 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
10 siblings, 0 replies; 130+ messages in thread
From: Andres Freund @ 2021-10-31 23:24 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Thomas Munro <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; John Naylor <[email protected]>; Andrew Dunstan <[email protected]>; Josef Šimánek <[email protected]>
Hi,
Attached is an updated version of the meson patchset.
Changes:
- support for remaining binaries in src/bin, contrib modules
- nearly all tests, including src/test/modules etc, are integrated.
- quite a few more, but not yet all, optional dependencies (most are
exercised in the included CI)
- runs tests on SIP enabled macos without needing a prior installation /
installation is relocatable
- support for building docs.
I couldn't get dbtoepub work in a vpath style build, so I changed that
to also use pandoc. No idea if anybody uses the epub rules?
- 32bit x86 [1], 64bit aarch64 builds
- cross-building windows from linux works
- error when building with meson against a source tree with an in-tree
autoconf build (leads to problems with pg_config.h etc)
- update-unicode, reformat-dat-files, expand-dat-files
Bigger missing pieces:
- pgxs (that's a *hard* one)
- NLS
- test / add support for platforms besides freebsd, linux, macos, windows
- remaining hardcoded configure tests (e.g. ACCEPT_TYPE_ARG*)
- win32 resource files only handled for two binaries, needs to be made
more compact
- ecpg
- fixing up flex output
- truckloads of polishing
- some tests (e.g. pg_upgrade, because of the upcoming tap conversion,
other tests that are shell scripts). Some tests are now run
unconditionally that previously were opt-in.
- what exactly gets installed where
- a "dist" target
- fix "ldap" build on macos
Greetings,
Andres Freund
[1] I had not defined SIZEOF_SIZE_T. Surprisingly that still results in
a successful 64bit build, but not a successful 32bit build.
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-11-15 18:34 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
10 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2021-11-15 18:34 UTC (permalink / raw)
To: pgsql-hackers; Tom Lane <[email protected]>
Hi,
FWIW, I tried building postgres on a few other operating systems using
meson, after I got access to the gcc compile farm. Here's the results:
- openbsd: Compiled fine. Hit one issue running tests:
openbsd has *completely* broken $ORIGIN support. It uses CWD as $ORIGIN
rpaths, which obviously breaks for binaries invoked via PATH. So there
goes the idea to only use $ORIGIN to run tests. Still seems worth to use
on other platforms, particularly because it works with SIP on macos
I understand not supporting $ORIGIN at all. But implementing it this way
seems insane.
I also ran into some problems with the semaphore limits. I had to switch to
USE_NAMED_POSIX_SEMAPHORES to make the tests pass at all.
- netbsd: Compiled fine after some minor fix. There's a bit more to fix around
many libraries not being in the normal library directory, but in
/usr/pkg/lib, which is not in the library search path (i.e. we need to add
an rpath for that in a few more places).
- AIX: Compiled and basic postgres runs fine after a few fixes (big endian
test, converting exports.txt into the right format). Doesn't yet
successfully run more than trivial tests, because I didn't implement the
necessary generation of import files for postgres, but that's just a bit of
work.
This is hampered by the fact that the vanilla postgres crashes for me. I
haven't quite figured out what's the problem. Might be a system issue -
lots of other tools, e.g. perl, segfault frequently.
One important thing to call out: Meson has support for the AIX linker, but
*not* the xlc compiler. I.e. one has to use gcc (or clang, but I didn't
try). I don't know if we'd require adding support for xlc to meson - xlc is
pretty buggy and it doesn't seem particularly crucial to support such an old
crufty compiler on a platform that's not used to a significant degree?
Greetings,
Andres
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-11-15 19:11 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Tom Lane @ 2021-11-15 19:11 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> One important thing to call out: Meson has support for the AIX linker, but
> *not* the xlc compiler. I.e. one has to use gcc (or clang, but I didn't
> try). I don't know if we'd require adding support for xlc to meson - xlc is
> pretty buggy and it doesn't seem particularly crucial to support such an old
> crufty compiler on a platform that's not used to a significant degree?
While I have no particular interest in AIX or xlc specifically, I do
worry about us becoming a builds-on-gcc-or-workalikes-only project.
I suppose MSVC provides a little bit of a cross-check, but I don't
really like giving up on other compilers. Discounting gcc+clang+MSVC
leaves just a few buildfarm animals, and the xlc ones are a significant
part of that population. (In fact, unless somebody renews fossa/husky's
icc license, the three xlc animals will be an outright majority of
them, because wrasse and anole are the only other active animals with
non-mainstream compilers.)
Having said that, I don't plan to be the one trying to get meson
to add xlc support ...
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-11-15 19:23 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 3 replies; 130+ messages in thread
From: Andres Freund @ 2021-11-15 19:23 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-11-15 14:11:25 -0500, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > One important thing to call out: Meson has support for the AIX linker, but
> > *not* the xlc compiler. I.e. one has to use gcc (or clang, but I didn't
> > try). I don't know if we'd require adding support for xlc to meson - xlc is
> > pretty buggy and it doesn't seem particularly crucial to support such an old
> > crufty compiler on a platform that's not used to a significant degree?
>
> While I have no particular interest in AIX or xlc specifically, I do
> worry about us becoming a builds-on-gcc-or-workalikes-only project.
> I suppose MSVC provides a little bit of a cross-check, but I don't
> really like giving up on other compilers. Discounting gcc+clang+MSVC
> leaves just a few buildfarm animals, and the xlc ones are a significant
> part of that population.
Yea, that's a reasonable concern. I wonder if there's some non-mainstream
compiler that actually works on, um, more easily available platforms that we
could utilize.
> (In fact, unless somebody renews fossa/husky's
> icc license, the three xlc animals will be an outright majority of
> them, because wrasse and anole are the only other active animals with
> non-mainstream compilers.)
It should probably be doable to get somebody to run another icc animal. Icc is
supported by meson, fwiw.
> Having said that, I don't plan to be the one trying to get meson
> to add xlc support ...
It'd probably not be too hard. But given that it's quite hard to get access to
AIX + xlc, I'm not sure it's something I want to propose. There's no resources
to run halfway regular tests on that I found...
It's good to make sure we're not growing too reliant on some compiler(s), but
imo only really makes sense if the alternative compilers are meaningfully
available and maintained.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-11-15 19:48 Robert Haas <[email protected]>
parent: Andres Freund <[email protected]>
2 siblings, 0 replies; 130+ messages in thread
From: Robert Haas @ 2021-11-15 19:48 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Mon, Nov 15, 2021 at 2:23 PM Andres Freund <[email protected]> wrote:
> It's good to make sure we're not growing too reliant on some compiler(s), but
> imo only really makes sense if the alternative compilers are meaningfully
> available and maintained.
That's a sensible position. I do worry that with this proposed move
we're going to be giving up some of the flexibility that we have right
now. I'm not sure exactly what that means in practice. But make is
just a way of running shell commands, and so you can run any shell
commands you want. The concept of some compiler not being supported
isn't really a thing that even makes sense in a world that is powered
by make. With a big enough hammer you can run any commands you like,
including any compilation commands you like. The whole thing is likely
to be a bit crufty which is a downside, and you might spend more time
fiddling with it than you really want. But nothing is really ever
blocked.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-11-15 22:08 Thomas Munro <[email protected]>
parent: Andres Freund <[email protected]>
2 siblings, 2 replies; 130+ messages in thread
From: Thomas Munro @ 2021-11-15 22:08 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Tue, Nov 16, 2021 at 8:23 AM Andres Freund <[email protected]> wrote:
> On 2021-11-15 14:11:25 -0500, Tom Lane wrote:
> > Having said that, I don't plan to be the one trying to get meson
> > to add xlc support ...
>
> It'd probably not be too hard. But given that it's quite hard to get access to
> AIX + xlc, I'm not sure it's something I want to propose. There's no resources
> to run halfway regular tests on that I found...
FWIW there's a free-as-in-beer edition of xlc for Linux (various
distros, POWER only) so you could use qemu, though of course there
will be differences WRT AIX especially around linking, and I suppose a
big part of that work would be precisely understanding stuff like
linker details.
It looks like we have two xlc 12.1 compilers in the farm, but those
compilers are EOL'd[1]. The current release is 16.1, and we have one
of those. The interesting thing about 16.1 is that you can invoke it
as xlclang to get the new clang frontend and, I think, possibly use
more clang/gcc-ish compiler switches[2].
[1] https://www.ibm.com/support/pages/lifecycle/search?q=xl%20c%2Fc%2B%2B
[2] https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=new-clang-based-front-end
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-11-15 22:34 Tom Lane <[email protected]>
parent: Thomas Munro <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Tom Lane @ 2021-11-15 22:34 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
Thomas Munro <[email protected]> writes:
> ... The interesting thing about 16.1 is that you can invoke it
> as xlclang to get the new clang frontend and, I think, possibly use
> more clang/gcc-ish compiler switches[2].
> [2] https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=new-clang-based-front-end
Ho, that's an earful ;-). Though I wonder whether that frontend
hides the AIX-specific linking issues you mentioned. (Also, although
I see /opt/IBM/xlc/16.1.0/ on gcc119, there's no xlclang there.
So whether we have useful access to it right now is unclear.)
This plays into something that was nagging at me while I wrote my
upthread screed about not giving up on non-gcc/clang compilers:
are those compilers outcompeting all the proprietary ones, to the
extent that the latter will be dead soon anyway? I think Microsoft
is rich enough and stubborn enough to keep on developing MSVC no
matter what, but other compiler vendors may see the handwriting
on the wall. Writing C compilers can't be a growth industry these
days.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-11-15 22:46 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Andres Freund @ 2021-11-15 22:46 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
Hi,
On 2021-11-15 17:34:33 -0500, Tom Lane wrote:
> Thomas Munro <[email protected]> writes:
> > ... The interesting thing about 16.1 is that you can invoke it
> > as xlclang to get the new clang frontend and, I think, possibly use
> > more clang/gcc-ish compiler switches[2].
> > [2] https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=new-clang-based-front-end
>
> Ho, that's an earful ;-). Though I wonder whether that frontend
> hides the AIX-specific linking issues you mentioned. (Also, although
> I see /opt/IBM/xlc/16.1.0/ on gcc119, there's no xlclang there.
> So whether we have useful access to it right now is unclear.)
It's actually available there, but in /opt/IBM/xlC/16.1.0/bin/xlclang++ (note
the upper case C).
It doesn't really hide the linking issues afaict. I think they're basically an
ABI rather than a linker invocation issue. It's not that hard to address them
though, it's basically making mkldexport.sh a tiny bit more general and
integrating it into src/backend/postgres' build.
We don't have to generate export files for shared libraries anymore though,
afaict, because there's 'expall', which suffices for our purposes. dlopen()
doesn't require an import file.
> This plays into something that was nagging at me while I wrote my
> upthread screed about not giving up on non-gcc/clang compilers:
> are those compilers outcompeting all the proprietary ones, to the
> extent that the latter will be dead soon anyway?
I think that's a pretty clear trend. The ones that aren't dying seem to be
incrementally onto more and more rebasing onto llvm tooling.
It doesn't help that most of those compilers are primarily for OSs that, uh,
aren't exactly growing. Which limits their potential usability significantly.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-11-15 22:52 Thomas Munro <[email protected]>
parent: Thomas Munro <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Thomas Munro @ 2021-11-15 22:52 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Tue, Nov 16, 2021 at 11:08 AM Thomas Munro <[email protected]> wrote:
> FWIW there's a free-as-in-beer edition of xlc for Linux (various
> distros, POWER only) so you could use qemu,
(It's also known to be possible to run AIX 7.2 on qemu, but the
install media is not made available to developers for testing/CI
without a hardware serial number. Boo.)
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2021-11-18 20:44 Thomas Munro <[email protected]>
parent: Andres Freund <[email protected]>
2 siblings, 0 replies; 130+ messages in thread
From: Thomas Munro @ 2021-11-18 20:44 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Tue, Nov 16, 2021 at 8:23 AM Andres Freund <[email protected]> wrote:
> On 2021-11-15 14:11:25 -0500, Tom Lane wrote:
> > (In fact, unless somebody renews fossa/husky's
> > icc license, the three xlc animals will be an outright majority of
> > them, because wrasse and anole are the only other active animals with
> > non-mainstream compilers.)
>
> It should probably be doable to get somebody to run another icc animal. Icc is
> supported by meson, fwiw.
FWIW, in case someone is interested in bringing ICC back to the farm,
some light googling tells me that newer editions of "classic" ICC (as
opposed to "data parallel" ICC, parts of some kind of rebrand) no
longer require regular licence bureaucracy, and can be installed in
modern easier to maintain ways. For example, I see that some people
add Intel's APT repository and apt-get install the compiler inside CI
jobs, on Ubuntu.
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-05-26 00:47 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
10 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2022-05-26 00:47 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi Tom,
in the meson unconference session you'd spotted flex flags for psqlscanslash.l
(I think) being "hardcoded". As far as I can tell that's largely just copied
from the Makefile):
src/backend/parser/Makefile:scan.c: FLEXFLAGS = -CF -p -p
src/backend/utils/adt/Makefile:jsonpath_scan.c: FLEXFLAGS = -CF -p -p
src/bin/psql/Makefile:psqlscanslash.c: FLEXFLAGS = -Cfe -p -p
src/fe_utils/Makefile:psqlscan.c: FLEXFLAGS = -Cfe -p -p
note that it's not even FLEXFLAGS += or such.
I honestly don't know enough about the various flex flags to judge what a
better approach would be? Looks like these flags are case specific? Perhaps we
could group them, i.e. have centrally defined "do compress" "don't compress"
flex flags?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-05-26 01:38 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Tom Lane @ 2022-05-26 01:38 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> in the meson unconference session you'd spotted flex flags for psqlscanslash.l
> (I think) being "hardcoded". As far as I can tell that's largely just copied
> from the Makefile):
> src/backend/parser/Makefile:scan.c: FLEXFLAGS = -CF -p -p
> src/backend/utils/adt/Makefile:jsonpath_scan.c: FLEXFLAGS = -CF -p -p
> src/bin/psql/Makefile:psqlscanslash.c: FLEXFLAGS = -Cfe -p -p
> src/fe_utils/Makefile:psqlscan.c: FLEXFLAGS = -Cfe -p -p
Hmm, OK. There *is* a FLEXFLAGS definition supplied by configure, and
I believe many of our scanners do use it, but evidently we're just
overriding it for the ones where we really care about using specific
flags. It also looks like the configure-supplied version is usually
empty, so the fact that this variable exists may be mostly a holdover
from Autoconf practice rather than something we ever cared about.
I think the main thing I didn't like about the way you have it in the
meson file is the loss of greppability. I could investigate this
question in a few seconds just now, but if we drop the use of
FLEXFLAGS as a macro it'll become much harder to figure out which
places use what.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-05-26 02:41 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2022-05-26 02:41 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2022-05-25 21:38:33 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > in the meson unconference session you'd spotted flex flags for psqlscanslash.l
> > (I think) being "hardcoded". As far as I can tell that's largely just copied
> > from the Makefile):
>
> > src/backend/parser/Makefile:scan.c: FLEXFLAGS = -CF -p -p
> > src/backend/utils/adt/Makefile:jsonpath_scan.c: FLEXFLAGS = -CF -p -p
> > src/bin/psql/Makefile:psqlscanslash.c: FLEXFLAGS = -Cfe -p -p
> > src/fe_utils/Makefile:psqlscan.c: FLEXFLAGS = -Cfe -p -p
>
> Hmm, OK. There *is* a FLEXFLAGS definition supplied by configure, and
> I believe many of our scanners do use it, but evidently we're just
> overriding it for the ones where we really care about using specific
> flags. It also looks like the configure-supplied version is usually
> empty, so the fact that this variable exists may be mostly a holdover
> from Autoconf practice rather than something we ever cared about.
Yea, it looks like that.
ISTM that it'd still be good to have something like FLEXFLAGS. But it doesn't
look great, nor really intentional, that FLEXFLAGS is overwritten rather than
appended?
> I think the main thing I didn't like about the way you have it in the
> meson file is the loss of greppability. I could investigate this
> question in a few seconds just now, but if we drop the use of
> FLEXFLAGS as a macro it'll become much harder to figure out which
> places use what.
I disliked a bunch of repetitiveness as I had it, so I'm polishing that part
just now.
What would you want to grep for? Places that specify additional flags? Or just
places using flex?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-05-26 02:58 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Tom Lane @ 2022-05-26 02:58 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2022-05-25 21:38:33 -0400, Tom Lane wrote:
>> I think the main thing I didn't like about the way you have it in the
>> meson file is the loss of greppability.
> What would you want to grep for? Places that specify additional flags? Or just
> places using flex?
Well, the consistency of having a single name for "flags given to
flex" seems to me to be worth something.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-05-26 08:47 Aleksander Alekseev <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Aleksander Alekseev @ 2022-05-26 08:47 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi Andres,
Thanks for working on this! I'm very enthusiastic about this effort and I was
glad to see on PGCon Unconference that the majority of the community seems
to be as well.
> The halfway decent list includes, I think:
> 1) cmake
> 2) bazel
> 3) meson
Was SCons considered as an option? It is widely adopted and written in Python
as well. Personally, I like the fact that with SCons you write config files
in pure Python, not some dialect you have to learn additionally. There is
a free e-book available [1].
What pros and cons do you see that make Meson a better choice?
[1]: https://scons.org/doc/production/PDF/scons-user.pdf
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-05-28 19:14 Andres Freund <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2022-05-28 19:14 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi,
On 2022-05-26 11:47:13 +0300, Aleksander Alekseev wrote:
> Thanks for working on this! I'm very enthusiastic about this effort and I was
> glad to see on PGCon Unconference that the majority of the community seems
> to be as well.
>
> > The halfway decent list includes, I think:
> > 1) cmake
> > 2) bazel
> > 3) meson
>
> Was SCons considered as an option?
> What pros and cons do you see that make Meson a better choice?
I looked at it and quickly discarded it. From what I could see there's not
been meaningful moves to it in the last couple years, if anything adoption has
been dropping. And I don't think we want to end up relying on yet another half
maintained tool.
Not having a ninja backend etc didn't strike me as great either - the builds
with scons I've done weren't fast at all.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-05-31 13:49 Aleksander Alekseev <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Aleksander Alekseev @ 2022-05-31 13:49 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi Andres,
> Not having a ninja backend etc didn't strike me as great either - the builds
> with scons I've done weren't fast at all.
I must admit, personally I never used Scons, I just know that it was considered
(an / the only?) alternative to CMake for many years. The Scons 4.3.0 release
notes say that Ninja is supported [1], but according to the user guide [2]
Ninja support is considered experimental.
Don't get me wrong, I don't insist on using Scons. I was just curious if it was
considered. Actually, a friend of mine pointed out that the fact that Scons
build files are literally a Python code could be a disadvantage. There is less
control of this code, basically it can do anything. It could complicate the
diagnosis of certain issues, etc.
Since you invested so much effort into Meson already let's just focus on it.
I tried the branch on GitHub on MacOS Monterey 12.3.1 and Ubuntu 20.04 LTS.
I was going to test it against several third party extensions, but it looks like
it is a bit early for this. On Ubuntu I got the following error:
```
../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
in a function)
332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
‘PG_KEYWORD’
32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
```
On MacOS I got multiple errors regarding LDAP:
```
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/
LDAP.framework/Headers/ldap.h:1:10: error: #include nested too deeply
#include <ldap.h>
../src/interfaces/libpq/fe-connect.c:4816:2: error: use of undeclared
identifier 'LDAP'
LDAP *ld = NULL;
^
../src/interfaces/libpq/fe-connect.c:4817:2: error: use of undeclared
identifier 'LDAPMessage'
LDAPMessage *res,
^
... etc...
```
I didn't invest much time into investigating these issues. For now I just
wanted to report them. Please let me know if you need any help with these
and/or additional information.
[1]: https://scons.org/scons-430-is-available.html
[2]: https://scons.org/doc/production/PDF/scons-user.pdf
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-05-31 19:25 Andres Freund <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2022-05-31 19:25 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi,
On 2022-05-31 16:49:17 +0300, Aleksander Alekseev wrote:
> I tried the branch on GitHub on MacOS Monterey 12.3.1 and Ubuntu 20.04 LTS.
> I was going to test it against several third party extensions, but it looks like
> it is a bit early for this. On Ubuntu I got the following error:
What do those extensions use to build? Since the unconference I added some
rudimentary PGXS compatibility, but it's definitely not complete yet.
> ```
> ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> in a function)
> 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
>
> ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> ‘PG_KEYWORD’
> 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> ```
Huh. I've not seen this before - could you provide a bit more detail about
what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
this to work.
> On MacOS I got multiple errors regarding LDAP:
Ah, yes. Sorry, that's an open issue that I need to fix. -Dldap=disabled for
the rescue. There's some crazy ordering dependency in macos framework
headers. The ldap framework contains an "ldap.h" header that includes
"ldap.h". So if you end up with the framework on the include path, you get
endless recursion.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-01 09:39 Aleksander Alekseev <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Aleksander Alekseev @ 2022-06-01 09:39 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi Andres,
> What do those extensions use to build? Since the unconference I added some
> rudimentary PGXS compatibility, but it's definitely not complete yet.
We mostly use CMake and Cargo, the Rust package manager. So I don't
anticipate many problems here, just want to make sure it's going to
work as expected.
> > ```
> > ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> > in a function)
> > 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
> >
> > ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> > ‘PG_KEYWORD’
> > 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> > ```
>
> Huh. I've not seen this before - could you provide a bit more detail about
> what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
> this to work.
I used PIP to install Meson, since the default APT package is too old, v0.53:
$ pip3 install --user meson
$ meson --version
0.62.1
$ ninja --version
1.10.0
The branch was checked out as it was described in the first email.
Then to reproduce the issue:
$ git status
On branch meson
Your branch is up to date with 'andres/meson'.
$ git fetch andres
$ git rebase -i andres/meson
$ meson setup build --buildtype debug
$ cd build
$ ninja
This is pretty much the default Ubuntu 20.04.4 LTS system with all the
recent updates installed, so it shouldn't be a problem to reproduce
the issue with a VM.
> > On MacOS I got multiple errors regarding LDAP:
>
> Ah, yes. Sorry, that's an open issue that I need to fix. -Dldap=disabled for
> the rescue. There's some crazy ordering dependency in macos framework
> headers. The ldap framework contains an "ldap.h" header that includes
> "ldap.h". So if you end up with the framework on the include path, you get
> endless recursion.
Thanks, this helped. I did the following:
$ meson configure -Dldap=disabled
$ meson configure -Dssl=openssl
$ meson configure -Dprefix=/Users/eax/pginstall
$ ninja
$ meson test
$ meson install
... and it terminated successfully. I was also able to configure and
run Postgres instance using my regular scripts, with some
modifications [1]
Then I decided to compile TimescaleDB against the newly installed
Postgres. Turns out there is a slight problem.
The extension uses CMake and also requires PostgreSQL to be compiled
with OpenSSL support. CMakeLists.txt looks for a
"--with-(ssl=)?openssl" regular expression in the "pg_config
--configure" output. The output is empty although Postgres was
compiled with OpenSSL support. The full output of pg_config looks like
this:
```
CONFIGURE =
CC = not recorded
CPPFLAGS = not recorded
CFLAGS = not recorded
CFLAGS_SL = not recorded
... etc ...
```
I get a bunch of errors from the compiler if I remove this particular
check from CMakeLists, but I have to investigate these a bit more
since the branch is based on PG15 and we don't officially support PG15
yet. It worked last time we checked a month or so ago, but the
situation may have changed.
[1]: https://github.com/afiskon/pgscripts/blob/master/single-install.sh
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-01 21:05 Andres Freund <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2022-06-01 21:05 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi,
On 2022-06-01 12:39:50 +0300, Aleksander Alekseev wrote:
> > > ```
> > > ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> > > in a function)
> > > 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
> > >
> > > ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> > > ‘PG_KEYWORD’
> > > 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> > > ```
> >
> > Huh. I've not seen this before - could you provide a bit more detail about
> > what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
> > this to work.
>
> I used PIP to install Meson, since the default APT package is too old, v0.53:
>
> $ pip3 install --user meson
> $ meson --version
> 0.62.1
> $ ninja --version
> 1.10.0
>
> The branch was checked out as it was described in the first email.
> Then to reproduce the issue:
>
> $ git status
> On branch meson
> Your branch is up to date with 'andres/meson'.
> $ git fetch andres
> $ git rebase -i andres/meson
> $ meson setup build --buildtype debug
> $ cd build
> $ ninja
>
> This is pretty much the default Ubuntu 20.04.4 LTS system with all the
> recent updates installed, so it shouldn't be a problem to reproduce
> the issue with a VM.
Will test.
> > > On MacOS I got multiple errors regarding LDAP:
> >
> > Ah, yes. Sorry, that's an open issue that I need to fix. -Dldap=disabled for
> > the rescue. There's some crazy ordering dependency in macos framework
> > headers. The ldap framework contains an "ldap.h" header that includes
> > "ldap.h". So if you end up with the framework on the include path, you get
> > endless recursion.
>
> Thanks, this helped.
Cool. I think I pushed a fix/workaround for the issue now. Still can't decide
whether it's apple's or meson's fault.
> I did the following:
>
> $ meson configure -Dldap=disabled
> $ meson configure -Dssl=openssl
> $ meson configure -Dprefix=/Users/eax/pginstall
FYI, you can set multiple options in one go ;)
> ... and it terminated successfully. I was also able to configure and
> run Postgres instance using my regular scripts, with some
> modifications [1]
Cool.
> Then I decided to compile TimescaleDB against the newly installed
> Postgres. Turns out there is a slight problem.
>
> The extension uses CMake and also requires PostgreSQL to be compiled
> with OpenSSL support. CMakeLists.txt looks for a
> "--with-(ssl=)?openssl" regular expression in the "pg_config
> --configure" output. The output is empty although Postgres was
> compiled with OpenSSL support.
Makes sense. Currently we don't fill the --configure thing, because there
configure wasn't used. We could try to generate something compatible from
meson options, but I'm not sure that's a good plan.
> The full output of pg_config looks like
> this:
>
> ```
> CONFIGURE =
> CC = not recorded
> CPPFLAGS = not recorded
> CFLAGS = not recorded
> CFLAGS_SL = not recorded
> ... etc ...
> ```
>
> I get a bunch of errors from the compiler if I remove this particular
> check from CMakeLists, but I have to investigate these a bit more
> since the branch is based on PG15 and we don't officially support PG15
> yet. It worked last time we checked a month or so ago, but the
> situation may have changed.
I suspect the errors might be due to CC/CPPFLAGS/... not being defined. I can
try to make it output something roughly compatible with the old output, for
most of those I already started to compute values for the PGXS compat stuff I
was hacking on recently.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-02 12:34 Aleksander Alekseev <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Aleksander Alekseev @ 2022-06-02 12:34 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi Andres,
> Cool. I think I pushed a fix/workaround for the issue now. Still can't decide
> whether it's apple's or meson's fault.
Many thanks! The fix solved the problem, I can compile with -Dldap=enabled now.
The code passes the tests too.
> > $ meson configure -Dldap=disabled
> > $ meson configure -Dssl=openssl
> > $ meson configure -Dprefix=/Users/eax/pginstall
>
> FYI, you can set multiple options in one go ;)
Thanks! ;)
> Makes sense. Currently we don't fill the --configure thing, because there
> configure wasn't used. We could try to generate something compatible from
> meson options, but I'm not sure that's a good plan.
If pg_config output was 100% backward compatible with Autotools one, that would
simplify the lives of the extension developers for sure. However, considering
that at PGCon we agreed that both Autotools and Meson will be maintained for
several releases, personally I wouldn't say that this compatibility is
necessary, nor is it realistically deliverable. Nevertheless, IMO there should
be a stable and documented way to determine the PostgreSQL version (this can be
done with `pg_config --version` for both Autotools and Meson), the build tool
used (no way to determine) and the build options (no way to determine
for Meson).
> I suspect the errors might be due to CC/CPPFLAGS/... not being defined. I can
> try to make it output something roughly compatible with the old output, for
> most of those I already started to compute values for the PGXS compat stuff I
> was hacking on recently.
Yes, that could explain the problem. Just for the record, I get several errors
regarding src/export.h in TimescaleDB code [1]:
```
/Users/eax/projects/c/timescaledb/src/export.h:26:5: error: pasting formed
')87628', an invalid preprocessing token [clang-diagnostic-error]
#if TS_EMPTY(PGDLLEXPORT)
^
/Users/eax/projects/c/timescaledb/src/export.h:17:22: note: expanded from
macro 'TS_EMPTY'
#define TS_EMPTY(x) (TS_CAT(x, 87628) == 87628)
^
/Users/eax/projects/c/timescaledb/src/export.h:15:23: note: expanded from
macro 'TS_CAT'
#define TS_CAT(x, y) x##y
^
/Users/eax/projects/c/timescaledb/src/export.h:26:14: error: function-like
macro '__attribute__' is not defined [clang-diagnostic-error]
#if TS_EMPTY(PGDLLEXPORT)
^
/Users/eax/pginstall/include/postgresql/server/c.h:1339:21: note: expanded
from macro 'PGDLLEXPORT'
#define PGDLLEXPORT __attribute__((visibility("default")))
^
/Users/eax/projects/c/timescaledb/src/export.h:30:2: error: "PGDLLEXPORT is
already defined" [clang-diagnostic-error]
#error "PGDLLEXPORT is already defined"
^
1 warning and 3 errors generated.
Error while processing /Users/eax/projects/c/timescaledb/src/extension.c
```
[1]: https://github.com/timescale/timescaledb/blob/main/src/export.h
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-02 15:22 Andres Freund <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Andres Freund @ 2022-06-02 15:22 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi,
On 2022-06-02 15:34:23 +0300, Aleksander Alekseev wrote:
> Hi Andres,
>
> > Cool. I think I pushed a fix/workaround for the issue now. Still can't decide
> > whether it's apple's or meson's fault.
>
> Many thanks! The fix solved the problem, I can compile with -Dldap=enabled now.
> The code passes the tests too.
Cool.
> > I suspect the errors might be due to CC/CPPFLAGS/... not being defined. I can
> > try to make it output something roughly compatible with the old output, for
> > most of those I already started to compute values for the PGXS compat stuff I
> > was hacking on recently.
>
> Yes, that could explain the problem. Just for the record, I get several errors
> regarding src/export.h in TimescaleDB code [1]:
>
I think this is timescale's issue. Why are you defining / undefining
PGDLLEXPORT?
Part of the patch series is to use visibility attributes, and your #if
TS_EMPTY(PGDLLEXPORT) thing can't handle that.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-02 16:39 Andres Freund <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2022-06-02 16:39 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi,
On 2022-06-01 12:39:50 +0300, Aleksander Alekseev wrote:
> > > ```
> > > ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> > > in a function)
> > > 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
> > >
> > > ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> > > ‘PG_KEYWORD’
> > > 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> > > ```
> >
> > Huh. I've not seen this before - could you provide a bit more detail about
> > what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
> > this to work.
>
> I used PIP to install Meson, since the default APT package is too old, v0.53:
>
> $ pip3 install --user meson
> $ meson --version
> 0.62.1
> $ ninja --version
> 1.10.0
>
> The branch was checked out as it was described in the first email.
> Then to reproduce the issue:
>
> $ git status
> On branch meson
> Your branch is up to date with 'andres/meson'.
> $ git fetch andres
> $ git rebase -i andres/meson
> $ meson setup build --buildtype debug
> $ cd build
> $ ninja
>
> This is pretty much the default Ubuntu 20.04.4 LTS system with all the
> recent updates installed, so it shouldn't be a problem to reproduce
> the issue with a VM.
Chatting with a colleague (who unbeknownst to me hit something similar in the
past) I think we figured it out. It's not due to Ubuntu 20.04 or such. It's
likely due to previously having an in-tree build with autoconf, doing make
clean, doing a git pull, then building with meson. The meson build doesn't yet
handle pre-existing flex / bison output.
I had tried to defend against conflicts with in-tree builds by detecting an
in-tree pg_config.h, but that doesn't help with files that aren't removed by
make clean. Like bison / flex output.
And I didn't notice this problem because it doesn't cause visible issues until
the lexer / grammar changes...
I'm not quite sure what the proper behaviour is when doing an out-of-tree
build with meson (all builds are out-of-tree), with a pre-existing flex /
bison output in the source tree that is out of date.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-02 17:08 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Tom Lane @ 2022-06-02 17:08 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> I'm not quite sure what the proper behaviour is when doing an out-of-tree
> build with meson (all builds are out-of-tree), with a pre-existing flex /
> bison output in the source tree that is out of date.
Definitely sounds like a gotcha.
On the one hand, there's been some discussion already of removing all
derived files from tarballs and just insisting that users provide all
needed tools when building from source. If we did that, it could be
sufficient for the meson build to check that no such files are present
in the source tree. (Checking a couple of them would be enough, likely.)
On the other hand, I'm not sure that I want such a change to be forced
by a toolchain change. It definitely seems a bit contrary to the plan
we'd formed of allowing meson and make-based builds to coexist for
a few years, because we'd be breaking at least some make-based build
processes.
Could we have the meson build check that, say, if gram.c exists it
is newer than gram.y? Or get it to ignore an in-tree gram.c?
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-02 17:26 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Andres Freund @ 2022-06-02 17:26 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-06-02 13:08:49 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > I'm not quite sure what the proper behaviour is when doing an out-of-tree
> > build with meson (all builds are out-of-tree), with a pre-existing flex /
> > bison output in the source tree that is out of date.
>
> Definitely sounds like a gotcha.
>
> On the one hand, there's been some discussion already of removing all
> derived files from tarballs and just insisting that users provide all
> needed tools when building from source. If we did that, it could be
> sufficient for the meson build to check that no such files are present
> in the source tree. (Checking a couple of them would be enough, likely.)
There already is a check for pg_config.h, so the most obvious source of this
is addressed. Just didn't think about the files that make clean doesn't remove
:/.
> On the other hand, I'm not sure that I want such a change to be forced
> by a toolchain change. It definitely seems a bit contrary to the plan
> we'd formed of allowing meson and make-based builds to coexist for
> a few years, because we'd be breaking at least some make-based build
> processes.
Agreed. I think it'd be pretty reasonable to not include flex / bison
output. They're not hard to acquire. The docs are perhaps another story.
I think it might be fine to say that make reallyclean (*) is required if
there's some conflicting in-source tree file?
> Could we have the meson build check that, say, if gram.c exists it
> is newer than gram.y? Or get it to ignore an in-tree gram.c?
I suspect the problem with ignoring is gram.h, that's probably a bit harder to
ignore. Right now I'm leaning towards either always erroring out if there's
bison/flex output in the source tree (with a hint towards make
reallyclean(*)), or erroring out if they're out of date (again with a hint
towards reallyclean)?
Alternatively we could just remove the generated .c/h files from the source
dir, as a part of regenerating them in the build dir? But I like the idea of
the source dir being readonly outside of explicit targets modifying sources
(e.g. update-unicode or such).
Greetings,
Andres Freund
(*) do we really not have a target that removes bison / flex output?
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-02 17:33 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Tom Lane @ 2022-06-02 17:33 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> (*) do we really not have a target that removes bison / flex output?
maintainer-clean
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-02 17:48 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2022-06-02 17:48 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-06-02 13:33:51 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > (*) do we really not have a target that removes bison / flex output?
>
> maintainer-clean
Don't think so:
# gram.c, gram.h, and scan.c are in the distribution tarball, so they
# are not cleaned here.
clean distclean maintainer-clean:
rm -f lex.backup
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-02 19:05 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Tom Lane @ 2022-06-02 19:05 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2022-06-02 13:33:51 -0400, Tom Lane wrote:
>> Andres Freund <[email protected]> writes:
>>> (*) do we really not have a target that removes bison / flex output?
>> maintainer-clean
> Don't think so:
See about line 300 in src/backend/Makefile. In any case, it's
easy to show by experiment that it does.
$ make maintainer-clean
...
$ git status --ignored
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-02 19:17 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2022-06-02 19:17 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-06-02 15:05:10 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > On 2022-06-02 13:33:51 -0400, Tom Lane wrote:
> >> Andres Freund <[email protected]> writes:
> >>> (*) do we really not have a target that removes bison / flex output?
>
> >> maintainer-clean
>
> > Don't think so:
>
> See about line 300 in src/backend/Makefile. In any case, it's
> easy to show by experiment that it does.
>
> $ make maintainer-clean
> ...
> $ git status --ignored
> On branch master
> Your branch is up to date with 'origin/master'.
>
> nothing to commit, working tree clean
Oh. I executed maintainer-clean inside src/backend/parser/, and thus didn't
see it getting cleaned up.
It seems pretty darn grotty that src/backend/parser/Makefile explicitly states
that gram.c ... aren't cleaned "here", but then src/backend/Makefile does
clean them up.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-02 19:53 Tom Lane <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Tom Lane @ 2022-06-02 19:53 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> Oh. I executed maintainer-clean inside src/backend/parser/, and thus didn't
> see it getting cleaned up.
> It seems pretty darn grotty that src/backend/parser/Makefile explicitly states
> that gram.c ... aren't cleaned "here", but then src/backend/Makefile does
> clean them up.
I agree the factorization of this ain't great. I'd think about improving
it, were it not that we're trying to get rid of it.
(But with meson, the whole idea of building or cleaning just part of the
tree is out the window anyway, no?)
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-02 21:13 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2022-06-02 21:13 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-06-02 15:53:50 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > Oh. I executed maintainer-clean inside src/backend/parser/, and thus didn't
> > see it getting cleaned up.
>
> > It seems pretty darn grotty that src/backend/parser/Makefile explicitly states
> > that gram.c ... aren't cleaned "here", but then src/backend/Makefile does
> > clean them up.
>
> I agree the factorization of this ain't great. I'd think about improving
> it, were it not that we're trying to get rid of it.
+1. I think I just wanted to excuse my confusion...
> (But with meson, the whole idea of building or cleaning just part of the
> tree is out the window anyway, no?)
Cleaning parts of the tree isn't supported as far as I know (not that I've
needed it). You can build parts of the tree by specifying the target
(e.g. ninja src/backend/postgres) or by specifying meta-targets (e.g. ninja
contrib backend). I've thought about contributing a patch to meson to
automatically generate targets for each directory that has sub-targets - it's
just a few lines.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-03 09:35 Aleksander Alekseev <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Aleksander Alekseev @ 2022-06-03 09:35 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
Hi hackers,
> See about line 300 in src/backend/Makefile. In any case, it's
> easy to show by experiment that it does.
`make maintainer-clean` did the trick, thanks. I suggest modifying meson.build
accordingly:
-run make distclean in the source tree.
+run `make maintainer-clean` in the source tree.
> I think this is timescale's issue. Why are you defining / undefining
> PGDLLEXPORT?
That's a great question.
As I understand some time ago the developers had a problem with a collision of
exported symbols on *nix platforms [1] and chose to solve it by re-defining
PGDLLEXPORT to __attribute__((visibility ("default"))) for GCC and CLang.
I agree that this is a questionable approach. Redefining a macro provided
by Postgres doesn't strike me as a good idea. I tried to remove this
re-definition, but it didn't go well [2]. So apparently it should be addressed
somehow differently.
> Part of the patch series is to use visibility attributes, and your #if
> TS_EMPTY(PGDLLEXPORT) thing can't handle that.
Out of curiosity, how come a patchset that adds an alternative build system
changes the visibility attributes? I would guess they should be the same
for both Autotools and Meson. Is it necessary in order to make Meson work?
If not, maybe it should be a separate patch.
[1]: https://github.com/timescale/timescaledb/commit/027b7b29420a742d7615c70d9f19b2b99c488c2c
[2]: https://github.com/timescale/timescaledb/pull/4413
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-06-03 16:23 Andres Freund <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Andres Freund @ 2022-06-03 16:23 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
Hi,
On 2022-06-03 12:35:45 +0300, Aleksander Alekseev wrote:
> > Part of the patch series is to use visibility attributes, and your #if
> > TS_EMPTY(PGDLLEXPORT) thing can't handle that.
>
> Out of curiosity, how come a patchset that adds an alternative build system
> changes the visibility attributes?
It was the simplest path - on windows (and AIx) extension symbols need to be
explicitly exported. We did that by building the objects constituting
extension libraries, collecting the symbols, generating an export file with
all the symbols, which then is passed to the linker. It was a lot less work
to just add the necessary PGDLLEXPORT annotations than make that export file
generation work for extensions.
> I would guess they should be the same for both Autotools and Meson.
It is, the patch adds it to both.
> Is it necessary in order to make Meson work?
Yes, or at least the simplest path.
> If not, maybe it should be a separate patch.
It is.
https://commitfest.postgresql.org/38/3396/
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-08-09 07:10 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
10 siblings, 1 reply; 130+ messages in thread
From: Andres Freund @ 2022-08-09 07:10 UTC (permalink / raw)
To: pgsql-hackers; Andrew Dunstan <[email protected]>; Magnus Hagander <[email protected]>
Hi,
I was looking at re-unifying gendef2.pl that the meson patchset had introduced
for temporary ease during hacking with gendef.pl. Testing that I noticed that
either I and my machine is very confused, or gendef.pl's check whether it can
skip work is bogus.
I noticed that, despite having code to avoid rerunning when the input files
are older than the .def file, it always runs.
# if the def file exists and is newer than all input object files, skip
# its creation
if (-f $deffile
&& (-M $deffile > max(map { -M } <$ARGV[0]/*.obj>)))
{
print "Not re-generating $defname.DEF, file already exists.\n";
exit(0);
}
My understanding of -M is that it returns the time delta between the file
modification and the start of the script. Which makes the use of max() bogus,
since it'll return the oldest time any input has been modified, not the
newest. And the condition needs to be inverted, because we want to skip the
work if $deffile is *newer*, right?
Am I missing something here?
I'm tempted to just remove the not-regenerating logic - gendef.pl shouldn't
run if there's nothing to do, and it'll e.g. not notice if there's an
additional input that wasn't there during the last invocation of gendef.pl.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-08-09 12:37 Andrew Dunstan <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: Andrew Dunstan @ 2022-08-09 12:37 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers; Magnus Hagander <[email protected]>
On 2022-08-09 Tu 03:10, Andres Freund wrote:
> Hi,
>
> I was looking at re-unifying gendef2.pl that the meson patchset had introduced
> for temporary ease during hacking with gendef.pl. Testing that I noticed that
> either I and my machine is very confused, or gendef.pl's check whether it can
> skip work is bogus.
>
> I noticed that, despite having code to avoid rerunning when the input files
> are older than the .def file, it always runs.
>
> # if the def file exists and is newer than all input object files, skip
> # its creation
> if (-f $deffile
> && (-M $deffile > max(map { -M } <$ARGV[0]/*.obj>)))
> {
> print "Not re-generating $defname.DEF, file already exists.\n";
> exit(0);
> }
>
> My understanding of -M is that it returns the time delta between the file
> modification and the start of the script. Which makes the use of max() bogus,
> since it'll return the oldest time any input has been modified, not the
> newest. And the condition needs to be inverted, because we want to skip the
> work if $deffile is *newer*, right?
>
> Am I missing something here?
No, you're right, this is bogus. Reversing the test and using min
instead of max is the obvious fix.
> I'm tempted to just remove the not-regenerating logic - gendef.pl shouldn't
> run if there's nothing to do, and it'll e.g. not notice if there's an
> additional input that wasn't there during the last invocation of gendef.pl.
>
Maybe, need to think about that more.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-08-10 17:19 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 1 reply; 130+ messages in thread
From: Andres Freund @ 2022-08-10 17:19 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-06-02 10:26:09 -0700, Andres Freund wrote:
> > Could we have the meson build check that, say, if gram.c exists it
> > is newer than gram.y? Or get it to ignore an in-tree gram.c?
>
> I suspect the problem with ignoring is gram.h, that's probably a bit harder to
> ignore.
I tried to ignore various generated files in the source tree, but I don't
think it's doable for all of them. Consider
e.g. src/backend/utils/misc/guc-file.c which is gets built via #include
"guc-file.c" from gram.c
Because it's a "" include, the search path starts in the current directory and
only then -I is searched. To my knowledge there's no way of changing
that. Quoting the gcc manpage:
-I dir
-iquote dir
-isystem dir
-idirafter dir
Add the directory dir to the list of directories to be searched for header files during preprocessing. If dir begins with = or $SYSROOT, then
the = or $SYSROOT is replaced by the sysroot prefix; see --sysroot and -isysroot.
Directories specified with -iquote apply only to the quote form of the directive, "#include "file"". Directories specified with -I, -isystem, or
-idirafter apply to lookup for both the "#include "file"" and "#include <file>" directives.
You can specify any number or combination of these options on the command line to search for header files in several directories. The lookup
order is as follows:
1. For the quote form of the include directive, the directory of the current file is searched first.
2. For the quote form of the include directive, the directories specified by -iquote options are searched in left-to-right order, as they appear
on the command line.
3. Directories specified with -I options are scanned in left-to-right order.
[...]
Except for copying guc.c from source to build tree before building, I don't
see a way of ignoring the in-build-tree guc-file.c.
Not sure what a good way of dealing with this is. For now I'll make it just
error out if there's any known such file in the source tree, but that's not a
good solution forever. If it were just "normal" build leftovers I'd propose
to (optionally) just remove them, but that's not good for tarballs.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-08-11 03:33 John Naylor <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 130+ messages in thread
From: John Naylor @ 2022-08-11 03:33 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
On Thu, Aug 11, 2022 at 12:19 AM Andres Freund <[email protected]> wrote:
> I tried to ignore various generated files in the source tree, but I don't
> think it's doable for all of them. Consider
> e.g. src/backend/utils/misc/guc-file.c which is gets built via #include
> "guc-file.c" from gram.c
With a bit of work, we could probably get rid of those includes. See
27199058d98ef7f for one example.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-08-11 03:37 Tom Lane <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 2 replies; 130+ messages in thread
From: Tom Lane @ 2022-08-11 03:37 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
John Naylor <[email protected]> writes:
> With a bit of work, we could probably get rid of those includes. See
> 27199058d98ef7f for one example.
Yeah --- it would mean creating gram.h files for all the bison grammars
not just a few of them, but it's certainly do-able if there's motivation
to make the changes. Most of the files that are done that way date
from before we knew about flex's %top.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-08-11 03:45 Tom Lane <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Tom Lane @ 2022-08-11 03:45 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
I wrote:
> John Naylor <[email protected]> writes:
>> With a bit of work, we could probably get rid of those includes. See
>> 27199058d98ef7f for one example.
> Yeah --- it would mean creating gram.h files for all the bison grammars
> not just a few of them, but it's certainly do-able if there's motivation
> to make the changes. Most of the files that are done that way date
> from before we knew about flex's %top.
BTW, 72b1e3a21 is another useful precedent in this area.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-08-11 03:57 John Naylor <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 2 replies; 130+ messages in thread
From: John Naylor @ 2022-08-11 03:57 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
On Thu, Aug 11, 2022 at 10:37 AM Tom Lane <[email protected]> wrote:
>
> John Naylor <[email protected]> writes:
> > With a bit of work, we could probably get rid of those includes. See
> > 27199058d98ef7f for one example.
>
> Yeah --- it would mean creating gram.h files for all the bison grammars
> not just a few of them, but it's certainly do-able if there's motivation
> to make the changes. Most of the files that are done that way date
> from before we knew about flex's %top.
I'll volunteer to work on this unless an easier solution happens to
come along in the next couple days. (aside: guc-file.l doesn't have a
grammar, so not yet sure if that makes the issue easier or harder...)
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-08-11 04:07 Andres Freund <[email protected]>
parent: John Naylor <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Andres Freund @ 2022-08-11 04:07 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Tom Lane <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-08-11 10:57:33 +0700, John Naylor wrote:
> I'll volunteer to work on this unless an easier solution happens to
> come along in the next couple days.
Cool!
> (aside: guc-file.l doesn't have a grammar, so not yet sure if that makes the
> issue easier or harder...)
I think we should consider compiling it separately from guc.c. guc.c already
compiles quite slowly (iirc beat only by ecpg and main grammar), and it's a
relatively commonly changed source file.
It might even be a good idea to split guc.c so it only contains the settings
arrays + direct dependencies...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-08-11 04:32 Tom Lane <[email protected]>
parent: John Naylor <[email protected]>
1 sibling, 0 replies; 130+ messages in thread
From: Tom Lane @ 2022-08-11 04:32 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
John Naylor <[email protected]> writes:
> I'll volunteer to work on this unless an easier solution happens to
> come along in the next couple days. (aside: guc-file.l doesn't have a
> grammar, so not yet sure if that makes the issue easier or harder...)
That one's probably mostly about the issue mentioned in the other
commit you identified. Without %top, it's impossible to make a
standalone flex module honor the rule about thou-shalt-have-no-
other-includes-before-postgres.h. So embedding it in some other
file was originally a necessity for that. Now that we know how
to fix that, it's just a matter of making sure that any other stuff
the scanner needs is available from a .h file.
regards, tom lane
^ permalink raw reply [nested|flat] 130+ messages in thread
* Re: [RFC] building postgres with meson
@ 2022-08-21 00:42 Andres Freund <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 0 replies; 130+ messages in thread
From: Andres Freund @ 2022-08-21 00:42 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers; Magnus Hagander <[email protected]>
Hi,
On 2022-08-09 08:37:16 -0400, Andrew Dunstan wrote:
> On 2022-08-09 Tu 03:10, Andres Freund wrote:
> > Hi,
> >
> > I was looking at re-unifying gendef2.pl that the meson patchset had introduced
> > for temporary ease during hacking with gendef.pl. Testing that I noticed that
> > either I and my machine is very confused, or gendef.pl's check whether it can
> > skip work is bogus.
> >
> > I noticed that, despite having code to avoid rerunning when the input files
> > are older than the .def file, it always runs.
> >
> > # if the def file exists and is newer than all input object files, skip
> > # its creation
> > if (-f $deffile
> > && (-M $deffile > max(map { -M } <$ARGV[0]/*.obj>)))
> > {
> > print "Not re-generating $defname.DEF, file already exists.\n";
> > exit(0);
> > }
> >
> > My understanding of -M is that it returns the time delta between the file
> > modification and the start of the script. Which makes the use of max() bogus,
> > since it'll return the oldest time any input has been modified, not the
> > newest. And the condition needs to be inverted, because we want to skip the
> > work if $deffile is *newer*, right?
> >
> > Am I missing something here?
>
>
> No, you're right, this is bogus. Reversing the test and using min
> instead of max is the obvious fix.
>
>
> > I'm tempted to just remove the not-regenerating logic - gendef.pl shouldn't
> > run if there's nothing to do, and it'll e.g. not notice if there's an
> > additional input that wasn't there during the last invocation of gendef.pl.
> >
>
> Maybe, need to think about that more.
Any thoughts?
I'd like to commit 0003 in
https://postgr.es/m/20220811002012.ju3rrz47i2e5tdha%40awork3.anarazel.de
fairly soon.
I did fix the bogus "die" message I added during some debugging since posting
that...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 130+ messages in thread
end of thread, other threads:[~2022-08-21 00:42 UTC | newest]
Thread overview: 130+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 09:08 ` Andres Freund <[email protected]>
2021-10-12 13:30 ` Peter Eisentraut <[email protected]>
2021-10-12 15:00 ` Robert Haas <[email protected]>
2021-10-12 15:47 ` Tom Lane <[email protected]>
2021-10-12 16:15 ` Andres Freund <[email protected]>
2021-10-12 19:01 ` Andres Freund <[email protected]>
2021-10-13 11:54 ` Daniel Gustafsson <[email protected]>
2021-10-12 15:08 ` Andrew Dunstan <[email protected]>
2021-10-12 15:21 ` Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Andres Freund <[email protected]>
2021-10-12 23:19 ` Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Andres Freund <[email protected]>
2021-10-13 21:58 ` Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 13:19 ` Josef Šimánek <[email protected]>
2021-10-14 13:29 ` Alvaro Herrera <[email protected]>
2021-10-14 17:24 ` Andres Freund <[email protected]>
2021-10-14 19:36 ` Josef Šimánek <[email protected]>
2021-10-14 22:08 ` Tom Lane <[email protected]>
2021-10-14 23:38 ` Andres Freund <[email protected]>
2021-10-14 23:51 ` Tom Lane <[email protected]>
2021-10-14 13:32 ` Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 19:29 ` Josef Šimánek <[email protected]>
2021-10-14 17:26 ` Andres Freund <[email protected]>
2021-10-12 15:28 ` Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Andres Freund <[email protected]>
2021-10-12 18:09 ` Andres Freund <[email protected]>
2021-10-12 18:37 ` Andrew Dunstan <[email protected]>
2021-10-12 19:16 ` Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Andres Freund <[email protected]>
2021-10-12 20:02 ` Andrew Dunstan <[email protected]>
2021-10-12 20:42 ` Andres Freund <[email protected]>
2021-10-13 01:03 ` Andres Freund <[email protected]>
2021-10-13 12:55 ` Andrew Dunstan <[email protected]>
2021-10-13 17:26 ` Andres Freund <[email protected]>
2021-10-13 20:06 ` Andrew Dunstan <[email protected]>
2021-10-13 21:46 ` Andres Freund <[email protected]>
2021-10-13 23:11 ` Andrew Dunstan <[email protected]>
2021-10-14 22:19 ` Andrew Dunstan <[email protected]>
2021-10-12 18:11 ` Andrew Dunstan <[email protected]>
2021-10-12 18:23 ` Andres Freund <[email protected]>
2021-10-12 18:42 ` Andrew Dunstan <[email protected]>
2021-10-12 19:55 ` John Naylor <[email protected]>
2021-10-12 20:59 ` Andres Freund <[email protected]>
2021-10-13 15:51 ` John Naylor <[email protected]>
2021-10-13 16:37 ` Andres Freund <[email protected]>
2021-10-13 17:19 ` John Naylor <[email protected]>
2021-10-13 17:42 ` Andres Freund <[email protected]>
2021-10-13 18:40 ` John Naylor <[email protected]>
2021-10-14 19:14 ` John Naylor <[email protected]>
2021-10-14 20:34 ` Andres Freund <[email protected]>
2021-10-14 20:54 ` John Naylor <[email protected]>
2021-10-14 21:16 ` John Naylor <[email protected]>
2021-10-14 21:48 ` Tom Lane <[email protected]>
2021-10-14 21:41 ` Sergey Shinderuk <[email protected]>
2021-10-14 22:00 ` Tom Lane <[email protected]>
2021-10-14 22:23 ` Thomas Munro <[email protected]>
2021-10-14 22:40 ` Tom Lane <[email protected]>
2021-10-14 23:04 ` Tom Lane <[email protected]>
2021-10-15 00:36 ` Thomas Munro <[email protected]>
2021-10-15 02:46 ` Tom Lane <[email protected]>
2021-10-15 03:20 ` Andres Freund <[email protected]>
2021-10-14 23:15 ` Andres Freund <[email protected]>
2021-10-14 23:23 ` Tom Lane <[email protected]>
2021-10-15 18:50 ` Andres Freund <[email protected]>
2021-10-15 22:36 ` Andres Freund <[email protected]>
2021-10-15 22:47 ` Andres Freund <[email protected]>
2021-10-14 22:55 ` Andres Freund <[email protected]>
2021-10-14 23:27 ` John Naylor <[email protected]>
2021-10-15 00:02 ` Andres Freund <[email protected]>
2021-10-14 04:27 ` Thomas Munro <[email protected]>
2021-10-21 21:48 ` Andres Freund <[email protected]>
2021-10-22 15:55 ` John Naylor <[email protected]>
2021-10-19 18:34 ` Andres Freund <[email protected]>
2021-10-19 19:22 ` Tom Lane <[email protected]>
2021-10-19 21:57 ` John Naylor <[email protected]>
2021-10-20 00:31 ` Andres Freund <[email protected]>
2021-10-20 01:10 ` Tom Lane <[email protected]>
2021-10-20 01:35 ` Andres Freund <[email protected]>
2021-10-20 01:08 ` Andres Freund <[email protected]>
2021-10-20 01:26 ` Tom Lane <[email protected]>
2021-10-20 01:49 ` Andres Freund <[email protected]>
2021-10-20 02:04 ` Tom Lane <[email protected]>
2021-10-20 02:41 ` Andres Freund <[email protected]>
2021-10-20 16:01 ` Andres Freund <[email protected]>
2021-10-31 23:24 ` Re: [RFC] building postgres with meson -v Andres Freund <[email protected]>
2021-11-15 18:34 ` Andres Freund <[email protected]>
2021-11-15 19:11 ` Tom Lane <[email protected]>
2021-11-15 19:23 ` Andres Freund <[email protected]>
2021-11-15 19:48 ` Robert Haas <[email protected]>
2021-11-15 22:08 ` Thomas Munro <[email protected]>
2021-11-15 22:34 ` Tom Lane <[email protected]>
2021-11-15 22:46 ` Andres Freund <[email protected]>
2021-11-15 22:52 ` Thomas Munro <[email protected]>
2021-11-18 20:44 ` Thomas Munro <[email protected]>
2022-05-26 00:47 ` Andres Freund <[email protected]>
2022-05-26 01:38 ` Tom Lane <[email protected]>
2022-05-26 02:41 ` Andres Freund <[email protected]>
2022-05-26 02:58 ` Tom Lane <[email protected]>
2022-05-26 08:47 ` Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Andres Freund <[email protected]>
2022-05-31 13:49 ` Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Andres Freund <[email protected]>
2022-06-01 09:39 ` Aleksander Alekseev <[email protected]>
2022-06-01 21:05 ` Andres Freund <[email protected]>
2022-06-02 12:34 ` Aleksander Alekseev <[email protected]>
2022-06-02 15:22 ` Andres Freund <[email protected]>
2022-06-02 16:39 ` Andres Freund <[email protected]>
2022-06-02 17:08 ` Tom Lane <[email protected]>
2022-06-02 17:26 ` Andres Freund <[email protected]>
2022-06-02 17:33 ` Tom Lane <[email protected]>
2022-06-02 17:48 ` Andres Freund <[email protected]>
2022-06-02 19:05 ` Tom Lane <[email protected]>
2022-06-02 19:17 ` Andres Freund <[email protected]>
2022-06-02 19:53 ` Tom Lane <[email protected]>
2022-06-02 21:13 ` Andres Freund <[email protected]>
2022-06-03 09:35 ` Aleksander Alekseev <[email protected]>
2022-06-03 16:23 ` Andres Freund <[email protected]>
2022-08-10 17:19 ` Andres Freund <[email protected]>
2022-08-11 03:33 ` John Naylor <[email protected]>
2022-08-11 03:37 ` Tom Lane <[email protected]>
2022-08-11 03:45 ` Tom Lane <[email protected]>
2022-08-11 03:57 ` John Naylor <[email protected]>
2022-08-11 04:07 ` Andres Freund <[email protected]>
2022-08-11 04:32 ` Tom Lane <[email protected]>
2022-08-09 07:10 ` Andres Freund <[email protected]>
2022-08-09 12:37 ` Andrew Dunstan <[email protected]>
2022-08-21 00:42 ` Andres Freund <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox