public inbox for [email protected]
help / color / mirror / Atom feed[RFC] building postgres with meson
139+ messages / 13 participants
[nested] [flat]
* [RFC] building postgres with meson
@ 2021-10-12 08:37 Andres Freund <[email protected]>
2021-10-12 09:08 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 13:30 ` Re: [RFC] building postgres with meson Peter Eisentraut <[email protected]>
2021-10-12 15:08 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-09 07:10 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 10 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-12 08:37 UTC (permalink / raw)
To: pgsql-hackers
Hi,
For the last year or so I've on and off tinkered with $subject. I think
it's in a state worth sharing now. First, let's look at a little
comparison.
My workstation:
non-cached configure:
current: 11.80s
meson: 6.67s
non-cached build (world-bin):
current: 40.46s
ninja: 7.31s
no-change build:
current: 1.17s
ninja: 0.06s
test world:
current: 105s
meson: 63s
What actually started to motivate me however were the long times windows
builds took to come back with testsresults. On CI, with the same machine
config:
build:
current: 202s (doesn't include genbki etc)
meson+ninja: 140s
meson+msbuild: 206s
test:
current: 1323s (many commands)
meson: 903s (single command)
(note that the test comparison isn't quite fair - there's a few tests
missing, but it's just small contrib ones afaik)
The biggest difference to me however is not the speed, but how readable
the output is.
Running the tests with meson in a terminal, shows the number of tests
that completed out of how many total, how much time has passed, how long
the currently running tests already have been running.
At the end of a testrun a count of tests is shown:
188/189 postgresql:tap+pg_basebackup / pg_basebackup/t/010_pg_basebackup.pl OK 39.51s 110 subtests passed
189/189 postgresql:isolation+snapshot_too_old / snapshot_too_old/isolation OK 62.93s
Ok: 188
Expected Fail: 0
Fail: 1
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Full log written to /tmp/meson/meson-logs/testlog.txt
The log has the output of the tests and ends with:
Summary of Failures:
120/189 postgresql:tap+recovery / recovery/t/007_sync_rep.pl ERROR 7.16s (exit status 255 or signal 127 SIGinvalid)
Quite the difference to make check-world -jnn output.
So, now that the teasing is done, let me explain a bit what lead me down
this path:
Autoconf + make is not being actively developed. Especially autoconf is
*barely* in maintenance mode - despite many shortcomings and bugs. It's
also technology that very few want to use - autoconf m4 is scary, and
it's scarier for people that started more recently than a lot of us
committers for example.
Recursive make as we use it is hard to get right. One reason the clean
make build is so slow compared to meson is that we had to resort to
.NOTPARALLEL to handle dependencies in a bunch of places. And despite
that, I quite regularly see incremental build failures that can be
resolved by retrying the build.
While we have incremental build via --enable-depend, they don't work
that reliable (i.e. misses necessary rebuilds) and yet is often too
aggressive. More modern build system can keep track of the precise
command used to build a target and rebuild it when that command changes.
We also don't just have the autoconf / make buildsystem, there's also
the msvc project generator - something most of us unix-y folks do not
like to touch. I think that, combined with there being no easy way to
run all tests, and it being just different, really hurt our windows
developer appeal (and subsequently the quality of postgres on
windows). I'm not saying this to ding the project generator - that was
well before there were decent "meta" buildsystems out there (and in some
ways it is a small one itself).
The last big issue I have with the current situation is that there's no
good test integration. make check-world output is essentially unreadable
/ not automatically parseable. Which led to the buildfarm having a
separate list of things it needs to test, so that failures can be
pinpointed and paired with appropriate logs. That approach unfortunately
doesn't scale well to multi-core CPUs, slowing down the buildfarm by a
fair bit.
This all led to me to experiment with improvements. I tried a few
somewhat crazy but incremental things like converting our buildsystem to
non-recursive make (I got it to build the backend, but it's too hard to
do manually I think), or to not run tests during the recursive make
check-world, but to append commands to a list of tests, that then is run
by a helper (can kinda be made to work). In the end I concluded that
the amount of time we'd need to invest to maintain our more-and-more
custom buildsystem going forward doesn't make sense.
Which lead me to look around and analyze which other buildsystems there
are that could make some sense for us. The halfway decent list includes,
I think:
1) cmake
2) bazel
3) meson
cmake would be a decent choice, I think. However, I just can't fully
warm up to it. Something about it just doesn't quite sit right with
me. That's not a good enough reason to prevent others from suggesting to
use it, but it's good enough to justify not investing a lot of time in
it myself.
Bazel has some nice architectural properties. But it requires a JVM to
run - I think that basically makes it insuitable for us. And the build
information seems quite arduous to maintain too.
Which left me with meson. It is a meta-buildsystem that can do the
actual work of building via ninja (the most common one, also targeted by
cmake), msbuild (visual studio project files, important for GUI work)
and xcode projects (I assume that's for a macos IDE, but I haven't tried
to use it). Meson roughly does what autoconf+automake did, in a
python-esque DSL, and outputs build-instructions for ninja / msbuild /
xcode. One interesting bit is that meson itself is written in python (
and fairly easy to contribute too - I got a few changes in now).
I don't think meson is perfect architecturally - e.g. its insistence on
not having functions ends up making it a bit harder to not end up
duplicating code. There's some user-interface oddities that are now hard
to fix fully, due to the faily wide usage. But all-in-all it's pretty
nice to use.
Its worth calling out that a lot of large open source projects have been
/ are migrating to meson. qemu/kvm, mesa (core part of graphics stack on
linux and also widely used in other platforms), a good chunk of GNOME,
and quite a few more. Due to that it seems unlikely to be abandoned
soon.
As far as I can tell the only OS that postgres currently supports that
meson doesn't support is HPUX. It'd likely be fairly easy to add
gcc-on-hpux support, a chunk more to add support for the proprietary
ones.
The attached patch (meson support is 0016, the rest is prerequisites
that aren't that interesting at this stage) converts most of postgres to
meson. There's a few missing contrib modules, only about half the
optional library dependencies are implemented, and I've only built on
x64. It builds on freebsd, linux, macos and windows (both ninja and
msbuild) and cross builds from linux to windows. Thomas helped make the
freebsd / macos pieces a reality, thanks!
I took a number of shortcuts (although there used to be a *lot*
more). So this shouldn't be reviewed to the normal standard of the
community - it's a prototype. But I think it's in a complete enough
shape that it allows to do a well-informed evaluation.
What doesn't yet work/ build:
- plenty optional libraries, contrib, NLS, docs build
- PGXS - and I don't yet know what to best do about it. One
backward-compatible way would be to continue use makefiles for pgxs,
but do the necessary replacement of Makefile.global.in via meson (and
not use that for postgres' own build). But that doesn't really
provide a nicer path for building postgres extensions on windows, so
it'd definitely not be a long-term path.
- JIT bitcode generation for anything but src/backend.
- anything but modern-ish x86. That's proably a small amount of work,
but something that needs to be done.
- exporting all symbols for extension modules on windows (the stuff for
postgres is implemented). Instead I marked the relevant symbols als
declspec(dllexport). I think we should do that regardless of the
buildsystem change. Restricting symbol visibility via gcc's
-fvisibility=hidden for extensions results in a substantially reduced
number of exported symbols, and even reduces object size (and I think
improves the code too). I'll send an email about that separately.
There's a lot more stuff to talk about, but I'll stop with a small bit
of instructions below:
Demo / instructions:
# Get code
git remote add andres [email protected]:anarazel/postgres.git
git fetch andres
git checkout --track andres/meson
# setup build directory
meson setup build --buildtype debug
cd build
# build (uses automatically as many cores as available)
ninja
# change configuration, build again
meson configure -Dssl=openssl
ninja
# run all tests
meson test
# run just recovery tests
meson test --suite setup --suite recovery
# list tests
meson test --list
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 09:08 ` Andres Freund <[email protected]>
9 siblings, 0 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-12 09:08 UTC (permalink / raw)
To: pgsql-hackers
Hi,
On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
> non-cached build (world-bin):
> current: 40.46s
> ninja: 7.31s
Interestingly this is pretty close to the minimum achievable on my
machine from the buildsystem perspective.
A build with -fuse-ld=lld, which the above didn't use, takes 6.979s. The
critical path is
bison gram.y -> gram.c 4.13s
gcc gram.c -> gram.o 2.05s
gcc postgres .... 0.317
A very helpful visualization is to transform ninja's build logs into a
tracefile with https://github.com/nico/ninjatracing
I attached an example - the trace.json.gz can be uploaded as-is to
https://ui.perfetto.dev/
It's quite a bit of of fun to look at imo.
There's a few other things quickly apparent:
- genbki prevents build progress due to dependencies on the generated
headers.
- the absolutely stupid way I implemented the python2->python3
regression test output conversion uses up a fair bit of resources
- tablecmds.c, pg_dump.c, xlog.c and a few other files are starting to
big enough to be problematic compile-time wise
Greetings,
Andres Freund
Attachments:
[application/gzip] trace.json.gz (23.6K, ../../[email protected]/2-trace.json.gz)
download
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 13:30 ` Peter Eisentraut <[email protected]>
2021-10-12 15:00 ` Re: [RFC] building postgres with meson Robert Haas <[email protected]>
2021-10-12 16:15 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
9 siblings, 2 replies; 139+ messages in thread
From: Peter Eisentraut @ 2021-10-12 13:30 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers
On 12.10.21 10:37, Andres Freund wrote:
> For the last year or so I've on and off tinkered with $subject. I think
> it's in a state worth sharing now. First, let's look at a little
> comparison.
I played with $subject a few years ago and liked it. I think, like you
said, meson is the best way forward. I support this project.
One problem I noticed back then was that some choices that we currently
determine ourselves in configure or the makefiles are hardcoded in
meson. For example, at the time, gcc on macOS was not supported. Meson
thought, if you are on macOS, you are surely using the Apple compiler,
and it supports these options. Fixing that required patches deep in the
bowels of the meson source code (and, in practice, waiting for a new
release etc.). I strongly suspect this isn't the only such problem.
For example, the shared library build behavior has been carefully tuned
in opinionated ways. With the autotools chain, one can override
anything with enough violence; so we have always felt free to do that.
I haven't followed it in a while, so I don't know what the situation is
now; but it is a concern, because we have always felt free to try new
and unusual build tools (Sun compiler, Intel compiler,
clang-when-it-was-new) early without waiting for anyone else.
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 13:30 ` Re: [RFC] building postgres with meson Peter Eisentraut <[email protected]>
@ 2021-10-12 15:00 ` Robert Haas <[email protected]>
2021-10-12 15:47 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Robert Haas @ 2021-10-12 15:00 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
On Tue, Oct 12, 2021 at 9:31 AM Peter Eisentraut
<[email protected]> wrote:
> One problem I noticed back then was that some choices that we currently
> determine ourselves in configure or the makefiles are hardcoded in
> meson. For example, at the time, gcc on macOS was not supported. Meson
> thought, if you are on macOS, you are surely using the Apple compiler,
> and it supports these options. Fixing that required patches deep in the
> bowels of the meson source code (and, in practice, waiting for a new
> release etc.). I strongly suspect this isn't the only such problem.
> For example, the shared library build behavior has been carefully tuned
> in opinionated ways. With the autotools chain, one can override
> anything with enough violence; so we have always felt free to do that.
> I haven't followed it in a while, so I don't know what the situation is
> now; but it is a concern, because we have always felt free to try new
> and unusual build tools (Sun compiler, Intel compiler,
> clang-when-it-was-new) early without waiting for anyone else.
I think we're going to need some solution to this problem. We have too
many people here with strong opinions about questions like this for me
to feel good about the idea that we're going to collectively be OK
with leaving these sorts of decisions up to some other project.
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 13:30 ` Re: [RFC] building postgres with meson Peter Eisentraut <[email protected]>
2021-10-12 15:00 ` Re: [RFC] building postgres with meson Robert Haas <[email protected]>
@ 2021-10-12 15:47 ` Tom Lane <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Tom Lane @ 2021-10-12 15:47 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
Robert Haas <[email protected]> writes:
> I think we're going to need some solution to this problem. We have too
> many people here with strong opinions about questions like this for me
> to feel good about the idea that we're going to collectively be OK
> with leaving these sorts of decisions up to some other project.
Agreed. I'm willing to put up with the costs of moving to some
other build system, but not if it dictates choices we don't want to
make about the end products.
> From my point of view, the time it takes to run configure is annoying,
> but the build time is pretty fine. On my system, configure takes about
> 33 seconds, and a full rebuild with 'make -j8' takes 14.5 seconds (I
> am using ccache). Moreover, most of the time when I run make, I'm only
> doing a partial rebuild, so it's near-instantaneous.
Read about Autoconf's --cache-file option. That and ccache are
absolutely essential tools IMO.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 13:30 ` Re: [RFC] building postgres with meson Peter Eisentraut <[email protected]>
@ 2021-10-12 16:15 ` Andres Freund <[email protected]>
2021-10-12 19:01 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-12 16:15 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 15:30:57 +0200, Peter Eisentraut wrote:
> I played with $subject a few years ago and liked it. I think, like you
> said, meson is the best way forward. I support this project.
Cool.
> One problem I noticed back then was that some choices that we currently
> determine ourselves in configure or the makefiles are hardcoded in meson.
Yea, there's some of that. I think some degree of reduction in flexibility is
needed to realistically target multiple "backend" build-system like visual
studio project files etc. but I wish there were a bit less of that
nonetheless.
> For example, at the time, gcc on macOS was not supported. Meson thought, if
> you are on macOS, you are surely using the Apple compiler, and it supports
> these options.
I'm pretty sure this one now can just be overridden with CC=gcc. It can on
linux and windows, but I don't have ready interactive access with a mac
(leaving cirrus asside, which now has a "start a terminal" option...).
> For example, the shared library build behavior has been carefully tuned in
> opinionated ways. With the autotools chain, one can override anything with
> enough violence; so we have always felt free to do that. I haven't followed
> it in a while, so I don't know what the situation is now; but it is a
> concern, because we have always felt free to try new and unusual build tools
> (Sun compiler, Intel compiler, clang-when-it-was-new) early without waiting
> for anyone else.
It's possible to just take over building e.g. shared libraries ourselves with
custom targets. Although it'd be a bit annoying to do. The bigger problem is
that that e.g. wouldn't play that nicely with generating visual studio
projects, which require to generate link steps in a certain way. It'd build,
but the GUI might loose some of its options. Etc.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 13:30 ` Re: [RFC] building postgres with meson Peter Eisentraut <[email protected]>
2021-10-12 16:15 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 19:01 ` Andres Freund <[email protected]>
2021-10-13 11:54 ` Re: [RFC] building postgres with meson Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-12 19:01 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 09:15:41 -0700, Andres Freund wrote:
> > For example, at the time, gcc on macOS was not supported. Meson thought, if
> > you are on macOS, you are surely using the Apple compiler, and it supports
> > these options.
>
> I'm pretty sure this one now can just be overridden with CC=gcc. It can on
> linux and windows, but I don't have ready interactive access with a mac
> (leaving cirrus asside, which now has a "start a terminal" option...).
It was a tad more complicated. But only because it took me a while to figure
out how to make gcc on macos actually work, independent of meson. Initially
gcc was always failing with errors about not finding the linker, and
installing binutils was a dead end.
Turns out just using a gcc at a specific path doesn't work, it ends up using
wrong internal binaries or something like that.
Once I got to that, the meson part was easy:
$ export PATH="/usr/local/opt/gcc/bin:$PATH"
$ CC=gcc-11 meson setup build-gcc
...
C compiler for the host machine: gcc-11 (gcc 11.2.0 "gcc-11 (Homebrew GCC 11.2.0) 11.2.0")
...
$ cd build-gcc
$ ninja test
...
181/181 postgresql:tap+subscription / subscription/t/100_bugs.pl OK 17.83s 5 subtests passed
Ok: 180
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 1
Timeout: 0
One thing that is nice with meson's testrunner is that it can parse the output
of tap tests and recognizes the number of completed / failed subtests. I
wonder whether we could make pg_regress' output tap compliant without the
output quality suffering too much.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 13:30 ` Re: [RFC] building postgres with meson Peter Eisentraut <[email protected]>
2021-10-12 16:15 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:01 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-13 11:54 ` Daniel Gustafsson <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Daniel Gustafsson @ 2021-10-13 11:54 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; pgsql-hackers
> On 12 Oct 2021, at 21:01, Andres Freund <[email protected]> wrote:
> One thing that is nice with meson's testrunner is that it can parse the output
> of tap tests and recognizes the number of completed / failed subtests. I
> wonder whether we could make pg_regress' output tap compliant without the
> output quality suffering too much.
I added a --tap option for TAP output to pg_regress together with Jinbao Chen
for giggles and killing some time a while back. It's not entirely done and
sort of PoC, but most of it works. Might not be of interest here, but in case
it is I've refreshed it slightly and rebased it. There might be better ways to
do it, but the aim was to make the diff against the guts of pg_regress small
and instead extract output functions for the different formats.
It omits the test timings, but that could be added either as a diagnostic line
following each status or as a YAML block in TAP 13 (the attached is standard
TAP, not version 13 but the change would be trivial).
If it's helpful and there's any interest for this I'm happy to finish it up now.
One thing that came out of this, is that we don't really handle the ignored
tests in the way the code thinks it does for normal output, the attached treats
ignored tests as SKIP tests.
--
Daniel Gustafsson https://vmware.com/
Attachments:
[application/octet-stream] pg_regress_tap.diff (14.1K, ../../[email protected]/2-pg_regress_tap.diff)
download | inline diff:
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index 05296f7ee1..dc60edf82e 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -95,6 +95,7 @@ static char *dlpath = PKGLIBDIR;
static char *user = NULL;
static _stringlist *extraroles = NULL;
static char *config_auth_datadir = NULL;
+static bool tap = false;
/* internal variables */
static const char *progname;
@@ -120,9 +121,71 @@ static int fail_ignore_count = 0;
static bool directory_exists(const char *dir);
static void make_directory(const char *dir);
-static void header(const char *fmt,...) pg_attribute_printf(1, 2);
+struct output_func
+{
+ void (*header)(const char *line);
+ void (*footer)(const char *difffilename, const char *logfilename);
+ void (*comment)(const char *comment);
+
+ void (*test_status_preamble)(const char *testname);
+
+ void (*test_status_ok)(const char *testname);
+ void (*test_status_failed)(const char *testname);
+ void (*test_status_ignored)(const char *testname);
+
+ void (*test_runtime)(const char *testname, double runtime);
+};
+
+
+void (*test_runtime)(const char *testname, double runtime);
+/* Text output format */
+static void header_text(const char *line);
+static void footer_text(const char *difffilename, const char *logfilename);
+static void comment_text(const char *comment);
+static void test_status_preamble_text(const char *testname);
+static void test_status_ok_text(const char *testname);
+static void test_status_failed_text(const char *testname);
+static void test_runtime_text(const char *testname, double runtime);
+
+struct output_func output_func_text =
+{
+ header_text,
+ footer_text,
+ comment_text,
+ test_status_preamble_text,
+ test_status_ok_text,
+ test_status_failed_text,
+ NULL,
+ test_runtime_text
+};
+
+/* TAP output format */
+static void header_tap(const char *line);
+static void footer_tap(const char *difffilename, const char *logfilename);
+static void comment_tap(const char *comment);
+static void test_status_ok_tap(const char *testname);
+static void test_status_failed_tap(const char *testname);
+static void test_status_ignored_tap(const char *testname);
+
+struct output_func output_func_tap =
+{
+ header_tap,
+ footer_tap,
+ comment_tap,
+ NULL,
+ test_status_ok_tap,
+ test_status_failed_tap,
+ test_status_ignored_tap,
+ NULL
+};
+
+struct output_func *output = &output_func_text;
+
+static void test_status_ok(const char *testname);
+
static void status(const char *fmt,...) pg_attribute_printf(1, 2);
static void psql_command(const char *database, const char *query,...) pg_attribute_printf(2, 3);
+static void status_end(void);
/*
* allow core files if possible.
@@ -206,18 +269,227 @@ split_to_stringlist(const char *s, const char *delim, _stringlist **listhead)
/*
* Print a progress banner on stdout.
*/
+static void
+header_text(const char *line)
+{
+ fprintf(stdout, "============== %-38s ==============\n", line);
+ fflush(stdout);
+}
+
+static void
+header_tap(const char *line)
+{
+ fprintf(stdout, "# %s\n", line);
+ fflush(stdout);
+}
+
static void
header(const char *fmt,...)
{
char tmp[64];
va_list ap;
+ if (!output->header)
+ return;
+
va_start(ap, fmt);
vsnprintf(tmp, sizeof(tmp), fmt, ap);
va_end(ap);
- fprintf(stdout, "============== %-38s ==============\n", tmp);
- fflush(stdout);
+ output->header(tmp);
+}
+
+static void
+footer_tap(const char *difffilename, const char *logfilename)
+{
+ status("1..%i\n", (fail_count + fail_ignore_count + success_count));
+ status_end();
+}
+
+static void
+footer(const char *difffilename, const char *logfilename)
+{
+ if (output->footer)
+ output->footer(difffilename, logfilename);
+}
+
+static void
+comment_text(const char *comment)
+{
+ status("%s", comment);
+}
+
+static void
+comment_tap(const char *comment)
+{
+ status("# %s", comment);
+}
+
+static void
+comment(const char *fmt,...)
+{
+ char tmp[256];
+ va_list ap;
+
+ if (!output->comment)
+ return;
+
+ va_start(ap, fmt);
+ vsnprintf(tmp, sizeof(tmp), fmt, ap);
+ va_end(ap);
+
+ output->comment(tmp);
+}
+
+static void
+test_status_preamble_text(const char *testname)
+{
+ status(_("test %-28s ... "), testname);
+}
+
+static void
+test_status_preamble(const char *testname)
+{
+ if (output->test_status_preamble)
+ output->test_status_preamble(testname);
+}
+
+static void
+test_status_ok_tap(const char *testname)
+{
+ /* There is no NLS translation here as "ok" is a protocol message */
+ status("ok %i - %s",
+ (fail_count + fail_ignore_count + success_count),
+ testname);
+}
+
+static void
+test_status_ok_text(const char *testname)
+{
+ (void) testname; /* unused */
+ status(_("ok ")); /* align with FAILED */
+}
+
+static void
+test_status_ok(const char *testname)
+{
+ success_count++;
+ if (output->test_status_ok)
+ output->test_status_ok(testname);
+}
+
+static void
+test_status_failed_tap(const char *testname)
+{
+ status("not ok %i - %s",
+ (fail_count + fail_ignore_count + success_count),
+ testname);
+}
+
+static void
+test_status_failed_text(const char *testname)
+{
+ status(_("FAILED"));
+}
+
+static void
+test_status_failed(const char *testname)
+{
+ fail_count++;
+ if (output->test_status_failed)
+ output->test_status_failed(testname);
+}
+
+static void
+test_status_ignored(const char *testname)
+{
+ fail_ignore_count++;
+ if (output->test_status_ignored)
+ output->test_status_ignored(testname);
+}
+
+static void
+test_status_ignored_tap(const char *testname)
+{
+ status("ok %i - %s # SKIP (ignored)",
+ (fail_count + fail_ignore_count + success_count),
+ testname);
+}
+
+static void
+test_runtime_text(const char *testname, double runtime)
+{
+ (void)testname;
+ status(_(" %8.0f ms"), runtime);
+}
+
+static void
+runtime(const char *testname, double runtime)
+{
+ if (output->test_runtime)
+ output->test_runtime(testname, runtime);
+}
+
+static void
+footer_text(const char *difffilename, const char *logfilename)
+{
+ char buf[256];
+
+ /*
+ * Emit nice-looking summary message
+ */
+ if (fail_count == 0 && fail_ignore_count == 0)
+ snprintf(buf, sizeof(buf),
+ _(" All %d tests passed. "),
+ success_count);
+ else if (fail_count == 0) /* fail_count=0, fail_ignore_count>0 */
+ snprintf(buf, sizeof(buf),
+ _(" %d of %d tests passed, %d failed test(s) ignored. "),
+ success_count,
+ success_count + fail_ignore_count,
+ fail_ignore_count);
+ else if (fail_ignore_count == 0) /* fail_count>0 && fail_ignore_count=0 */
+ snprintf(buf, sizeof(buf),
+ _(" %d of %d tests failed. "),
+ fail_count,
+ success_count + fail_count);
+ else
+ /* fail_count>0 && fail_ignore_count>0 */
+ snprintf(buf, sizeof(buf),
+ _(" %d of %d tests failed, %d of these failures ignored. "),
+ fail_count + fail_ignore_count,
+ success_count + fail_count + fail_ignore_count,
+ fail_ignore_count);
+
+ putchar('\n');
+ for (int i = strlen(buf); i > 0; i--)
+ putchar('=');
+ printf("\n%s\n", buf);
+ for (int i = strlen(buf); i > 0; i--)
+ putchar('=');
+ putchar('\n');
+ putchar('\n');
+
+ if (difffilename && logfilename)
+ {
+ printf(_("The differences that caused some tests to fail can be viewed in the\n"
+ "file \"%s\". A copy of the test summary that you see\n"
+ "above is saved in the file \"%s\".\n\n"),
+ difffilename, logfilename);
+ }
+}
+
+static void
+status_start(bool single, const char *testname)
+{
+ /* TAP only outputs after the test has finished */
+ if (tap)
+ return;
+
+ if (single)
+ status(_("test %-24s ... "), testname);
+ else
+ status(_(" %-24s ... "), testname);
}
/*
@@ -917,13 +1189,13 @@ initialize_environment(void)
#endif
if (pghost && pgport)
- printf(_("(using postmaster on %s, port %s)\n"), pghost, pgport);
+ comment(_("(using postmaster on %s, port %s)\n"), pghost, pgport);
if (pghost && !pgport)
- printf(_("(using postmaster on %s, default port)\n"), pghost);
+ comment(_("(using postmaster on %s, default port)\n"), pghost);
if (!pghost && pgport)
- printf(_("(using postmaster on Unix socket, port %s)\n"), pgport);
+ comment(_("(using postmaster on Unix socket, port %s)\n"), pgport);
if (!pghost && !pgport)
- printf(_("(using postmaster on Unix socket, default port)\n"));
+ comment(_("(using postmaster on Unix socket, default port)\n"));
}
convert_sourcefiles();
@@ -1167,9 +1439,10 @@ psql_command(const char *database, const char *query,...)
/* And now we can build and execute the shell command */
snprintf(psql_cmd, sizeof(psql_cmd),
- "\"%s%spsql\" -X -c \"%s\" \"%s\"",
+ "\"%s%spsql\" %s -X -c \"%s\" \"%s\"",
bindir ? bindir : "",
bindir ? "/" : "",
+ tap ? "-q" : "",
query_escaped,
database);
@@ -1704,6 +1977,9 @@ run_schedule(const char *schedule, test_start_function startfunc,
c++;
add_stringlist_item(&ignorelist, c);
+ test_status_ignored(c);
+ status_end();
+
/*
* Note: ignore: lines do not run the test, they just say that
* failure of this test when run later on is to be ignored. A bit
@@ -1762,7 +2038,7 @@ run_schedule(const char *schedule, test_start_function startfunc,
if (num_tests == 1)
{
- status(_("test %-28s ... "), tests[0]);
+ test_status_preamble(tests[0]);
pids[0] = (startfunc) (tests[0], &resultfiles[0], &expectfiles[0], &tags[0]);
INSTR_TIME_SET_CURRENT(starttimes[0]);
wait_for_tests(pids, statuses, stoptimes, NULL, 1);
@@ -1778,8 +2054,8 @@ run_schedule(const char *schedule, test_start_function startfunc,
{
int oldest = 0;
- status(_("parallel group (%d tests, in groups of %d): "),
- num_tests, max_connections);
+ comment(_("parallel group (%d tests, in groups of %d): "),
+ num_tests, max_connections);
for (i = 0; i < num_tests; i++)
{
if (i - oldest >= max_connections)
@@ -1799,7 +2075,7 @@ run_schedule(const char *schedule, test_start_function startfunc,
}
else
{
- status(_("parallel group (%d tests): "), num_tests);
+ comment(_("parallel group (%d tests): "), num_tests);
for (i = 0; i < num_tests; i++)
{
pids[i] = (startfunc) (tests[i], &resultfiles[i], &expectfiles[i], &tags[i]);
@@ -1818,7 +2094,7 @@ run_schedule(const char *schedule, test_start_function startfunc,
bool differ = false;
if (num_tests > 1)
- status(_(" %-28s ... "), tests[i]);
+ test_status_preamble(tests[i]);
/*
* Advance over all three lists simultaneously.
@@ -1858,27 +2134,18 @@ run_schedule(const char *schedule, test_start_function startfunc,
}
}
if (ignore)
- {
- status(_("failed (ignored)"));
- fail_ignore_count++;
- }
+ test_status_ignored(tests[i]);
else
- {
- status(_("FAILED"));
- fail_count++;
- }
+ test_status_failed(tests[i]);
}
else
- {
- status(_("ok ")); /* align with FAILED */
- success_count++;
- }
+ test_status_ok(tests[i]);
if (statuses[i] != 0)
log_child_failure(statuses[i]);
INSTR_TIME_SUBTRACT(stoptimes[i], starttimes[i]);
- status(_(" %8.0f ms"), INSTR_TIME_GET_MILLISEC(stoptimes[i]));
+ runtime(tests[i], INSTR_TIME_GET_MILLISEC(stoptimes[i]));
status_end();
}
@@ -1917,7 +2184,7 @@ run_single_test(const char *test, test_start_function startfunc,
*tl;
bool differ = false;
- status(_("test %-28s ... "), test);
+ test_status_preamble(test);
pid = (startfunc) (test, &resultfiles, &expectfiles, &tags);
INSTR_TIME_SET_CURRENT(starttime);
wait_for_tests(&pid, &exit_status, &stoptime, NULL, 1);
@@ -1947,15 +2214,9 @@ run_single_test(const char *test, test_start_function startfunc,
}
if (differ)
- {
- status(_("FAILED"));
- fail_count++;
- }
+ test_status_failed(test);
else
- {
- status(_("ok ")); /* align with FAILED */
- success_count++;
- }
+ test_status_ok(test);
if (exit_status != 0)
log_child_failure(exit_status);
@@ -2152,6 +2413,7 @@ regression_main(int argc, char *argv[],
{"config-auth", required_argument, NULL, 24},
{"max-concurrent-tests", required_argument, NULL, 25},
{"make-testtablespace-dir", no_argument, NULL, 26},
+ {"tap", no_argument, NULL, 27},
{NULL, 0, NULL, 0}
};
@@ -2285,6 +2547,9 @@ regression_main(int argc, char *argv[],
case 26:
make_testtablespace_dir = true;
break;
+ case 27:
+ tap = true;
+ break;
default:
/* getopt_long already emitted a complaint */
fprintf(stderr, _("\nTry \"%s -h\" for more information.\n"),
@@ -2311,6 +2576,9 @@ regression_main(int argc, char *argv[],
exit(0);
}
+ if (tap)
+ output = &output_func_tap;
+
if (temp_instance && !port_specified_by_user)
/*
@@ -2636,54 +2904,20 @@ regression_main(int argc, char *argv[],
fclose(logfile);
- /*
- * Emit nice-looking summary message
- */
- if (fail_count == 0 && fail_ignore_count == 0)
- snprintf(buf, sizeof(buf),
- _(" All %d tests passed. "),
- success_count);
- else if (fail_count == 0) /* fail_count=0, fail_ignore_count>0 */
- snprintf(buf, sizeof(buf),
- _(" %d of %d tests passed, %d failed test(s) ignored. "),
- success_count,
- success_count + fail_ignore_count,
- fail_ignore_count);
- else if (fail_ignore_count == 0) /* fail_count>0 && fail_ignore_count=0 */
- snprintf(buf, sizeof(buf),
- _(" %d of %d tests failed. "),
- fail_count,
- success_count + fail_count);
- else
- /* fail_count>0 && fail_ignore_count>0 */
- snprintf(buf, sizeof(buf),
- _(" %d of %d tests failed, %d of these failures ignored. "),
- fail_count + fail_ignore_count,
- success_count + fail_count + fail_ignore_count,
- fail_ignore_count);
-
- putchar('\n');
- for (i = strlen(buf); i > 0; i--)
- putchar('=');
- printf("\n%s\n", buf);
- for (i = strlen(buf); i > 0; i--)
- putchar('=');
- putchar('\n');
- putchar('\n');
-
- if (file_size(difffilename) > 0)
- {
- printf(_("The differences that caused some tests to fail can be viewed in the\n"
- "file \"%s\". A copy of the test summary that you see\n"
- "above is saved in the file \"%s\".\n\n"),
- difffilename, logfilename);
- }
- else
+ if (file_size(difffilename) <= 0)
{
unlink(difffilename);
unlink(logfilename);
+
+ free(difffilename);
+ difffilename = NULL;
+ free(logfilename);
+ logfilename = NULL;
}
+ footer(difffilename, logfilename);
+ status_end();
+
if (fail_count != 0)
exit(1);
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 15:08 ` Andrew Dunstan <[email protected]>
9 siblings, 0 replies; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-12 15:08 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers
On 10/12/21 4:37 AM, Andres Freund wrote:
> git remote add andres [email protected]:anarazel/postgres.git
ITYM:
git remote add andres git://github.com/anarazel/postgres.git
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 15:21 ` Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
9 siblings, 1 reply; 139+ messages in thread
From: Josef Šimánek @ 2021-10-12 15:21 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
.
út 12. 10. 2021 v 10:37 odesílatel Andres Freund <[email protected]> napsal:
>
> Hi,
>
> For the last year or so I've on and off tinkered with $subject. I think
> it's in a state worth sharing now. First, let's look at a little
> comparison.
>
> My workstation:
>
> non-cached configure:
> current: 11.80s
> meson: 6.67s
>
> non-cached build (world-bin):
> current: 40.46s
> ninja: 7.31s
>
> no-change build:
> current: 1.17s
> ninja: 0.06s
>
> test world:
> current: 105s
> meson: 63s
>
>
> What actually started to motivate me however were the long times windows
> builds took to come back with testsresults. On CI, with the same machine
> config:
>
> build:
> current: 202s (doesn't include genbki etc)
> meson+ninja: 140s
> meson+msbuild: 206s
>
>
> test:
> current: 1323s (many commands)
> meson: 903s (single command)
>
> (note that the test comparison isn't quite fair - there's a few tests
> missing, but it's just small contrib ones afaik)
>
>
> The biggest difference to me however is not the speed, but how readable
> the output is.
>
> Running the tests with meson in a terminal, shows the number of tests
> that completed out of how many total, how much time has passed, how long
> the currently running tests already have been running.
>
> At the end of a testrun a count of tests is shown:
>
> 188/189 postgresql:tap+pg_basebackup / pg_basebackup/t/010_pg_basebackup.pl OK 39.51s 110 subtests passed
> 189/189 postgresql:isolation+snapshot_too_old / snapshot_too_old/isolation OK 62.93s
>
>
> Ok: 188
> Expected Fail: 0
> Fail: 1
> Unexpected Pass: 0
> Skipped: 0
> Timeout: 0
>
> Full log written to /tmp/meson/meson-logs/testlog.txt
>
>
> The log has the output of the tests and ends with:
>
> Summary of Failures:
> 120/189 postgresql:tap+recovery / recovery/t/007_sync_rep.pl ERROR 7.16s (exit status 255 or signal 127 SIGinvalid)
>
>
> Quite the difference to make check-world -jnn output.
>
>
> So, now that the teasing is done, let me explain a bit what lead me down
> this path:
>
> Autoconf + make is not being actively developed. Especially autoconf is
> *barely* in maintenance mode - despite many shortcomings and bugs. It's
> also technology that very few want to use - autoconf m4 is scary, and
> it's scarier for people that started more recently than a lot of us
> committers for example.
>
> Recursive make as we use it is hard to get right. One reason the clean
> make build is so slow compared to meson is that we had to resort to
> .NOTPARALLEL to handle dependencies in a bunch of places. And despite
> that, I quite regularly see incremental build failures that can be
> resolved by retrying the build.
>
> While we have incremental build via --enable-depend, they don't work
> that reliable (i.e. misses necessary rebuilds) and yet is often too
> aggressive. More modern build system can keep track of the precise
> command used to build a target and rebuild it when that command changes.
>
>
> We also don't just have the autoconf / make buildsystem, there's also
> the msvc project generator - something most of us unix-y folks do not
> like to touch. I think that, combined with there being no easy way to
> run all tests, and it being just different, really hurt our windows
> developer appeal (and subsequently the quality of postgres on
> windows). I'm not saying this to ding the project generator - that was
> well before there were decent "meta" buildsystems out there (and in some
> ways it is a small one itself).
>
>
> The last big issue I have with the current situation is that there's no
> good test integration. make check-world output is essentially unreadable
> / not automatically parseable. Which led to the buildfarm having a
> separate list of things it needs to test, so that failures can be
> pinpointed and paired with appropriate logs. That approach unfortunately
> doesn't scale well to multi-core CPUs, slowing down the buildfarm by a
> fair bit.
>
>
> This all led to me to experiment with improvements. I tried a few
> somewhat crazy but incremental things like converting our buildsystem to
> non-recursive make (I got it to build the backend, but it's too hard to
> do manually I think), or to not run tests during the recursive make
> check-world, but to append commands to a list of tests, that then is run
> by a helper (can kinda be made to work). In the end I concluded that
> the amount of time we'd need to invest to maintain our more-and-more
> custom buildsystem going forward doesn't make sense.
>
>
> Which lead me to look around and analyze which other buildsystems there
> are that could make some sense for us. The halfway decent list includes,
> I think:
> 1) cmake
> 2) bazel
> 3) meson
>
>
> cmake would be a decent choice, I think. However, I just can't fully
> warm up to it. Something about it just doesn't quite sit right with
> me. That's not a good enough reason to prevent others from suggesting to
> use it, but it's good enough to justify not investing a lot of time in
> it myself.
>
> Bazel has some nice architectural properties. But it requires a JVM to
> run - I think that basically makes it insuitable for us. And the build
> information seems quite arduous to maintain too.
>
> Which left me with meson. It is a meta-buildsystem that can do the
> actual work of building via ninja (the most common one, also targeted by
> cmake), msbuild (visual studio project files, important for GUI work)
> and xcode projects (I assume that's for a macos IDE, but I haven't tried
> to use it). Meson roughly does what autoconf+automake did, in a
> python-esque DSL, and outputs build-instructions for ninja / msbuild /
> xcode. One interesting bit is that meson itself is written in python (
> and fairly easy to contribute too - I got a few changes in now).
>
>
> I don't think meson is perfect architecturally - e.g. its insistence on
> not having functions ends up making it a bit harder to not end up
> duplicating code. There's some user-interface oddities that are now hard
> to fix fully, due to the faily wide usage. But all-in-all it's pretty
> nice to use.
>
>
> Its worth calling out that a lot of large open source projects have been
> / are migrating to meson. qemu/kvm, mesa (core part of graphics stack on
> linux and also widely used in other platforms), a good chunk of GNOME,
> and quite a few more. Due to that it seems unlikely to be abandoned
> soon.
>
>
> As far as I can tell the only OS that postgres currently supports that
> meson doesn't support is HPUX. It'd likely be fairly easy to add
> gcc-on-hpux support, a chunk more to add support for the proprietary
> ones.
>
>
> The attached patch (meson support is 0016, the rest is prerequisites
> that aren't that interesting at this stage) converts most of postgres to
> meson. There's a few missing contrib modules, only about half the
> optional library dependencies are implemented, and I've only built on
> x64. It builds on freebsd, linux, macos and windows (both ninja and
> msbuild) and cross builds from linux to windows. Thomas helped make the
> freebsd / macos pieces a reality, thanks!
>
> I took a number of shortcuts (although there used to be a *lot*
> more). So this shouldn't be reviewed to the normal standard of the
> community - it's a prototype. But I think it's in a complete enough
> shape that it allows to do a well-informed evaluation.
>
> What doesn't yet work/ build:
>
> - plenty optional libraries, contrib, NLS, docs build
>
> - PGXS - and I don't yet know what to best do about it. One
> backward-compatible way would be to continue use makefiles for pgxs,
> but do the necessary replacement of Makefile.global.in via meson (and
> not use that for postgres' own build). But that doesn't really
> provide a nicer path for building postgres extensions on windows, so
> it'd definitely not be a long-term path.
>
> - JIT bitcode generation for anything but src/backend.
>
> - anything but modern-ish x86. That's proably a small amount of work,
> but something that needs to be done.
>
> - exporting all symbols for extension modules on windows (the stuff for
> postgres is implemented). Instead I marked the relevant symbols als
> declspec(dllexport). I think we should do that regardless of the
> buildsystem change. Restricting symbol visibility via gcc's
> -fvisibility=hidden for extensions results in a substantially reduced
> number of exported symbols, and even reduces object size (and I think
> improves the code too). I'll send an email about that separately.
>
>
>
>
> There's a lot more stuff to talk about, but I'll stop with a small bit
> of instructions below:
>
>
> Demo / instructions:
> # Get code
> git remote add andres [email protected]:anarazel/postgres.git
> git fetch andres
> git checkout --track andres/meson
>
> # setup build directory
> meson setup build --buildtype debug
> cd build
>
> # build (uses automatically as many cores as available)
> ninja
I'm getting errors at this step. You can find my output at
https://pastebin.com/Ar5VqfFG. Setup went well without errors. Is that
expected for now?
> # change configuration, build again
> meson configure -Dssl=openssl
> ninja
>
> # run all tests
> meson test
>
> # run just recovery tests
> meson test --suite setup --suite recovery
>
> # list tests
> meson test --list
>
>
> Greetings,
>
> Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
@ 2021-10-12 17:17 ` Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-12 17:17 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 17:21:50 +0200, Josef Šimánek wrote:
> > # build (uses automatically as many cores as available)
> > ninja
>
> I'm getting errors at this step. You can find my output at
> https://pastebin.com/Ar5VqfFG. Setup went well without errors. Is that
> expected for now?
Thanks, that's helpful. And no, that's not expected (*), it should be fixed.
What OS / distribution / version is this?
Can you build postgres "normally" with --with-gss? Seems like we're ending up
with a version of gssapi that we're not compatible with.
You should be able to get past this by disabling gss using meson configure
-Dgssapi=disabled.
Greetings,
Andres Freund
* except kinda, in the sense that I'd expect it to be buggy, given that I've
run it only on a few machines and it's very, uh, bleeding edge
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 23:19 ` Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Josef Šimánek @ 2021-10-12 23:19 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
út 12. 10. 2021 v 19:17 odesílatel Andres Freund <[email protected]> napsal:
>
> Hi,
>
> On 2021-10-12 17:21:50 +0200, Josef Šimánek wrote:
> > > # build (uses automatically as many cores as available)
> > > ninja
> >
> > I'm getting errors at this step. You can find my output at
> > https://pastebin.com/Ar5VqfFG. Setup went well without errors. Is that
> > expected for now?
>
> Thanks, that's helpful. And no, that's not expected (*), it should be fixed.
>
> What OS / distribution / version is this?
Fedora 34 (64 bit)
> Can you build postgres "normally" with --with-gss? Seems like we're ending up
> with a version of gssapi that we're not compatible with.
Yes, I can.
> You should be able to get past this by disabling gss using meson configure
> -Dgssapi=disabled.
I tried to clean and start from scratch, but I'm getting different
error probably related to wrongly configured JIT (LLVM wasn't found
during meson setup). I'll debug on my side to provide more info.
Whole build error could be found at https://pastebin.com/hCFqcPvZ.
Setup log could be found at https://pastebin.com/wjbE1w56.
> Greetings,
>
> Andres Freund
>
> * except kinda, in the sense that I'd expect it to be buggy, given that I've
> run it only on a few machines and it's very, uh, bleeding edge
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
@ 2021-10-12 23:54 ` Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-12 23:54 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 01:19:27 +0200, Josef Šimánek wrote:
> I tried to clean and start from scratch, but I'm getting different
> error probably related to wrongly configured JIT (LLVM wasn't found
> during meson setup). I'll debug on my side to provide more info.
../src/backend/jit/jit.c:91:73: error: ‘DLSUFFIX’ undeclared (first use in this function)
91 | snprintf(path, MAXPGPATH, "%s/%s%s", pkglib_path, jit_provider, DLSUFFIX);
| ^~~~~~~~
This *very* likely is related to building in a source tree that also contains
a "non-meson" build "in place". The problem is that the meson build picks up
the pg_config.h generated by ./configure in the "normal" build, rather than
the one meson generated itself.
You'd need to execute make distclean or such, or use a separate git checkout.
I forgot about this issue because I only ever build postgres from outside the
source-tree (by invoking configure from a separate directory), so there's
never build products in it. I think at least I need to make the build emit a
warning / error if there's a pg_config.h in the source tree...
This is the part of the jit code that's built regardless of llvm availability
- you'd get the same error in a few other places unrelated to jit.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-13 21:58 ` Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Josef Šimánek @ 2021-10-13 21:58 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
st 13. 10. 2021 v 1:54 odesílatel Andres Freund <[email protected]> napsal:
>
> Hi,
>
> On 2021-10-13 01:19:27 +0200, Josef Šimánek wrote:
> > I tried to clean and start from scratch, but I'm getting different
> > error probably related to wrongly configured JIT (LLVM wasn't found
> > during meson setup). I'll debug on my side to provide more info.
>
> ../src/backend/jit/jit.c:91:73: error: ‘DLSUFFIX’ undeclared (first use in this function)
> 91 | snprintf(path, MAXPGPATH, "%s/%s%s", pkglib_path, jit_provider, DLSUFFIX);
> | ^~~~~~~~
>
> This *very* likely is related to building in a source tree that also contains
> a "non-meson" build "in place". The problem is that the meson build picks up
> the pg_config.h generated by ./configure in the "normal" build, rather than
> the one meson generated itself.
>
> You'd need to execute make distclean or such, or use a separate git checkout.
>
> I forgot about this issue because I only ever build postgres from outside the
> source-tree (by invoking configure from a separate directory), so there's
> never build products in it. I think at least I need to make the build emit a
> warning / error if there's a pg_config.h in the source tree...
Hello, thanks for the hint. I can finally build using meson and run
regress tests.
The only problem I do have currently is auto-detection of perl. I'm
getting error related to missing "Opcode.pm". PERL is autodetected and
enabled (https://pastebin.com/xfRRrDcU).
I do get the same error when I enforce perl for current master build
(./configure --with-perl). Using ./configure with perl autodetection
skips plperl extension on my system.
Disabling perl manually for meson build (meson setup build
--reconfigure --buildtype debug -Dperl=disabled) works for me.
>
> This is the part of the jit code that's built regardless of llvm availability
> - you'd get the same error in a few other places unrelated to jit.
>
> Greetings,
>
> Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
@ 2021-10-14 13:14 ` Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 13:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Dagfinn Ilmari Mannsåker @ 2021-10-14 13:14 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
Josef Šimánek <[email protected]> writes:
> The only problem I do have currently is auto-detection of perl. I'm
> getting error related to missing "Opcode.pm". PERL is autodetected and
> enabled (https://pastebin.com/xfRRrDcU).
Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
core module, and should be in /usr/lib64/perl5, judging by the paths in
the error message.
Which OS is this? Some Linux distributions have separate packages for
the interpreter itself and the included modules, and the packages can be
named confusingly. E.g. on older Redhat/Fedora versions you have to
install the 'perl-core' package to get all the modules, 'perl' is just
the interpreter and the bare minimum set of strictily necessary modules.
They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
minimal one.
- ilmari
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
@ 2021-10-14 13:19 ` Josef Šimánek <[email protected]>
2021-10-14 13:29 ` Re: [RFC] building postgres with meson Alvaro Herrera <[email protected]>
2021-10-14 13:32 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Josef Šimánek @ 2021-10-14 13:19 UTC (permalink / raw)
To: Dagfinn Ilmari Mannsåker <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
čt 14. 10. 2021 v 15:14 odesílatel Dagfinn Ilmari Mannsåker
<[email protected]> napsal:
>
> Josef Šimánek <[email protected]> writes:
>
> > The only problem I do have currently is auto-detection of perl. I'm
> > getting error related to missing "Opcode.pm". PERL is autodetected and
> > enabled (https://pastebin.com/xfRRrDcU).
>
> Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
> core module, and should be in /usr/lib64/perl5, judging by the paths in
> the error message.
>
> Which OS is this? Some Linux distributions have separate packages for
> the interpreter itself and the included modules, and the packages can be
> named confusingly. E.g. on older Redhat/Fedora versions you have to
> install the 'perl-core' package to get all the modules, 'perl' is just
> the interpreter and the bare minimum set of strictily necessary modules.
>
> They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
> that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
> minimal one.
I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
package. Anyway it behaves differently with autoconf tools and the
meson build system. Is perl disabled by default in the current build
system?
>
> - ilmari
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 13:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
@ 2021-10-14 13:29 ` Alvaro Herrera <[email protected]>
2021-10-14 17:24 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Alvaro Herrera @ 2021-10-14 13:29 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: Dagfinn Ilmari Mannsåker <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
On 2021-Oct-14, Josef Šimánek wrote:
> I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> package. Anyway it behaves differently with autoconf tools and the
> meson build system. Is perl disabled by default in the current build
> system?
Yes, you have to use --with-perl in order to get it.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
"Puedes vivir sólo una vez, pero si lo haces bien, una vez es suficiente"
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 13:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:29 ` Re: [RFC] building postgres with meson Alvaro Herrera <[email protected]>
@ 2021-10-14 17:24 ` Andres Freund <[email protected]>
2021-10-14 19:36 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 22:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-14 17:24 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Josef Šimánek <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 10:29:42 -0300, Alvaro Herrera wrote:
> On 2021-Oct-14, Josef Šimánek wrote:
>
> > I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> > package. Anyway it behaves differently with autoconf tools and the
> > meson build system. Is perl disabled by default in the current build
> > system?
Hm, so it seems we should make the test separately verify that perl -M{Opcode,
ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
detection with a useful message?
> Yes, you have to use --with-perl in order to get it.
With the meson prototype I set most optional features to "auto", except for
LLVM, as that increases compile times noticeably.
For configure we didn't/don't want to do much auto-detection, because that
makes life harder for distributors. But meson has one switch controlling all
features set to 'auto' and not explicitly enabled/disabled:
--auto-features {enabled,disabled,auto} Override value of all 'auto' features (default: auto).
so the argument doesn't apply to the same degree there. We could default
auto-features to something else too.
There were two other reasons:
1) I got tired of needing to disable zlib, readline to be able to build on
windows.
2) Exercising all the dependency detection / checking seems important at this
stage
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 13:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:29 ` Re: [RFC] building postgres with meson Alvaro Herrera <[email protected]>
2021-10-14 17:24 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-14 19:36 ` Josef Šimánek <[email protected]>
1 sibling, 0 replies; 139+ messages in thread
From: Josef Šimánek @ 2021-10-14 19:36 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; pgsql-hackers
čt 14. 10. 2021 v 19:24 odesílatel Andres Freund <[email protected]> napsal:
>
> Hi,
>
> On 2021-10-14 10:29:42 -0300, Alvaro Herrera wrote:
> > On 2021-Oct-14, Josef Šimánek wrote:
> >
> > > I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> > > package. Anyway it behaves differently with autoconf tools and the
> > > meson build system. Is perl disabled by default in the current build
> > > system?
>
> Hm, so it seems we should make the test separately verify that perl -M{Opcode,
> ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
> detection with a useful message?
I can confirm "perl -MOpcode" fails. ExtUtils::Embed and
ExtUtils::ParseXS are present. Looking at the local system history of
perl-interpreter package, it seems to be installed by default on
Fedora 34. Friendly error message would be welcomed.
>
> > Yes, you have to use --with-perl in order to get it.
>
> With the meson prototype I set most optional features to "auto", except for
> LLVM, as that increases compile times noticeably.
>
> For configure we didn't/don't want to do much auto-detection, because that
> makes life harder for distributors. But meson has one switch controlling all
> features set to 'auto' and not explicitly enabled/disabled:
> --auto-features {enabled,disabled,auto} Override value of all 'auto' features (default: auto).
> so the argument doesn't apply to the same degree there. We could default
> auto-features to something else too.
>
> There were two other reasons:
>
> 1) I got tired of needing to disable zlib, readline to be able to build on
> windows.
> 2) Exercising all the dependency detection / checking seems important at this
> stage
Clear, thanks for the info.
> Greetings,
>
> Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 13:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:29 ` Re: [RFC] building postgres with meson Alvaro Herrera <[email protected]>
2021-10-14 17:24 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-14 22:08 ` Tom Lane <[email protected]>
2021-10-14 23:38 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Tom Lane @ 2021-10-14 22:08 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Josef Šimánek <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> Hm, so it seems we should make the test separately verify that perl -M{Opcode,
> ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
> detection with a useful message?
Our existing policy is that we should check this at configure time,
not later. Since plperl won't work at all without Opcode, it seems
appropriate to add a check there if you say --with-perl. I wasn't
aware that Red Hat had unbundled that from the minimal perl
installation :-(.
OTOH, if they've not unbundled ExtUtils::Embed or ExtUtils::ParseXS,
I doubt it's worth the configure cycles to check for those separately.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 13:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:29 ` Re: [RFC] building postgres with meson Alvaro Herrera <[email protected]>
2021-10-14 17:24 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-10-14 23:38 ` Andres Freund <[email protected]>
2021-10-14 23:51 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-14 23:38 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Josef Šimánek <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 18:08:58 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > Hm, so it seems we should make the test separately verify that perl -M{Opcode,
> > ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
> > detection with a useful message?
>
> Our existing policy is that we should check this at configure time,
> not later.
Yea, I was thinking of configure (and meson's equivalent) as well.
> Since plperl won't work at all without Opcode, it seems
> appropriate to add a check there if you say --with-perl. I wasn't
> aware that Red Hat had unbundled that from the minimal perl
> installation :-(.
>
> OTOH, if they've not unbundled ExtUtils::Embed or ExtUtils::ParseXS,
> I doubt it's worth the configure cycles to check for those separately.
On debian the perl binary, with a sparse set of modules is in
perl-base. ExtUtils::Embed and ExtUtils::ParseXS are in
perl-modules-x.yy. Whereas Opcode is in libperlx.yy. But libperlx.yy depends
on perl-modules-x.yy so I guess an Opcode.pm check would suffice.
Seems we can just check all of them at once with with something like
perl -MOpcode -MExtUtils::Embed -MExtUtils::ParseXSNotAvailable -e ''
Can't locate ExtUtils/ParseXSNotAvailable.pm in @INC (you may need to install the ExtUtils::ParseXS3 module) (@INC contains: /home/andres/bin/perl5/lib/perl5/x86_64-linux-gnu-thread-multi /home/andres/bin/perl5/lib/perl5 /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1 /usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32 /usr/share/perl/5.32 /usr/local/lib/site_perl).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 13:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:29 ` Re: [RFC] building postgres with meson Alvaro Herrera <[email protected]>
2021-10-14 17:24 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 23:38 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-14 23:51 ` Tom Lane <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Tom Lane @ 2021-10-14 23:51 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Josef Šimánek <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2021-10-14 18:08:58 -0400, Tom Lane wrote:
>> Andres Freund <[email protected]> writes:
>>> Hm, so it seems we should make the test separately verify that perl -M{Opcode,
>>> ExtUtils::Embed, ExtUtils::ParseXS} doesn't fail, so that we can fail perl
>>> detection with a useful message?
>> Our existing policy is that we should check this at configure time,
>> not later.
> Yea, I was thinking of configure (and meson's equivalent) as well.
Ah, sorry, I misunderstood what you meant by "test".
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 13:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
@ 2021-10-14 13:32 ` Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 19:29 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Dagfinn Ilmari Mannsåker @ 2021-10-14 13:32 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
Josef Šimánek <[email protected]> writes:
> čt 14. 10. 2021 v 15:14 odesílatel Dagfinn Ilmari Mannsåker
> <[email protected]> napsal:
>>
>> Josef Šimánek <[email protected]> writes:
>>
>> > The only problem I do have currently is auto-detection of perl. I'm
>> > getting error related to missing "Opcode.pm". PERL is autodetected and
>> > enabled (https://pastebin.com/xfRRrDcU).
>>
>> Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
>> core module, and should be in /usr/lib64/perl5, judging by the paths in
>> the error message.
>>
>> Which OS is this? Some Linux distributions have separate packages for
>> the interpreter itself and the included modules, and the packages can be
>> named confusingly. E.g. on older Redhat/Fedora versions you have to
>> install the 'perl-core' package to get all the modules, 'perl' is just
>> the interpreter and the bare minimum set of strictily necessary modules.
>>
>> They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
>> that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
>> minimal one.
>
> I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> package.`
Yes, it's a separate package, but the 'perl' package depends on all the
core module packages, so installing that should fix things. You appear
to only have 'perl-interpreter' installed.
> Anyway it behaves differently with autoconf tools and the meson build
> system. Is perl disabled by default in the current build system?
configure doesn't auto-detect any optional features, they have to be
explicitly enabled using --with-foo switches.
- ilmari
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 13:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-14 13:32 ` Re: [RFC] building postgres with meson Dagfinn Ilmari Mannsåker <[email protected]>
@ 2021-10-14 19:29 ` Josef Šimánek <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Josef Šimánek @ 2021-10-14 19:29 UTC (permalink / raw)
To: Dagfinn Ilmari Mannsåker <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
čt 14. 10. 2021 v 15:32 odesílatel Dagfinn Ilmari Mannsåker
<[email protected]> napsal:
>
> Josef Šimánek <[email protected]> writes:
>
> > čt 14. 10. 2021 v 15:14 odesílatel Dagfinn Ilmari Mannsåker
> > <[email protected]> napsal:
> >>
> >> Josef Šimánek <[email protected]> writes:
> >>
> >> > The only problem I do have currently is auto-detection of perl. I'm
> >> > getting error related to missing "Opcode.pm". PERL is autodetected and
> >> > enabled (https://pastebin.com/xfRRrDcU).
> >>
> >> Your Perl (not PERL) installation seems to be incomplete. Opcode.pm is a
> >> core module, and should be in /usr/lib64/perl5, judging by the paths in
> >> the error message.
> >>
> >> Which OS is this? Some Linux distributions have separate packages for
> >> the interpreter itself and the included modules, and the packages can be
> >> named confusingly. E.g. on older Redhat/Fedora versions you have to
> >> install the 'perl-core' package to get all the modules, 'perl' is just
> >> the interpreter and the bare minimum set of strictily necessary modules.
> >>
> >> They've fixed this in recent versions (Fedora 34 and Redhat 8, IIRC), so
> >> that 'perl' gives you the hole bundle, and 'perl-interpeter' is the
> >> minimal one.
> >
> > I'm using Fedora 34 and I still see perl-Opcode.x86_64 as a separate
> > package.`
>
> Yes, it's a separate package, but the 'perl' package depends on all the
> core module packages, so installing that should fix things. You appear
> to only have 'perl-interpreter' installed.
You're right. Installing "perl" or "perl-Opcode" manually fixes this
problem. Currently I only have "perl-interpreter" installed.
> > Anyway it behaves differently with autoconf tools and the meson build
> > system. Is perl disabled by default in the current build system?
>
> configure doesn't auto-detect any optional features, they have to be
> explicitly enabled using --with-foo switches.
>
> - ilmari
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:21 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 23:19 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 21:58 ` Re: [RFC] building postgres with meson Josef Šimánek <[email protected]>
@ 2021-10-14 17:26 ` Andres Freund <[email protected]>
1 sibling, 0 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-14 17:26 UTC (permalink / raw)
To: Josef Šimánek <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 23:58:12 +0200, Josef Šimánek wrote:
> st 13. 10. 2021 v 1:54 odesílatel Andres Freund <[email protected]> napsal:
> > This *very* likely is related to building in a source tree that also contains
> > a "non-meson" build "in place". The problem is that the meson build picks up
> > the pg_config.h generated by ./configure in the "normal" build, rather than
> > the one meson generated itself.
> >
> > You'd need to execute make distclean or such, or use a separate git checkout.
> >
> > I forgot about this issue because I only ever build postgres from outside the
> > source-tree (by invoking configure from a separate directory), so there's
> > never build products in it. I think at least I need to make the build emit a
> > warning / error if there's a pg_config.h in the source tree...
>
> Hello, thanks for the hint. I can finally build using meson and run
> regress tests.
I yesterday pushed code that should detect this case (with an error). Should
now detect the situation both when you first run configure in tree, and then
meson, and the other way round (by the dirty hack of ./configure touch'ing
meson.build at the end for in-tree builds).
> The only problem I do have currently is auto-detection of perl. I'm
> getting error related to missing "Opcode.pm". PERL is autodetected and
> enabled (https://pastebin.com/xfRRrDcU).
>
> I do get the same error when I enforce perl for current master build
> (./configure --with-perl). Using ./configure with perl autodetection
> skips plperl extension on my system.
>
> Disabling perl manually for meson build (meson setup build
> --reconfigure --buildtype debug -Dperl=disabled) works for me.
Yay, thanks for testing!
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 15:28 ` Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
9 siblings, 1 reply; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-12 15:28 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers
On 10/12/21 4:37 AM, Andres Freund wrote:
> # setup build directory
> meson setup build --buildtype debug
I took this for an outing on msys2 and it just seems to hang. If it's not hanging it's unbelievably slow.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
@ 2021-10-12 15:50 ` Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-12 15:50 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers
On 10/12/21 11:28 AM, Andrew Dunstan wrote:
> On 10/12/21 4:37 AM, Andres Freund wrote:
>> # setup build directory
>> meson setup build --buildtype debug
> I took this for an outing on msys2 and it just seems to hang. If it's not hanging it's unbelievably slow.
>
>
It hung because it expected the compiler to be 'ccache cc'. Hanging in
such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
but it then errored out looking for perl libs. I think msys2 is going to
be a bit difficult here :-(
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
@ 2021-10-12 16:59 ` Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:11 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-12 16:59 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
> It hung because it expected the compiler to be 'ccache cc'. Hanging in
> such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
> but it then errored out looking for perl libs. I think msys2 is going to
> be a bit difficult here :-(
Hm. Yea, the perl thing is my fault - you should be able to get past it with
-Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
I can't reproduce the hanging though. I needed to install bison, flex and
ninja and disable perl as described above, but then it built just fine.
It does seems to crash somewhere in the main regression tests though, I think
I don't do the "set stack depth" dance correctly for msys.
If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?
(*) I've for now made most dependencies autodetected, unless you pass
--auto-features disabled to collectively disable all the auto-detected
features. Initially I had mirrored the autoconf behaviour, but I got sick of
forgetting to turn off readline or zlib on windows. And then it was useful to
test on multiple operating systems...
For working on windows meson's wraps are quite useful. I've not added that to
the git branch, but if you manually do
mkdir subprojects
meson wrap install lz4
meson wrap install zlib
building with -Dzlib=enabled -Dlz4=enabled will fall back to building lz4,
zlib as-needed.
I was wondering about adding a binary wrap for e.g. bison, flex on windows, so
that the process of getting a build going isn't as arduous.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 18:09 ` Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-12 18:09 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 09:59:26 -0700, Andres Freund wrote:
> On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
> > It hung because it expected the compiler to be 'ccache cc'. Hanging in
> > such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
> > but it then errored out looking for perl libs. I think msys2 is going to
> > be a bit difficult here :-(
>
> Hm. Yea, the perl thing is my fault - you should be able to get past it with
> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
This is a weird one. I don't know much about msys, so it's probably related to
that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
shell commands that exists, but not according to msys's own python
$ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:', os.path.exists(p))"
does /usr/lib/perl5/core_perl/CORE exist: False
$ ls -ld /usr/lib/perl5/core_perl/CORE
drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE
So it's not too surprising that that doesn't work out. It's easy enough to
work around, but still pretty weird.
I pushed a workaround for the config-time error, but it doesn't yet recognize
msys perl correctly. But at least it's not alone in that - configure doesn't
seem to either, so I'm probably doing something wrong :)
> I can't reproduce the hanging though. I needed to install bison, flex and
> ninja and disable perl as described above, but then it built just fine.
>
> It does seems to crash somewhere in the main regression tests though, I think
> I don't do the "set stack depth" dance correctly for msys.
That was it - just hadn't ported setting -Wl,--stack=... for !msvc
windows. Pushed the fix for that out.
I guess I should figure out how to commandline install msys and add it to CI.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 18:37 ` Andrew Dunstan <[email protected]>
2021-10-12 19:16 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-12 18:37 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 2:09 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-12 09:59:26 -0700, Andres Freund wrote:
>> On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
>>> It hung because it expected the compiler to be 'ccache cc'. Hanging in
>>> such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
>>> but it then errored out looking for perl libs. I think msys2 is going to
>>> be a bit difficult here :-(
>> Hm. Yea, the perl thing is my fault - you should be able to get past it with
>> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
> This is a weird one. I don't know much about msys, so it's probably related to
> that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
> shell commands that exists, but not according to msys's own python
>
> $ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:', os.path.exists(p))"
> does /usr/lib/perl5/core_perl/CORE exist: False
>
> $ ls -ld /usr/lib/perl5/core_perl/CORE
> drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE
Looks to me like a python issue:
# perl -e 'my $p = "/usr/lib/perl5/core_perl/CORE"; print qq(does $p
exist: ), -e $p, qq{\n};'
does /usr/lib/perl5/core_perl/CORE exist: 1
# python -c "import os; p = '/usr/lib/perl5/core_perl/CORE';
print(f'does {p} exist:', os.path.exists(p))"
does /usr/lib/perl5/core_perl/CORE exist: False
# cygpath -m /usr/lib/perl5/core_perl/CORE
C:/tools/msys64/usr/lib/perl5/core_perl/CORE
# python -c "import os; p =
'C:/tools/msys64/usr/lib/perl5/core_perl/CORE'; print(f'does {p}
exist:', os.path.exists(p))"
does C:/tools/msys64/usr/lib/perl5/core_perl/CORE exist: True
Clearly python is not understanding msys virtualized paths.
>
>
> I guess I should figure out how to commandline install msys and add it to CI.
>
here's what I do:
# msys2 outputs esc-[3J which clears the screen's scroll buffer. Nasty.
# so we redirect the output
# find the log in c:\Windows\System32 if needed
choco install -y --no-progress --limit-output msys2 > msys2inst.log
c:\tools\msys64\usr\bin\bash -l
'/c/vfiles/windows-uploads/msys2-packages.sh'
Here's what's in msys-packages.sh:
pacman -S --needed --noconfirm \
base-devel \
msys/git \
msys/ccache \
msys/vim \
msys/perl-Crypt-SSLeay \
mingw-w64-clang-x86_64-toolchain \
mingw-w64-x86_64-toolchain
# could do: pacman -S --needed --noconfirm development
# this is more economical. These should cover most of the things you
might
# want to configure with
pacman -S --needed --noconfirm \
msys/gettext-devel \
msys/icu-devel \
msys/libiconv-devel \
msys/libreadline-devel \
msys/libxml2-devel \
msys/libxslt-devel \
msys/openssl-devel \
msys/zlib-devel
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
@ 2021-10-12 19:16 ` Andrew Dunstan <[email protected]>
1 sibling, 0 replies; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-12 19:16 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 2:37 PM, Andrew Dunstan wrote:
> On 10/12/21 2:09 PM, Andres Freund wrote:
>> Hi,
>>
>> On 2021-10-12 09:59:26 -0700, Andres Freund wrote:
>>> On 2021-10-12 11:50:03 -0400, Andrew Dunstan wrote:
>>>> It hung because it expected the compiler to be 'ccache cc'. Hanging in
>>>> such a case is kinda unforgivable. I remedied that by setting 'CC=gcc'
>>>> but it then errored out looking for perl libs. I think msys2 is going to
>>>> be a bit difficult here :-(
>>> Hm. Yea, the perl thing is my fault - you should be able to get past it with
>>> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
>> This is a weird one. I don't know much about msys, so it's probably related to
>> that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
>> shell commands that exists, but not according to msys's own python
>>
>> $ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:', os.path.exists(p))"
>> does /usr/lib/perl5/core_perl/CORE exist: False
>>
>> $ ls -ld /usr/lib/perl5/core_perl/CORE
>> drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE
>
> Looks to me like a python issue:
>
>
> # perl -e 'my $p = "/usr/lib/perl5/core_perl/CORE"; print qq(does $p
> exist: ), -e $p, qq{\n};'
> does /usr/lib/perl5/core_perl/CORE exist: 1
>
> # python -c "import os; p = '/usr/lib/perl5/core_perl/CORE';
> print(f'does {p} exist:', os.path.exists(p))"
> does /usr/lib/perl5/core_perl/CORE exist: False
>
> # cygpath -m /usr/lib/perl5/core_perl/CORE
> C:/tools/msys64/usr/lib/perl5/core_perl/CORE
>
> # python -c "import os; p =
> 'C:/tools/msys64/usr/lib/perl5/core_perl/CORE'; print(f'does {p}
> exist:', os.path.exists(p))"
> does C:/tools/msys64/usr/lib/perl5/core_perl/CORE exist: True
>
>
> Clearly python is not understanding msys virtualized paths.
It's a matter of which python you use. The one that understands msys
paths is msys/python. The mingw64 packages are normally pure native
windows and so don't understand msys paths. I know it's confusing :-(
# /usr/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE';
print(f'does {p} exist:', os.path.exists(p))"
does /usr/lib/perl5/core_perl/CORE exist: True
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
@ 2021-10-12 19:29 ` Andres Freund <[email protected]>
2021-10-12 20:02 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-12 19:29 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 14:37:04 -0400, Andrew Dunstan wrote:
> On 10/12/21 2:09 PM, Andres Freund wrote:
> >> Hm. Yea, the perl thing is my fault - you should be able to get past it with
> >> -Dperl=disabled, and I'll take a look at fixing the perl detection. (*)
> > This is a weird one. I don't know much about msys, so it's probably related to
> > that. Perl spits out /usr/lib/perl5/core_perl/ as its archlibexp. According to
> > shell commands that exists, but not according to msys's own python
> >
> > $ /mingw64/bin/python -c "import os; p = '/usr/lib/perl5/core_perl/CORE'; print(f'does {p} exist:', os.path.exists(p))"
> > does /usr/lib/perl5/core_perl/CORE exist: False
> >
> > $ ls -ld /usr/lib/perl5/core_perl/CORE
> > drwxr-xr-x 1 anfreund anfreund 0 Oct 10 10:19 /usr/lib/perl5/core_perl/CORE
> Looks to me like a python issue:
> Clearly python is not understanding msys virtualized paths.
Ah, it's a question of the *wrong* python being used :/. I somehow ended up
with both a mingw and an msys python, with the mingw python taking preference
over the msys one. The latter one does understand such paths.
> > I guess I should figure out how to commandline install msys and add it to CI.
> here's what I do:
Thanks!
Does that recipe get you to a build where ./configure --with-perl succeeds?
I see this here:
checking for Perl archlibexp... /usr/lib/perl5/core_perl
checking for Perl privlibexp... /usr/share/perl5/core_perl
checking for Perl useshrplib... true
checking for CFLAGS recommended by Perl... -DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -fno-strict-aliasing -fstack-protector-strong
checking for CFLAGS to compile embedded Perl... -DPERL_USE_SAFE_PUTENV
checking for flags to link embedded Perl... no
configure: error: could not determine flags for linking embedded Perl.
This probably means that ExtUtils::Embed or ExtUtils::MakeMaker is not
installed.
If I just include perl.h from a test file with gcc using the above flags it
fails to compile:
$ echo '#include <perl.h>' > test.c
$ gcc -DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -fno-strict-aliasing -fstack-protector-strong test.c -c -I /c/dev/msys64/usr/lib/perl5/core_perl/CORE
In file included from test.c:1:
C:/dev/msys64/usr/lib/perl5/core_perl/CORE/perl.h:1003:13: fatal error: sys/wait.h: No such file or directory
1003 | # include <sys/wait.h>
and ldopts bleats
$ perl -MExtUtils::Embed -e ldopts
Warning (mostly harmless): No library found for -lpthread
Warning (mostly harmless): No library found for -ldl
-Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--enable-auto-image-base -fstack-protector-strong -L/usr/lib/perl5/core_perl/CORE -lperl -lcrypt
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 20:02 ` Andrew Dunstan <[email protected]>
2021-10-12 20:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-12 20:02 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 3:29 PM, Andres Freund wrote:
>
> Does that recipe get you to a build where ./configure --with-perl succeeds?
>
> I see this here:
>
> checking for Perl archlibexp... /usr/lib/perl5/core_perl
> checking for Perl privlibexp... /usr/share/perl5/core_perl
> checking for Perl useshrplib... true
> checking for CFLAGS recommended by Perl... -DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -fno-strict-aliasing -fstack-protector-strong
> checking for CFLAGS to compile embedded Perl... -DPERL_USE_SAFE_PUTENV
> checking for flags to link embedded Perl... no
> configure: error: could not determine flags for linking embedded Perl.
> This probably means that ExtUtils::Embed or ExtUtils::MakeMaker is not
> installed.
>
> If I just include perl.h from a test file with gcc using the above flags it
> fails to compile:
You need to build against a native perl, like Strawberry or ActiveState.
(I have had mixed success with Strawberry) You do that by putting a path
to it at the start of the PATH. The wrinkle in this is that you need
prove to point to one that understands virtual paths. So you do
something like this:
PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 20:02 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
@ 2021-10-12 20:42 ` Andres Freund <[email protected]>
2021-10-13 01:03 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-12 20:42 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 16:02:14 -0400, Andrew Dunstan wrote:
> You need to build against a native perl, like Strawberry or ActiveState.
> (I have had mixed success with Strawberry)
Do you understand why that is needed?
> You do that by putting a path to it at the start of the PATH. The wrinkle in
> this is that you need prove to point to one that understands virtual
> paths. So you do something like this:
>
>
> PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...
Oh my.
I'll try that later... I wonder if we could make this easier from our side?
This is a lot of magic to know.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 20:02 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 20:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-13 01:03 ` Andres Freund <[email protected]>
2021-10-13 12:55 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-13 01:03 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 13:42:56 -0700, Andres Freund wrote:
> On 2021-10-12 16:02:14 -0400, Andrew Dunstan wrote:
> > You do that by putting a path to it at the start of the PATH. The wrinkle in
> > this is that you need prove to point to one that understands virtual
> > paths. So you do something like this:
> >
> >
> > PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...
>
> Oh my.
>
> I'll try that later... I wonder if we could make this easier from our side?
> This is a lot of magic to know.
I managed to get this working. At first it failed because I don't have
pexports - it's not available inside msys as far as I could tell. And seems to
be unmaintained. But replacing pexports with gendef fixed that.
There's this comment in src/pl/plperl/GNUmakefile
# Perl on win32 ships with import libraries only for Microsoft Visual C++,
# which are not compatible with mingw gcc. Therefore we need to build a
# new import library to link with.
but I seem to be able to link fine without going through that song-and-dance?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 20:02 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 20:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 01:03 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-13 12:55 ` Andrew Dunstan <[email protected]>
2021-10-13 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-13 12:55 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 9:03 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-12 13:42:56 -0700, Andres Freund wrote:
>> On 2021-10-12 16:02:14 -0400, Andrew Dunstan wrote:
>>> You do that by putting a path to it at the start of the PATH. The wrinkle in
>>> this is that you need prove to point to one that understands virtual
>>> paths. So you do something like this:
>>>
>>>
>>> PATH="/c/perl/bin:$PATH" PROVE=/bin/core_perl/prove configure ...
>> Oh my.
>>
>> I'll try that later... I wonder if we could make this easier from our side?
>> This is a lot of magic to know.
> I managed to get this working. At first it failed because I don't have
> pexports - it's not available inside msys as far as I could tell. And seems to
> be unmaintained. But replacing pexports with gendef fixed that.
>
> There's this comment in src/pl/plperl/GNUmakefile
>
> # Perl on win32 ships with import libraries only for Microsoft Visual C++,
> # which are not compatible with mingw gcc. Therefore we need to build a
> # new import library to link with.
>
> but I seem to be able to link fine without going through that song-and-dance?
>
It looks like you're not building a native postgres, but rather one
targeted at msys. To build one that's native (i.e. runs without any
presence of msys) you need to do these things before building:
MSYSTEM=MINGW64
MSYSTEM_CHOST=x86_64-w64-mingw32
PATH="/mingw64/bin:$PATH"
pexports will be in the resulting path, and the build will use the
native compiler.
You can use fairywren's config as a guide.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 20:02 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 20:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 01:03 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 12:55 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
@ 2021-10-13 17:26 ` Andres Freund <[email protected]>
2021-10-13 20:06 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-13 17:26 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 08:55:38 -0400, Andrew Dunstan wrote:
> On 10/12/21 9:03 PM, Andres Freund wrote:
> > I managed to get this working. At first it failed because I don't have
> > pexports - it's not available inside msys as far as I could tell. And seems to
> > be unmaintained. But replacing pexports with gendef fixed that.
> >
> > There's this comment in src/pl/plperl/GNUmakefile
> >
> > # Perl on win32 ships with import libraries only for Microsoft Visual C++,
> > # which are not compatible with mingw gcc. Therefore we need to build a
> > # new import library to link with.
> >
> > but I seem to be able to link fine without going through that song-and-dance?
> >
>
>
> It looks like you're not building a native postgres, but rather one
> targeted at msys. To build one that's native (i.e. runs without any
> presence of msys) you need to do these things before building:
>
> MSYSTEM=MINGW64
> MSYSTEM_CHOST=x86_64-w64-mingw32
> PATH="/mingw64/bin:$PATH"
I had a config equivalent to this (slight difference in PATH, but the same gcc
being picked), and I just verified that it still works if I set up PATH like
that. I get a working plperl out of it. Without msys on PATH or such.
where perl526.dll
C:\perl\strawberry-5.26.3.1-64bit\perl\bin\perl526.dll
dumpbin /imports 'C:/Users/anfreund/src/pg-meson/build-mingw/tmp_install/lib/plperl.dll'|grep dll
Dump of file C:\Users\anfreund\src\pg-meson\build-mingw\tmp_install\lib\plperl.dll
KERNEL32.dll
msvcrt.dll
perl526.dll
dumpbin /imports .\build-mingw\tmp_install\bin\postgres.exe|grep dll
ADVAPI32.dll
KERNEL32.dll
msvcrt.dll
Secur32.dll
WLDAP32.dll
WS2_32.dll
do $$elog(NOTICE, "blob");$$ language plperl;
NOTICE: blob
DO
To me this looks like it's a plperl built without the import file recreation,
without being msys dependent?
> pexports will be in the resulting path, and the build will use the
> native compiler.
I don't see pexports anywhere in the msys installation. I can see it available
on sourceforge, and I see a few others asking where to get it from in the
context of msys, and being pointed to manually downloading it.
Seems like we should consider using gendef instead of pexports, given it's
available in msys?
$ pacman -Fy
$ pacman -F gendef.exe
...
mingw64/mingw-w64-x86_64-tools-git 9.0.0.6316.acdc7adc9-1 (mingw-w64-x86_64-toolchain) [installed]
mingw64/bin/gendef.exe
..
$ pacman -F pexports.exe
$ pacman -Fx pexports
<bunch of packages containing smtpexports.h>
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 20:02 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 20:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 01:03 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 12:55 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-13 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-13 20:06 ` Andrew Dunstan <[email protected]>
2021-10-13 21:46 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-13 20:06 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/13/21 1:26 PM, Andres Freund wrote:
>
>> pexports will be in the resulting path, and the build will use the
>> native compiler.
> I don't see pexports anywhere in the msys installation. I can see it available
> on sourceforge, and I see a few others asking where to get it from in the
> context of msys, and being pointed to manually downloading it.
Weird. fairywren has it, which means that it must have been removed from
the packages at some stage, fairly recently as fairywren isn't that old.
I just confirmed the absence on a 100% fresh install.
It is in Strawberry's c/bin directory.
>
> Seems like we should consider using gendef instead of pexports, given it's
> available in msys?
Yeah. It's missing on my ancient msys animal (frogmouth), but it doesn't
build --with-perl.
jacana seems to have it.
If you prep a patch I'll test it.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 20:02 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 20:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 01:03 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 12:55 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-13 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 20:06 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
@ 2021-10-13 21:46 ` Andres Freund <[email protected]>
2021-10-13 23:11 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-13 21:46 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 16:06:32 -0400, Andrew Dunstan wrote:
> If you prep a patch I'll test it.
Well, right now I'm wondering if the better fix is to just remove the whole
win32 block. I don't know how far back, but afaict it's not needed. Seems to
have been needed for narwhal at some point, according to 02b61dd08f99. But
narwhal is long dead.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 20:02 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 20:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 01:03 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 12:55 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-13 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 20:06 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-13 21:46 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-13 23:11 ` Andrew Dunstan <[email protected]>
2021-10-14 22:19 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-13 23:11 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/13/21 5:46 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-13 16:06:32 -0400, Andrew Dunstan wrote:
>> If you prep a patch I'll test it.
> Well, right now I'm wondering if the better fix is to just remove the whole
> win32 block. I don't know how far back, but afaict it's not needed. Seems to
> have been needed for narwhal at some point, according to 02b61dd08f99. But
> narwhal is long dead.
>
Ok, I'll test it out.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:09 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 20:02 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 20:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 01:03 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 12:55 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-13 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 20:06 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-13 21:46 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 23:11 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
@ 2021-10-14 22:19 ` Andrew Dunstan <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-14 22:19 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/13/21 7:11 PM, Andrew Dunstan wrote:
> On 10/13/21 5:46 PM, Andres Freund wrote:
>> Hi,
>>
>> On 2021-10-13 16:06:32 -0400, Andrew Dunstan wrote:
>>> If you prep a patch I'll test it.
>> Well, right now I'm wondering if the better fix is to just remove the whole
>> win32 block. I don't know how far back, but afaict it's not needed. Seems to
>> have been needed for narwhal at some point, according to 02b61dd08f99. But
>> narwhal is long dead.
>>
> Ok, I'll test it out.
>
confirmed that jacana doesn't need this code to build or test plperl
(all I did was change the test from win32 to win32x). There would still
be work to do to fix the contrib bool_plperl, jsonb_plperl and
hstore_plperl modules.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 18:11 ` Andrew Dunstan <[email protected]>
2021-10-12 18:23 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-12 18:11 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 12:59 PM, Andres Freund wrote:
>
>
> If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?
Here's the entire thing
# cat
C:/tools/msys64/home/Administrator/postgresql/build/meson-logs/meson-log.txt
Build started at 2021-10-12T18:08:34.387568
Main binary: C:/tools/msys64/mingw64/bin/python.exe
Build Options: -Dbuildtype=debug
Python system: Windows
The Meson build system
Version: 0.59.1
Source dir: C:/tools/msys64/home/Administrator/postgresql
Build dir: C:/tools/msys64/home/Administrator/postgresql/build
Build type: native build
Project name: postgresql
Project version: 15devel
Sanity testing C compiler: ccache cc
Is cross compiler: False.
Sanity check compiler command line: ccache cc sanitycheckc.c -o
sanitycheckc.exe -D_FILE_OFFSET_BITS=64
Sanity check compile stdout:
-----
Sanity check compile stderr:
-----
meson.build:1:0: ERROR: Compiler ccache cc can not compile programs.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:11 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
@ 2021-10-12 18:23 ` Andres Freund <[email protected]>
2021-10-12 18:42 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-12 18:23 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 14:11:39 -0400, Andrew Dunstan wrote:
> On 10/12/21 12:59 PM, Andres Freund wrote:
> > If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?
> Here's the entire thing
> Sanity check compiler command line: ccache cc sanitycheckc.c -o
> sanitycheckc.exe -D_FILE_OFFSET_BITS=64
> Sanity check compile stdout:
>
> -----
> Sanity check compile stderr:
>
> -----
>
> meson.build:1:0: ERROR: Compiler ccache cc can not compile programs.
Huh, it's not a question of gcc vs cc, it's that meson automatically uses
ccache. And it looks like msys's ccache is broken at the moment (installed
yesterday):
$ ccache --version
ccache version 4.4.1
...
$ echo > test.c
$ ccache cc -c test.c
Segmentation fault (core dumped)
..
not sure how that leads to hanging, but it's not too surprising that things
don't work out after that...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 15:28 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 18:11 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
2021-10-12 18:23 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 18:42 ` Andrew Dunstan <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Andrew Dunstan @ 2021-10-12 18:42 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On 10/12/21 2:23 PM, Andres Freund wrote:
> Hi,
>
> On 2021-10-12 14:11:39 -0400, Andrew Dunstan wrote:
>> On 10/12/21 12:59 PM, Andres Freund wrote:
>>> If you repro the hanging, what's the last bit in meson-logs/meson-log.txt?
>> Here's the entire thing
>> Sanity check compiler command line: ccache cc sanitycheckc.c -o
>> sanitycheckc.exe -D_FILE_OFFSET_BITS=64
>> Sanity check compile stdout:
>>
>> -----
>> Sanity check compile stderr:
>>
>> -----
>>
>> meson.build:1:0: ERROR: Compiler ccache cc can not compile programs.
> Huh, it's not a question of gcc vs cc, it's that meson automatically uses
> ccache. And it looks like msys's ccache is broken at the moment (installed
> yesterday):
>
> $ ccache --version
> ccache version 4.4.1
> ...
>
> $ echo > test.c
> $ ccache cc -c test.c
> Segmentation fault (core dumped)
> ..
>
> not sure how that leads to hanging, but it's not too surprising that things
> don't work out after that...
>
Yes, I've had to disable ccache on fairywren.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-12 19:55 ` John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-21 21:48 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
9 siblings, 2 replies; 139+ messages in thread
From: John Naylor @ 2021-10-12 19:55 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Tue, Oct 12, 2021 at 4:37 AM Andres Freund <[email protected]> wrote:
[Meson prototype]
The build code looks pretty approachable for someone with no prior
exposure, and feels pretty nice when running it (I couldn't get a build
working but I'll leave that aside for now).
> As far as I can tell the only OS that postgres currently supports that
> meson doesn't support is HPUX. It'd likely be fairly easy to add
> gcc-on-hpux support, a chunk more to add support for the proprietary
> ones.
That would also have to work for all the dependencies, which were displayed
to me as:
ninja, gdbm, ca-certificates, [email protected], readline, sqlite and [email protected]
Also, could utility makefile targets be made to work? I'm thinking in
particular of update-unicode and reformat-dat-files, for example.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-12 20:59 ` Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-12 20:59 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 15:55:22 -0400, John Naylor wrote:
> On Tue, Oct 12, 2021 at 4:37 AM Andres Freund <[email protected]> wrote:
> The build code looks pretty approachable for someone with no prior
> exposure, and feels pretty nice when running it
That's part of what attracted me...
> (I couldn't get a build working but I'll leave that aside for now).
If you want to do that separately, I'll try to fix it.
> > As far as I can tell the only OS that postgres currently supports that
> > meson doesn't support is HPUX. It'd likely be fairly easy to add
> > gcc-on-hpux support, a chunk more to add support for the proprietary
> > ones.
>
> That would also have to work for all the dependencies, which were displayed
> to me as:
>
> ninja, gdbm, ca-certificates, [email protected], readline, sqlite and [email protected]
meson does depend on ninja (to execute the build) and of course python. But
the rest should be optional dependencies. ninja builds without any
dependencies as long as you don't change its parser sources. python builds on
aix, hpux etc.
Not sure what way gdbm [email protected] and sqlite are pulled in? I assume readline
is for python...
> Also, could utility makefile targets be made to work? I'm thinking in
> particular of update-unicode and reformat-dat-files, for example.
Yes, that shouldn't be a problem. You can run arbitrary code in targets
(there's plenty need for that already in what I have so far).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-13 15:51 ` John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 04:27 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: John Naylor @ 2021-10-13 15:51 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Tue, Oct 12, 2021 at 4:59 PM Andres Freund <[email protected]> wrote:
> On 2021-10-12 15:55:22 -0400, John Naylor wrote:
> > (I couldn't get a build working but I'll leave that aside for now).
>
> If you want to do that separately, I'll try to fix it.
Okay, I pulled the latest commits and tried again:
[51/950] Compiling C object
src/interfaces/libpq/libpq.5.dylib.p/fe-connect.c.o
FAILED: src/interfaces/libpq/libpq.5.dylib.p/fe-connect.c.o
ccache cc -Isrc/interfaces/libpq/libpq.5.dylib.p -Isrc/interfaces/libpq
-I../src/interfaces/libpq -Isrc/port -I../src/port -Isrc/include
-I../src/include
-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers
-I/usr/local/opt/readline/include -I/usr/local/opt/gettext/include
-I/usr/local/opt/zlib/include -I/usr/local/opt/openssl/include
-fcolor-diagnostics -Wall -Winvalid-pch -Wextra -O0 -g -isysroot
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fno-strict-aliasing
-fwrapv -Wmissing-prototypes -Wpointer-arith -Werror=vla -Wendif-labels
-Wmissing-format-attribute -Wformat-security -Wdeclaration-after-statement
-Wno-unused-command-line-argument -Wno-missing-field-initializers
-Wno-sign-compare -Wno-unused-parameter -msse4.2
-F/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework
-DFRONTEND -MD -MQ src/interfaces/libpq/libpq.5.dylib.p/fe-connect.c.o -MF
src/interfaces/libpq/libpq.5.dylib.p/fe-connect.c.o.d -o
src/interfaces/libpq/libpq.5.dylib.p/fe-connect.c.o -c
../src/interfaces/libpq/fe-connect.c
In file included from ../src/interfaces/libpq/fe-connect.c:72:
In file included from
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers/ldap.h:1:
[the last line is repeated a bunch of times, then...]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers/ldap.h:1:10:
error: #include nested too deeply
#include <ldap.h>
^
Then the expected "undeclared identifier" errors that would arise from a
missing header. I tried compiling --with-ldap with the Make build, and only
got warnings about deprecated declarations -- that build completed.
I tried disabling ldap with the Meson build but I'll spare the details of
what went wrong there in case I did something wrong, so we can take things
one step at a time.
> > That would also have to work for all the dependencies, which were
displayed
> > to me as:
> >
> > ninja, gdbm, ca-certificates, [email protected], readline, sqlite and
[email protected]
>
> meson does depend on ninja (to execute the build) and of course python.
But
> the rest should be optional dependencies. ninja builds without any
> dependencies as long as you don't change its parser sources. python
builds on
> aix, hpux etc.
>
> Not sure what way gdbm [email protected] and sqlite are pulled in? I assume
readline
> is for python...
Hmm, weird.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-13 16:37 ` Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-13 16:37 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 11:51:03 -0400, John Naylor wrote:
> On Tue, Oct 12, 2021 at 4:59 PM Andres Freund <[email protected]> wrote:
>
> > On 2021-10-12 15:55:22 -0400, John Naylor wrote:
> > > (I couldn't get a build working but I'll leave that aside for now).
> >
> > If you want to do that separately, I'll try to fix it.
>
> Okay, I pulled the latest commits and tried again:
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers/ldap.h:1:
>
> [the last line is repeated a bunch of times, then...]
Oh. I actually saw that on CI at some point... That one is definitely
odd. Currently CI for OSX builds like
- brew install make coreutils ccache icu4c lz4 tcl-tk openldap
- brew install meson ninja [email protected]
..
PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig:$PKG_CONFIG_PATH"
PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH"
PKG_CONFIG_PATH="/usr/local/opt/openldap/lib/pkgconfig:$PKG_CONFIG_PATH"
export PKG_CONFIG_PATH
meson setup --buildtype debug -Dcassert=true -Dssl=openssl build
but I set that up knowing little about macos.
For the autoconf build CI currently does something similar via
LIBS="/usr/local/lib:$LIBS"
INCLUDES="/usr/local/include:$INCLUDES"
...
LIBS="/usr/local/opt/openldap/lib:$LIBS"
INCLUDES="/usr/local/opt/openldap/include:$INCLUDES"
...
--with-includes="$INCLUDES" \
--with-libs="$LIBS" \
are you doing something like that? Or does it work for you without? I vaguely
recall hitting a similar problem as you report when not passing
/usr/local/... to configure.
> i tried disabling ldap with the meson build but i'll spare the details of
> what went wrong there in case i did something wrong, so we can take things
> one step at a time.
you can change it for an existing builddir with
meson configure -dldap=disabled or when setting up a new builddir by passing
-dldap=disabled at that time.
> > > ninja, gdbm, ca-certificates, [email protected], readline, sqlite and
> [email protected]
> >
> > meson does depend on ninja (to execute the build) and of course python.
> but
> > the rest should be optional dependencies. ninja builds without any
> > dependencies as long as you don't change its parser sources. python
> builds on
> > aix, hpux etc.
> >
> > not sure what way gdbm [email protected] and sqlite are pulled in? i assume
> readline
> > is for python...
>
> Hmm, weird.
They're homebrew python deps: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/[email protected]#L28
which are optional things enabled explicitly:
https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/[email protected]#L123
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-13 17:19 ` John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: John Naylor @ 2021-10-13 17:19 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Wed, Oct 13, 2021 at 12:37 PM Andres Freund <[email protected]> wrote:
> For the autoconf build CI currently does something similar via
> LIBS="/usr/local/lib:$LIBS"
> INCLUDES="/usr/local/include:$INCLUDES"
> ...
> LIBS="/usr/local/opt/openldap/lib:$LIBS"
> INCLUDES="/usr/local/opt/openldap/include:$INCLUDES"
> ...
> --with-includes="$INCLUDES" \
> --with-libs="$LIBS" \
>
> are you doing something like that? Or does it work for you without? I
vaguely
> recall hitting a similar problem as you report when not passing
> /usr/local/... to configure.
I didn't do anything like that for the autoconf build. I have in the past
done things retail, like
--with-icu ICU_CFLAGS='-I/usr/local/opt/icu4c/include/'
ICU_LIBS='-L/usr/local/opt/icu4c/lib/ -licui18n -licuuc -licudata'
> > i tried disabling ldap with the meson build but i'll spare the details
of
> > what went wrong there in case i did something wrong, so we can take
things
> > one step at a time.
>
> you can change it for an existing builddir with
> meson configure -dldap=disabled or when setting up a new builddir by
passing
> -dldap=disabled at that time.
Somehow our emails got lower-cased down here, but I tried it with capital D:
meson configure -Dldap=disabled
inside the build dir and got this:
../meson.build:278:2: ERROR: Tried to assign the invalid value "None" of
type NoneType to variable.
Line 278 is
ldap_r = ldap = dependency('', required : false)
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-13 17:42 ` Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-13 17:42 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-13 13:19:36 -0400, John Naylor wrote:
> On Wed, Oct 13, 2021 at 12:37 PM Andres Freund <[email protected]> wrote:
>
> > For the autoconf build CI currently does something similar via
> > LIBS="/usr/local/lib:$LIBS"
> > INCLUDES="/usr/local/include:$INCLUDES"
> > ...
> > LIBS="/usr/local/opt/openldap/lib:$LIBS"
> > INCLUDES="/usr/local/opt/openldap/include:$INCLUDES"
> > ...
> > --with-includes="$INCLUDES" \
> > --with-libs="$LIBS" \
> >
> > are you doing something like that? Or does it work for you without? I
> vaguely
> > recall hitting a similar problem as you report when not passing
> > /usr/local/... to configure.
>
> I didn't do anything like that for the autoconf build. I have in the past
> done things retail, like
I'll try to see how this works / what causes the breakage.
> Somehow our emails got lower-cased down here, but I tried it with capital D:
:)
> meson configure -Dldap=disabled
>
> inside the build dir and got this:
>
> ../meson.build:278:2: ERROR: Tried to assign the invalid value "None" of
> type NoneType to variable.
>
> Line 278 is
>
> ldap_r = ldap = dependency('', required : false)
Oops, I broke that when trying to clean things up. I guess I write too much C
;). It needs to be two lines.
I pushed the fix for that.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-13 18:40 ` John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: John Naylor @ 2021-10-13 18:40 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Wed, Oct 13, 2021 at 1:42 PM Andres Freund <[email protected]> wrote:
> I pushed the fix for that.
Ok great, it builds now! :-) Now something's off with dynamic loading.
There are libraries in ./tmp_install/usr/local/lib/ but apparently initdb
doesn't know to look for them there:
$ cat /Users/john/pgdev/meson/build/testrun/main/pg_regress/log/initdb.log
dyld: Library not loaded: /usr/local/lib/libpq.5.dylib
Referenced from:
/Users/john/pgdev/meson/build/tmp_install/usr/local/bin/initdb
Reason: image not found
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-14 19:14 ` John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: John Naylor @ 2021-10-14 19:14 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
I wrote:
> Ok great, it builds now! :-) Now something's off with dynamic loading.
There are libraries in ./tmp_install/usr/local/lib/ but apparently initdb
doesn't know to look for them there:
>
> $ cat /Users/john/pgdev/meson/build/testrun/main/pg_regress/log/initdb.log
> dyld: Library not loaded: /usr/local/lib/libpq.5.dylib
> Referenced from:
/Users/john/pgdev/meson/build/tmp_install/usr/local/bin/initdb
> Reason: image not found
After poking a bit more, this only happens when trying to run the tests. If
I specify a prefix, I can install, init, and start the server just fine, so
that much works.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-14 20:34 ` Andres Freund <[email protected]>
2021-10-14 20:54 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-14 20:34 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: pgsql-hackers
Hi,
On October 14, 2021 12:14:16 PM PDT, John Naylor <[email protected]> wrote:
>I wrote:
>
>> Ok great, it builds now! :-) Now something's off with dynamic loading.
>There are libraries in ./tmp_install/usr/local/lib/ but apparently initdb
>doesn't know to look for them there:
>>
>> $ cat /Users/john/pgdev/meson/build/testrun/main/pg_regress/log/initdb.log
>> dyld: Library not loaded: /usr/local/lib/libpq.5.dylib
>> Referenced from:
>/Users/john/pgdev/meson/build/tmp_install/usr/local/bin/initdb
>> Reason: image not found
>
>After poking a bit more, this only happens when trying to run the tests. If
>I specify a prefix, I can install, init, and start the server just fine, so
>that much works.
Is this a Mac with SIP enabled? The Mac CI presumably has that disabled, which is why I didn't see this issue there. Probably need to implement whatever Tom figured out to do about that for the current way of running tests.
Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-14 20:54 ` John Naylor <[email protected]>
2021-10-14 21:16 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 21:41 ` Re: [RFC] building postgres with meson Sergey Shinderuk <[email protected]>
1 sibling, 2 replies; 139+ messages in thread
From: John Naylor @ 2021-10-14 20:54 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Thu, Oct 14, 2021 at 4:34 PM Andres Freund <[email protected]> wrote:
> Is this a Mac with SIP enabled? The Mac CI presumably has that disabled,
which is why I didn't see this issue there. Probably need to implement
whatever Tom figured out to do about that for the current way of running
tests.
System Information says it's disabled. Running "csrutil status" complains
of an unsupported configuration, which doesn't sound good, so I should
probably go fix that independent of anything else. :-/
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 20:54 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-14 21:16 ` John Naylor <[email protected]>
2021-10-14 21:48 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: John Naylor @ 2021-10-14 21:16 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
I wrote:
> > Is this a Mac with SIP enabled? The Mac CI presumably has that
disabled, which is why I didn't see this issue there. Probably need to
implement whatever Tom figured out to do about that for the current way of
running tests.
>
> System Information says it's disabled. Running "csrutil status" complains
of an unsupported configuration, which doesn't sound good, so I should
probably go fix that independent of anything else. :-/
Looking online, I wonder if the "unsupported" message might be overly
cautious. In any case, I do remember turning something off to allow a
debugger to run. Here are all the settings, in case it matters:
Apple Internal: disabled
Kext Signing: enabled
Filesystem Protections: enabled
Debugging Restrictions: disabled
DTrace Restrictions: enabled
NVRAM Protections: enabled
BaseSystem Verification: enabled
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 20:54 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 21:16 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-14 21:48 ` Tom Lane <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Tom Lane @ 2021-10-14 21:48 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
John Naylor <[email protected]> writes:
>> System Information says it's disabled. Running "csrutil status" complains
>> of an unsupported configuration, which doesn't sound good, so I should
>> probably go fix that independent of anything else. :-/
> Looking online, I wonder if the "unsupported" message might be overly
> cautious. In any case, I do remember turning something off to allow a
> debugger to run. Here are all the settings, in case it matters:
> Apple Internal: disabled
> Kext Signing: enabled
> Filesystem Protections: enabled
> Debugging Restrictions: disabled
> DTrace Restrictions: enabled
> NVRAM Protections: enabled
> BaseSystem Verification: enabled
I remember having seen that report too, after some previous software
upgrade that had started from a "SIP disabled" status. I'm mostly
guessing here, but my guess is that
(a) csrutil only considers the all-enabled and all-disabled states
of these individual flags to be "supported" cases.
(b) some one or more of these flags came along in a macOS update,
and if you did the update starting from a "disabled" state, you
nonetheless ended up with the new flags enabled, leading to the
mixed state that csrutil complains about.
I've lost count of the number of times I've seen macOS updates
be sloppy about preserving non-default settings, so I don't find
theory (b) to be even slightly surprising.
Whether the mixed state is actually problematic in any way,
I dunno. I don't recall having had any problems before noticing
that that was what I had.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 20:54 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-14 21:41 ` Sergey Shinderuk <[email protected]>
1 sibling, 0 replies; 139+ messages in thread
From: Sergey Shinderuk @ 2021-10-14 21:41 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
Hi,
On 14.10.2021 23:54, John Naylor wrote:
> On Thu, Oct 14, 2021 at 4:34 PM Andres Freund <[email protected]
> <mailto:[email protected]>> wrote:
>
> > Is this a Mac with SIP enabled? The Mac CI presumably has that
> disabled, which is why I didn't see this issue there. Probably need to
> implement whatever Tom figured out to do about that for the current way
> of running tests.
>
> System Information says it's disabled. Running "csrutil status"
> complains of an unsupported configuration, which doesn't sound good, so
> I should probably go fix that independent of anything else. :-/
Maybe you could check that DYLD_LIBRARY_PATH is working for you?
% DYLD_FALLBACK_LIBRARY_PATH=
DYLD_LIBRARY_PATH=./tmp_install/usr/local/lib
./tmp_install/usr/local/bin/psql --version
psql (PostgreSQL) 15devel
Without DYLD_LIBRARY_PATH I get the error, as expected:
% DYLD_FALLBACK_LIBRARY_PATH= ./tmp_install/usr/local/bin/psql --version
dyld: Library not loaded: /usr/local/lib/libpq.5.dylib
Referenced from:
/Users/shinderuk/src/postgres-meson/build/./tmp_install/usr/local/bin/psql
Reason: image not found
I add "DYLD_FALLBACK_LIBRARY_PATH=" because otherwise dyld falls back to
/usr/lib/libpq.5.dylib provided by Apple (I am testing on Catalina).
% DYLD_PRINT_LIBRARIES=1 ./tmp_install/usr/local/bin/psql --version 2>&1
| grep libpq
dyld: loaded: <4EDF735E-2104-32AD-BE7B-B400ABFCF57C> /usr/lib/libpq.5.dylib
Regards,
--
Sergey Shinderuk https://postgrespro.com/
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-14 22:00 ` Tom Lane <[email protected]>
2021-10-14 22:23 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-10-14 22:55 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
1 sibling, 2 replies; 139+ messages in thread
From: Tom Lane @ 2021-10-14 22:00 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: John Naylor <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> Is this a Mac with SIP enabled? The Mac CI presumably has that disabled, which is why I didn't see this issue there. Probably need to implement whatever Tom figured out to do about that for the current way of running tests.
AFAIR the only cases we've made work are
(1) disable SIP
(2) avoid the need for (1) by always doing "make install" before
"make check".
Peter E. did some hacking towards another solution awhile ago,
but IIRC it involved changing the built binaries, and I think
we concluded that the benefits didn't justify that.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-10-14 22:23 ` Thomas Munro <[email protected]>
2021-10-14 22:40 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 23:15 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
1 sibling, 2 replies; 139+ messages in thread
From: Thomas Munro @ 2021-10-14 22:23 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
On Fri, Oct 15, 2021 at 11:00 AM Tom Lane <[email protected]> wrote:
> Peter E. did some hacking towards another solution awhile ago,
> but IIRC it involved changing the built binaries, and I think
> we concluded that the benefits didn't justify that.
Yeah, by now there are lots of useful blogs from various projects
figuring out that you can use the install_name_tool to adjust the
paths it uses to be absolute or relative to certain magic words, like
@executable_path/../lib/blah.dylib, which is tempting, but...
realistically, for serious hacking on a Mac, SIP is so annoying that
it isn't the only reason you'll want to turn it off: it stops
dtrace/dtruss/... from working, and somehow prevents debuggers from
working when you've ssh'd in from a remote machine with a proper
keyboard, and probably more things that I'm forgetting.
I wish I could find the Xnu source that shows exactly how and when the
environment is suppressed in this way to understand better, but it
doesn't jump out of Apple's github; maybe it's hiding in closed source
machinery...
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:23 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
@ 2021-10-14 22:40 ` Tom Lane <[email protected]>
2021-10-14 23:04 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Tom Lane @ 2021-10-14 22:40 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Andres Freund <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Thomas Munro <[email protected]> writes:
> I wish I could find the Xnu source that shows exactly how and when the
> environment is suppressed in this way to understand better, but it
> doesn't jump out of Apple's github; maybe it's hiding in closed source
> machinery...
I recall that we figured out awhile ago that the environment gets trimmed
when make (or whatever) executes some command via the shell; seemingly,
Apple has decided that /bin/sh is a security-critical program that mustn't
be run with a non-default DYLD_LIBRARY_PATH. Dunno if that helps you
find where the damage is done exactly.
(The silliness of this policy, when you pair it with the fact that they
don't reset PATH at the same time, seems blindingly obvious to me. But
apparently not to Apple.)
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:23 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-10-14 22:40 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-10-14 23:04 ` Tom Lane <[email protected]>
2021-10-15 00:36 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Tom Lane @ 2021-10-14 23:04 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Andres Freund <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
I wrote:
> I recall that we figured out awhile ago that the environment gets trimmed
> when make (or whatever) executes some command via the shell; seemingly,
> Apple has decided that /bin/sh is a security-critical program that mustn't
> be run with a non-default DYLD_LIBRARY_PATH. Dunno if that helps you
> find where the damage is done exactly.
BTW, here's the evidence for this theory:
[tgl@pro ~]$ cat checkenv.c
#include <stdio.h>
#include <stdlib.h>
int
main(int argc, char **argv)
{
char *pth = getenv("DYLD_LIBRARY_PATH");
if (pth)
printf("DYLD_LIBRARY_PATH = %s\n", pth);
else
printf("DYLD_LIBRARY_PATH is unset\n");
return 0;
}
[tgl@pro ~]$ gcc checkenv.c
[tgl@pro ~]$ ./a.out
DYLD_LIBRARY_PATH is unset
[tgl@pro ~]$ export DYLD_LIBRARY_PATH=/Users/tgl/pginstall/lib
[tgl@pro ~]$ ./a.out
DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
[tgl@pro ~]$ sh -c ./a.out
DYLD_LIBRARY_PATH is unset
[tgl@pro ~]$ ./a.out
DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
[tgl@pro ~]$ bash -c ./a.out
DYLD_LIBRARY_PATH is unset
You have to check the environment using an "unprivileged" program.
If you try to examine the environment using, say, "env", you will get
very misleading results. AFAICT, /usr/bin/env is *also* considered
security-critical, because I cannot get it to ever report that
DYLD_LIBRARY_PATH is set.
Hmm ... /usr/bin/perl seems to act the same way. It can see
ENV{'PATH'} but not ENV{'DYLD_LIBRARY_PATH'}.
This may indicate that they've applied this policy on a blanket
basis to everything in /bin and /usr/bin (and other system
directories, maybe), rather than singling out the shell.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:23 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-10-14 22:40 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 23:04 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-10-15 00:36 ` Thomas Munro <[email protected]>
2021-10-15 02:46 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Thomas Munro @ 2021-10-15 00:36 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
On Fri, Oct 15, 2021 at 12:04 PM Tom Lane <[email protected]> wrote:
> [tgl@pro ~]$ cat checkenv.c
> #include <stdio.h>
> #include <stdlib.h>
>
> int
> main(int argc, char **argv)
> {
> char *pth = getenv("DYLD_LIBRARY_PATH");
>
> if (pth)
> printf("DYLD_LIBRARY_PATH = %s\n", pth);
> else
> printf("DYLD_LIBRARY_PATH is unset\n");
>
> return 0;
> }
> [tgl@pro ~]$ gcc checkenv.c
> [tgl@pro ~]$ ./a.out
> DYLD_LIBRARY_PATH is unset
> [tgl@pro ~]$ export DYLD_LIBRARY_PATH=/Users/tgl/pginstall/lib
> [tgl@pro ~]$ ./a.out
> DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
> [tgl@pro ~]$ sh -c ./a.out
> DYLD_LIBRARY_PATH is unset
> [tgl@pro ~]$ ./a.out
> DYLD_LIBRARY_PATH = /Users/tgl/pginstall/lib
> [tgl@pro ~]$ bash -c ./a.out
> DYLD_LIBRARY_PATH is unset
>
> You have to check the environment using an "unprivileged" program.
> If you try to examine the environment using, say, "env", you will get
> very misleading results. AFAICT, /usr/bin/env is *also* considered
> security-critical, because I cannot get it to ever report that
> DYLD_LIBRARY_PATH is set.
>
> Hmm ... /usr/bin/perl seems to act the same way. It can see
> ENV{'PATH'} but not ENV{'DYLD_LIBRARY_PATH'}.
>
> This may indicate that they've applied this policy on a blanket
> basis to everything in /bin and /usr/bin (and other system
> directories, maybe), rather than singling out the shell.
Looks like it. If I've found the right code here, it looks like where
any common-or-garden Unix runtime linker would ignore LD_LIBRARY_PATH
for a setuid binary, they've trained theirs to whack DYLD_*, and also
for code-signed and __RESTRICT-marked executables.
https://github.com/opensource-apple/dyld/blob/master/src/dyld.cpp#L1681
I suppose you could point SHELL at an unsigned copy of sh (codesign
--remove-signature, or something from brew/ports/x) so that GNU make
should respect, but I don't know how many other exec("/bin/sh") calls
might be hiding around the place (I guess perl calls system()?) and
might require some kind of LD_PRELOAD hackery... not much fun.
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:23 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-10-14 22:40 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 23:04 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-15 00:36 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
@ 2021-10-15 02:46 ` Tom Lane <[email protected]>
2021-10-15 03:20 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Tom Lane @ 2021-10-15 02:46 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Andres Freund <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Thomas Munro <[email protected]> writes:
> On Fri, Oct 15, 2021 at 12:04 PM Tom Lane <[email protected]> wrote:
>> This may indicate that they've applied this policy on a blanket
>> basis to everything in /bin and /usr/bin (and other system
>> directories, maybe), rather than singling out the shell.
> Looks like it. If I've found the right code here, it looks like where
> any common-or-garden Unix runtime linker would ignore LD_LIBRARY_PATH
> for a setuid binary, they've trained theirs to whack DYLD_*, and also
> for code-signed and __RESTRICT-marked executables.
> https://github.com/opensource-apple/dyld/blob/master/src/dyld.cpp#L1681
Ugh. That explains it, all right.
> I suppose you could point SHELL at an unsigned copy of sh (codesign
> --remove-signature, or something from brew/ports/x) so that GNU make
> should respect, but I don't know how many other exec("/bin/sh") calls
> might be hiding around the place (I guess perl calls system()?) and
> might require some kind of LD_PRELOAD hackery... not much fun.
Yeah. I thought about invoking everything via a small wrapper
that restores the correct setting of DYLD_LIBRARY_PATH. We could
perhaps make that work for the invocations of test postmasters
and psqls from "make" and TAP scripts, but hacking up our code's
sundry uses of system(3) like that seems like it'd be very messy,
if feasible at all.
BTW, the POSIX spec explicitly discourages letting SHELL affect the
behavior of system(3), so I bet that wouldn't help.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:23 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-10-14 22:40 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 23:04 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-15 00:36 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-10-15 02:46 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-10-15 03:20 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-15 03:20 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Thomas Munro <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 22:46:07 -0400, Tom Lane wrote:
> Thomas Munro <[email protected]> writes:
> > I suppose you could point SHELL at an unsigned copy of sh (codesign
> > --remove-signature, or something from brew/ports/x) so that GNU make
> > should respect, but I don't know how many other exec("/bin/sh") calls
> > might be hiding around the place (I guess perl calls system()?) and
> > might require some kind of LD_PRELOAD hackery... not much fun.
>
> Yeah. I thought about invoking everything via a small wrapper
> that restores the correct setting of DYLD_LIBRARY_PATH. We could
> perhaps make that work for the invocations of test postmasters
> and psqls from "make" and TAP scripts, but hacking up our code's
> sundry uses of system(3) like that seems like it'd be very messy,
> if feasible at all.
It does sound like using relative rpaths might be the thing we want - and like
they've been available since 10.5 or something.
Is there a reason we're using absolute rpaths on a bunch of platforms, rather
than relative ones, which'd allow relocation?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:23 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
@ 2021-10-14 23:15 ` Andres Freund <[email protected]>
2021-10-14 23:23 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-14 23:15 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Tom Lane <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-15 11:23:00 +1300, Thomas Munro wrote:
> On Fri, Oct 15, 2021 at 11:00 AM Tom Lane <[email protected]> wrote:
> > Peter E. did some hacking towards another solution awhile ago,
> > but IIRC it involved changing the built binaries, and I think
> > we concluded that the benefits didn't justify that.
>
> Yeah, by now there are lots of useful blogs from various projects
> figuring out that you can use the install_name_tool to adjust the
> paths it uses to be absolute or relative to certain magic words, like
> @executable_path/../lib/blah.dylib, which is tempting, but...
> realistically, for serious hacking on a Mac, SIP is so annoying that
> it isn't the only reason you'll want to turn it off: it stops
> dtrace/dtruss/... from working, and somehow prevents debuggers from
> working when you've ssh'd in from a remote machine with a proper
> keyboard, and probably more things that I'm forgetting.
Meson has support for using install_name_tool to remove "build time" rpaths
and set "install time" rpaths during the installation process - which uses
install_name_tool on mac.
If, and perhaps that's too big an if, relative rpaths actually work despite
SIP, it might be worth setting a relative install_rpath, because afaict that
should then work both for a "real" installation and our temporary test one.
If absolute rpaths are required, it'd make the process a bit more expensive,
because we'd probably need to change a configure time option during the temporary
install. No actual rebuilds would be required, but still.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:23 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-10-14 23:15 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-14 23:23 ` Tom Lane <[email protected]>
2021-10-15 18:50 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Tom Lane @ 2021-10-14 23:23 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Thomas Munro <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> If, and perhaps that's too big an if, relative rpaths actually work despite
> SIP, it might be worth setting a relative install_rpath, because afaict that
> should then work both for a "real" installation and our temporary test one.
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:23 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-10-14 23:15 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 23:23 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-10-15 18:50 ` Andres Freund <[email protected]>
2021-10-15 22:36 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-15 18:50 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: Thomas Munro <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 19:23:58 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > If, and perhaps that's too big an if, relative rpaths actually work despite
> > SIP, it might be worth setting a relative install_rpath, because afaict that
> > should then work both for a "real" installation and our temporary test one.
>
> From what we know so far, it seems like SIP wouldn't interfere with
> that (if it works at all). I think what SIP desires to prevent is
> messing with a program's execution by setting DYLD_LIBRARY_PATH.
> As long as the program executable itself is saying where to find
> the library, I don't see why they should interfere with that.
Well, there's *some* danger with relative rpaths, because they might
accidentally be pointing somewhere non-existing and user-creatable. Not a huge
risk, but as you say:
> (Again, it seems blindingly stupid to forbid this while not blocking
> PATH or any of the other environment variables that have always affected
> execution. But what do I know.)
these aren't necessarily carefuly weighed considerations :/
But it seems to work well from what I gather.
> > If absolute rpaths are required, it'd make the process a bit more expensive,
>
> It'd also put the kibosh on relocatable install trees, though I dunno how
> much people really care about that.
We currently use absolute rpaths, or something equivalent.
The reason that running tests on macos works is that we set the "install_name"
of shared libraries to the intended installed location, using an absolute
path:
LINK.shared = $(COMPILER) -dynamiclib -install_name '$(libdir)/lib$(NAME).$(SO_MAJOR_VERSION)$(DLSUFFIX)' $(version_link) $(exported_symbols_list) -multiply_defined suppress
which on macos means that all libraries linking to that dylib reference it
under that absolute path.
On most other platforms we set an absolute rpath to the installation
directory, which has an equivalent effect:
rpathdir = $(libdir)
It seems to work quite well to change our own references to libpq in binaries
/ shared libs to be relative, but to leave the install_name of the libraries
intact. In combination with adding an rpath of @loader_path/../lib/ to
binaries and @loader_path/ to shlibs, the install will re relocatable.
It doesn't work as well to actually have a non-absolute install_name for
libraries (e.g. @rpath/libpq.dylib), because then external code linking to
libpq needs to add an rpath to the installation to make it work.
The advantage of this approach over Peter's is that it's not temp-install
specific - due to the relative paths, it makes installations relocatable
without relying [DY]LD_LIBRARY_PATH.
On other unixoid systems this whole mess is simpler, because we can just add
$ORIGIN to shared libraries and $ORIGIN/../lib/ to binaries. We don't need to
leave some absolute path in the libraries themself intact.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:23 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-10-14 23:15 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 23:23 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-15 18:50 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-15 22:36 ` Andres Freund <[email protected]>
2021-10-15 22:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-15 22:36 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: Thomas Munro <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-15 11:50:30 -0700, Andres Freund wrote:
> It seems to work quite well to change our own references to libpq in binaries
> / shared libs to be relative, but to leave the install_name of the libraries
> intact. In combination with adding an rpath of @loader_path/../lib/ to
> binaries and @loader_path/ to shlibs, the install will re relocatable.
>
> It doesn't work as well to actually have a non-absolute install_name for
> libraries (e.g. @rpath/libpq.dylib), because then external code linking to
> libpq needs to add an rpath to the installation to make it work.
>
> The advantage of this approach over Peter's is that it's not temp-install
> specific - due to the relative paths, it makes installations relocatable
> without relying [DY]LD_LIBRARY_PATH.
>
> On other unixoid systems this whole mess is simpler, because we can just add
> $ORIGIN to shared libraries and $ORIGIN/../lib/ to binaries. We don't need to
> leave some absolute path in the libraries themself intact.
I implemented this for the meson build, and it seems to work nicely. The macos
part was harder than I hoped due to the install_name stuff, which meson
doesn't solve.
https://github.com/anarazel/postgres/commit/a35379c28989469cc4b701a8d7a22422e6302e09
After that the build directory is relocatale.
I don't immediately see a way to do this reasonably for the autoconf
build. We'd need a list of our own shared libraries from somewhere, and then
replace the references after building the objects?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:23 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-10-14 23:15 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 23:23 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-15 18:50 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-15 22:36 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-15 22:47 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-15 22:47 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: Thomas Munro <[email protected]>; John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-15 15:36:16 -0700, Andres Freund wrote:
> On 2021-10-15 11:50:30 -0700, Andres Freund wrote:
> > It seems to work quite well to change our own references to libpq in binaries
> > / shared libs to be relative, but to leave the install_name of the libraries
> > intact. In combination with adding an rpath of @loader_path/../lib/ to
> > binaries and @loader_path/ to shlibs, the install will re relocatable.
> >
> > It doesn't work as well to actually have a non-absolute install_name for
> > libraries (e.g. @rpath/libpq.dylib), because then external code linking to
> > libpq needs to add an rpath to the installation to make it work.
> >
> > The advantage of this approach over Peter's is that it's not temp-install
> > specific - due to the relative paths, it makes installations relocatable
> > without relying [DY]LD_LIBRARY_PATH.
> >
> > On other unixoid systems this whole mess is simpler, because we can just add
> > $ORIGIN to shared libraries and $ORIGIN/../lib/ to binaries. We don't need to
> > leave some absolute path in the libraries themself intact.
>
> I implemented this for the meson build, and it seems to work nicely. The macos
> part was harder than I hoped due to the install_name stuff, which meson
> doesn't solve.
>
> https://github.com/anarazel/postgres/commit/a35379c28989469cc4b701a8d7a22422e6302e09
>
> After that the build directory is relocatale.
Well, now that I think about it, it's still only relocatable in the sense that
postgres itself will continue to work. Outside code linking to e.g. libpq will
get the wrong path after relocation the source tree, due to the absolute
install_name.
But that doesn't seem solvable, unless we make the installed install_name to
be '@rpath/libpq...dylib' and require code linking to libpq to pass
-Wl,-rpath,/path/to/libpq when linking to libpq.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-10-14 22:55 ` Andres Freund <[email protected]>
2021-10-14 23:27 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-14 22:55 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: John Naylor <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 18:00:49 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > Is this a Mac with SIP enabled? The Mac CI presumably has that disabled, which is why I didn't see this issue there. Probably need to implement whatever Tom figured out to do about that for the current way of running tests.
>
> AFAIR the only cases we've made work are
>
> (1) disable SIP
>
> (2) avoid the need for (1) by always doing "make install" before
> "make check".
Ah, I thought it was more than that. In that case, John, does meson's test
succeed after you did the "proper" install? Assuming it's in a path that's
allowed to provide shared libraries?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:55 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-14 23:27 ` John Naylor <[email protected]>
2021-10-15 00:02 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: John Naylor @ 2021-10-14 23:27 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Thu, Oct 14, 2021 at 6:55 PM Andres Freund <[email protected]> wrote:
> Ah, I thought it was more than that. In that case, John, does meson's test
> succeed after you did the "proper" install? Assuming it's in a path that's
> allowed to provide shared libraries?
Oh, it can act like installcheck? [checks] Yep, "meson test" ran fine (*).
It still ran the temp install first, but in any case it worked. The full
"configure step" was
meson setup build --buildtype debug -Dldap=disabled -Dcassert=true
-Dprefix=$(pwd)/inst
* (all passed but skipped subscription/t/012_collation.pl)
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 16:37 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 17:19 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-13 17:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 18:40 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 19:14 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-14 20:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 22:00 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-14 22:55 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-14 23:27 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-15 00:02 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-15 00:02 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
Hi,
On 2021-10-14 19:27:17 -0400, John Naylor wrote:
> On Thu, Oct 14, 2021 at 6:55 PM Andres Freund <[email protected]> wrote:
> > Ah, I thought it was more than that. In that case, John, does meson's test
> > succeed after you did the "proper" install? Assuming it's in a path that's
> > allowed to provide shared libraries?
>
> Oh, it can act like installcheck? [checks] Yep, "meson test" ran fine (*).
> It still ran the temp install first, but in any case it worked.
As far as I understand Tom, our normal make check only works on OSX if
previously you ran make install. Which will have installed libpq into the
"proper" install location. Because all our binaries will, by default, have an
rpath to the library directory embedded, that then allows binaries in the
temporary install to work. But using the wrong libpq - which most of the time
turns out to be harmless, because libpq doesn't change that rapidly.
> * (all passed but skipped subscription/t/012_collation.pl)
That test requires ICU, so that's fine. I guess we could prevent the test from
being executed in the first place, but I don't think we've done that for cases
where it's one specific test in a t/ directory, where others in the same
directory do not have such dependencies.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-12 20:59 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-13 15:51 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-14 04:27 ` Thomas Munro <[email protected]>
1 sibling, 0 replies; 139+ messages in thread
From: Thomas Munro @ 2021-10-14 04:27 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
On Thu, Oct 14, 2021 at 4:51 AM John Naylor
<[email protected]> wrote:
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers/ldap.h:1:10: error: #include nested too deeply
> #include <ldap.h>
> ^
I vaguely recall that PostgreSQL should build OK against Apple's copy
of OpenLDAP. That recursive include loop is coming from a "framework"
header that contains just a couple of lines like #include <ldap.h> to
try to include the real header, which should also be in the include
path, somewhere like
/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include/ldap.h.
I think we'd need to figure out where that
-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers
directive is coming from and get rid of it, so we can include the real
header directly.
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-21 21:48 ` Andres Freund <[email protected]>
2021-10-22 15:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-21 21:48 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-12 15:55:22 -0400, John Naylor wrote:
> Also, could utility makefile targets be made to work? I'm thinking in
> particular of update-unicode and reformat-dat-files, for example.
Implementing reformat-dat-files was trivial:
https://github.com/anarazel/postgres/commit/29c1ce1ad4731290714978da5ce81e99ef051bec
However, update-unicode is a bit harder. Partially not directly because of
meson, but because update-unicode as-is afaict doesn't support VPATH builds,
and meson enforces those.
make update-unicode
...
make -C src/common/unicode update-unicode
'/usr/bin/perl' generate-unicode_norm_table.pl
Can't open perl script "generate-unicode_norm_table.pl": No such file or directory
It's not too hard to fix. See attached for the minimal stuff that I
immediately found to be needed. There's likely more,
e.g. src/backend/utils/mb/Unicode - but I didn't immediately see where that's
invoked from.
The slightly bigger issue making update-unicode work with meson is that meson
doesn't provide support for invoking build targets in specific directories
(because it doesn't map nicely to e.g. msbuild). But scripts like
src/common/unicode/generate-unicode_norm_table.pl rely on CWD. It's not hard
to work around that, but IMO it's better for such scripts to not rely on CWD.
Greetings,
Andres Freund
Attachments:
[text/x-diff] update-unicode.diff (2.2K, ../../[email protected]/2-update-unicode.diff)
download | inline diff:
diff --git i/src/common/unicode/Makefile w/src/common/unicode/Makefile
index a3683dd86b9..e69054d4671 100644
--- i/src/common/unicode/Makefile
+++ w/src/common/unicode/Makefile
@@ -12,14 +12,14 @@ subdir = src/common/unicode
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
+override CPPFLAGS := -DFRONTEND -I$(abs_top_builddir)/src/common/unicode $(CPPFLAGS)
LIBS += $(PTHREAD_LIBS)
# By default, do nothing.
all:
update-unicode: unicode_norm_table.h unicode_combining_table.h unicode_east_asian_fw_table.h unicode_normprops_table.h unicode_norm_hashfunc.h
- mv $^ ../../../src/include/common/
+ mv $^ $(top_srcdir)/src/include/common/
$(MAKE) normalization-check
# These files are part of the Unicode Character Database. Download
@@ -33,7 +33,7 @@ UnicodeData.txt EastAsianWidth.txt DerivedNormalizationProps.txt CompositionExcl
unicode_norm_hashfunc.h: unicode_norm_table.h
unicode_norm_table.h: generate-unicode_norm_table.pl UnicodeData.txt CompositionExclusions.txt
- $(PERL) generate-unicode_norm_table.pl
+ $(PERL) $^
unicode_combining_table.h: generate-unicode_combining_table.pl UnicodeData.txt
$(PERL) $^ >$@
@@ -58,7 +58,7 @@ submake-common:
$(MAKE) -C .. all
norm_test_table.h: generate-norm_test_table.pl NormalizationTest.txt
- perl generate-norm_test_table.pl NormalizationTest.txt $@
+ perl $^ $@
.PHONY: normalization-check
diff --git i/contrib/unaccent/Makefile w/contrib/unaccent/Makefile
index b8307d1601e..d6c466e07ad 100644
--- i/contrib/unaccent/Makefile
+++ w/contrib/unaccent/Makefile
@@ -27,12 +27,12 @@ include $(top_builddir)/src/Makefile.global
include $(top_srcdir)/contrib/contrib-global.mk
endif
-update-unicode: unaccent.rules
+update-unicode: $(srcdir)/unaccent.rules
# Allow running this even without --with-python
PYTHON ?= python
-unaccent.rules: generate_unaccent_rules.py ../../src/common/unicode/UnicodeData.txt Latin-ASCII.xml
+$(srcdir)/unaccent.rules: generate_unaccent_rules.py ../../src/common/unicode/UnicodeData.txt Latin-ASCII.xml
$(PYTHON) $< --unicode-data-file $(word 2,$^) --latin-ascii-file $(word 3,$^) >$@
# Only download it once; dependencies must match src/common/unicode/
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 19:55 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-21 21:48 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-22 15:55 ` John Naylor <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: John Naylor @ 2021-10-22 15:55 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
On Thu, Oct 21, 2021 at 5:48 PM Andres Freund <[email protected]> wrote:
> However, update-unicode is a bit harder. Partially not directly because
of
> meson, but because update-unicode as-is afaict doesn't support VPATH
builds,
> and meson enforces those.
> make update-unicode
> ...
> make -C src/common/unicode update-unicode
> '/usr/bin/perl' generate-unicode_norm_table.pl
> Can't open perl script "generate-unicode_norm_table.pl": No such file or
directory
>
> It's not too hard to fix. See attached for the minimal stuff that I
> immediately found to be needed.
Thanks for doing that, it works well enough for demonstration. With your
patch, and using an autoconf VPATH build, the unicode tables work fine, but
it complains of a permission error in generate_unaccent_rules.py. That
seems to be because the script is invoked directly rather than as an
argument to the python interpreter.
> The slightly bigger issue making update-unicode work with meson is that
meson
> doesn't provide support for invoking build targets in specific directories
> (because it doesn't map nicely to e.g. msbuild). But scripts like
> src/common/unicode/generate-unicode_norm_table.pl rely on CWD. It's not
hard
> to work around that, but IMO it's better for such scripts to not rely on
CWD.
Yeah. I encountered a further issue: With autoconf on HEAD, with a source
tree build executed in contrib/unaccent:
$ touch generate_unaccent_rules.py
$ make update-unicode
generate_unaccent_rules.py --unicode-data-file
../../src/common/unicode/UnicodeData.txt --latin-ascii-file Latin-ASCII.xml
>unaccent.rules
/bin/sh: generate_unaccent_rules.py: command not found
make: *** [unaccent.rules] Error 127
make: *** Deleting file `unaccent.rules'
...so in this case it seems not to know to use CWD here.
Anyway, this can be put off until the very end, since it's not run often.
You've demonstrated how these targets would work, and that's good enough
for now.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-19 18:34 ` Andres Freund <[email protected]>
2021-10-19 19:22 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
9 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-19 18:34 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi Tom,
On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
> As far as I can tell the only OS that postgres currently supports that
> meson doesn't support is HPUX. It'd likely be fairly easy to add
> gcc-on-hpux support, a chunk more to add support for the proprietary
> ones.
Tom, wrt HPUX on pa-risc, what are your thoughts there? IIRC we gave up
supporting HP's compiler on pa-risc a while ago.
As I said it'd probably not be too hard to add meson support for hpux on hppa,
it's probably just a few branches. But that'd require access somewhere. The
gcc compile farm does not have a hppa member anymore...
I did notice that gcc will declare hppa-hpux obsolete in gcc 12 and will
remove at some point:
"The hppa[12]*-*-hpux10* and hppa[12]*-*-hpux11* configurations targeting 32-bit PA-RISC with HP-UX have been obsoleted and will be removed in a future release."
https://gcc.gnu.org/gcc-12/changes.html
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-19 19:22 ` Tom Lane <[email protected]>
2021-10-19 21:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-20 01:08 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Tom Lane @ 2021-10-19 19:22 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
>> As far as I can tell the only OS that postgres currently supports that
>> meson doesn't support is HPUX. It'd likely be fairly easy to add
>> gcc-on-hpux support, a chunk more to add support for the proprietary
>> ones.
> Tom, wrt HPUX on pa-risc, what are your thoughts there? IIRC we gave up
> supporting HP's compiler on pa-risc a while ago.
Right. I am still testing with gcc on HP-PA. I'd kind of like to
keep it running just as an edge case for our spinlock support, but
I'm not sure that I want to do any huge amount of work on meson
to keep that going.
I do have a functioning OpenBSD installation on that machine, so
one alternative if the porting costs look too high is to replace
gaur with an OpenBSD animal. However, last I checked, OpenBSD
was about half the speed of HPUX on that hardware, so I'm not
real eager to go that way. gaur's already about the slowest
animal in the farm :-(
> As I said it'd probably not be too hard to add meson support for hpux on hppa,
> it's probably just a few branches. But that'd require access somewhere. The
> gcc compile farm does not have a hppa member anymore...
If you've got an idea where to look, I could add that to my
to-do queue.
In any case, I don't think we need to consider HPUX as a blocker
for the meson approach. The value-add from keeping gaur going
probably isn't terribly much. I'm more concerned about the
effort involved in getting meson going on some other old animals,
such as prairiedog.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 19:22 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-10-19 21:57 ` John Naylor <[email protected]>
2021-10-20 00:31 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: John Naylor @ 2021-10-19 21:57 UTC (permalink / raw)
To: Andres Freund <[email protected]>; Thomas Munro <[email protected]>; +Cc: pgsql-hackers
Hi,
I know this is still in the evaluation stage, but I did notice some
discrepencies in the Flex flags. With the attached patch, the read-only
data segment seems to match up pretty well now.
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] sync-flex-flags-with-autoconf-build.patch (5.1K, ../../CAFBsxsF3qasUjUrpsV5Ld1Q41A1YERJeUaUoc4zu0aGBfWOUrA@mail.gmail.com/3-sync-flex-flags-with-autoconf-build.patch)
download | inline diff:
diff --git a/contrib/cube/meson.build b/contrib/cube/meson.build
index 49276aed64..3cf7ebdd8e 100644
--- a/contrib/cube/meson.build
+++ b/contrib/cube/meson.build
@@ -6,7 +6,7 @@ cube_sources = files(
cubescan = custom_target('cubescan',
input: ['cubescan.l'],
output: ['cubescan.c'],
- command: [flex, '-CFe', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
+ command: [flex, '-o', '@OUTPUT@', '@INPUT@'])
cube_sources += custom_target('cubeparse',
input: 'cubeparse.y',
diff --git a/src/backend/parser/meson.build b/src/backend/parser/meson.build
index 491eacf20b..5ce4d09f31 100644
--- a/src/backend/parser/meson.build
+++ b/src/backend/parser/meson.build
@@ -28,7 +28,7 @@ parser_sources = [files('parser.c')]
backend_scanner = custom_target('scan',
input: ['scan.l'],
output: ['scan.c'],
- command: [flex, '-CF', '-p', '-p', '-o', '@OUTPUT@', '@INPUT0@'])
+ command: [flex, '-b', '-CF', '-p', '-p', '-o', '@OUTPUT@', '@INPUT0@'])
parser_sources += backend_scanner[0]
parser_sources += backend_parser_header[0]
diff --git a/src/backend/replication/meson.build b/src/backend/replication/meson.build
index 2573f166d7..ee12c6d49d 100644
--- a/src/backend/replication/meson.build
+++ b/src/backend/replication/meson.build
@@ -17,7 +17,7 @@ backend_sources += files(
repl_scanner = custom_target('repl_scanner',
input : files('repl_scanner.l'),
output : ['repl_scanner.c'],
- command : [flex, '-CF', '-p', '-p', '-o', '@OUTPUT0@', '@INPUT@']
+ command: [flex, '-o', '@OUTPUT@', '@INPUT@']
)
generated_backend_sources += custom_target('repl_gram',
@@ -30,7 +30,7 @@ generated_backend_sources += custom_target('repl_gram',
syncrep_scanner = custom_target('syncrep_scanner',
input: 'syncrep_scanner.l',
output: 'syncrep_scanner.c',
- command: [flex, '-CF', '-p', '-p', '-o', '@OUTPUT0@', '@INPUT@'])
+ command: [flex, '-o', '@OUTPUT@', '@INPUT@'])
generated_backend_sources += custom_target('syncrep_gram',
input: 'syncrep_gram.y',
diff --git a/src/backend/utils/adt/meson.build b/src/backend/utils/adt/meson.build
index 086fde8ff0..e1cea1eb4e 100644
--- a/src/backend/utils/adt/meson.build
+++ b/src/backend/utils/adt/meson.build
@@ -109,7 +109,7 @@ backend_sources += files(
jsonpath_scan = custom_target('jsonpath_scan',
input: ['jsonpath_scan.l'],
output: ['jsonpath_scan.c'],
- command: [flex, '-CF', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
+ command: [flex, '-b', '-CF', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
# jsonpath_scan is compiled as part of jsonpath_gram
generated_backend_sources += custom_target('jsonpath_parse',
diff --git a/src/backend/utils/misc/meson.build b/src/backend/utils/misc/meson.build
index 5274c8aa1a..2c0090ad33 100644
--- a/src/backend/utils/misc/meson.build
+++ b/src/backend/utils/misc/meson.build
@@ -18,7 +18,7 @@ backend_sources += files(
guc_scan = custom_target('guc_scan',
input: ['guc-file.l'],
output: ['guc-file.c.h'],
- command: [flex, '-CF', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
+ command: [flex, '-o', '@OUTPUT@', '@INPUT@'])
generated_backend_sources += guc_scan
diff --git a/src/bin/pgbench/meson.build b/src/bin/pgbench/meson.build
index 5c4a778ff3..bc135abebf 100644
--- a/src/bin/pgbench/meson.build
+++ b/src/bin/pgbench/meson.build
@@ -9,7 +9,7 @@ pgbench_sources = files(
exprscan = custom_target('exprscan',
input : files('exprscan.l'),
output : ['exprscan.c'],
- command : [flex, '-CF', '-p', '-p', '-o', '@OUTPUT0@', '@INPUT@']
+ command : [flex, '-o', '@OUTPUT0@', '@INPUT@']
)
exprparse = custom_target('exprparse',
diff --git a/src/bin/psql/meson.build b/src/bin/psql/meson.build
index 98921f801d..e56beb28e1 100644
--- a/src/bin/psql/meson.build
+++ b/src/bin/psql/meson.build
@@ -18,7 +18,7 @@ psql_sources = files(
psql_sources += custom_target('psqlscanslash',
input: ['psqlscanslash.l'],
output: ['psqlscanslash.c'],
- command: [flex, '-CFe', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
+ command: [flex, '-b', '-Cfe', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
psql_sources += custom_target('psql_help',
input: ['create_help.pl'],
diff --git a/src/fe_utils/meson.build b/src/fe_utils/meson.build
index b305727d96..e3f0b34cf1 100644
--- a/src/fe_utils/meson.build
+++ b/src/fe_utils/meson.build
@@ -16,7 +16,7 @@ fe_utils_sources = files(
fe_utils_sources += custom_target('psqlscan',
input: ['psqlscan.l'],
output: ['psqlscan.c'],
- command: [flex, '-Cfe', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
+ command: [flex, '-b', '-Cfe', '-p', '-p', '-o', '@OUTPUT@', '@INPUT@'])
fe_utils = static_library('fe_utils',
fe_utils_sources + generated_headers,
diff --git a/src/test/isolation/meson.build b/src/test/isolation/meson.build
index 637b480755..ea8baa2063 100644
--- a/src/test/isolation/meson.build
+++ b/src/test/isolation/meson.build
@@ -8,7 +8,7 @@ isolation_sources = pg_regress_c + files(
spec_scanner = custom_target('specscanner',
input : files('specscanner.l'),
output : ['specscanner.c'],
- command : [flex, '-CF', '-p', '-p', '-o', '@OUTPUT0@', '@INPUT@']
+ command : [flex, '-o', '@OUTPUT0@', '@INPUT@']
)
isolationtester_sources = files('isolationtester.c')
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 19:22 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-19 21:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2021-10-20 00:31 ` Andres Freund <[email protected]>
2021-10-20 01:10 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-20 01:35 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-20 00:31 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
Hi,
On 2021-10-19 17:57:31 -0400, John Naylor wrote:
> I know this is still in the evaluation stage, but I did notice some
> discrepencies in the Flex flags. With the attached patch, the read-only
> data segment seems to match up pretty well now.
Good catch. I think I just copied them around...
I wish we had a bit more consistency in the flags, so we could centralize
them. Seems there's no reason to not use -p -p and -b everywhere?
I also need to make meson use our flex wrapper for the relevant versions... I
can see the warning that'd be fixed by it on macos CI. Will do that and push
it out to my github repo together with your changes.
Thanks!
Andres
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 19:22 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-19 21:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-20 00:31 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-20 01:10 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 139+ messages in thread
From: Tom Lane @ 2021-10-20 01:10 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: John Naylor <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> I wish we had a bit more consistency in the flags, so we could centralize
> them. Seems there's no reason to not use -p -p and -b everywhere?
I don't think we care enough about performance of most of the scanners
to make them all backup-free, so -1 to that idea.
We could possibly replace the command line switches with %option
entries in the files themselves. But I think the reason we haven't
done so for -b is that the Makefile still needs to know about it
so as to know what to do with the lex.backup output file.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 19:22 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-19 21:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2021-10-20 00:31 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-20 01:35 ` Andres Freund <[email protected]>
1 sibling, 0 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-20 01:35 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
Hi,
On 2021-10-19 17:31:22 -0700, Andres Freund wrote:
> I also need to make meson use our flex wrapper for the relevant versions... I
> can see the warning that'd be fixed by it on macos CI. Will do that and push
> it out to my github repo together with your changes.
That turned out to be more work than I anticipated, so I pushed your changes
out separately.
There's this bit in plflex.pl that talks about adjusting yywrap() for msvc. I
didn't implement that and didn't see any compilation problems. Looks like that
originally hails from 2011, in 08a0c2dabc3b9d59d72d7a79ed867b8e37d275a7
Hm. Seems not worth carrying forward unless it actually causes trouble?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 19:22 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-10-20 01:08 ` Andres Freund <[email protected]>
2021-10-20 01:26 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-20 01:08 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-19 15:22:15 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > On 2021-10-12 01:37:21 -0700, Andres Freund wrote:
> >> As far as I can tell the only OS that postgres currently supports that
> >> meson doesn't support is HPUX. It'd likely be fairly easy to add
> >> gcc-on-hpux support, a chunk more to add support for the proprietary
> >> ones.
>
> > Tom, wrt HPUX on pa-risc, what are your thoughts there? IIRC we gave up
> > supporting HP's compiler on pa-risc a while ago.
>
> Right. I am still testing with gcc on HP-PA. I'd kind of like to
> keep it running just as an edge case for our spinlock support, but
> I'm not sure that I want to do any huge amount of work on meson
> to keep that going.
Makes sense. While that does test an odd special case for our spinlock
implementation, it's also the only supported platform with that edge case, and
it seems extremely unlikely that there ever will be a new platform with such
odd/limited atomic operations.
> I do have a functioning OpenBSD installation on that machine, so
> one alternative if the porting costs look too high is to replace
> gaur with an OpenBSD animal. However, last I checked, OpenBSD
> was about half the speed of HPUX on that hardware, so I'm not
> real eager to go that way. gaur's already about the slowest
> animal in the farm :-(
Yea, that doesn't sound enticing. Seems like we either should keep it running
on hp-ux or just drop parisc support?
> > As I said it'd probably not be too hard to add meson support for hpux on hppa,
> > it's probably just a few branches. But that'd require access somewhere. The
> > gcc compile farm does not have a hppa member anymore...
>
> If you've got an idea where to look, I could add that to my to-do queue.
It might even just work. Looks like meson does have pa-risc detection. While
it doesn't have any specifically for hpux, it just falls back to python's
sys.platform in that case. python3 -c 'import sys;print(sys.platform)'
meson generates output for ninja to execute (basically a faster make that's
partially faster by being much less flexible. Intended to be output by more
user-friendly buildsystems ). Ninja can be built by a minimal python script,
or with cmake. The former doesn't seem to have hpux support, the latter does I
think.
https://github.com/ninja-build/ninja
So it could be interesting to see if ninja builds.
I've not taught the PG meson the necessary stuff for a 32 bit build. So
there's no point is trying whether meson works that much. I'll try to do that,
and let you know.
> I'm more concerned about the effort involved in getting meson going on some
> other old animals, such as prairiedog.
Yea, that's an *old* OS version. One version too old to have support for
@rpath, added in 10.5 :(. Is there a reason to run 10.4 specifically?
According to wikipedia 10.5 is the last version to support ppc.
Looks like python still supports building back to 10.4.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 19:22 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-20 01:08 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-20 01:26 ` Tom Lane <[email protected]>
2021-10-20 01:49 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Tom Lane @ 2021-10-20 01:26 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2021-10-19 15:22:15 -0400, Tom Lane wrote:
>> I'm more concerned about the effort involved in getting meson going on some
>> other old animals, such as prairiedog.
> Yea, that's an *old* OS version. One version too old to have support for
> @rpath, added in 10.5 :(. Is there a reason to run 10.4 specifically?
> According to wikipedia 10.5 is the last version to support ppc.
My notes say
Currently running OSX 10.4.11 (last release of Tiger); although 10.5 Leopard
supports PPCs, it refuses to install if CPU speed < 867MHz, well beyond the
Cube's ability. Wikipedia does suggest it's possible to run Leopard, but...
https://en.wikipedia.org/wiki/Mac_OS_X_Leopard#Usage_on_unsupported_hardware
I'm not sure that I have install media for 10.5 anymore, either --- ISTR
some machine's CD drive failing and not letting me get the CD back out.
If I did have it, I don't think there'd be a way to update past 10.5.0
(surely Apple no longer has those updaters on-line?), so on the whole
I think that path is a nonstarter.
I do have 10.5 running on an old G4 PowerMac, but that machine is (a)
noisy (b) power-hungry and (c) getting flaky, so I'm uneager to spin up
a buildfarm animal on it.
As with the HPPA, a potential compromise is to spin up some newer
BSD-ish system on it. I agree that OSX 10.4 is uninteresting as a
software platform, but I'd like to keep 32-bit PPC represented in
the farm.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 19:22 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-20 01:08 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-20 01:26 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-10-20 01:49 ` Andres Freund <[email protected]>
2021-10-20 02:04 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-20 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-20 01:49 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-19 21:26:53 -0400, Tom Lane wrote:
> My notes say
>
> Currently running OSX 10.4.11 (last release of Tiger); although 10.5 Leopard
> supports PPCs, it refuses to install if CPU speed < 867MHz, well beyond the
> Cube's ability. Wikipedia does suggest it's possible to run Leopard, but...
> https://en.wikipedia.org/wiki/Mac_OS_X_Leopard#Usage_on_unsupported_hardware
>
> I'm not sure that I have install media for 10.5 anymore, either --- ISTR
> some machine's CD drive failing and not letting me get the CD back out.
> If I did have it, I don't think there'd be a way to update past 10.5.0
> (surely Apple no longer has those updaters on-line?), so on the whole
> I think that path is a nonstarter.
That does indeed sound like a nonstarter.
> I do have 10.5 running on an old G4 PowerMac, but that machine is (a)
> noisy (b) power-hungry and (c) getting flaky, so I'm uneager to spin up
> a buildfarm animal on it.
Understandable.
> As with the HPPA, a potential compromise is to spin up some newer
> BSD-ish system on it. I agree that OSX 10.4 is uninteresting as a
> software platform, but I'd like to keep 32-bit PPC represented in
> the farm.
I assume the reason 32-bit PPC is interesting is that it's commonly run big
endian?
I wonder when it'll be faster to run 32bit ppc via qemu than natively :)
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 19:22 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-20 01:08 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-20 01:26 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-20 01:49 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-20 02:04 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 139+ messages in thread
From: Tom Lane @ 2021-10-20 02:04 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2021-10-19 21:26:53 -0400, Tom Lane wrote:
>> As with the HPPA, a potential compromise is to spin up some newer
>> BSD-ish system on it. I agree that OSX 10.4 is uninteresting as a
>> software platform, but I'd like to keep 32-bit PPC represented in
>> the farm.
> I assume the reason 32-bit PPC is interesting is that it's commonly run big
> endian?
Aside from bit width and endianness, I believe it's a somewhat smaller
instruction set than the newer CPUs.
> I wonder when it'll be faster to run 32bit ppc via qemu than natively :)
I think qemu would have a ways to go for that. More to the point,
I've found that its emulation is not as precise as one might wish...
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 19:22 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-20 01:08 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-20 01:26 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-20 01:49 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-20 02:41 ` Andres Freund <[email protected]>
2021-10-20 16:01 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-10-20 02:41 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-19 18:49:43 -0700, Andres Freund wrote:
> I wonder when it'll be faster to run 32bit ppc via qemu than natively :)
Freebsd didn't seem to want to boot, but surprisingly a debian buster image
started at least the installer without problems... Will probably take a while
to see if it actually works.
I assume to make it acceptable from a build-speed perspective one would have
to use distcc with the compiler running outside.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-19 19:22 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-20 01:08 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-20 01:26 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-10-20 01:49 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-20 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-10-20 16:01 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Andres Freund @ 2021-10-20 16:01 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-10-19 19:41:56 -0700, Andres Freund wrote:
> On 2021-10-19 18:49:43 -0700, Andres Freund wrote:
> > I wonder when it'll be faster to run 32bit ppc via qemu than natively :)
>
> Freebsd didn't seem to want to boot, but surprisingly a debian buster image
> started at least the installer without problems... Will probably take a while
> to see if it actually works.
The build was quite slow (cold ccache cache, only 1 cpu):
real 106m33.418s
user 86m36.363s
sys 17m33.830s
But the actual test time wasn't *too* bad, compared to the 32bit ppc animals
real 12m14.944s
user 0m51.622s
sys 0m44.743s
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-11-15 18:34 ` Andres Freund <[email protected]>
2021-11-15 19:11 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
9 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2021-11-15 18:34 UTC (permalink / raw)
To: pgsql-hackers; Tom Lane <[email protected]>
Hi,
FWIW, I tried building postgres on a few other operating systems using
meson, after I got access to the gcc compile farm. Here's the results:
- openbsd: Compiled fine. Hit one issue running tests:
openbsd has *completely* broken $ORIGIN support. It uses CWD as $ORIGIN
rpaths, which obviously breaks for binaries invoked via PATH. So there
goes the idea to only use $ORIGIN to run tests. Still seems worth to use
on other platforms, particularly because it works with SIP on macos
I understand not supporting $ORIGIN at all. But implementing it this way
seems insane.
I also ran into some problems with the semaphore limits. I had to switch to
USE_NAMED_POSIX_SEMAPHORES to make the tests pass at all.
- netbsd: Compiled fine after some minor fix. There's a bit more to fix around
many libraries not being in the normal library directory, but in
/usr/pkg/lib, which is not in the library search path (i.e. we need to add
an rpath for that in a few more places).
- AIX: Compiled and basic postgres runs fine after a few fixes (big endian
test, converting exports.txt into the right format). Doesn't yet
successfully run more than trivial tests, because I didn't implement the
necessary generation of import files for postgres, but that's just a bit of
work.
This is hampered by the fact that the vanilla postgres crashes for me. I
haven't quite figured out what's the problem. Might be a system issue -
lots of other tools, e.g. perl, segfault frequently.
One important thing to call out: Meson has support for the AIX linker, but
*not* the xlc compiler. I.e. one has to use gcc (or clang, but I didn't
try). I don't know if we'd require adding support for xlc to meson - xlc is
pretty buggy and it doesn't seem particularly crucial to support such an old
crufty compiler on a platform that's not used to a significant degree?
Greetings,
Andres
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-11-15 19:11 ` Tom Lane <[email protected]>
2021-11-15 19:23 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Tom Lane @ 2021-11-15 19:11 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> One important thing to call out: Meson has support for the AIX linker, but
> *not* the xlc compiler. I.e. one has to use gcc (or clang, but I didn't
> try). I don't know if we'd require adding support for xlc to meson - xlc is
> pretty buggy and it doesn't seem particularly crucial to support such an old
> crufty compiler on a platform that's not used to a significant degree?
While I have no particular interest in AIX or xlc specifically, I do
worry about us becoming a builds-on-gcc-or-workalikes-only project.
I suppose MSVC provides a little bit of a cross-check, but I don't
really like giving up on other compilers. Discounting gcc+clang+MSVC
leaves just a few buildfarm animals, and the xlc ones are a significant
part of that population. (In fact, unless somebody renews fossa/husky's
icc license, the three xlc animals will be an outright majority of
them, because wrasse and anole are the only other active animals with
non-mainstream compilers.)
Having said that, I don't plan to be the one trying to get meson
to add xlc support ...
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 19:11 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-11-15 19:23 ` Andres Freund <[email protected]>
2021-11-15 19:48 ` Re: [RFC] building postgres with meson Robert Haas <[email protected]>
2021-11-15 22:08 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-11-18 20:44 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
0 siblings, 3 replies; 139+ messages in thread
From: Andres Freund @ 2021-11-15 19:23 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2021-11-15 14:11:25 -0500, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > One important thing to call out: Meson has support for the AIX linker, but
> > *not* the xlc compiler. I.e. one has to use gcc (or clang, but I didn't
> > try). I don't know if we'd require adding support for xlc to meson - xlc is
> > pretty buggy and it doesn't seem particularly crucial to support such an old
> > crufty compiler on a platform that's not used to a significant degree?
>
> While I have no particular interest in AIX or xlc specifically, I do
> worry about us becoming a builds-on-gcc-or-workalikes-only project.
> I suppose MSVC provides a little bit of a cross-check, but I don't
> really like giving up on other compilers. Discounting gcc+clang+MSVC
> leaves just a few buildfarm animals, and the xlc ones are a significant
> part of that population.
Yea, that's a reasonable concern. I wonder if there's some non-mainstream
compiler that actually works on, um, more easily available platforms that we
could utilize.
> (In fact, unless somebody renews fossa/husky's
> icc license, the three xlc animals will be an outright majority of
> them, because wrasse and anole are the only other active animals with
> non-mainstream compilers.)
It should probably be doable to get somebody to run another icc animal. Icc is
supported by meson, fwiw.
> Having said that, I don't plan to be the one trying to get meson
> to add xlc support ...
It'd probably not be too hard. But given that it's quite hard to get access to
AIX + xlc, I'm not sure it's something I want to propose. There's no resources
to run halfway regular tests on that I found...
It's good to make sure we're not growing too reliant on some compiler(s), but
imo only really makes sense if the alternative compilers are meaningfully
available and maintained.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 19:11 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-11-15 19:23 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-11-15 19:48 ` Robert Haas <[email protected]>
2 siblings, 0 replies; 139+ messages in thread
From: Robert Haas @ 2021-11-15 19:48 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Mon, Nov 15, 2021 at 2:23 PM Andres Freund <[email protected]> wrote:
> It's good to make sure we're not growing too reliant on some compiler(s), but
> imo only really makes sense if the alternative compilers are meaningfully
> available and maintained.
That's a sensible position. I do worry that with this proposed move
we're going to be giving up some of the flexibility that we have right
now. I'm not sure exactly what that means in practice. But make is
just a way of running shell commands, and so you can run any shell
commands you want. The concept of some compiler not being supported
isn't really a thing that even makes sense in a world that is powered
by make. With a big enough hammer you can run any commands you like,
including any compilation commands you like. The whole thing is likely
to be a bit crufty which is a downside, and you might spend more time
fiddling with it than you really want. But nothing is really ever
blocked.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 19:11 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-11-15 19:23 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-11-15 22:08 ` Thomas Munro <[email protected]>
2021-11-15 22:34 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-11-15 22:52 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2 siblings, 2 replies; 139+ messages in thread
From: Thomas Munro @ 2021-11-15 22:08 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Tue, Nov 16, 2021 at 8:23 AM Andres Freund <[email protected]> wrote:
> On 2021-11-15 14:11:25 -0500, Tom Lane wrote:
> > Having said that, I don't plan to be the one trying to get meson
> > to add xlc support ...
>
> It'd probably not be too hard. But given that it's quite hard to get access to
> AIX + xlc, I'm not sure it's something I want to propose. There's no resources
> to run halfway regular tests on that I found...
FWIW there's a free-as-in-beer edition of xlc for Linux (various
distros, POWER only) so you could use qemu, though of course there
will be differences WRT AIX especially around linking, and I suppose a
big part of that work would be precisely understanding stuff like
linker details.
It looks like we have two xlc 12.1 compilers in the farm, but those
compilers are EOL'd[1]. The current release is 16.1, and we have one
of those. The interesting thing about 16.1 is that you can invoke it
as xlclang to get the new clang frontend and, I think, possibly use
more clang/gcc-ish compiler switches[2].
[1] https://www.ibm.com/support/pages/lifecycle/search?q=xl%20c%2Fc%2B%2B
[2] https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=new-clang-based-front-end
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 19:11 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-11-15 19:23 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 22:08 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
@ 2021-11-15 22:34 ` Tom Lane <[email protected]>
2021-11-15 22:46 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Tom Lane @ 2021-11-15 22:34 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers
Thomas Munro <[email protected]> writes:
> ... The interesting thing about 16.1 is that you can invoke it
> as xlclang to get the new clang frontend and, I think, possibly use
> more clang/gcc-ish compiler switches[2].
> [2] https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=new-clang-based-front-end
Ho, that's an earful ;-). Though I wonder whether that frontend
hides the AIX-specific linking issues you mentioned. (Also, although
I see /opt/IBM/xlc/16.1.0/ on gcc119, there's no xlclang there.
So whether we have useful access to it right now is unclear.)
This plays into something that was nagging at me while I wrote my
upthread screed about not giving up on non-gcc/clang compilers:
are those compilers outcompeting all the proprietary ones, to the
extent that the latter will be dead soon anyway? I think Microsoft
is rich enough and stubborn enough to keep on developing MSVC no
matter what, but other compiler vendors may see the handwriting
on the wall. Writing C compilers can't be a growth industry these
days.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 19:11 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-11-15 19:23 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 22:08 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
2021-11-15 22:34 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2021-11-15 22:46 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Andres Freund @ 2021-11-15 22:46 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
Hi,
On 2021-11-15 17:34:33 -0500, Tom Lane wrote:
> Thomas Munro <[email protected]> writes:
> > ... The interesting thing about 16.1 is that you can invoke it
> > as xlclang to get the new clang frontend and, I think, possibly use
> > more clang/gcc-ish compiler switches[2].
> > [2] https://www.ibm.com/docs/en/xl-c-and-cpp-aix/16.1?topic=new-clang-based-front-end
>
> Ho, that's an earful ;-). Though I wonder whether that frontend
> hides the AIX-specific linking issues you mentioned. (Also, although
> I see /opt/IBM/xlc/16.1.0/ on gcc119, there's no xlclang there.
> So whether we have useful access to it right now is unclear.)
It's actually available there, but in /opt/IBM/xlC/16.1.0/bin/xlclang++ (note
the upper case C).
It doesn't really hide the linking issues afaict. I think they're basically an
ABI rather than a linker invocation issue. It's not that hard to address them
though, it's basically making mkldexport.sh a tiny bit more general and
integrating it into src/backend/postgres' build.
We don't have to generate export files for shared libraries anymore though,
afaict, because there's 'expall', which suffices for our purposes. dlopen()
doesn't require an import file.
> This plays into something that was nagging at me while I wrote my
> upthread screed about not giving up on non-gcc/clang compilers:
> are those compilers outcompeting all the proprietary ones, to the
> extent that the latter will be dead soon anyway?
I think that's a pretty clear trend. The ones that aren't dying seem to be
incrementally onto more and more rebasing onto llvm tooling.
It doesn't help that most of those compilers are primarily for OSs that, uh,
aren't exactly growing. Which limits their potential usability significantly.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 19:11 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-11-15 19:23 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 22:08 ` Re: [RFC] building postgres with meson Thomas Munro <[email protected]>
@ 2021-11-15 22:52 ` Thomas Munro <[email protected]>
1 sibling, 0 replies; 139+ messages in thread
From: Thomas Munro @ 2021-11-15 22:52 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Tue, Nov 16, 2021 at 11:08 AM Thomas Munro <[email protected]> wrote:
> FWIW there's a free-as-in-beer edition of xlc for Linux (various
> distros, POWER only) so you could use qemu,
(It's also known to be possible to run AIX 7.2 on qemu, but the
install media is not made available to developers for testing/CI
without a hardware serial number. Boo.)
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 18:34 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2021-11-15 19:11 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2021-11-15 19:23 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2021-11-18 20:44 ` Thomas Munro <[email protected]>
2 siblings, 0 replies; 139+ messages in thread
From: Thomas Munro @ 2021-11-18 20:44 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Tue, Nov 16, 2021 at 8:23 AM Andres Freund <[email protected]> wrote:
> On 2021-11-15 14:11:25 -0500, Tom Lane wrote:
> > (In fact, unless somebody renews fossa/husky's
> > icc license, the three xlc animals will be an outright majority of
> > them, because wrasse and anole are the only other active animals with
> > non-mainstream compilers.)
>
> It should probably be doable to get somebody to run another icc animal. Icc is
> supported by meson, fwiw.
FWIW, in case someone is interested in bringing ICC back to the farm,
some light googling tells me that newer editions of "classic" ICC (as
opposed to "data parallel" ICC, parts of some kind of rebrand) no
longer require regular licence bureaucracy, and can be installed in
modern easier to maintain ways. For example, I see that some people
add Intel's APT repository and apt-get install the compiler inside CI
jobs, on Ubuntu.
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-05-26 00:47 ` Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
9 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-05-26 00:47 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi Tom,
in the meson unconference session you'd spotted flex flags for psqlscanslash.l
(I think) being "hardcoded". As far as I can tell that's largely just copied
from the Makefile):
src/backend/parser/Makefile:scan.c: FLEXFLAGS = -CF -p -p
src/backend/utils/adt/Makefile:jsonpath_scan.c: FLEXFLAGS = -CF -p -p
src/bin/psql/Makefile:psqlscanslash.c: FLEXFLAGS = -Cfe -p -p
src/fe_utils/Makefile:psqlscan.c: FLEXFLAGS = -Cfe -p -p
note that it's not even FLEXFLAGS += or such.
I honestly don't know enough about the various flex flags to judge what a
better approach would be? Looks like these flags are case specific? Perhaps we
could group them, i.e. have centrally defined "do compress" "don't compress"
flex flags?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-05-26 01:38 ` Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Tom Lane @ 2022-05-26 01:38 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> in the meson unconference session you'd spotted flex flags for psqlscanslash.l
> (I think) being "hardcoded". As far as I can tell that's largely just copied
> from the Makefile):
> src/backend/parser/Makefile:scan.c: FLEXFLAGS = -CF -p -p
> src/backend/utils/adt/Makefile:jsonpath_scan.c: FLEXFLAGS = -CF -p -p
> src/bin/psql/Makefile:psqlscanslash.c: FLEXFLAGS = -Cfe -p -p
> src/fe_utils/Makefile:psqlscan.c: FLEXFLAGS = -Cfe -p -p
Hmm, OK. There *is* a FLEXFLAGS definition supplied by configure, and
I believe many of our scanners do use it, but evidently we're just
overriding it for the ones where we really care about using specific
flags. It also looks like the configure-supplied version is usually
empty, so the fact that this variable exists may be mostly a holdover
from Autoconf practice rather than something we ever cared about.
I think the main thing I didn't like about the way you have it in the
meson file is the loss of greppability. I could investigate this
question in a few seconds just now, but if we drop the use of
FLEXFLAGS as a macro it'll become much harder to figure out which
places use what.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2022-05-26 02:41 ` Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-05-26 02:41 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2022-05-25 21:38:33 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > in the meson unconference session you'd spotted flex flags for psqlscanslash.l
> > (I think) being "hardcoded". As far as I can tell that's largely just copied
> > from the Makefile):
>
> > src/backend/parser/Makefile:scan.c: FLEXFLAGS = -CF -p -p
> > src/backend/utils/adt/Makefile:jsonpath_scan.c: FLEXFLAGS = -CF -p -p
> > src/bin/psql/Makefile:psqlscanslash.c: FLEXFLAGS = -Cfe -p -p
> > src/fe_utils/Makefile:psqlscan.c: FLEXFLAGS = -Cfe -p -p
>
> Hmm, OK. There *is* a FLEXFLAGS definition supplied by configure, and
> I believe many of our scanners do use it, but evidently we're just
> overriding it for the ones where we really care about using specific
> flags. It also looks like the configure-supplied version is usually
> empty, so the fact that this variable exists may be mostly a holdover
> from Autoconf practice rather than something we ever cared about.
Yea, it looks like that.
ISTM that it'd still be good to have something like FLEXFLAGS. But it doesn't
look great, nor really intentional, that FLEXFLAGS is overwritten rather than
appended?
> I think the main thing I didn't like about the way you have it in the
> meson file is the loss of greppability. I could investigate this
> question in a few seconds just now, but if we drop the use of
> FLEXFLAGS as a macro it'll become much harder to figure out which
> places use what.
I disliked a bunch of repetitiveness as I had it, so I'm polishing that part
just now.
What would you want to grep for? Places that specify additional flags? Or just
places using flex?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-05-26 02:58 ` Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Tom Lane @ 2022-05-26 02:58 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2022-05-25 21:38:33 -0400, Tom Lane wrote:
>> I think the main thing I didn't like about the way you have it in the
>> meson file is the loss of greppability.
> What would you want to grep for? Places that specify additional flags? Or just
> places using flex?
Well, the consistency of having a single name for "flags given to
flex" seems to me to be worth something.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2022-05-26 08:47 ` Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Aleksander Alekseev @ 2022-05-26 08:47 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi Andres,
Thanks for working on this! I'm very enthusiastic about this effort and I was
glad to see on PGCon Unconference that the majority of the community seems
to be as well.
> The halfway decent list includes, I think:
> 1) cmake
> 2) bazel
> 3) meson
Was SCons considered as an option? It is widely adopted and written in Python
as well. Personally, I like the fact that with SCons you write config files
in pure Python, not some dialect you have to learn additionally. There is
a free e-book available [1].
What pros and cons do you see that make Meson a better choice?
[1]: https://scons.org/doc/production/PDF/scons-user.pdf
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
@ 2022-05-28 19:14 ` Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-05-28 19:14 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi,
On 2022-05-26 11:47:13 +0300, Aleksander Alekseev wrote:
> Thanks for working on this! I'm very enthusiastic about this effort and I was
> glad to see on PGCon Unconference that the majority of the community seems
> to be as well.
>
> > The halfway decent list includes, I think:
> > 1) cmake
> > 2) bazel
> > 3) meson
>
> Was SCons considered as an option?
> What pros and cons do you see that make Meson a better choice?
I looked at it and quickly discarded it. From what I could see there's not
been meaningful moves to it in the last couple years, if anything adoption has
been dropping. And I don't think we want to end up relying on yet another half
maintained tool.
Not having a ninja backend etc didn't strike me as great either - the builds
with scons I've done weren't fast at all.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-05-31 13:49 ` Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Aleksander Alekseev @ 2022-05-31 13:49 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi Andres,
> Not having a ninja backend etc didn't strike me as great either - the builds
> with scons I've done weren't fast at all.
I must admit, personally I never used Scons, I just know that it was considered
(an / the only?) alternative to CMake for many years. The Scons 4.3.0 release
notes say that Ninja is supported [1], but according to the user guide [2]
Ninja support is considered experimental.
Don't get me wrong, I don't insist on using Scons. I was just curious if it was
considered. Actually, a friend of mine pointed out that the fact that Scons
build files are literally a Python code could be a disadvantage. There is less
control of this code, basically it can do anything. It could complicate the
diagnosis of certain issues, etc.
Since you invested so much effort into Meson already let's just focus on it.
I tried the branch on GitHub on MacOS Monterey 12.3.1 and Ubuntu 20.04 LTS.
I was going to test it against several third party extensions, but it looks like
it is a bit early for this. On Ubuntu I got the following error:
```
../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
in a function)
332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
‘PG_KEYWORD’
32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
```
On MacOS I got multiple errors regarding LDAP:
```
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/
LDAP.framework/Headers/ldap.h:1:10: error: #include nested too deeply
#include <ldap.h>
../src/interfaces/libpq/fe-connect.c:4816:2: error: use of undeclared
identifier 'LDAP'
LDAP *ld = NULL;
^
../src/interfaces/libpq/fe-connect.c:4817:2: error: use of undeclared
identifier 'LDAPMessage'
LDAPMessage *res,
^
... etc...
```
I didn't invest much time into investigating these issues. For now I just
wanted to report them. Please let me know if you need any help with these
and/or additional information.
[1]: https://scons.org/scons-430-is-available.html
[2]: https://scons.org/doc/production/PDF/scons-user.pdf
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
@ 2022-05-31 19:25 ` Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-05-31 19:25 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi,
On 2022-05-31 16:49:17 +0300, Aleksander Alekseev wrote:
> I tried the branch on GitHub on MacOS Monterey 12.3.1 and Ubuntu 20.04 LTS.
> I was going to test it against several third party extensions, but it looks like
> it is a bit early for this. On Ubuntu I got the following error:
What do those extensions use to build? Since the unconference I added some
rudimentary PGXS compatibility, but it's definitely not complete yet.
> ```
> ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> in a function)
> 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
>
> ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> ‘PG_KEYWORD’
> 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> ```
Huh. I've not seen this before - could you provide a bit more detail about
what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
this to work.
> On MacOS I got multiple errors regarding LDAP:
Ah, yes. Sorry, that's an open issue that I need to fix. -Dldap=disabled for
the rescue. There's some crazy ordering dependency in macos framework
headers. The ldap framework contains an "ldap.h" header that includes
"ldap.h". So if you end up with the framework on the include path, you get
endless recursion.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-06-01 09:39 ` Aleksander Alekseev <[email protected]>
2022-06-01 21:05 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Aleksander Alekseev @ 2022-06-01 09:39 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi Andres,
> What do those extensions use to build? Since the unconference I added some
> rudimentary PGXS compatibility, but it's definitely not complete yet.
We mostly use CMake and Cargo, the Rust package manager. So I don't
anticipate many problems here, just want to make sure it's going to
work as expected.
> > ```
> > ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> > in a function)
> > 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
> >
> > ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> > ‘PG_KEYWORD’
> > 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> > ```
>
> Huh. I've not seen this before - could you provide a bit more detail about
> what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
> this to work.
I used PIP to install Meson, since the default APT package is too old, v0.53:
$ pip3 install --user meson
$ meson --version
0.62.1
$ ninja --version
1.10.0
The branch was checked out as it was described in the first email.
Then to reproduce the issue:
$ git status
On branch meson
Your branch is up to date with 'andres/meson'.
$ git fetch andres
$ git rebase -i andres/meson
$ meson setup build --buildtype debug
$ cd build
$ ninja
This is pretty much the default Ubuntu 20.04.4 LTS system with all the
recent updates installed, so it shouldn't be a problem to reproduce
the issue with a VM.
> > On MacOS I got multiple errors regarding LDAP:
>
> Ah, yes. Sorry, that's an open issue that I need to fix. -Dldap=disabled for
> the rescue. There's some crazy ordering dependency in macos framework
> headers. The ldap framework contains an "ldap.h" header that includes
> "ldap.h". So if you end up with the framework on the include path, you get
> endless recursion.
Thanks, this helped. I did the following:
$ meson configure -Dldap=disabled
$ meson configure -Dssl=openssl
$ meson configure -Dprefix=/Users/eax/pginstall
$ ninja
$ meson test
$ meson install
... and it terminated successfully. I was also able to configure and
run Postgres instance using my regular scripts, with some
modifications [1]
Then I decided to compile TimescaleDB against the newly installed
Postgres. Turns out there is a slight problem.
The extension uses CMake and also requires PostgreSQL to be compiled
with OpenSSL support. CMakeLists.txt looks for a
"--with-(ssl=)?openssl" regular expression in the "pg_config
--configure" output. The output is empty although Postgres was
compiled with OpenSSL support. The full output of pg_config looks like
this:
```
CONFIGURE =
CC = not recorded
CPPFLAGS = not recorded
CFLAGS = not recorded
CFLAGS_SL = not recorded
... etc ...
```
I get a bunch of errors from the compiler if I remove this particular
check from CMakeLists, but I have to investigate these a bit more
since the branch is based on PG15 and we don't officially support PG15
yet. It worked last time we checked a month or so ago, but the
situation may have changed.
[1]: https://github.com/afiskon/pgscripts/blob/master/single-install.sh
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
@ 2022-06-01 21:05 ` Andres Freund <[email protected]>
2022-06-02 12:34 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-06-01 21:05 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi,
On 2022-06-01 12:39:50 +0300, Aleksander Alekseev wrote:
> > > ```
> > > ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> > > in a function)
> > > 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
> > >
> > > ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> > > ‘PG_KEYWORD’
> > > 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> > > ```
> >
> > Huh. I've not seen this before - could you provide a bit more detail about
> > what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
> > this to work.
>
> I used PIP to install Meson, since the default APT package is too old, v0.53:
>
> $ pip3 install --user meson
> $ meson --version
> 0.62.1
> $ ninja --version
> 1.10.0
>
> The branch was checked out as it was described in the first email.
> Then to reproduce the issue:
>
> $ git status
> On branch meson
> Your branch is up to date with 'andres/meson'.
> $ git fetch andres
> $ git rebase -i andres/meson
> $ meson setup build --buildtype debug
> $ cd build
> $ ninja
>
> This is pretty much the default Ubuntu 20.04.4 LTS system with all the
> recent updates installed, so it shouldn't be a problem to reproduce
> the issue with a VM.
Will test.
> > > On MacOS I got multiple errors regarding LDAP:
> >
> > Ah, yes. Sorry, that's an open issue that I need to fix. -Dldap=disabled for
> > the rescue. There's some crazy ordering dependency in macos framework
> > headers. The ldap framework contains an "ldap.h" header that includes
> > "ldap.h". So if you end up with the framework on the include path, you get
> > endless recursion.
>
> Thanks, this helped.
Cool. I think I pushed a fix/workaround for the issue now. Still can't decide
whether it's apple's or meson's fault.
> I did the following:
>
> $ meson configure -Dldap=disabled
> $ meson configure -Dssl=openssl
> $ meson configure -Dprefix=/Users/eax/pginstall
FYI, you can set multiple options in one go ;)
> ... and it terminated successfully. I was also able to configure and
> run Postgres instance using my regular scripts, with some
> modifications [1]
Cool.
> Then I decided to compile TimescaleDB against the newly installed
> Postgres. Turns out there is a slight problem.
>
> The extension uses CMake and also requires PostgreSQL to be compiled
> with OpenSSL support. CMakeLists.txt looks for a
> "--with-(ssl=)?openssl" regular expression in the "pg_config
> --configure" output. The output is empty although Postgres was
> compiled with OpenSSL support.
Makes sense. Currently we don't fill the --configure thing, because there
configure wasn't used. We could try to generate something compatible from
meson options, but I'm not sure that's a good plan.
> The full output of pg_config looks like
> this:
>
> ```
> CONFIGURE =
> CC = not recorded
> CPPFLAGS = not recorded
> CFLAGS = not recorded
> CFLAGS_SL = not recorded
> ... etc ...
> ```
>
> I get a bunch of errors from the compiler if I remove this particular
> check from CMakeLists, but I have to investigate these a bit more
> since the branch is based on PG15 and we don't officially support PG15
> yet. It worked last time we checked a month or so ago, but the
> situation may have changed.
I suspect the errors might be due to CC/CPPFLAGS/... not being defined. I can
try to make it output something roughly compatible with the old output, for
most of those I already started to compute values for the PGXS compat stuff I
was hacking on recently.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-01 21:05 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-06-02 12:34 ` Aleksander Alekseev <[email protected]>
2022-06-02 15:22 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Aleksander Alekseev @ 2022-06-02 12:34 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi Andres,
> Cool. I think I pushed a fix/workaround for the issue now. Still can't decide
> whether it's apple's or meson's fault.
Many thanks! The fix solved the problem, I can compile with -Dldap=enabled now.
The code passes the tests too.
> > $ meson configure -Dldap=disabled
> > $ meson configure -Dssl=openssl
> > $ meson configure -Dprefix=/Users/eax/pginstall
>
> FYI, you can set multiple options in one go ;)
Thanks! ;)
> Makes sense. Currently we don't fill the --configure thing, because there
> configure wasn't used. We could try to generate something compatible from
> meson options, but I'm not sure that's a good plan.
If pg_config output was 100% backward compatible with Autotools one, that would
simplify the lives of the extension developers for sure. However, considering
that at PGCon we agreed that both Autotools and Meson will be maintained for
several releases, personally I wouldn't say that this compatibility is
necessary, nor is it realistically deliverable. Nevertheless, IMO there should
be a stable and documented way to determine the PostgreSQL version (this can be
done with `pg_config --version` for both Autotools and Meson), the build tool
used (no way to determine) and the build options (no way to determine
for Meson).
> I suspect the errors might be due to CC/CPPFLAGS/... not being defined. I can
> try to make it output something roughly compatible with the old output, for
> most of those I already started to compute values for the PGXS compat stuff I
> was hacking on recently.
Yes, that could explain the problem. Just for the record, I get several errors
regarding src/export.h in TimescaleDB code [1]:
```
/Users/eax/projects/c/timescaledb/src/export.h:26:5: error: pasting formed
')87628', an invalid preprocessing token [clang-diagnostic-error]
#if TS_EMPTY(PGDLLEXPORT)
^
/Users/eax/projects/c/timescaledb/src/export.h:17:22: note: expanded from
macro 'TS_EMPTY'
#define TS_EMPTY(x) (TS_CAT(x, 87628) == 87628)
^
/Users/eax/projects/c/timescaledb/src/export.h:15:23: note: expanded from
macro 'TS_CAT'
#define TS_CAT(x, y) x##y
^
/Users/eax/projects/c/timescaledb/src/export.h:26:14: error: function-like
macro '__attribute__' is not defined [clang-diagnostic-error]
#if TS_EMPTY(PGDLLEXPORT)
^
/Users/eax/pginstall/include/postgresql/server/c.h:1339:21: note: expanded
from macro 'PGDLLEXPORT'
#define PGDLLEXPORT __attribute__((visibility("default")))
^
/Users/eax/projects/c/timescaledb/src/export.h:30:2: error: "PGDLLEXPORT is
already defined" [clang-diagnostic-error]
#error "PGDLLEXPORT is already defined"
^
1 warning and 3 errors generated.
Error while processing /Users/eax/projects/c/timescaledb/src/extension.c
```
[1]: https://github.com/timescale/timescaledb/blob/main/src/export.h
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-01 21:05 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 12:34 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
@ 2022-06-02 15:22 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Andres Freund @ 2022-06-02 15:22 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi,
On 2022-06-02 15:34:23 +0300, Aleksander Alekseev wrote:
> Hi Andres,
>
> > Cool. I think I pushed a fix/workaround for the issue now. Still can't decide
> > whether it's apple's or meson's fault.
>
> Many thanks! The fix solved the problem, I can compile with -Dldap=enabled now.
> The code passes the tests too.
Cool.
> > I suspect the errors might be due to CC/CPPFLAGS/... not being defined. I can
> > try to make it output something roughly compatible with the old output, for
> > most of those I already started to compute values for the PGXS compat stuff I
> > was hacking on recently.
>
> Yes, that could explain the problem. Just for the record, I get several errors
> regarding src/export.h in TimescaleDB code [1]:
>
I think this is timescale's issue. Why are you defining / undefining
PGDLLEXPORT?
Part of the patch series is to use visibility attributes, and your #if
TS_EMPTY(PGDLLEXPORT) thing can't handle that.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
@ 2022-06-02 16:39 ` Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-06-02 16:39 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Tom Lane <[email protected]>
Hi,
On 2022-06-01 12:39:50 +0300, Aleksander Alekseev wrote:
> > > ```
> > > ../src/include/parser/kwlist.h:332:25: error: ‘PARAMETER’ undeclared here (not
> > > in a function)
> > > 332 | PG_KEYWORD("parameter", PARAMETER, UNRESERVED_KEYWORD, BARE_LABEL)
> > >
> > > ../src/interfaces/ecpg/preproc/keywords.c:32:55: note: in definition of macro
> > > ‘PG_KEYWORD’
> > > 32 | #define PG_KEYWORD(kwname, value, category, collabel) value,
> > > ```
> >
> > Huh. I've not seen this before - could you provide a bit more detail about
> > what you did? CI isn't testing ubuntu, but it is testing Debian, so I'd expect
> > this to work.
>
> I used PIP to install Meson, since the default APT package is too old, v0.53:
>
> $ pip3 install --user meson
> $ meson --version
> 0.62.1
> $ ninja --version
> 1.10.0
>
> The branch was checked out as it was described in the first email.
> Then to reproduce the issue:
>
> $ git status
> On branch meson
> Your branch is up to date with 'andres/meson'.
> $ git fetch andres
> $ git rebase -i andres/meson
> $ meson setup build --buildtype debug
> $ cd build
> $ ninja
>
> This is pretty much the default Ubuntu 20.04.4 LTS system with all the
> recent updates installed, so it shouldn't be a problem to reproduce
> the issue with a VM.
Chatting with a colleague (who unbeknownst to me hit something similar in the
past) I think we figured it out. It's not due to Ubuntu 20.04 or such. It's
likely due to previously having an in-tree build with autoconf, doing make
clean, doing a git pull, then building with meson. The meson build doesn't yet
handle pre-existing flex / bison output.
I had tried to defend against conflicts with in-tree builds by detecting an
in-tree pg_config.h, but that doesn't help with files that aren't removed by
make clean. Like bison / flex output.
And I didn't notice this problem because it doesn't cause visible issues until
the lexer / grammar changes...
I'm not quite sure what the proper behaviour is when doing an out-of-tree
build with meson (all builds are out-of-tree), with a pre-existing flex /
bison output in the source tree that is out of date.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-06-02 17:08 ` Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Tom Lane @ 2022-06-02 17:08 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> I'm not quite sure what the proper behaviour is when doing an out-of-tree
> build with meson (all builds are out-of-tree), with a pre-existing flex /
> bison output in the source tree that is out of date.
Definitely sounds like a gotcha.
On the one hand, there's been some discussion already of removing all
derived files from tarballs and just insisting that users provide all
needed tools when building from source. If we did that, it could be
sufficient for the meson build to check that no such files are present
in the source tree. (Checking a couple of them would be enough, likely.)
On the other hand, I'm not sure that I want such a change to be forced
by a toolchain change. It definitely seems a bit contrary to the plan
we'd formed of allowing meson and make-based builds to coexist for
a few years, because we'd be breaking at least some make-based build
processes.
Could we have the meson build check that, say, if gram.c exists it
is newer than gram.y? Or get it to ignore an in-tree gram.c?
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2022-06-02 17:26 ` Andres Freund <[email protected]>
2022-06-02 17:33 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Andres Freund @ 2022-06-02 17:26 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-06-02 13:08:49 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > I'm not quite sure what the proper behaviour is when doing an out-of-tree
> > build with meson (all builds are out-of-tree), with a pre-existing flex /
> > bison output in the source tree that is out of date.
>
> Definitely sounds like a gotcha.
>
> On the one hand, there's been some discussion already of removing all
> derived files from tarballs and just insisting that users provide all
> needed tools when building from source. If we did that, it could be
> sufficient for the meson build to check that no such files are present
> in the source tree. (Checking a couple of them would be enough, likely.)
There already is a check for pg_config.h, so the most obvious source of this
is addressed. Just didn't think about the files that make clean doesn't remove
:/.
> On the other hand, I'm not sure that I want such a change to be forced
> by a toolchain change. It definitely seems a bit contrary to the plan
> we'd formed of allowing meson and make-based builds to coexist for
> a few years, because we'd be breaking at least some make-based build
> processes.
Agreed. I think it'd be pretty reasonable to not include flex / bison
output. They're not hard to acquire. The docs are perhaps another story.
I think it might be fine to say that make reallyclean (*) is required if
there's some conflicting in-source tree file?
> Could we have the meson build check that, say, if gram.c exists it
> is newer than gram.y? Or get it to ignore an in-tree gram.c?
I suspect the problem with ignoring is gram.h, that's probably a bit harder to
ignore. Right now I'm leaning towards either always erroring out if there's
bison/flex output in the source tree (with a hint towards make
reallyclean(*)), or erroring out if they're out of date (again with a hint
towards reallyclean)?
Alternatively we could just remove the generated .c/h files from the source
dir, as a part of regenerating them in the build dir? But I like the idea of
the source dir being readonly outside of explicit targets modifying sources
(e.g. update-unicode or such).
Greetings,
Andres Freund
(*) do we really not have a target that removes bison / flex output?
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-06-02 17:33 ` Tom Lane <[email protected]>
2022-06-02 17:48 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Tom Lane @ 2022-06-02 17:33 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> (*) do we really not have a target that removes bison / flex output?
maintainer-clean
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:33 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2022-06-02 17:48 ` Andres Freund <[email protected]>
2022-06-02 19:05 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-06-02 17:48 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-06-02 13:33:51 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > (*) do we really not have a target that removes bison / flex output?
>
> maintainer-clean
Don't think so:
# gram.c, gram.h, and scan.c are in the distribution tarball, so they
# are not cleaned here.
clean distclean maintainer-clean:
rm -f lex.backup
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:33 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:48 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-06-02 19:05 ` Tom Lane <[email protected]>
2022-06-02 19:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Tom Lane @ 2022-06-02 19:05 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2022-06-02 13:33:51 -0400, Tom Lane wrote:
>> Andres Freund <[email protected]> writes:
>>> (*) do we really not have a target that removes bison / flex output?
>> maintainer-clean
> Don't think so:
See about line 300 in src/backend/Makefile. In any case, it's
easy to show by experiment that it does.
$ make maintainer-clean
...
$ git status --ignored
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:33 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:48 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 19:05 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2022-06-02 19:17 ` Andres Freund <[email protected]>
2022-06-02 19:53 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-06-02 19:17 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-06-02 15:05:10 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > On 2022-06-02 13:33:51 -0400, Tom Lane wrote:
> >> Andres Freund <[email protected]> writes:
> >>> (*) do we really not have a target that removes bison / flex output?
>
> >> maintainer-clean
>
> > Don't think so:
>
> See about line 300 in src/backend/Makefile. In any case, it's
> easy to show by experiment that it does.
>
> $ make maintainer-clean
> ...
> $ git status --ignored
> On branch master
> Your branch is up to date with 'origin/master'.
>
> nothing to commit, working tree clean
Oh. I executed maintainer-clean inside src/backend/parser/, and thus didn't
see it getting cleaned up.
It seems pretty darn grotty that src/backend/parser/Makefile explicitly states
that gram.c ... aren't cleaned "here", but then src/backend/Makefile does
clean them up.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:33 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:48 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 19:05 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 19:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-06-02 19:53 ` Tom Lane <[email protected]>
2022-06-02 21:13 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Tom Lane @ 2022-06-02 19:53 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> Oh. I executed maintainer-clean inside src/backend/parser/, and thus didn't
> see it getting cleaned up.
> It seems pretty darn grotty that src/backend/parser/Makefile explicitly states
> that gram.c ... aren't cleaned "here", but then src/backend/Makefile does
> clean them up.
I agree the factorization of this ain't great. I'd think about improving
it, were it not that we're trying to get rid of it.
(But with meson, the whole idea of building or cleaning just part of the
tree is out the window anyway, no?)
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:33 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:48 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 19:05 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 19:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 19:53 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2022-06-02 21:13 ` Andres Freund <[email protected]>
2022-06-03 09:35 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-06-02 21:13 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-06-02 15:53:50 -0400, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > Oh. I executed maintainer-clean inside src/backend/parser/, and thus didn't
> > see it getting cleaned up.
>
> > It seems pretty darn grotty that src/backend/parser/Makefile explicitly states
> > that gram.c ... aren't cleaned "here", but then src/backend/Makefile does
> > clean them up.
>
> I agree the factorization of this ain't great. I'd think about improving
> it, were it not that we're trying to get rid of it.
+1. I think I just wanted to excuse my confusion...
> (But with meson, the whole idea of building or cleaning just part of the
> tree is out the window anyway, no?)
Cleaning parts of the tree isn't supported as far as I know (not that I've
needed it). You can build parts of the tree by specifying the target
(e.g. ninja src/backend/postgres) or by specifying meta-targets (e.g. ninja
contrib backend). I've thought about contributing a patch to meson to
automatically generate targets for each directory that has sub-targets - it's
just a few lines.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:33 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:48 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 19:05 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 19:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 19:53 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 21:13 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-06-03 09:35 ` Aleksander Alekseev <[email protected]>
2022-06-03 16:23 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Aleksander Alekseev @ 2022-06-03 09:35 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
Hi hackers,
> See about line 300 in src/backend/Makefile. In any case, it's
> easy to show by experiment that it does.
`make maintainer-clean` did the trick, thanks. I suggest modifying meson.build
accordingly:
-run make distclean in the source tree.
+run `make maintainer-clean` in the source tree.
> I think this is timescale's issue. Why are you defining / undefining
> PGDLLEXPORT?
That's a great question.
As I understand some time ago the developers had a problem with a collision of
exported symbols on *nix platforms [1] and chose to solve it by re-defining
PGDLLEXPORT to __attribute__((visibility ("default"))) for GCC and CLang.
I agree that this is a questionable approach. Redefining a macro provided
by Postgres doesn't strike me as a good idea. I tried to remove this
re-definition, but it didn't go well [2]. So apparently it should be addressed
somehow differently.
> Part of the patch series is to use visibility attributes, and your #if
> TS_EMPTY(PGDLLEXPORT) thing can't handle that.
Out of curiosity, how come a patchset that adds an alternative build system
changes the visibility attributes? I would guess they should be the same
for both Autotools and Meson. Is it necessary in order to make Meson work?
If not, maybe it should be a separate patch.
[1]: https://github.com/timescale/timescaledb/commit/027b7b29420a742d7615c70d9f19b2b99c488c2c
[2]: https://github.com/timescale/timescaledb/pull/4413
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:33 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:48 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 19:05 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 19:17 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 19:53 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 21:13 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-03 09:35 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
@ 2022-06-03 16:23 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Andres Freund @ 2022-06-03 16:23 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
Hi,
On 2022-06-03 12:35:45 +0300, Aleksander Alekseev wrote:
> > Part of the patch series is to use visibility attributes, and your #if
> > TS_EMPTY(PGDLLEXPORT) thing can't handle that.
>
> Out of curiosity, how come a patchset that adds an alternative build system
> changes the visibility attributes?
It was the simplest path - on windows (and AIx) extension symbols need to be
explicitly exported. We did that by building the objects constituting
extension libraries, collecting the symbols, generating an export file with
all the symbols, which then is passed to the linker. It was a lot less work
to just add the necessary PGDLLEXPORT annotations than make that export file
generation work for extensions.
> I would guess they should be the same for both Autotools and Meson.
It is, the patch adds it to both.
> Is it necessary in order to make Meson work?
Yes, or at least the simplest path.
> If not, maybe it should be a separate patch.
It is.
https://commitfest.postgresql.org/38/3396/
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-08-10 17:19 ` Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-08-10 17:19 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-06-02 10:26:09 -0700, Andres Freund wrote:
> > Could we have the meson build check that, say, if gram.c exists it
> > is newer than gram.y? Or get it to ignore an in-tree gram.c?
>
> I suspect the problem with ignoring is gram.h, that's probably a bit harder to
> ignore.
I tried to ignore various generated files in the source tree, but I don't
think it's doable for all of them. Consider
e.g. src/backend/utils/misc/guc-file.c which is gets built via #include
"guc-file.c" from gram.c
Because it's a "" include, the search path starts in the current directory and
only then -I is searched. To my knowledge there's no way of changing
that. Quoting the gcc manpage:
-I dir
-iquote dir
-isystem dir
-idirafter dir
Add the directory dir to the list of directories to be searched for header files during preprocessing. If dir begins with = or $SYSROOT, then
the = or $SYSROOT is replaced by the sysroot prefix; see --sysroot and -isysroot.
Directories specified with -iquote apply only to the quote form of the directive, "#include "file"". Directories specified with -I, -isystem, or
-idirafter apply to lookup for both the "#include "file"" and "#include <file>" directives.
You can specify any number or combination of these options on the command line to search for header files in several directories. The lookup
order is as follows:
1. For the quote form of the include directive, the directory of the current file is searched first.
2. For the quote form of the include directive, the directories specified by -iquote options are searched in left-to-right order, as they appear
on the command line.
3. Directories specified with -I options are scanned in left-to-right order.
[...]
Except for copying guc.c from source to build tree before building, I don't
see a way of ignoring the in-build-tree guc-file.c.
Not sure what a good way of dealing with this is. For now I'll make it just
error out if there's any known such file in the source tree, but that's not a
good solution forever. If it were just "normal" build leftovers I'd propose
to (optionally) just remove them, but that's not good for tarballs.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-08-11 03:33 ` John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: John Naylor @ 2022-08-11 03:33 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
On Thu, Aug 11, 2022 at 12:19 AM Andres Freund <[email protected]> wrote:
> I tried to ignore various generated files in the source tree, but I don't
> think it's doable for all of them. Consider
> e.g. src/backend/utils/misc/guc-file.c which is gets built via #include
> "guc-file.c" from gram.c
With a bit of work, we could probably get rid of those includes. See
27199058d98ef7f for one example.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2022-08-11 03:37 ` Tom Lane <[email protected]>
2022-08-11 03:45 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
0 siblings, 2 replies; 139+ messages in thread
From: Tom Lane @ 2022-08-11 03:37 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
John Naylor <[email protected]> writes:
> With a bit of work, we could probably get rid of those includes. See
> 27199058d98ef7f for one example.
Yeah --- it would mean creating gram.h files for all the bison grammars
not just a few of them, but it's certainly do-able if there's motivation
to make the changes. Most of the files that are done that way date
from before we knew about flex's %top.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2022-08-11 03:45 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 139+ messages in thread
From: Tom Lane @ 2022-08-11 03:45 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
I wrote:
> John Naylor <[email protected]> writes:
>> With a bit of work, we could probably get rid of those includes. See
>> 27199058d98ef7f for one example.
> Yeah --- it would mean creating gram.h files for all the bison grammars
> not just a few of them, but it's certainly do-able if there's motivation
> to make the changes. Most of the files that are done that way date
> from before we knew about flex's %top.
BTW, 72b1e3a21 is another useful precedent in this area.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
@ 2022-08-11 03:57 ` John Naylor <[email protected]>
2022-08-11 04:07 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 04:32 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
1 sibling, 2 replies; 139+ messages in thread
From: John Naylor @ 2022-08-11 03:57 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andres Freund <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
On Thu, Aug 11, 2022 at 10:37 AM Tom Lane <[email protected]> wrote:
>
> John Naylor <[email protected]> writes:
> > With a bit of work, we could probably get rid of those includes. See
> > 27199058d98ef7f for one example.
>
> Yeah --- it would mean creating gram.h files for all the bison grammars
> not just a few of them, but it's certainly do-able if there's motivation
> to make the changes. Most of the files that are done that way date
> from before we knew about flex's %top.
I'll volunteer to work on this unless an easier solution happens to
come along in the next couple days. (aside: guc-file.l doesn't have a
grammar, so not yet sure if that makes the issue easier or harder...)
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2022-08-11 04:07 ` Andres Freund <[email protected]>
2022-08-12 06:01 ` build remaining Flex files standalone John Naylor <[email protected]>
1 sibling, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-08-11 04:07 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Tom Lane <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-08-11 10:57:33 +0700, John Naylor wrote:
> I'll volunteer to work on this unless an easier solution happens to
> come along in the next couple days.
Cool!
> (aside: guc-file.l doesn't have a grammar, so not yet sure if that makes the
> issue easier or harder...)
I think we should consider compiling it separately from guc.c. guc.c already
compiles quite slowly (iirc beat only by ecpg and main grammar), and it's a
relatively commonly changed source file.
It might even be a good idea to split guc.c so it only contains the settings
arrays + direct dependencies...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* build remaining Flex files standalone
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 04:07 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-08-12 06:01 ` John Naylor <[email protected]>
2022-08-13 08:39 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: John Naylor @ 2022-08-12 06:01 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
Starting a new thread to control clutter. [was: Re: [RFC] building
postgres with meson]
motivation: https://www.postgresql.org/message-id/20220810171935.7k5zgnjwqzalzmtm%40awork3.anarazel.de
On Thu, Aug 11, 2022 at 11:07 AM Andres Freund <[email protected]> wrote:
> I think we should consider compiling it separately from guc.c. guc.c already
> compiles quite slowly (iirc beat only by ecpg and main grammar), and it's a
> relatively commonly changed source file.
Done in the attached, and will do the rest in time. It seemed most
straightforward to put ProcessConfigFileInternal() in guc.c since
that's where most of its callers are, and it relies on some vars and
types declared there. There are a couple new extern declarations in
guc.h that are only for guc.c and guc-file.c:
+/* functions shared between guc.c and guc-file.l */
+extern int guc_name_compare(const char *namea, const char *nameb);
+extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
+ bool applySettings, int elevel);
+extern void record_config_file_error(const char *errmsg,
+ const char *config_file,
+ int lineno,
+ ConfigVariable **head_p,
+ ConfigVariable **tail_p);
These might be better placed in a new guc_internal.h. Thoughts?
> It might even be a good idea to split guc.c so it only contains the settings
> arrays + direct dependencies...
Perhaps this can be a TODO item, one which falls under "[E] marks
items that are easier to implement". I've been slacking on removing
the old/intractable cruft from the TODO list, but we should also be
sticking small nice-but-not-necessary things in there. That said, if
this idea has any bearing on the guc_internal.h idea, it might be
better dealt with now.
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[text/x-patch] v1-0001-Build-guc-file.c-standalone.patch (27.4K, ../../CAFBsxsF8Gc2StS3haXofshHCzqNMRXiSxvQEYGwnFsTmsdwNeg@mail.gmail.com/2-v1-0001-Build-guc-file.c-standalone.patch)
download | inline diff:
From d723ba14acf56fd432e9e263db937fcc13fc0355 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Thu, 11 Aug 2022 19:38:37 +0700
Subject: [PATCH v1] Build guc-file.c standalone
The proposed Meson build system will need a way to ignore certain
generated files in order to coexist with the autoconf build system,
and #include'd C files generated by Flex make this more difficult.
Build guc-file.c separately from guc.c, as was done in 72b1e3a21.
TODO: other Flex-generated files
Discussion: https://www.postgresql.org/message-id/20220810171935.7k5zgnjwqzalzmtm%40awork3.anarazel.de
---
src/backend/utils/misc/Makefile | 5 +-
src/backend/utils/misc/guc-file.l | 367 +-----------------------------
src/backend/utils/misc/guc.c | 362 ++++++++++++++++++++++++++++-
src/include/utils/guc.h | 9 +
4 files changed, 370 insertions(+), 373 deletions(-)
diff --git a/src/backend/utils/misc/Makefile b/src/backend/utils/misc/Makefile
index 1d5327cf64..cf7ce9bc83 100644
--- a/src/backend/utils/misc/Makefile
+++ b/src/backend/utils/misc/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
OBJS = \
guc.o \
+ guc-file.o \
help_config.o \
pg_config.o \
pg_controldata.o \
@@ -37,10 +38,6 @@ endif
include $(top_srcdir)/src/backend/common.mk
-# guc-file is compiled as part of guc
-guc.o: guc-file.c
-
# Note: guc-file.c is not deleted by 'make clean',
# since we want to ship it in distribution tarballs.
clean:
- @rm -f lex.yy.c
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index ce5633844c..08adb454de 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -7,7 +7,7 @@
* src/backend/utils/misc/guc-file.l
*/
-%{
+%top{
#include "postgres.h"
@@ -17,9 +17,12 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "storage/fd.h"
+#include <sys/stat.h>
#include "utils/guc.h"
+#include "utils/memutils.h"
+}
-
+%{
/*
* flex emits a yy_fatal_error() function that it calls in response to
* critical errors like malloc failure, file I/O errors, and detection of
@@ -48,12 +51,6 @@ static sigjmp_buf *GUC_flex_fatal_jmp;
static void FreeConfigVariable(ConfigVariable *item);
-static void record_config_file_error(const char *errmsg,
- const char *config_file,
- int lineno,
- ConfigVariable **head_p,
- ConfigVariable **tail_p);
-
static int GUC_flex_fatal(const char *msg);
/* LCOV_EXCL_START */
@@ -159,358 +156,6 @@ ProcessConfigFile(GucContext context)
MemoryContextDelete(config_cxt);
}
-/*
- * This function handles both actual config file (re)loads and execution of
- * show_all_file_settings() (i.e., the pg_file_settings view). In the latter
- * case we don't apply any of the settings, but we make all the usual validity
- * checks, and we return the ConfigVariable list so that it can be printed out
- * by show_all_file_settings().
- */
-static ConfigVariable *
-ProcessConfigFileInternal(GucContext context, bool applySettings, int elevel)
-{
- bool error = false;
- bool applying = false;
- const char *ConfFileWithError;
- ConfigVariable *item,
- *head,
- *tail;
- int i;
-
- /* Parse the main config file into a list of option names and values */
- ConfFileWithError = ConfigFileName;
- head = tail = NULL;
-
- if (!ParseConfigFile(ConfigFileName, true,
- NULL, 0, 0, elevel,
- &head, &tail))
- {
- /* Syntax error(s) detected in the file, so bail out */
- error = true;
- goto bail_out;
- }
-
- /*
- * Parse the PG_AUTOCONF_FILENAME file, if present, after the main file to
- * replace any parameters set by ALTER SYSTEM command. Because this file
- * is in the data directory, we can't read it until the DataDir has been
- * set.
- */
- if (DataDir)
- {
- if (!ParseConfigFile(PG_AUTOCONF_FILENAME, false,
- NULL, 0, 0, elevel,
- &head, &tail))
- {
- /* Syntax error(s) detected in the file, so bail out */
- error = true;
- ConfFileWithError = PG_AUTOCONF_FILENAME;
- goto bail_out;
- }
- }
- else
- {
- /*
- * If DataDir is not set, the PG_AUTOCONF_FILENAME file cannot be
- * read. In this case, we don't want to accept any settings but
- * data_directory from postgresql.conf, because they might be
- * overwritten with settings in the PG_AUTOCONF_FILENAME file which
- * will be read later. OTOH, since data_directory isn't allowed in the
- * PG_AUTOCONF_FILENAME file, it will never be overwritten later.
- */
- ConfigVariable *newlist = NULL;
-
- /*
- * Prune all items except the last "data_directory" from the list.
- */
- for (item = head; item; item = item->next)
- {
- if (!item->ignore &&
- strcmp(item->name, "data_directory") == 0)
- newlist = item;
- }
-
- if (newlist)
- newlist->next = NULL;
- head = tail = newlist;
-
- /*
- * Quick exit if data_directory is not present in file.
- *
- * We need not do any further processing, in particular we don't set
- * PgReloadTime; that will be set soon by subsequent full loading of
- * the config file.
- */
- if (head == NULL)
- goto bail_out;
- }
-
- /*
- * Mark all extant GUC variables as not present in the config file. We
- * need this so that we can tell below which ones have been removed from
- * the file since we last processed it.
- */
- for (i = 0; i < num_guc_variables; i++)
- {
- struct config_generic *gconf = guc_variables[i];
-
- gconf->status &= ~GUC_IS_IN_FILE;
- }
-
- /*
- * Check if all the supplied option names are valid, as an additional
- * quasi-syntactic check on the validity of the config file. It is
- * important that the postmaster and all backends agree on the results of
- * this phase, else we will have strange inconsistencies about which
- * processes accept a config file update and which don't. Hence, unknown
- * custom variable names have to be accepted without complaint. For the
- * same reason, we don't attempt to validate the options' values here.
- *
- * In addition, the GUC_IS_IN_FILE flag is set on each existing GUC
- * variable mentioned in the file; and we detect duplicate entries in the
- * file and mark the earlier occurrences as ignorable.
- */
- for (item = head; item; item = item->next)
- {
- struct config_generic *record;
-
- /* Ignore anything already marked as ignorable */
- if (item->ignore)
- continue;
-
- /*
- * Try to find the variable; but do not create a custom placeholder if
- * it's not there already.
- */
- record = find_option(item->name, false, true, elevel);
-
- if (record)
- {
- /* If it's already marked, then this is a duplicate entry */
- if (record->status & GUC_IS_IN_FILE)
- {
- /*
- * Mark the earlier occurrence(s) as dead/ignorable. We could
- * avoid the O(N^2) behavior here with some additional state,
- * but it seems unlikely to be worth the trouble.
- */
- ConfigVariable *pitem;
-
- for (pitem = head; pitem != item; pitem = pitem->next)
- {
- if (!pitem->ignore &&
- strcmp(pitem->name, item->name) == 0)
- pitem->ignore = true;
- }
- }
- /* Now mark it as present in file */
- record->status |= GUC_IS_IN_FILE;
- }
- else if (!valid_custom_variable_name(item->name))
- {
- /* Invalid non-custom variable, so complain */
- ereport(elevel,
- (errcode(ERRCODE_UNDEFINED_OBJECT),
- errmsg("unrecognized configuration parameter \"%s\" in file \"%s\" line %d",
- item->name,
- item->filename, item->sourceline)));
- item->errmsg = pstrdup("unrecognized configuration parameter");
- error = true;
- ConfFileWithError = item->filename;
- }
- }
-
- /*
- * If we've detected any errors so far, we don't want to risk applying any
- * changes.
- */
- if (error)
- goto bail_out;
-
- /* Otherwise, set flag that we're beginning to apply changes */
- applying = true;
-
- /*
- * Check for variables having been removed from the config file, and
- * revert their reset values (and perhaps also effective values) to the
- * boot-time defaults. If such a variable can't be changed after startup,
- * report that and continue.
- */
- for (i = 0; i < num_guc_variables; i++)
- {
- struct config_generic *gconf = guc_variables[i];
- GucStack *stack;
-
- if (gconf->reset_source != PGC_S_FILE ||
- (gconf->status & GUC_IS_IN_FILE))
- continue;
- if (gconf->context < PGC_SIGHUP)
- {
- /* The removal can't be effective without a restart */
- gconf->status |= GUC_PENDING_RESTART;
- ereport(elevel,
- (errcode(ERRCODE_CANT_CHANGE_RUNTIME_PARAM),
- errmsg("parameter \"%s\" cannot be changed without restarting the server",
- gconf->name)));
- record_config_file_error(psprintf("parameter \"%s\" cannot be changed without restarting the server",
- gconf->name),
- NULL, 0,
- &head, &tail);
- error = true;
- continue;
- }
-
- /* No more to do if we're just doing show_all_file_settings() */
- if (!applySettings)
- continue;
-
- /*
- * Reset any "file" sources to "default", else set_config_option will
- * not override those settings.
- */
- if (gconf->reset_source == PGC_S_FILE)
- gconf->reset_source = PGC_S_DEFAULT;
- if (gconf->source == PGC_S_FILE)
- gconf->source = PGC_S_DEFAULT;
- for (stack = gconf->stack; stack; stack = stack->prev)
- {
- if (stack->source == PGC_S_FILE)
- stack->source = PGC_S_DEFAULT;
- }
-
- /* Now we can re-apply the wired-in default (i.e., the boot_val) */
- if (set_config_option(gconf->name, NULL,
- context, PGC_S_DEFAULT,
- GUC_ACTION_SET, true, 0, false) > 0)
- {
- /* Log the change if appropriate */
- if (context == PGC_SIGHUP)
- ereport(elevel,
- (errmsg("parameter \"%s\" removed from configuration file, reset to default",
- gconf->name)));
- }
- }
-
- /*
- * Restore any variables determined by environment variables or
- * dynamically-computed defaults. This is a no-op except in the case
- * where one of these had been in the config file and is now removed.
- *
- * In particular, we *must not* do this during the postmaster's initial
- * loading of the file, since the timezone functions in particular should
- * be run only after initialization is complete.
- *
- * XXX this is an unmaintainable crock, because we have to know how to set
- * (or at least what to call to set) every non-PGC_INTERNAL variable that
- * could potentially have PGC_S_DYNAMIC_DEFAULT or PGC_S_ENV_VAR source.
- */
- if (context == PGC_SIGHUP && applySettings)
- {
- InitializeGUCOptionsFromEnvironment();
- pg_timezone_abbrev_initialize();
- /* this selects SQL_ASCII in processes not connected to a database */
- SetConfigOption("client_encoding", GetDatabaseEncodingName(),
- PGC_BACKEND, PGC_S_DYNAMIC_DEFAULT);
- }
-
- /*
- * Now apply the values from the config file.
- */
- for (item = head; item; item = item->next)
- {
- char *pre_value = NULL;
- int scres;
-
- /* Ignore anything marked as ignorable */
- if (item->ignore)
- continue;
-
- /* In SIGHUP cases in the postmaster, we want to report changes */
- if (context == PGC_SIGHUP && applySettings && !IsUnderPostmaster)
- {
- const char *preval = GetConfigOption(item->name, true, false);
-
- /* If option doesn't exist yet or is NULL, treat as empty string */
- if (!preval)
- preval = "";
- /* must dup, else might have dangling pointer below */
- pre_value = pstrdup(preval);
- }
-
- scres = set_config_option(item->name, item->value,
- context, PGC_S_FILE,
- GUC_ACTION_SET, applySettings, 0, false);
- if (scres > 0)
- {
- /* variable was updated, so log the change if appropriate */
- if (pre_value)
- {
- const char *post_value = GetConfigOption(item->name, true, false);
-
- if (!post_value)
- post_value = "";
- if (strcmp(pre_value, post_value) != 0)
- ereport(elevel,
- (errmsg("parameter \"%s\" changed to \"%s\"",
- item->name, item->value)));
- }
- item->applied = true;
- }
- else if (scres == 0)
- {
- error = true;
- item->errmsg = pstrdup("setting could not be applied");
- ConfFileWithError = item->filename;
- }
- else
- {
- /* no error, but variable's active value was not changed */
- item->applied = true;
- }
-
- /*
- * We should update source location unless there was an error, since
- * even if the active value didn't change, the reset value might have.
- * (In the postmaster, there won't be a difference, but it does matter
- * in backends.)
- */
- if (scres != 0 && applySettings)
- set_config_sourcefile(item->name, item->filename,
- item->sourceline);
-
- if (pre_value)
- pfree(pre_value);
- }
-
- /* Remember when we last successfully loaded the config file. */
- if (applySettings)
- PgReloadTime = GetCurrentTimestamp();
-
-bail_out:
- if (error && applySettings)
- {
- /* During postmaster startup, any error is fatal */
- if (context == PGC_POSTMASTER)
- ereport(ERROR,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors",
- ConfFileWithError)));
- else if (applying)
- ereport(elevel,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
- ConfFileWithError)));
- else
- ereport(elevel,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors; no changes were applied",
- ConfFileWithError)));
- }
-
- /* Successful or otherwise, return the collected data list */
- return head;
-}
-
/*
* Given a configuration file or directory location that may be a relative
* path, return an absolute one. We consider the location to be relative to
@@ -659,7 +304,7 @@ cleanup:
* Capture an error message in the ConfigVariable list returned by
* config file parsing.
*/
-static void
+void
record_config_file_error(const char *errmsg,
const char *config_file,
int lineno,
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 5db5df6285..e4a00f1205 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -243,10 +243,6 @@ static void assign_recovery_target_lsn(const char *newval, void *extra);
static bool check_primary_slot_name(char **newval, void **extra, GucSource source);
static bool check_default_with_oids(bool *newval, void **extra, GucSource source);
-/* Private functions in guc-file.l that need to be called from guc.c */
-static ConfigVariable *ProcessConfigFileInternal(GucContext context,
- bool applySettings, int elevel);
-
/*
* Track whether there were any deferred checks for custom resource managers
* specified in wal_consistency_checking.
@@ -5164,8 +5160,8 @@ static bool report_needed; /* true if any GUC_REPORT reports are needed */
static int GUCNestLevel = 0; /* 1 when in main transaction */
+static struct config_generic *find_option(const char *name, bool create_placeholders, bool skip_errors, int elevel);
static int guc_var_compare(const void *a, const void *b);
-static int guc_name_compare(const char *namea, const char *nameb);
static void InitializeGUCOptionsFromEnvironment(void);
static void InitializeOneGUCOption(struct config_generic *gconf);
static void push_old_value(struct config_generic *gconf, GucAction action);
@@ -5184,7 +5180,359 @@ static bool validate_option_array_item(const char *name, const char *value,
static void write_auto_conf_file(int fd, const char *filename, ConfigVariable *head_p);
static void replace_auto_config_value(ConfigVariable **head_p, ConfigVariable **tail_p,
const char *name, const char *value);
+static bool valid_custom_variable_name(const char *name);
+
+/*
+ * This function handles both actual config file (re)loads and execution of
+ * show_all_file_settings() (i.e., the pg_file_settings view). In the latter
+ * case we don't apply any of the settings, but we make all the usual validity
+ * checks, and we return the ConfigVariable list so that it can be printed out
+ * by show_all_file_settings().
+ */
+ConfigVariable *
+ProcessConfigFileInternal(GucContext context, bool applySettings, int elevel)
+{
+ bool error = false;
+ bool applying = false;
+ const char *ConfFileWithError;
+ ConfigVariable *item,
+ *head,
+ *tail;
+ int i;
+
+ /* Parse the main config file into a list of option names and values */
+ ConfFileWithError = ConfigFileName;
+ head = tail = NULL;
+
+ if (!ParseConfigFile(ConfigFileName, true,
+ NULL, 0, 0, elevel,
+ &head, &tail))
+ {
+ /* Syntax error(s) detected in the file, so bail out */
+ error = true;
+ goto bail_out;
+ }
+
+ /*
+ * Parse the PG_AUTOCONF_FILENAME file, if present, after the main file to
+ * replace any parameters set by ALTER SYSTEM command. Because this file
+ * is in the data directory, we can't read it until the DataDir has been
+ * set.
+ */
+ if (DataDir)
+ {
+ if (!ParseConfigFile(PG_AUTOCONF_FILENAME, false,
+ NULL, 0, 0, elevel,
+ &head, &tail))
+ {
+ /* Syntax error(s) detected in the file, so bail out */
+ error = true;
+ ConfFileWithError = PG_AUTOCONF_FILENAME;
+ goto bail_out;
+ }
+ }
+ else
+ {
+ /*
+ * If DataDir is not set, the PG_AUTOCONF_FILENAME file cannot be
+ * read. In this case, we don't want to accept any settings but
+ * data_directory from postgresql.conf, because they might be
+ * overwritten with settings in the PG_AUTOCONF_FILENAME file which
+ * will be read later. OTOH, since data_directory isn't allowed in the
+ * PG_AUTOCONF_FILENAME file, it will never be overwritten later.
+ */
+ ConfigVariable *newlist = NULL;
+
+ /*
+ * Prune all items except the last "data_directory" from the list.
+ */
+ for (item = head; item; item = item->next)
+ {
+ if (!item->ignore &&
+ strcmp(item->name, "data_directory") == 0)
+ newlist = item;
+ }
+
+ if (newlist)
+ newlist->next = NULL;
+ head = tail = newlist;
+
+ /*
+ * Quick exit if data_directory is not present in file.
+ *
+ * We need not do any further processing, in particular we don't set
+ * PgReloadTime; that will be set soon by subsequent full loading of
+ * the config file.
+ */
+ if (head == NULL)
+ goto bail_out;
+ }
+
+ /*
+ * Mark all extant GUC variables as not present in the config file. We
+ * need this so that we can tell below which ones have been removed from
+ * the file since we last processed it.
+ */
+ for (i = 0; i < num_guc_variables; i++)
+ {
+ struct config_generic *gconf = guc_variables[i];
+
+ gconf->status &= ~GUC_IS_IN_FILE;
+ }
+
+ /*
+ * Check if all the supplied option names are valid, as an additional
+ * quasi-syntactic check on the validity of the config file. It is
+ * important that the postmaster and all backends agree on the results of
+ * this phase, else we will have strange inconsistencies about which
+ * processes accept a config file update and which don't. Hence, unknown
+ * custom variable names have to be accepted without complaint. For the
+ * same reason, we don't attempt to validate the options' values here.
+ *
+ * In addition, the GUC_IS_IN_FILE flag is set on each existing GUC
+ * variable mentioned in the file; and we detect duplicate entries in the
+ * file and mark the earlier occurrences as ignorable.
+ */
+ for (item = head; item; item = item->next)
+ {
+ struct config_generic *record;
+
+ /* Ignore anything already marked as ignorable */
+ if (item->ignore)
+ continue;
+
+ /*
+ * Try to find the variable; but do not create a custom placeholder if
+ * it's not there already.
+ */
+ record = find_option(item->name, false, true, elevel);
+
+ if (record)
+ {
+ /* If it's already marked, then this is a duplicate entry */
+ if (record->status & GUC_IS_IN_FILE)
+ {
+ /*
+ * Mark the earlier occurrence(s) as dead/ignorable. We could
+ * avoid the O(N^2) behavior here with some additional state,
+ * but it seems unlikely to be worth the trouble.
+ */
+ ConfigVariable *pitem;
+
+ for (pitem = head; pitem != item; pitem = pitem->next)
+ {
+ if (!pitem->ignore &&
+ strcmp(pitem->name, item->name) == 0)
+ pitem->ignore = true;
+ }
+ }
+ /* Now mark it as present in file */
+ record->status |= GUC_IS_IN_FILE;
+ }
+ else if (!valid_custom_variable_name(item->name))
+ {
+ /* Invalid non-custom variable, so complain */
+ ereport(elevel,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("unrecognized configuration parameter \"%s\" in file \"%s\" line %d",
+ item->name,
+ item->filename, item->sourceline)));
+ item->errmsg = pstrdup("unrecognized configuration parameter");
+ error = true;
+ ConfFileWithError = item->filename;
+ }
+ }
+
+ /*
+ * If we've detected any errors so far, we don't want to risk applying any
+ * changes.
+ */
+ if (error)
+ goto bail_out;
+
+ /* Otherwise, set flag that we're beginning to apply changes */
+ applying = true;
+
+ /*
+ * Check for variables having been removed from the config file, and
+ * revert their reset values (and perhaps also effective values) to the
+ * boot-time defaults. If such a variable can't be changed after startup,
+ * report that and continue.
+ */
+ for (i = 0; i < num_guc_variables; i++)
+ {
+ struct config_generic *gconf = guc_variables[i];
+ GucStack *stack;
+
+ if (gconf->reset_source != PGC_S_FILE ||
+ (gconf->status & GUC_IS_IN_FILE))
+ continue;
+ if (gconf->context < PGC_SIGHUP)
+ {
+ /* The removal can't be effective without a restart */
+ gconf->status |= GUC_PENDING_RESTART;
+ ereport(elevel,
+ (errcode(ERRCODE_CANT_CHANGE_RUNTIME_PARAM),
+ errmsg("parameter \"%s\" cannot be changed without restarting the server",
+ gconf->name)));
+ record_config_file_error(psprintf("parameter \"%s\" cannot be changed without restarting the server",
+ gconf->name),
+ NULL, 0,
+ &head, &tail);
+ error = true;
+ continue;
+ }
+
+ /* No more to do if we're just doing show_all_file_settings() */
+ if (!applySettings)
+ continue;
+
+ /*
+ * Reset any "file" sources to "default", else set_config_option will
+ * not override those settings.
+ */
+ if (gconf->reset_source == PGC_S_FILE)
+ gconf->reset_source = PGC_S_DEFAULT;
+ if (gconf->source == PGC_S_FILE)
+ gconf->source = PGC_S_DEFAULT;
+ for (stack = gconf->stack; stack; stack = stack->prev)
+ {
+ if (stack->source == PGC_S_FILE)
+ stack->source = PGC_S_DEFAULT;
+ }
+
+ /* Now we can re-apply the wired-in default (i.e., the boot_val) */
+ if (set_config_option(gconf->name, NULL,
+ context, PGC_S_DEFAULT,
+ GUC_ACTION_SET, true, 0, false) > 0)
+ {
+ /* Log the change if appropriate */
+ if (context == PGC_SIGHUP)
+ ereport(elevel,
+ (errmsg("parameter \"%s\" removed from configuration file, reset to default",
+ gconf->name)));
+ }
+ }
+
+ /*
+ * Restore any variables determined by environment variables or
+ * dynamically-computed defaults. This is a no-op except in the case
+ * where one of these had been in the config file and is now removed.
+ *
+ * In particular, we *must not* do this during the postmaster's initial
+ * loading of the file, since the timezone functions in particular should
+ * be run only after initialization is complete.
+ *
+ * XXX this is an unmaintainable crock, because we have to know how to set
+ * (or at least what to call to set) every non-PGC_INTERNAL variable that
+ * could potentially have PGC_S_DYNAMIC_DEFAULT or PGC_S_ENV_VAR source.
+ */
+ if (context == PGC_SIGHUP && applySettings)
+ {
+ InitializeGUCOptionsFromEnvironment();
+ pg_timezone_abbrev_initialize();
+ /* this selects SQL_ASCII in processes not connected to a database */
+ SetConfigOption("client_encoding", GetDatabaseEncodingName(),
+ PGC_BACKEND, PGC_S_DYNAMIC_DEFAULT);
+ }
+ /*
+ * Now apply the values from the config file.
+ */
+ for (item = head; item; item = item->next)
+ {
+ char *pre_value = NULL;
+ int scres;
+
+ /* Ignore anything marked as ignorable */
+ if (item->ignore)
+ continue;
+
+ /* In SIGHUP cases in the postmaster, we want to report changes */
+ if (context == PGC_SIGHUP && applySettings && !IsUnderPostmaster)
+ {
+ const char *preval = GetConfigOption(item->name, true, false);
+
+ /* If option doesn't exist yet or is NULL, treat as empty string */
+ if (!preval)
+ preval = "";
+ /* must dup, else might have dangling pointer below */
+ pre_value = pstrdup(preval);
+ }
+
+ scres = set_config_option(item->name, item->value,
+ context, PGC_S_FILE,
+ GUC_ACTION_SET, applySettings, 0, false);
+ if (scres > 0)
+ {
+ /* variable was updated, so log the change if appropriate */
+ if (pre_value)
+ {
+ const char *post_value = GetConfigOption(item->name, true, false);
+
+ if (!post_value)
+ post_value = "";
+ if (strcmp(pre_value, post_value) != 0)
+ ereport(elevel,
+ (errmsg("parameter \"%s\" changed to \"%s\"",
+ item->name, item->value)));
+ }
+ item->applied = true;
+ }
+ else if (scres == 0)
+ {
+ error = true;
+ item->errmsg = pstrdup("setting could not be applied");
+ ConfFileWithError = item->filename;
+ }
+ else
+ {
+ /* no error, but variable's active value was not changed */
+ item->applied = true;
+ }
+
+ /*
+ * We should update source location unless there was an error, since
+ * even if the active value didn't change, the reset value might have.
+ * (In the postmaster, there won't be a difference, but it does matter
+ * in backends.)
+ */
+ if (scres != 0 && applySettings)
+ set_config_sourcefile(item->name, item->filename,
+ item->sourceline);
+
+ if (pre_value)
+ pfree(pre_value);
+ }
+
+ /* Remember when we last successfully loaded the config file. */
+ if (applySettings)
+ PgReloadTime = GetCurrentTimestamp();
+
+bail_out:
+ if (error && applySettings)
+ {
+ /* During postmaster startup, any error is fatal */
+ if (context == PGC_POSTMASTER)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors",
+ ConfFileWithError)));
+ else if (applying)
+ ereport(elevel,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
+ ConfFileWithError)));
+ else
+ ereport(elevel,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors; no changes were applied",
+ ConfFileWithError)));
+ }
+
+ /* Successful or otherwise, return the collected data list */
+ return head;
+}
/*
* Some infrastructure for checking malloc/strdup/realloc calls
@@ -5741,7 +6089,7 @@ guc_var_compare(const void *a, const void *b)
/*
* the bare comparison function for GUC names
*/
-static int
+int
guc_name_compare(const char *namea, const char *nameb)
{
/*
@@ -12988,5 +13336,3 @@ check_default_with_oids(bool *newval, void **extra, GucSource source)
return true;
}
-
-#include "guc-file.c"
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index e734493a48..aae071cd82 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -442,6 +442,15 @@ extern void GUC_check_errcode(int sqlerrcode);
pre_format_elog_string(errno, TEXTDOMAIN), \
GUC_check_errhint_string = format_elog_string
+/* functions shared between guc.c and guc-file.l */
+extern int guc_name_compare(const char *namea, const char *nameb);
+extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
+ bool applySettings, int elevel);
+extern void record_config_file_error(const char *errmsg,
+ const char *config_file,
+ int lineno,
+ ConfigVariable **head_p,
+ ConfigVariable **tail_p);
/*
* The following functions are not in guc.c, but are declared here to avoid
--
2.36.1
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: build remaining Flex files standalone
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 04:07 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-12 06:01 ` build remaining Flex files standalone John Naylor <[email protected]>
@ 2022-08-13 08:39 ` John Naylor <[email protected]>
2022-08-15 18:11 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: John Naylor @ 2022-08-13 08:39 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
Here are the rest. Most of it was pretty straightforward, with the
main exception of jsonpath_scan.c, which is not quite finished. That
one passes tests but still has one compiler warning. I'm unsure how
much of what is there already is really necessary or was cargo-culted
from elsewhere without explanation. For starters, I'm not sure why the
grammar has a forward declaration of "union YYSTYPE". It's noteworthy
that it used to compile standalone, but with a bit more stuff, and
that was reverted in 550b9d26f80fa30. I can hack on it some more later
but I ran out of steam today.
Other questions thus far:
- "BISONFLAGS += -d" is now in every make file with a .y file -- can
we just force that everywhere?
- Include order seems to matter for the grammar's .h file. I didn't
test if that was the case every time, and after a few miscompiles just
always made it the last inclusion, but I'm wondering if we should keep
those inclusions outside %top{} and put it at the start of the next
%{} ?
- contrib/cubeparse.y now has a global variable -- not terrific, but I
wanted to get something working first.
- I'm actually okay with guc-file.c now, but I'll still welcome
comments on that.
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[text/x-patch] v201-0003-Build-repl_scanner.c-standalone.patch (4.1K, ../../CAFBsxsGq=jLMofR3BN1S1xJcaJsYcQnkWFRtRDObYVtDST0+Vg@mail.gmail.com/2-v201-0003-Build-repl_scanner.c-standalone.patch)
download | inline diff:
From da2b610b8608e6759f5ed9cc32b487ea8e750ce4 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Fri, 12 Aug 2022 17:09:45 +0700
Subject: [PATCH v201 3/9] Build repl_scanner.c standalone
---
src/backend/replication/Makefile | 11 +++++++++--
src/backend/replication/repl_gram.y | 2 --
src/backend/replication/repl_scanner.l | 27 +++++++++++++++-----------
3 files changed, 25 insertions(+), 15 deletions(-)
diff --git a/src/backend/replication/Makefile b/src/backend/replication/Makefile
index 2bffac58c0..bc8170418f 100644
--- a/src/backend/replication/Makefile
+++ b/src/backend/replication/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
OBJS = \
repl_gram.o \
+ repl_scanner.o \
slot.o \
slotfuncs.o \
syncrep.o \
@@ -28,8 +29,14 @@ SUBDIRS = logical
include $(top_srcdir)/src/backend/common.mk
-# repl_scanner is compiled as part of repl_gram
-repl_gram.o: repl_scanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+repl_gram.h: repl_gram.c
+ touch $@
+
+repl_gram.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+repl_gram.o repl_scanner.o: repl_gram.h
# syncrep_scanner is compiled as part of syncrep_gram
syncrep_gram.o: syncrep_scanner.c
diff --git a/src/backend/replication/repl_gram.y b/src/backend/replication/repl_gram.y
index 4cf087e602..b343f108d3 100644
--- a/src/backend/replication/repl_gram.y
+++ b/src/backend/replication/repl_gram.y
@@ -416,5 +416,3 @@ ident_or_keyword:
;
%%
-
-#include "repl_scanner.c"
diff --git a/src/backend/replication/repl_scanner.l b/src/backend/replication/repl_scanner.l
index 586f0d3a5c..95a933c9a8 100644
--- a/src/backend/replication/repl_scanner.l
+++ b/src/backend/replication/repl_scanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* repl_scanner.l
@@ -17,7 +17,12 @@
#include "utils/builtins.h"
#include "parser/scansup.h"
+#include "replication/walsender_private.h"
+
+#include "repl_gram.h"
+}
+%{
/* Avoid exit() on fatal scanner errors (a bit ugly -- see yy_fatal_error) */
#undef fprintf
#define fprintf(file, fmt, msg) fprintf_to_ereport(fmt, msg)
@@ -130,7 +135,7 @@ WAIT { return K_WAIT; }
{space}+ { /* do nothing */ }
{digit}+ {
- yylval.uintval = strtoul(yytext, NULL, 10);
+ replication_yylval.uintval = strtoul(yytext, NULL, 10);
return UCONST;
}
@@ -138,8 +143,8 @@ WAIT { return K_WAIT; }
uint32 hi,
lo;
if (sscanf(yytext, "%X/%X", &hi, &lo) != 2)
- yyerror("invalid streaming start location");
- yylval.recptr = ((uint64) hi) << 32 | lo;
+ replication_yyerror("invalid streaming start location");
+ replication_yylval.recptr = ((uint64) hi) << 32 | lo;
return RECPTR;
}
@@ -151,7 +156,7 @@ WAIT { return K_WAIT; }
<xq>{quotestop} {
yyless(1);
BEGIN(INITIAL);
- yylval.str = litbufdup();
+ replication_yylval.str = litbufdup();
return SCONST;
}
@@ -173,9 +178,9 @@ WAIT { return K_WAIT; }
yyless(1);
BEGIN(INITIAL);
- yylval.str = litbufdup();
- len = strlen(yylval.str);
- truncate_identifier(yylval.str, len, true);
+ replication_yylval.str = litbufdup();
+ len = strlen(replication_yylval.str);
+ truncate_identifier(replication_yylval.str, len, true);
return IDENT;
}
@@ -186,7 +191,7 @@ WAIT { return K_WAIT; }
{identifier} {
int len = strlen(yytext);
- yylval.str = downcase_truncate_identifier(yytext, len, true);
+ replication_yylval.str = downcase_truncate_identifier(yytext, len, true);
return IDENT;
}
@@ -195,7 +200,7 @@ WAIT { return K_WAIT; }
return yytext[0];
}
-<xq,xd><<EOF>> { yyerror("unterminated quoted string"); }
+<xq,xd><<EOF>> { replication_yyerror("unterminated quoted string"); }
<<EOF>> {
@@ -231,7 +236,7 @@ addlitchar(unsigned char ychar)
}
void
-yyerror(const char *message)
+replication_yyerror(const char *message)
{
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
--
2.36.1
[text/x-patch] v201-0001-Build-guc-file.c-standalone.patch (27.5K, ../../CAFBsxsGq=jLMofR3BN1S1xJcaJsYcQnkWFRtRDObYVtDST0+Vg@mail.gmail.com/3-v201-0001-Build-guc-file.c-standalone.patch)
download | inline diff:
From d723ba14acf56fd432e9e263db937fcc13fc0355 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Thu, 11 Aug 2022 19:38:37 +0700
Subject: [PATCH v201 1/9] Build guc-file.c standalone
The proposed Meson build system will need a way to ignore certain
generated files in order to coexist with the autoconf build system,
and #include'd C files generated by Flex make this more difficult.
Build guc-file.c separately from guc.c, as was done in 72b1e3a21.
TODO: other Flex-generated files
Discussion: https://www.postgresql.org/message-id/20220810171935.7k5zgnjwqzalzmtm%40awork3.anarazel.de
---
src/backend/utils/misc/Makefile | 5 +-
src/backend/utils/misc/guc-file.l | 367 +-----------------------------
src/backend/utils/misc/guc.c | 362 ++++++++++++++++++++++++++++-
src/include/utils/guc.h | 9 +
4 files changed, 370 insertions(+), 373 deletions(-)
diff --git a/src/backend/utils/misc/Makefile b/src/backend/utils/misc/Makefile
index 1d5327cf64..cf7ce9bc83 100644
--- a/src/backend/utils/misc/Makefile
+++ b/src/backend/utils/misc/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
OBJS = \
guc.o \
+ guc-file.o \
help_config.o \
pg_config.o \
pg_controldata.o \
@@ -37,10 +38,6 @@ endif
include $(top_srcdir)/src/backend/common.mk
-# guc-file is compiled as part of guc
-guc.o: guc-file.c
-
# Note: guc-file.c is not deleted by 'make clean',
# since we want to ship it in distribution tarballs.
clean:
- @rm -f lex.yy.c
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index ce5633844c..08adb454de 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -7,7 +7,7 @@
* src/backend/utils/misc/guc-file.l
*/
-%{
+%top{
#include "postgres.h"
@@ -17,9 +17,12 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "storage/fd.h"
+#include <sys/stat.h>
#include "utils/guc.h"
+#include "utils/memutils.h"
+}
-
+%{
/*
* flex emits a yy_fatal_error() function that it calls in response to
* critical errors like malloc failure, file I/O errors, and detection of
@@ -48,12 +51,6 @@ static sigjmp_buf *GUC_flex_fatal_jmp;
static void FreeConfigVariable(ConfigVariable *item);
-static void record_config_file_error(const char *errmsg,
- const char *config_file,
- int lineno,
- ConfigVariable **head_p,
- ConfigVariable **tail_p);
-
static int GUC_flex_fatal(const char *msg);
/* LCOV_EXCL_START */
@@ -159,358 +156,6 @@ ProcessConfigFile(GucContext context)
MemoryContextDelete(config_cxt);
}
-/*
- * This function handles both actual config file (re)loads and execution of
- * show_all_file_settings() (i.e., the pg_file_settings view). In the latter
- * case we don't apply any of the settings, but we make all the usual validity
- * checks, and we return the ConfigVariable list so that it can be printed out
- * by show_all_file_settings().
- */
-static ConfigVariable *
-ProcessConfigFileInternal(GucContext context, bool applySettings, int elevel)
-{
- bool error = false;
- bool applying = false;
- const char *ConfFileWithError;
- ConfigVariable *item,
- *head,
- *tail;
- int i;
-
- /* Parse the main config file into a list of option names and values */
- ConfFileWithError = ConfigFileName;
- head = tail = NULL;
-
- if (!ParseConfigFile(ConfigFileName, true,
- NULL, 0, 0, elevel,
- &head, &tail))
- {
- /* Syntax error(s) detected in the file, so bail out */
- error = true;
- goto bail_out;
- }
-
- /*
- * Parse the PG_AUTOCONF_FILENAME file, if present, after the main file to
- * replace any parameters set by ALTER SYSTEM command. Because this file
- * is in the data directory, we can't read it until the DataDir has been
- * set.
- */
- if (DataDir)
- {
- if (!ParseConfigFile(PG_AUTOCONF_FILENAME, false,
- NULL, 0, 0, elevel,
- &head, &tail))
- {
- /* Syntax error(s) detected in the file, so bail out */
- error = true;
- ConfFileWithError = PG_AUTOCONF_FILENAME;
- goto bail_out;
- }
- }
- else
- {
- /*
- * If DataDir is not set, the PG_AUTOCONF_FILENAME file cannot be
- * read. In this case, we don't want to accept any settings but
- * data_directory from postgresql.conf, because they might be
- * overwritten with settings in the PG_AUTOCONF_FILENAME file which
- * will be read later. OTOH, since data_directory isn't allowed in the
- * PG_AUTOCONF_FILENAME file, it will never be overwritten later.
- */
- ConfigVariable *newlist = NULL;
-
- /*
- * Prune all items except the last "data_directory" from the list.
- */
- for (item = head; item; item = item->next)
- {
- if (!item->ignore &&
- strcmp(item->name, "data_directory") == 0)
- newlist = item;
- }
-
- if (newlist)
- newlist->next = NULL;
- head = tail = newlist;
-
- /*
- * Quick exit if data_directory is not present in file.
- *
- * We need not do any further processing, in particular we don't set
- * PgReloadTime; that will be set soon by subsequent full loading of
- * the config file.
- */
- if (head == NULL)
- goto bail_out;
- }
-
- /*
- * Mark all extant GUC variables as not present in the config file. We
- * need this so that we can tell below which ones have been removed from
- * the file since we last processed it.
- */
- for (i = 0; i < num_guc_variables; i++)
- {
- struct config_generic *gconf = guc_variables[i];
-
- gconf->status &= ~GUC_IS_IN_FILE;
- }
-
- /*
- * Check if all the supplied option names are valid, as an additional
- * quasi-syntactic check on the validity of the config file. It is
- * important that the postmaster and all backends agree on the results of
- * this phase, else we will have strange inconsistencies about which
- * processes accept a config file update and which don't. Hence, unknown
- * custom variable names have to be accepted without complaint. For the
- * same reason, we don't attempt to validate the options' values here.
- *
- * In addition, the GUC_IS_IN_FILE flag is set on each existing GUC
- * variable mentioned in the file; and we detect duplicate entries in the
- * file and mark the earlier occurrences as ignorable.
- */
- for (item = head; item; item = item->next)
- {
- struct config_generic *record;
-
- /* Ignore anything already marked as ignorable */
- if (item->ignore)
- continue;
-
- /*
- * Try to find the variable; but do not create a custom placeholder if
- * it's not there already.
- */
- record = find_option(item->name, false, true, elevel);
-
- if (record)
- {
- /* If it's already marked, then this is a duplicate entry */
- if (record->status & GUC_IS_IN_FILE)
- {
- /*
- * Mark the earlier occurrence(s) as dead/ignorable. We could
- * avoid the O(N^2) behavior here with some additional state,
- * but it seems unlikely to be worth the trouble.
- */
- ConfigVariable *pitem;
-
- for (pitem = head; pitem != item; pitem = pitem->next)
- {
- if (!pitem->ignore &&
- strcmp(pitem->name, item->name) == 0)
- pitem->ignore = true;
- }
- }
- /* Now mark it as present in file */
- record->status |= GUC_IS_IN_FILE;
- }
- else if (!valid_custom_variable_name(item->name))
- {
- /* Invalid non-custom variable, so complain */
- ereport(elevel,
- (errcode(ERRCODE_UNDEFINED_OBJECT),
- errmsg("unrecognized configuration parameter \"%s\" in file \"%s\" line %d",
- item->name,
- item->filename, item->sourceline)));
- item->errmsg = pstrdup("unrecognized configuration parameter");
- error = true;
- ConfFileWithError = item->filename;
- }
- }
-
- /*
- * If we've detected any errors so far, we don't want to risk applying any
- * changes.
- */
- if (error)
- goto bail_out;
-
- /* Otherwise, set flag that we're beginning to apply changes */
- applying = true;
-
- /*
- * Check for variables having been removed from the config file, and
- * revert their reset values (and perhaps also effective values) to the
- * boot-time defaults. If such a variable can't be changed after startup,
- * report that and continue.
- */
- for (i = 0; i < num_guc_variables; i++)
- {
- struct config_generic *gconf = guc_variables[i];
- GucStack *stack;
-
- if (gconf->reset_source != PGC_S_FILE ||
- (gconf->status & GUC_IS_IN_FILE))
- continue;
- if (gconf->context < PGC_SIGHUP)
- {
- /* The removal can't be effective without a restart */
- gconf->status |= GUC_PENDING_RESTART;
- ereport(elevel,
- (errcode(ERRCODE_CANT_CHANGE_RUNTIME_PARAM),
- errmsg("parameter \"%s\" cannot be changed without restarting the server",
- gconf->name)));
- record_config_file_error(psprintf("parameter \"%s\" cannot be changed without restarting the server",
- gconf->name),
- NULL, 0,
- &head, &tail);
- error = true;
- continue;
- }
-
- /* No more to do if we're just doing show_all_file_settings() */
- if (!applySettings)
- continue;
-
- /*
- * Reset any "file" sources to "default", else set_config_option will
- * not override those settings.
- */
- if (gconf->reset_source == PGC_S_FILE)
- gconf->reset_source = PGC_S_DEFAULT;
- if (gconf->source == PGC_S_FILE)
- gconf->source = PGC_S_DEFAULT;
- for (stack = gconf->stack; stack; stack = stack->prev)
- {
- if (stack->source == PGC_S_FILE)
- stack->source = PGC_S_DEFAULT;
- }
-
- /* Now we can re-apply the wired-in default (i.e., the boot_val) */
- if (set_config_option(gconf->name, NULL,
- context, PGC_S_DEFAULT,
- GUC_ACTION_SET, true, 0, false) > 0)
- {
- /* Log the change if appropriate */
- if (context == PGC_SIGHUP)
- ereport(elevel,
- (errmsg("parameter \"%s\" removed from configuration file, reset to default",
- gconf->name)));
- }
- }
-
- /*
- * Restore any variables determined by environment variables or
- * dynamically-computed defaults. This is a no-op except in the case
- * where one of these had been in the config file and is now removed.
- *
- * In particular, we *must not* do this during the postmaster's initial
- * loading of the file, since the timezone functions in particular should
- * be run only after initialization is complete.
- *
- * XXX this is an unmaintainable crock, because we have to know how to set
- * (or at least what to call to set) every non-PGC_INTERNAL variable that
- * could potentially have PGC_S_DYNAMIC_DEFAULT or PGC_S_ENV_VAR source.
- */
- if (context == PGC_SIGHUP && applySettings)
- {
- InitializeGUCOptionsFromEnvironment();
- pg_timezone_abbrev_initialize();
- /* this selects SQL_ASCII in processes not connected to a database */
- SetConfigOption("client_encoding", GetDatabaseEncodingName(),
- PGC_BACKEND, PGC_S_DYNAMIC_DEFAULT);
- }
-
- /*
- * Now apply the values from the config file.
- */
- for (item = head; item; item = item->next)
- {
- char *pre_value = NULL;
- int scres;
-
- /* Ignore anything marked as ignorable */
- if (item->ignore)
- continue;
-
- /* In SIGHUP cases in the postmaster, we want to report changes */
- if (context == PGC_SIGHUP && applySettings && !IsUnderPostmaster)
- {
- const char *preval = GetConfigOption(item->name, true, false);
-
- /* If option doesn't exist yet or is NULL, treat as empty string */
- if (!preval)
- preval = "";
- /* must dup, else might have dangling pointer below */
- pre_value = pstrdup(preval);
- }
-
- scres = set_config_option(item->name, item->value,
- context, PGC_S_FILE,
- GUC_ACTION_SET, applySettings, 0, false);
- if (scres > 0)
- {
- /* variable was updated, so log the change if appropriate */
- if (pre_value)
- {
- const char *post_value = GetConfigOption(item->name, true, false);
-
- if (!post_value)
- post_value = "";
- if (strcmp(pre_value, post_value) != 0)
- ereport(elevel,
- (errmsg("parameter \"%s\" changed to \"%s\"",
- item->name, item->value)));
- }
- item->applied = true;
- }
- else if (scres == 0)
- {
- error = true;
- item->errmsg = pstrdup("setting could not be applied");
- ConfFileWithError = item->filename;
- }
- else
- {
- /* no error, but variable's active value was not changed */
- item->applied = true;
- }
-
- /*
- * We should update source location unless there was an error, since
- * even if the active value didn't change, the reset value might have.
- * (In the postmaster, there won't be a difference, but it does matter
- * in backends.)
- */
- if (scres != 0 && applySettings)
- set_config_sourcefile(item->name, item->filename,
- item->sourceline);
-
- if (pre_value)
- pfree(pre_value);
- }
-
- /* Remember when we last successfully loaded the config file. */
- if (applySettings)
- PgReloadTime = GetCurrentTimestamp();
-
-bail_out:
- if (error && applySettings)
- {
- /* During postmaster startup, any error is fatal */
- if (context == PGC_POSTMASTER)
- ereport(ERROR,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors",
- ConfFileWithError)));
- else if (applying)
- ereport(elevel,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
- ConfFileWithError)));
- else
- ereport(elevel,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors; no changes were applied",
- ConfFileWithError)));
- }
-
- /* Successful or otherwise, return the collected data list */
- return head;
-}
-
/*
* Given a configuration file or directory location that may be a relative
* path, return an absolute one. We consider the location to be relative to
@@ -659,7 +304,7 @@ cleanup:
* Capture an error message in the ConfigVariable list returned by
* config file parsing.
*/
-static void
+void
record_config_file_error(const char *errmsg,
const char *config_file,
int lineno,
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 5db5df6285..e4a00f1205 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -243,10 +243,6 @@ static void assign_recovery_target_lsn(const char *newval, void *extra);
static bool check_primary_slot_name(char **newval, void **extra, GucSource source);
static bool check_default_with_oids(bool *newval, void **extra, GucSource source);
-/* Private functions in guc-file.l that need to be called from guc.c */
-static ConfigVariable *ProcessConfigFileInternal(GucContext context,
- bool applySettings, int elevel);
-
/*
* Track whether there were any deferred checks for custom resource managers
* specified in wal_consistency_checking.
@@ -5164,8 +5160,8 @@ static bool report_needed; /* true if any GUC_REPORT reports are needed */
static int GUCNestLevel = 0; /* 1 when in main transaction */
+static struct config_generic *find_option(const char *name, bool create_placeholders, bool skip_errors, int elevel);
static int guc_var_compare(const void *a, const void *b);
-static int guc_name_compare(const char *namea, const char *nameb);
static void InitializeGUCOptionsFromEnvironment(void);
static void InitializeOneGUCOption(struct config_generic *gconf);
static void push_old_value(struct config_generic *gconf, GucAction action);
@@ -5184,7 +5180,359 @@ static bool validate_option_array_item(const char *name, const char *value,
static void write_auto_conf_file(int fd, const char *filename, ConfigVariable *head_p);
static void replace_auto_config_value(ConfigVariable **head_p, ConfigVariable **tail_p,
const char *name, const char *value);
+static bool valid_custom_variable_name(const char *name);
+
+/*
+ * This function handles both actual config file (re)loads and execution of
+ * show_all_file_settings() (i.e., the pg_file_settings view). In the latter
+ * case we don't apply any of the settings, but we make all the usual validity
+ * checks, and we return the ConfigVariable list so that it can be printed out
+ * by show_all_file_settings().
+ */
+ConfigVariable *
+ProcessConfigFileInternal(GucContext context, bool applySettings, int elevel)
+{
+ bool error = false;
+ bool applying = false;
+ const char *ConfFileWithError;
+ ConfigVariable *item,
+ *head,
+ *tail;
+ int i;
+
+ /* Parse the main config file into a list of option names and values */
+ ConfFileWithError = ConfigFileName;
+ head = tail = NULL;
+
+ if (!ParseConfigFile(ConfigFileName, true,
+ NULL, 0, 0, elevel,
+ &head, &tail))
+ {
+ /* Syntax error(s) detected in the file, so bail out */
+ error = true;
+ goto bail_out;
+ }
+
+ /*
+ * Parse the PG_AUTOCONF_FILENAME file, if present, after the main file to
+ * replace any parameters set by ALTER SYSTEM command. Because this file
+ * is in the data directory, we can't read it until the DataDir has been
+ * set.
+ */
+ if (DataDir)
+ {
+ if (!ParseConfigFile(PG_AUTOCONF_FILENAME, false,
+ NULL, 0, 0, elevel,
+ &head, &tail))
+ {
+ /* Syntax error(s) detected in the file, so bail out */
+ error = true;
+ ConfFileWithError = PG_AUTOCONF_FILENAME;
+ goto bail_out;
+ }
+ }
+ else
+ {
+ /*
+ * If DataDir is not set, the PG_AUTOCONF_FILENAME file cannot be
+ * read. In this case, we don't want to accept any settings but
+ * data_directory from postgresql.conf, because they might be
+ * overwritten with settings in the PG_AUTOCONF_FILENAME file which
+ * will be read later. OTOH, since data_directory isn't allowed in the
+ * PG_AUTOCONF_FILENAME file, it will never be overwritten later.
+ */
+ ConfigVariable *newlist = NULL;
+
+ /*
+ * Prune all items except the last "data_directory" from the list.
+ */
+ for (item = head; item; item = item->next)
+ {
+ if (!item->ignore &&
+ strcmp(item->name, "data_directory") == 0)
+ newlist = item;
+ }
+
+ if (newlist)
+ newlist->next = NULL;
+ head = tail = newlist;
+
+ /*
+ * Quick exit if data_directory is not present in file.
+ *
+ * We need not do any further processing, in particular we don't set
+ * PgReloadTime; that will be set soon by subsequent full loading of
+ * the config file.
+ */
+ if (head == NULL)
+ goto bail_out;
+ }
+
+ /*
+ * Mark all extant GUC variables as not present in the config file. We
+ * need this so that we can tell below which ones have been removed from
+ * the file since we last processed it.
+ */
+ for (i = 0; i < num_guc_variables; i++)
+ {
+ struct config_generic *gconf = guc_variables[i];
+
+ gconf->status &= ~GUC_IS_IN_FILE;
+ }
+
+ /*
+ * Check if all the supplied option names are valid, as an additional
+ * quasi-syntactic check on the validity of the config file. It is
+ * important that the postmaster and all backends agree on the results of
+ * this phase, else we will have strange inconsistencies about which
+ * processes accept a config file update and which don't. Hence, unknown
+ * custom variable names have to be accepted without complaint. For the
+ * same reason, we don't attempt to validate the options' values here.
+ *
+ * In addition, the GUC_IS_IN_FILE flag is set on each existing GUC
+ * variable mentioned in the file; and we detect duplicate entries in the
+ * file and mark the earlier occurrences as ignorable.
+ */
+ for (item = head; item; item = item->next)
+ {
+ struct config_generic *record;
+
+ /* Ignore anything already marked as ignorable */
+ if (item->ignore)
+ continue;
+
+ /*
+ * Try to find the variable; but do not create a custom placeholder if
+ * it's not there already.
+ */
+ record = find_option(item->name, false, true, elevel);
+
+ if (record)
+ {
+ /* If it's already marked, then this is a duplicate entry */
+ if (record->status & GUC_IS_IN_FILE)
+ {
+ /*
+ * Mark the earlier occurrence(s) as dead/ignorable. We could
+ * avoid the O(N^2) behavior here with some additional state,
+ * but it seems unlikely to be worth the trouble.
+ */
+ ConfigVariable *pitem;
+
+ for (pitem = head; pitem != item; pitem = pitem->next)
+ {
+ if (!pitem->ignore &&
+ strcmp(pitem->name, item->name) == 0)
+ pitem->ignore = true;
+ }
+ }
+ /* Now mark it as present in file */
+ record->status |= GUC_IS_IN_FILE;
+ }
+ else if (!valid_custom_variable_name(item->name))
+ {
+ /* Invalid non-custom variable, so complain */
+ ereport(elevel,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("unrecognized configuration parameter \"%s\" in file \"%s\" line %d",
+ item->name,
+ item->filename, item->sourceline)));
+ item->errmsg = pstrdup("unrecognized configuration parameter");
+ error = true;
+ ConfFileWithError = item->filename;
+ }
+ }
+
+ /*
+ * If we've detected any errors so far, we don't want to risk applying any
+ * changes.
+ */
+ if (error)
+ goto bail_out;
+
+ /* Otherwise, set flag that we're beginning to apply changes */
+ applying = true;
+
+ /*
+ * Check for variables having been removed from the config file, and
+ * revert their reset values (and perhaps also effective values) to the
+ * boot-time defaults. If such a variable can't be changed after startup,
+ * report that and continue.
+ */
+ for (i = 0; i < num_guc_variables; i++)
+ {
+ struct config_generic *gconf = guc_variables[i];
+ GucStack *stack;
+
+ if (gconf->reset_source != PGC_S_FILE ||
+ (gconf->status & GUC_IS_IN_FILE))
+ continue;
+ if (gconf->context < PGC_SIGHUP)
+ {
+ /* The removal can't be effective without a restart */
+ gconf->status |= GUC_PENDING_RESTART;
+ ereport(elevel,
+ (errcode(ERRCODE_CANT_CHANGE_RUNTIME_PARAM),
+ errmsg("parameter \"%s\" cannot be changed without restarting the server",
+ gconf->name)));
+ record_config_file_error(psprintf("parameter \"%s\" cannot be changed without restarting the server",
+ gconf->name),
+ NULL, 0,
+ &head, &tail);
+ error = true;
+ continue;
+ }
+
+ /* No more to do if we're just doing show_all_file_settings() */
+ if (!applySettings)
+ continue;
+
+ /*
+ * Reset any "file" sources to "default", else set_config_option will
+ * not override those settings.
+ */
+ if (gconf->reset_source == PGC_S_FILE)
+ gconf->reset_source = PGC_S_DEFAULT;
+ if (gconf->source == PGC_S_FILE)
+ gconf->source = PGC_S_DEFAULT;
+ for (stack = gconf->stack; stack; stack = stack->prev)
+ {
+ if (stack->source == PGC_S_FILE)
+ stack->source = PGC_S_DEFAULT;
+ }
+
+ /* Now we can re-apply the wired-in default (i.e., the boot_val) */
+ if (set_config_option(gconf->name, NULL,
+ context, PGC_S_DEFAULT,
+ GUC_ACTION_SET, true, 0, false) > 0)
+ {
+ /* Log the change if appropriate */
+ if (context == PGC_SIGHUP)
+ ereport(elevel,
+ (errmsg("parameter \"%s\" removed from configuration file, reset to default",
+ gconf->name)));
+ }
+ }
+
+ /*
+ * Restore any variables determined by environment variables or
+ * dynamically-computed defaults. This is a no-op except in the case
+ * where one of these had been in the config file and is now removed.
+ *
+ * In particular, we *must not* do this during the postmaster's initial
+ * loading of the file, since the timezone functions in particular should
+ * be run only after initialization is complete.
+ *
+ * XXX this is an unmaintainable crock, because we have to know how to set
+ * (or at least what to call to set) every non-PGC_INTERNAL variable that
+ * could potentially have PGC_S_DYNAMIC_DEFAULT or PGC_S_ENV_VAR source.
+ */
+ if (context == PGC_SIGHUP && applySettings)
+ {
+ InitializeGUCOptionsFromEnvironment();
+ pg_timezone_abbrev_initialize();
+ /* this selects SQL_ASCII in processes not connected to a database */
+ SetConfigOption("client_encoding", GetDatabaseEncodingName(),
+ PGC_BACKEND, PGC_S_DYNAMIC_DEFAULT);
+ }
+ /*
+ * Now apply the values from the config file.
+ */
+ for (item = head; item; item = item->next)
+ {
+ char *pre_value = NULL;
+ int scres;
+
+ /* Ignore anything marked as ignorable */
+ if (item->ignore)
+ continue;
+
+ /* In SIGHUP cases in the postmaster, we want to report changes */
+ if (context == PGC_SIGHUP && applySettings && !IsUnderPostmaster)
+ {
+ const char *preval = GetConfigOption(item->name, true, false);
+
+ /* If option doesn't exist yet or is NULL, treat as empty string */
+ if (!preval)
+ preval = "";
+ /* must dup, else might have dangling pointer below */
+ pre_value = pstrdup(preval);
+ }
+
+ scres = set_config_option(item->name, item->value,
+ context, PGC_S_FILE,
+ GUC_ACTION_SET, applySettings, 0, false);
+ if (scres > 0)
+ {
+ /* variable was updated, so log the change if appropriate */
+ if (pre_value)
+ {
+ const char *post_value = GetConfigOption(item->name, true, false);
+
+ if (!post_value)
+ post_value = "";
+ if (strcmp(pre_value, post_value) != 0)
+ ereport(elevel,
+ (errmsg("parameter \"%s\" changed to \"%s\"",
+ item->name, item->value)));
+ }
+ item->applied = true;
+ }
+ else if (scres == 0)
+ {
+ error = true;
+ item->errmsg = pstrdup("setting could not be applied");
+ ConfFileWithError = item->filename;
+ }
+ else
+ {
+ /* no error, but variable's active value was not changed */
+ item->applied = true;
+ }
+
+ /*
+ * We should update source location unless there was an error, since
+ * even if the active value didn't change, the reset value might have.
+ * (In the postmaster, there won't be a difference, but it does matter
+ * in backends.)
+ */
+ if (scres != 0 && applySettings)
+ set_config_sourcefile(item->name, item->filename,
+ item->sourceline);
+
+ if (pre_value)
+ pfree(pre_value);
+ }
+
+ /* Remember when we last successfully loaded the config file. */
+ if (applySettings)
+ PgReloadTime = GetCurrentTimestamp();
+
+bail_out:
+ if (error && applySettings)
+ {
+ /* During postmaster startup, any error is fatal */
+ if (context == PGC_POSTMASTER)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors",
+ ConfFileWithError)));
+ else if (applying)
+ ereport(elevel,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
+ ConfFileWithError)));
+ else
+ ereport(elevel,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors; no changes were applied",
+ ConfFileWithError)));
+ }
+
+ /* Successful or otherwise, return the collected data list */
+ return head;
+}
/*
* Some infrastructure for checking malloc/strdup/realloc calls
@@ -5741,7 +6089,7 @@ guc_var_compare(const void *a, const void *b)
/*
* the bare comparison function for GUC names
*/
-static int
+int
guc_name_compare(const char *namea, const char *nameb)
{
/*
@@ -12988,5 +13336,3 @@ check_default_with_oids(bool *newval, void **extra, GucSource source)
return true;
}
-
-#include "guc-file.c"
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index e734493a48..aae071cd82 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -442,6 +442,15 @@ extern void GUC_check_errcode(int sqlerrcode);
pre_format_elog_string(errno, TEXTDOMAIN), \
GUC_check_errhint_string = format_elog_string
+/* functions shared between guc.c and guc-file.l */
+extern int guc_name_compare(const char *namea, const char *nameb);
+extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
+ bool applySettings, int elevel);
+extern void record_config_file_error(const char *errmsg,
+ const char *config_file,
+ int lineno,
+ ConfigVariable **head_p,
+ ConfigVariable **tail_p);
/*
* The following functions are not in guc.c, but are declared here to avoid
--
2.36.1
[text/x-patch] v201-0005-Build-specscanner.c-standalone.patch (4.5K, ../../CAFBsxsGq=jLMofR3BN1S1xJcaJsYcQnkWFRtRDObYVtDST0+Vg@mail.gmail.com/4-v201-0005-Build-specscanner.c-standalone.patch)
download | inline diff:
From 4c105b7b415f0937e7fa42e8313c9452a8db1ad4 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 09:34:17 +0700
Subject: [PATCH v201 5/9] Build specscanner.c standalone
---
src/test/isolation/.gitignore | 1 +
src/test/isolation/Makefile | 15 +++++++++++----
src/test/isolation/specparse.y | 2 --
src/test/isolation/specscanner.l | 25 ++++++++++++++++---------
4 files changed, 28 insertions(+), 15 deletions(-)
diff --git a/src/test/isolation/.gitignore b/src/test/isolation/.gitignore
index 870dac4d28..2c13b4bf98 100644
--- a/src/test/isolation/.gitignore
+++ b/src/test/isolation/.gitignore
@@ -3,6 +3,7 @@
/pg_isolation_regress
# Local generated source files
+/specparse.h
/specparse.c
/specscanner.c
diff --git a/src/test/isolation/Makefile b/src/test/isolation/Makefile
index 0d452c89d4..b8738b7c1b 100644
--- a/src/test/isolation/Makefile
+++ b/src/test/isolation/Makefile
@@ -15,7 +15,8 @@ override CPPFLAGS := -I. -I$(srcdir) -I$(libpq_srcdir) \
OBJS = \
$(WIN32RES) \
isolationtester.o \
- specparse.o
+ specparse.o \
+ specscanner.o
all: isolationtester$(X) pg_isolation_regress$(X)
@@ -44,8 +45,14 @@ isolationtester$(X): $(OBJS) | submake-libpq submake-libpgport
distprep: specparse.c specscanner.c
-# specscanner is compiled as part of specparse
-specparse.o: specscanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+specparse.h: specparse.c
+ touch $@
+
+specparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+specparse.o specscanner.o: specparse.h
# specparse.c and specscanner.c are in the distribution tarball,
# so do not clean them here
@@ -55,7 +62,7 @@ clean distclean:
rm -rf $(pg_regress_clean_files)
maintainer-clean: distclean
- rm -f specparse.c specscanner.c
+ rm -f specparse.h specparse.c specscanner.c
installcheck: all
$(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule
diff --git a/src/test/isolation/specparse.y b/src/test/isolation/specparse.y
index eb368184b8..657285cc23 100644
--- a/src/test/isolation/specparse.y
+++ b/src/test/isolation/specparse.y
@@ -276,5 +276,3 @@ blocker:
;
%%
-
-#include "specscanner.c"
diff --git a/src/test/isolation/specscanner.l b/src/test/isolation/specscanner.l
index aa6e89268e..2dc292c21d 100644
--- a/src/test/isolation/specscanner.l
+++ b/src/test/isolation/specscanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* specscanner.l
@@ -9,7 +9,14 @@
*
*-------------------------------------------------------------------------
*/
+#include "postgres_fe.h"
+
+#include "isolationtester.h"
+#include "specparse.h"
+}
+
+%{
static int yyline = 1; /* line number for error reporting */
#define LITBUF_INIT 1024 /* initial size of litbuf */
@@ -75,7 +82,7 @@ teardown { return TEARDOWN; }
/* Plain identifiers */
{identifier} {
- yylval.str = pg_strdup(yytext);
+ spec_yylval.str = pg_strdup(yytext);
return(identifier);
}
@@ -87,13 +94,13 @@ teardown { return TEARDOWN; }
<qident>\"\" { addlitchar(yytext[0]); }
<qident>\" {
litbuf[litbufpos] = '\0';
- yylval.str = pg_strdup(litbuf);
+ spec_yylval.str = pg_strdup(litbuf);
BEGIN(INITIAL);
return(identifier);
}
<qident>. { addlitchar(yytext[0]); }
-<qident>\n { yyerror("unexpected newline in quoted identifier"); }
-<qident><<EOF>> { yyerror("unterminated quoted identifier"); }
+<qident>\n { spec_yyerror("unexpected newline in quoted identifier"); }
+<qident><<EOF>> { spec_yyerror("unterminated quoted identifier"); }
/* SQL blocks: { UPDATE ... } */
/* We trim leading/trailing whitespace, otherwise they're unprocessed */
@@ -104,7 +111,7 @@ teardown { return TEARDOWN; }
}
<sql>{space}*"}" {
litbuf[litbufpos] = '\0';
- yylval.str = pg_strdup(litbuf);
+ spec_yylval.str = pg_strdup(litbuf);
BEGIN(INITIAL);
return(sqlblock);
}
@@ -116,12 +123,12 @@ teardown { return TEARDOWN; }
addlitchar(yytext[0]);
}
<sql><<EOF>> {
- yyerror("unterminated sql block");
+ spec_yyerror("unterminated sql block");
}
/* Numbers and punctuation */
{digit}+ {
- yylval.integer = atoi(yytext);
+ spec_yylval.integer = atoi(yytext);
return INTEGER;
}
@@ -150,7 +157,7 @@ addlitchar(char c)
}
void
-yyerror(const char *message)
+spec_yyerror(const char *message)
{
fprintf(stderr, "%s at line %d\n", message, yyline);
exit(1);
--
2.36.1
[text/x-patch] v201-0004-Build-syncrep_scanner.c-standalone.patch (3.5K, ../../CAFBsxsGq=jLMofR3BN1S1xJcaJsYcQnkWFRtRDObYVtDST0+Vg@mail.gmail.com/5-v201-0004-Build-syncrep_scanner.c-standalone.patch)
download | inline diff:
From 92d48ac8354bce8bb57a43e01448835e1cd75871 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Fri, 12 Aug 2022 18:27:39 +0700
Subject: [PATCH v201 4/9] Build syncrep_scanner.c standalone
---
src/backend/replication/.gitignore | 1 +
src/backend/replication/Makefile | 11 +++++++++--
src/backend/replication/syncrep_gram.y | 2 --
src/backend/replication/syncrep_scanner.l | 17 +++++++++++------
4 files changed, 21 insertions(+), 10 deletions(-)
diff --git a/src/backend/replication/.gitignore b/src/backend/replication/.gitignore
index d1df6147bd..ad138c0c98 100644
--- a/src/backend/replication/.gitignore
+++ b/src/backend/replication/.gitignore
@@ -1,4 +1,5 @@
/repl_gram.c
/repl_scanner.c
+/syncrep_gram.h
/syncrep_gram.c
/syncrep_scanner.c
diff --git a/src/backend/replication/Makefile b/src/backend/replication/Makefile
index bc8170418f..23f29ba545 100644
--- a/src/backend/replication/Makefile
+++ b/src/backend/replication/Makefile
@@ -21,6 +21,7 @@ OBJS = \
slotfuncs.o \
syncrep.o \
syncrep_gram.o \
+ syncrep_scanner.o \
walreceiver.o \
walreceiverfuncs.o \
walsender.o
@@ -38,8 +39,14 @@ repl_gram.c: BISONFLAGS += -d
# Force these dependencies to be known even without dependency info built:
repl_gram.o repl_scanner.o: repl_gram.h
-# syncrep_scanner is compiled as part of syncrep_gram
-syncrep_gram.o: syncrep_scanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+syncrep_gram.h: syncrep_gram.c
+ touch $@
+
+syncrep_gram.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+syncrep_gram.o syncrep_scanner.o: syncrep_gram.h
# repl_gram.c, repl_scanner.c, syncrep_gram.c and syncrep_scanner.c
# are in the distribution tarball, so they are not cleaned here.
diff --git a/src/backend/replication/syncrep_gram.y b/src/backend/replication/syncrep_gram.y
index d932f2cda3..4fc3647da1 100644
--- a/src/backend/replication/syncrep_gram.y
+++ b/src/backend/replication/syncrep_gram.y
@@ -112,5 +112,3 @@ create_syncrep_config(const char *num_sync, List *members, uint8 syncrep_method)
return config;
}
-
-#include "syncrep_scanner.c"
diff --git a/src/backend/replication/syncrep_scanner.l b/src/backend/replication/syncrep_scanner.l
index 1952c8c6e0..8f38cb4613 100644
--- a/src/backend/replication/syncrep_scanner.l
+++ b/src/backend/replication/syncrep_scanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* syncrep_scanner.l
@@ -16,7 +16,12 @@
#include "postgres.h"
#include "lib/stringinfo.h"
+#include "replication/syncrep.h"
+
+#include "syncrep_gram.h"
+}
+%{
/* Avoid exit() on fatal scanner errors (a bit ugly -- see yy_fatal_error) */
#undef fprintf
#define fprintf(file, fmt, msg) fprintf_to_ereport(fmt, msg)
@@ -82,28 +87,28 @@ xdinside [^"]+
appendStringInfoString(&xdbuf, yytext);
}
<xd>{xdstop} {
- yylval.str = xdbuf.data;
+ syncrep_yylval.str = xdbuf.data;
xdbuf.data = NULL;
BEGIN(INITIAL);
return NAME;
}
<xd><<EOF>> {
- yyerror("unterminated quoted identifier");
+ syncrep_yyerror("unterminated quoted identifier");
return JUNK;
}
{identifier} {
- yylval.str = pstrdup(yytext);
+ syncrep_yylval.str = pstrdup(yytext);
return NAME;
}
{digit}+ {
- yylval.str = pstrdup(yytext);
+ syncrep_yylval.str = pstrdup(yytext);
return NUM;
}
"*" {
- yylval.str = "*";
+ syncrep_yylval.str = "*";
return NAME;
}
--
2.36.1
[text/x-patch] v201-0002-Build-booscanner.c-standalone.patch (5.8K, ../../CAFBsxsGq=jLMofR3BN1S1xJcaJsYcQnkWFRtRDObYVtDST0+Vg@mail.gmail.com/6-v201-0002-Build-booscanner.c-standalone.patch)
download | inline diff:
From 7d4ecfcb3e91f3b45e94b9e64c7c40f1bbd22aa8 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Fri, 12 Aug 2022 15:45:24 +0700
Subject: [PATCH v201 2/9] Build booscanner.c standalone
---
src/backend/bootstrap/.gitignore | 1 +
src/backend/bootstrap/Makefile | 11 +++++-
src/backend/bootstrap/bootparse.y | 2 -
src/backend/bootstrap/bootscanner.l | 57 +++++++++++++++--------------
4 files changed, 40 insertions(+), 31 deletions(-)
diff --git a/src/backend/bootstrap/.gitignore b/src/backend/bootstrap/.gitignore
index 1ffe8ca39e..6351b920fd 100644
--- a/src/backend/bootstrap/.gitignore
+++ b/src/backend/bootstrap/.gitignore
@@ -1,2 +1,3 @@
+/bootparse.h
/bootparse.c
/bootscanner.c
diff --git a/src/backend/bootstrap/Makefile b/src/backend/bootstrap/Makefile
index 6421efb227..c39eb7089c 100644
--- a/src/backend/bootstrap/Makefile
+++ b/src/backend/bootstrap/Makefile
@@ -14,12 +14,19 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
OBJS = \
bootparse.o \
+ bootscanner.o \
bootstrap.o
include $(top_srcdir)/src/backend/common.mk
-# bootscanner is compiled as part of bootparse
-bootparse.o: bootscanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+bootparse.h: bootparse.c
+ touch $@
+
+bootparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+bootparse.o bootscan.o: bootparse.h
# bootparse.c and bootscanner.c are in the distribution tarball, so
# they are not cleaned here.
diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y
index 7d7655d295..c45ddde67f 100644
--- a/src/backend/bootstrap/bootparse.y
+++ b/src/backend/bootstrap/bootparse.y
@@ -488,5 +488,3 @@ boot_ident:
| XNULL { $$ = pstrdup($1); }
;
%%
-
-#include "bootscanner.c"
diff --git a/src/backend/bootstrap/bootscanner.l b/src/backend/bootstrap/bootscanner.l
index 3094ccb93f..3f916fbb93 100644
--- a/src/backend/bootstrap/bootscanner.l
+++ b/src/backend/bootstrap/bootscanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* bootscanner.l
@@ -18,8 +18,11 @@
#include "bootstrap/bootstrap.h"
#include "utils/guc.h"
-/* Not needed now that this file is compiled as part of bootparse. */
-/* #include "bootparse.h" */
+/* XXX must be included after bootstrap.h */
+#include "bootparse.h"
+}
+
+%{
/* LCOV_EXCL_START */
@@ -52,7 +55,7 @@ id [-A-Za-z0-9_]+
sid \'([^']|\'\')*\'
/*
- * Keyword tokens return the keyword text (as a constant string) in yylval.kw,
+ * Keyword tokens return the keyword text (as a constant string) in boot_yylval.kw,
* just in case that's needed because we want to treat the keyword as an
* unreserved identifier. Note that _null_ is not treated as a keyword
* for this purpose; it's the one "reserved word" in the bootstrap syntax.
@@ -60,23 +63,23 @@ sid \'([^']|\'\')*\'
* Notice that all the keywords are case-sensitive, and for historical
* reasons some must be upper case.
*
- * String tokens return a palloc'd string in yylval.str.
+ * String tokens return a palloc'd string in boot_yylval.str.
*/
%%
-open { yylval.kw = "open"; return OPEN; }
+open { boot_yylval.kw = "open"; return OPEN; }
-close { yylval.kw = "close"; return XCLOSE; }
+close { boot_yylval.kw = "close"; return XCLOSE; }
-create { yylval.kw = "create"; return XCREATE; }
+create { boot_yylval.kw = "create"; return XCREATE; }
-OID { yylval.kw = "OID"; return OBJ_ID; }
-bootstrap { yylval.kw = "bootstrap"; return XBOOTSTRAP; }
-shared_relation { yylval.kw = "shared_relation"; return XSHARED_RELATION; }
-rowtype_oid { yylval.kw = "rowtype_oid"; return XROWTYPE_OID; }
+OID { boot_yylval.kw = "OID"; return OBJ_ID; }
+bootstrap { boot_yylval.kw = "bootstrap"; return XBOOTSTRAP; }
+shared_relation { boot_yylval.kw = "shared_relation"; return XSHARED_RELATION; }
+rowtype_oid { boot_yylval.kw = "rowtype_oid"; return XROWTYPE_OID; }
-insert { yylval.kw = "insert"; return INSERT_TUPLE; }
+insert { boot_yylval.kw = "insert"; return INSERT_TUPLE; }
_null_ { return NULLVAL; }
@@ -90,25 +93,25 @@ _null_ { return NULLVAL; }
^\#[^\n]* ; /* drop everything after "#" for comments */
-declare { yylval.kw = "declare"; return XDECLARE; }
-build { yylval.kw = "build"; return XBUILD; }
-indices { yylval.kw = "indices"; return INDICES; }
-unique { yylval.kw = "unique"; return UNIQUE; }
-index { yylval.kw = "index"; return INDEX; }
-on { yylval.kw = "on"; return ON; }
-using { yylval.kw = "using"; return USING; }
-toast { yylval.kw = "toast"; return XTOAST; }
-FORCE { yylval.kw = "FORCE"; return XFORCE; }
-NOT { yylval.kw = "NOT"; return XNOT; }
-NULL { yylval.kw = "NULL"; return XNULL; }
+declare { boot_yylval.kw = "declare"; return XDECLARE; }
+build { boot_yylval.kw = "build"; return XBUILD; }
+indices { boot_yylval.kw = "indices"; return INDICES; }
+unique { boot_yylval.kw = "unique"; return UNIQUE; }
+index { boot_yylval.kw = "index"; return INDEX; }
+on { boot_yylval.kw = "on"; return ON; }
+using { boot_yylval.kw = "using"; return USING; }
+toast { boot_yylval.kw = "toast"; return XTOAST; }
+FORCE { boot_yylval.kw = "FORCE"; return XFORCE; }
+NOT { boot_yylval.kw = "NOT"; return XNOT; }
+NULL { boot_yylval.kw = "NULL"; return XNULL; }
{id} {
- yylval.str = pstrdup(yytext);
+ boot_yylval.str = pstrdup(yytext);
return ID;
}
{sid} {
/* strip quotes and escapes */
- yylval.str = DeescapeQuotedString(yytext);
+ boot_yylval.str = DeescapeQuotedString(yytext);
return ID;
}
@@ -121,7 +124,7 @@ NULL { yylval.kw = "NULL"; return XNULL; }
/* LCOV_EXCL_STOP */
void
-yyerror(const char *message)
+boot_yyerror(const char *message)
{
elog(ERROR, "%s at line %d", message, yyline);
}
--
2.36.1
[text/x-patch] v201-0006-Build-cubescan.c-standalone.patch (5.1K, ../../CAFBsxsGq=jLMofR3BN1S1xJcaJsYcQnkWFRtRDObYVtDST0+Vg@mail.gmail.com/7-v201-0006-Build-cubescan.c-standalone.patch)
download | inline diff:
From 75168640fbd4e2a23f2765b2918cd6a0676d0f74 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 11:18:06 +0700
Subject: [PATCH v201 6/9] Build cubescan.c standalone
---
contrib/cube/.gitignore | 1 +
contrib/cube/Makefile | 16 ++++++++++------
contrib/cube/cubedata.h | 4 ++++
contrib/cube/cubeparse.y | 9 ++-------
contrib/cube/cubescan.l | 33 +++++++++++++++++++--------------
5 files changed, 36 insertions(+), 27 deletions(-)
diff --git a/contrib/cube/.gitignore b/contrib/cube/.gitignore
index cb4c989fff..f788440c79 100644
--- a/contrib/cube/.gitignore
+++ b/contrib/cube/.gitignore
@@ -1,3 +1,4 @@
+/cubeparse.h
/cubeparse.c
/cubescan.c
# Generated subdirectories
diff --git a/contrib/cube/Makefile b/contrib/cube/Makefile
index cf195506c7..4fd19aac35 100644
--- a/contrib/cube/Makefile
+++ b/contrib/cube/Makefile
@@ -4,7 +4,8 @@ MODULE_big = cube
OBJS = \
$(WIN32RES) \
cube.o \
- cubeparse.o
+ cubeparse.o \
+ cubescan.o
EXTENSION = cube
DATA = cube--1.2.sql cube--1.2--1.3.sql cube--1.3--1.4.sql cube--1.4--1.5.sql \
@@ -15,8 +16,6 @@ HEADERS = cubedata.h
REGRESS = cube cube_sci
-EXTRA_CLEAN = y.tab.c y.tab.h
-
SHLIB_LINK += $(filter -lm, $(LIBS))
ifdef USE_PGXS
@@ -30,11 +29,16 @@ include $(top_builddir)/src/Makefile.global
include $(top_srcdir)/contrib/contrib-global.mk
endif
+# See notes in src/backend/parser/Makefile about the following two rules
+cubeparse.h: cubeparse.c
+ touch $@
+
+cubeparse.c: BISONFLAGS += -d
-# cubescan is compiled as part of cubeparse
-cubeparse.o: cubescan.c
+# Force these dependencies to be known even without dependency info built:
+cubeparse.o cubescan.o: cubeparse.h
distprep: cubeparse.c cubescan.c
maintainer-clean:
- rm -f cubeparse.c cubescan.c
+ rm -f cubeparse.h cubeparse.c cubescan.c
diff --git a/contrib/cube/cubedata.h b/contrib/cube/cubedata.h
index dbe7d4f742..0b373048b5 100644
--- a/contrib/cube/cubedata.h
+++ b/contrib/cube/cubedata.h
@@ -67,3 +67,7 @@ extern void cube_scanner_finish(void);
/* in cubeparse.y */
extern int cube_yyparse(NDBOX **result);
+
+/* All grammar constructs return strings */
+#define YYSTYPE char *
+extern int scanbuflen;
diff --git a/contrib/cube/cubeparse.y b/contrib/cube/cubeparse.y
index 7577c4515c..d3fd1fb475 100644
--- a/contrib/cube/cubeparse.y
+++ b/contrib/cube/cubeparse.y
@@ -9,9 +9,6 @@
#include "cubedata.h"
#include "utils/float.h"
-/* All grammar constructs return strings */
-#define YYSTYPE char *
-
/*
* Bison doesn't allocate anything that needs to live across parser calls,
* so we can easily have it use palloc instead of malloc. This prevents
@@ -23,8 +20,8 @@
#define YYMALLOC palloc
#define YYFREE pfree
-static char *scanbuf;
-static int scanbuflen;
+// TODO: get rid of global var
+int scanbuflen;
static int item_count(const char *s, char delim);
static NDBOX *write_box(int dim, char *str1, char *str2);
@@ -265,5 +262,3 @@ write_point_as_box(int dim, char *str)
return bp;
}
-
-#include "cubescan.c"
diff --git a/contrib/cube/cubescan.l b/contrib/cube/cubescan.l
index bd400e3684..b7d35c6f78 100644
--- a/contrib/cube/cubescan.l
+++ b/contrib/cube/cubescan.l
@@ -1,9 +1,16 @@
-%{
+%top{
/*
* A scanner for EMP-style numeric ranges
* contrib/cube/cubescan.l
*/
+#include "postgres.h"
+#include "cubedata.h"
+
+#include "cubeparse.h"
+}
+
+%{
/* LCOV_EXCL_START */
/* No reason to constrain amount of data slurped */
@@ -21,9 +28,7 @@ fprintf_to_ereport(const char *fmt, const char *msg)
/* Handles to the buffer that the lexer uses internally */
static YY_BUFFER_STATE scanbufhandle;
-/* this is now declared in cubeparse.y: */
-/* static char *scanbuf; */
-/* static int scanbuflen; */
+static char *scanbuf;
%}
%option 8bit
@@ -45,14 +50,14 @@ NaN [nN][aA][nN]
%%
-{float} yylval = yytext; return CUBEFLOAT;
-{infinity} yylval = yytext; return CUBEFLOAT;
-{NaN} yylval = yytext; return CUBEFLOAT;
-\[ yylval = "("; return O_BRACKET;
-\] yylval = ")"; return C_BRACKET;
-\( yylval = "("; return O_PAREN;
-\) yylval = ")"; return C_PAREN;
-\, yylval = ","; return COMMA;
+{float} cube_yylval = yytext; return CUBEFLOAT;
+{infinity} cube_yylval = yytext; return CUBEFLOAT;
+{NaN} cube_yylval = yytext; return CUBEFLOAT;
+\[ cube_yylval = "("; return O_BRACKET;
+\] cube_yylval = ")"; return C_BRACKET;
+\( cube_yylval = "("; return O_PAREN;
+\) cube_yylval = ")"; return C_PAREN;
+\, cube_yylval = ","; return COMMA;
[ \t\n\r\f]+ /* discard spaces */
. return yytext[0]; /* alert parser of the garbage */
@@ -62,7 +67,7 @@ NaN [nN][aA][nN]
/* result is not used, but Bison expects this signature */
void
-yyerror(NDBOX **result, const char *message)
+cube_yyerror(NDBOX **result, const char *message)
{
if (*yytext == YY_END_OF_BUFFER_CHAR)
{
@@ -89,7 +94,7 @@ yyerror(NDBOX **result, const char *message)
void
cube_scanner_init(const char *str)
{
- Size slen = strlen(str);
+ Size slen = strlen(str);
/*
* Might be left over after ereport()
--
2.36.1
[text/x-patch] v201-0007-Build-segscan.c-standalone.patch (3.7K, ../../CAFBsxsGq=jLMofR3BN1S1xJcaJsYcQnkWFRtRDObYVtDST0+Vg@mail.gmail.com/8-v201-0007-Build-segscan.c-standalone.patch)
download | inline diff:
From 5170d8fac8bbb9e101e10f18ed280e446b305e26 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 12:00:33 +0700
Subject: [PATCH v201 7/9] Build segscan.c standalone
---
contrib/seg/.gitignore | 1 +
contrib/seg/Makefile | 15 +++++++++++----
contrib/seg/segparse.y | 3 ---
contrib/seg/segscan.l | 25 +++++++++++++++----------
4 files changed, 27 insertions(+), 17 deletions(-)
diff --git a/contrib/seg/.gitignore b/contrib/seg/.gitignore
index 69e73d2096..fa247a4e67 100644
--- a/contrib/seg/.gitignore
+++ b/contrib/seg/.gitignore
@@ -1,3 +1,4 @@
+/segparse.h
/segparse.c
/segscan.c
# Generated subdirectories
diff --git a/contrib/seg/Makefile b/contrib/seg/Makefile
index bb63e83506..c6c134b8f1 100644
--- a/contrib/seg/Makefile
+++ b/contrib/seg/Makefile
@@ -4,7 +4,8 @@ MODULE_big = seg
OBJS = \
$(WIN32RES) \
seg.o \
- segparse.o
+ segparse.o \
+ segscan.o
EXTENSION = seg
DATA = seg--1.1.sql seg--1.1--1.2.sql seg--1.2--1.3.sql seg--1.3--1.4.sql \
@@ -29,10 +30,16 @@ include $(top_srcdir)/contrib/contrib-global.mk
endif
-# segscan is compiled as part of segparse
-segparse.o: segscan.c
+# See notes in src/backend/parser/Makefile about the following two rules
+segparse.h: segparse.c
+ touch $@
+
+segparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+segparse.o segscan.o: segparse.h
distprep: segparse.c segscan.c
maintainer-clean:
- rm -f segparse.c segscan.c
+ rm -f segparse.h segparse.c segscan.c
diff --git a/contrib/seg/segparse.y b/contrib/seg/segparse.y
index 33e3a9f35f..637eacd1a6 100644
--- a/contrib/seg/segparse.y
+++ b/contrib/seg/segparse.y
@@ -160,6 +160,3 @@ seg_atof(const char *value)
datum = DirectFunctionCall1(float4in, CStringGetDatum(value));
return DatumGetFloat4(datum);
}
-
-
-#include "segscan.c"
diff --git a/contrib/seg/segscan.l b/contrib/seg/segscan.l
index 5f6595e9eb..db0db1aa70 100644
--- a/contrib/seg/segscan.l
+++ b/contrib/seg/segscan.l
@@ -1,8 +1,15 @@
-%{
+%top{
/*
* A scanner for EMP-style numeric ranges
*/
+#include "postgres.h"
+
+#include "segdata.h"
+#include "segparse.h"
+}
+
+%{
/* LCOV_EXCL_START */
/* No reason to constrain amount of data slurped */
@@ -21,7 +28,6 @@ fprintf_to_ereport(const char *fmt, const char *msg)
/* Handles to the buffer that the lexer uses internally */
static YY_BUFFER_STATE scanbufhandle;
static char *scanbuf;
-static int scanbuflen;
%}
%option 8bit
@@ -42,12 +48,12 @@ float ({integer}|{real})([eE]{integer})?
%%
-{range} yylval.text = yytext; return RANGE;
-{plumin} yylval.text = yytext; return PLUMIN;
-{float} yylval.text = yytext; return SEGFLOAT;
-\< yylval.text = "<"; return EXTENSION;
-\> yylval.text = ">"; return EXTENSION;
-\~ yylval.text = "~"; return EXTENSION;
+{range} seg_yylval.text = yytext; return RANGE;
+{plumin} seg_yylval.text = yytext; return PLUMIN;
+{float} seg_yylval.text = yytext; return SEGFLOAT;
+\< seg_yylval.text = "<"; return EXTENSION;
+\> seg_yylval.text = ">"; return EXTENSION;
+\~ seg_yylval.text = "~"; return EXTENSION;
[ \t\n\r\f]+ /* discard spaces */
. return yytext[0]; /* alert parser of the garbage */
@@ -56,7 +62,7 @@ float ({integer}|{real})([eE]{integer})?
/* LCOV_EXCL_STOP */
void
-yyerror(SEG *result, const char *message)
+seg_yyerror(SEG *result, const char *message)
{
if (*yytext == YY_END_OF_BUFFER_CHAR)
{
@@ -94,7 +100,6 @@ seg_scanner_init(const char *str)
/*
* Make a scan buffer with special termination needed by flex.
*/
- scanbuflen = slen;
scanbuf = palloc(slen + 2);
memcpy(scanbuf, str, slen);
scanbuf[slen] = scanbuf[slen + 1] = YY_END_OF_BUFFER_CHAR;
--
2.36.1
[text/x-patch] v201-0008-Build-jsonpath_scan.c-standalone.patch (6.6K, ../../CAFBsxsGq=jLMofR3BN1S1xJcaJsYcQnkWFRtRDObYVtDST0+Vg@mail.gmail.com/9-v201-0008-Build-jsonpath_scan.c-standalone.patch)
download | inline diff:
From 4d16b395978e8bc830e91d363cf9cadda0c00365 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 12:35:55 +0700
Subject: [PATCH v201 8/9] Build jsonpath_scan.c standalone
XXX: warnings about missing jsonpath_yylex
---
src/backend/utils/adt/.gitignore | 1 +
src/backend/utils/adt/Makefile | 11 +++++++++--
src/backend/utils/adt/jsonpath_gram.y | 23 -----------------------
src/backend/utils/adt/jsonpath_scan.l | 25 +++++++++++++++----------
src/include/utils/jsonpath.h | 13 +++++++++++++
5 files changed, 38 insertions(+), 35 deletions(-)
diff --git a/src/backend/utils/adt/.gitignore b/src/backend/utils/adt/.gitignore
index 48cf941a52..7fab054407 100644
--- a/src/backend/utils/adt/.gitignore
+++ b/src/backend/utils/adt/.gitignore
@@ -1,2 +1,3 @@
+/jsonpath_gram.h
/jsonpath_gram.c
/jsonpath_scan.c
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 7c722ea2ce..d03f897478 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -57,6 +57,7 @@ OBJS = \
jsonpath.o \
jsonpath_exec.o \
jsonpath_gram.o \
+ jsonpath_scan.o \
like.o \
like_support.o \
lockfuncs.o \
@@ -119,11 +120,17 @@ OBJS = \
xid8funcs.o \
xml.o
+# See notes in src/backend/parser/Makefile about the following two rules
+jsonpath_gram.h: jsonpath_gram.c
+ touch $@
+
+jsonpath_gram.c: BISONFLAGS += -d
+
jsonpath_scan.c: FLEXFLAGS = -CF -p -p
jsonpath_scan.c: FLEX_NO_BACKUP=yes
-# jsonpath_scan is compiled as part of jsonpath_gram
-jsonpath_gram.o: jsonpath_scan.c
+# Force these dependencies to be known even without dependency info built:
+jsonpath_gram.o jsonpath_gram.o jsonpath_parser.o: jsonpath_gram.h
# jsonpath_gram.c and jsonpath_scan.c are in the distribution tarball,
# so they are not cleaned here.
diff --git a/src/backend/utils/adt/jsonpath_gram.y b/src/backend/utils/adt/jsonpath_gram.y
index f903dba3e3..a7557d325e 100644
--- a/src/backend/utils/adt/jsonpath_gram.y
+++ b/src/backend/utils/adt/jsonpath_gram.y
@@ -24,21 +24,8 @@
#include "utils/builtins.h"
#include "utils/jsonpath.h"
-/* struct JsonPathString is shared between scan and gram */
-typedef struct JsonPathString
-{
- char *val;
- int len;
- int total;
-} JsonPathString;
-
union YYSTYPE;
-/* flex 2.5.4 doesn't bother with a decl for this */
-int jsonpath_yylex(union YYSTYPE *yylval_param);
-int jsonpath_yyparse(JsonPathParseResult **result);
-void jsonpath_yyerror(JsonPathParseResult **result, const char *message);
-
static JsonPathParseItem *makeItemType(JsonPathItemType type);
static JsonPathParseItem *makeItemString(JsonPathString *s);
static JsonPathParseItem *makeItemVariable(JsonPathString *s);
@@ -593,13 +580,3 @@ jspConvertRegexFlags(uint32 xflags)
return cflags;
}
-
-/*
- * jsonpath_scan.l is compiled as part of jsonpath_gram.y. Currently, this is
- * unavoidable because jsonpath_gram does not create a .h file to export its
- * token symbols. If these files ever grow large enough to be worth compiling
- * separately, that could be fixed; but for now it seems like useless
- * complication.
- */
-
-#include "jsonpath_scan.c"
diff --git a/src/backend/utils/adt/jsonpath_scan.l b/src/backend/utils/adt/jsonpath_scan.l
index 4351f6ec98..edd9d1c706 100644
--- a/src/backend/utils/adt/jsonpath_scan.l
+++ b/src/backend/utils/adt/jsonpath_scan.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* jsonpath_scan.l
@@ -17,9 +17,14 @@
#include "postgres.h"
+#include "utils/jsonpath.h"
#include "mb/pg_wchar.h"
#include "nodes/pg_list.h"
+#include "jsonpath_gram.h"
+}
+
+%{
static JsonPathString scanstring;
/* Handles to the buffer that the lexer uses internally */
@@ -142,9 +147,9 @@ hex_fail \\x{hex_dig}{0,1}
<xnq,xq,xvq>{hex_char} { parseHexChar(yytext); }
-<xnq,xq,xvq>{unicode}*{unicodefail} { yyerror(NULL, "invalid unicode sequence"); }
+<xnq,xq,xvq>{unicode}*{unicodefail} { jsonpath_yyerror(NULL, "invalid unicode sequence"); }
-<xnq,xq,xvq>{hex_fail} { yyerror(NULL, "invalid hex character sequence"); }
+<xnq,xq,xvq>{hex_fail} { jsonpath_yyerror(NULL, "invalid hex character sequence"); }
<xnq,xq,xvq>{unicode}+\\ {
/* throw back the \\, and treat as unicode */
@@ -154,9 +159,9 @@ hex_fail \\x{hex_dig}{0,1}
<xnq,xq,xvq>\\. { addchar(false, yytext[1]); }
-<xnq,xq,xvq>\\ { yyerror(NULL, "unexpected end after backslash"); }
+<xnq,xq,xvq>\\ { jsonpath_yyerror(NULL, "unexpected end after backslash"); }
-<xq,xvq><<EOF>> { yyerror(NULL, "unexpected end of quoted string"); }
+<xq,xvq><<EOF>> { jsonpath_yyerror(NULL, "unexpected end of quoted string"); }
<xq>\" {
yylval->str = scanstring;
@@ -178,7 +183,7 @@ hex_fail \\x{hex_dig}{0,1}
<xc>\* { }
-<xc><<EOF>> { yyerror(NULL, "unexpected end of comment"); }
+<xc><<EOF>> { jsonpath_yyerror(NULL, "unexpected end of comment"); }
\&\& { return AND_P; }
@@ -244,10 +249,10 @@ hex_fail \\x{hex_dig}{0,1}
return INT_P;
}
-{realfail} { yyerror(NULL, "invalid numeric literal"); }
-{integer_junk} { yyerror(NULL, "trailing junk after numeric literal"); }
-{decimal_junk} { yyerror(NULL, "trailing junk after numeric literal"); }
-{real_junk} { yyerror(NULL, "trailing junk after numeric literal"); }
+{realfail} { jsonpath_yyerror(NULL, "invalid numeric literal"); }
+{integer_junk} { jsonpath_yyerror(NULL, "trailing junk after numeric literal"); }
+{decimal_junk} { jsonpath_yyerror(NULL, "trailing junk after numeric literal"); }
+{real_junk} { jsonpath_yyerror(NULL, "trailing junk after numeric literal"); }
\" {
addchar(true, '\0');
diff --git a/src/include/utils/jsonpath.h b/src/include/utils/jsonpath.h
index 8e79b8dc9f..b9b56209b2 100644
--- a/src/include/utils/jsonpath.h
+++ b/src/include/utils/jsonpath.h
@@ -21,6 +21,14 @@
#include "utils/jsonb.h"
#include "utils/jsonfuncs.h"
+/* struct JsonPathString is shared between scan and gram */
+typedef struct JsonPathString
+{
+ char *val;
+ int len;
+ int total;
+} JsonPathString;
+
typedef struct
{
int32 vl_len_; /* varlena header (do not touch directly!) */
@@ -250,6 +258,11 @@ typedef struct JsonPathParseResult
extern JsonPathParseResult *parsejsonpath(const char *str, int len);
+/* flex 2.5.4 doesn't bother with a decl for this */
+//int jsonpath_yylex(union YYSTYPE *yylval_param);
+//int jsonpath_yyparse(JsonPathParseResult **result);
+void jsonpath_yyerror(JsonPathParseResult **result, const char *message);
+
extern int jspConvertRegexFlags(uint32 xflags);
/*
--
2.36.1
[text/x-patch] v201-0009-Build-exprscan.c-standalone.patch (3.2K, ../../CAFBsxsGq=jLMofR3BN1S1xJcaJsYcQnkWFRtRDObYVtDST0+Vg@mail.gmail.com/10-v201-0009-Build-exprscan.c-standalone.patch)
download | inline diff:
From cae4dccc5aaee93e0ed258448f60d4178163ec1c Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 13:35:14 +0700
Subject: [PATCH v201 9/9] Build exprscan.c standalone
---
src/bin/pgbench/.gitignore | 1 +
src/bin/pgbench/Makefile | 13 ++++++++++---
src/bin/pgbench/exprparse.y | 15 ---------------
src/bin/pgbench/exprscan.l | 9 ++++++++-
4 files changed, 19 insertions(+), 19 deletions(-)
diff --git a/src/bin/pgbench/.gitignore b/src/bin/pgbench/.gitignore
index 983a3cd7a6..07492a993c 100644
--- a/src/bin/pgbench/.gitignore
+++ b/src/bin/pgbench/.gitignore
@@ -1,3 +1,4 @@
+/exprparse.h
/exprparse.c
/exprscan.c
/pgbench
diff --git a/src/bin/pgbench/Makefile b/src/bin/pgbench/Makefile
index f402fe7b91..6647c9fe97 100644
--- a/src/bin/pgbench/Makefile
+++ b/src/bin/pgbench/Makefile
@@ -10,6 +10,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
$(WIN32RES) \
exprparse.o \
+ exprscan.o \
pgbench.o
override CPPFLAGS := -I. -I$(srcdir) -I$(libpq_srcdir) $(CPPFLAGS)
@@ -26,8 +27,14 @@ all: pgbench
pgbench: $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-# exprscan is compiled as part of exprparse
-exprparse.o: exprscan.c
+# See notes in src/backend/parser/Makefile about the following two rules
+exprparse.h: exprparse.c
+ touch $@
+
+exprparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+exprparse.o exprscan.o: exprparse.h
distprep: exprparse.c exprscan.c
@@ -45,7 +52,7 @@ clean distclean:
rm -rf tmp_check
maintainer-clean: distclean
- rm -f exprparse.c exprscan.c
+ rm -f exprparse.h exprparse.c exprscan.c
check:
$(prove_check)
diff --git a/src/bin/pgbench/exprparse.y b/src/bin/pgbench/exprparse.y
index b5592d4b97..ade2ecdaab 100644
--- a/src/bin/pgbench/exprparse.y
+++ b/src/bin/pgbench/exprparse.y
@@ -526,18 +526,3 @@ make_case(yyscan_t yyscanner, PgBenchExprList *when_then_list, PgBenchExpr *else
find_func(yyscanner, "!case_end"),
make_elist(else_part, when_then_list));
}
-
-/*
- * exprscan.l is compiled as part of exprparse.y. Currently, this is
- * unavoidable because exprparse does not create a .h file to export
- * its token symbols. If these files ever grow large enough to be
- * worth compiling separately, that could be fixed; but for now it
- * seems like useless complication.
- */
-
-/* First, get rid of "#define yyscan_t" from pgbench.h */
-#undef yyscan_t
-/* ... and the yylval macro, which flex will have its own definition for */
-#undef yylval
-
-#include "exprscan.c"
diff --git a/src/bin/pgbench/exprscan.l b/src/bin/pgbench/exprscan.l
index 4f63818606..6e9d949dcf 100644
--- a/src/bin/pgbench/exprscan.l
+++ b/src/bin/pgbench/exprscan.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* exprscan.l
@@ -23,8 +23,15 @@
*-------------------------------------------------------------------------
*/
+#include "postgres_fe.h"
+
#include "fe_utils/psqlscan_int.h"
+#include "pgbench.h"
+#include "exprparse.h"
+}
+
+%{
/* context information for reporting errors in expressions */
static const char *expr_source = NULL;
static int expr_lineno = 0;
--
2.36.1
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: build remaining Flex files standalone
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 04:07 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-12 06:01 ` build remaining Flex files standalone John Naylor <[email protected]>
2022-08-13 08:39 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
@ 2022-08-15 18:11 ` Andres Freund <[email protected]>
2022-08-16 10:41 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-08-15 18:11 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
Hi,
Thanks for your work on this!
On 2022-08-13 15:39:06 +0700, John Naylor wrote:
> Here are the rest. Most of it was pretty straightforward, with the
> main exception of jsonpath_scan.c, which is not quite finished. That
> one passes tests but still has one compiler warning. I'm unsure how
> much of what is there already is really necessary or was cargo-culted
> from elsewhere without explanation. For starters, I'm not sure why the
> grammar has a forward declaration of "union YYSTYPE". It's noteworthy
> that it used to compile standalone, but with a bit more stuff, and
> that was reverted in 550b9d26f80fa30. I can hack on it some more later
> but I ran out of steam today.
I'm not sure either...
> Other questions thus far:
>
> - "BISONFLAGS += -d" is now in every make file with a .y file -- can
> we just force that everywhere?
Hm. Not sure it's worth it, extensions might use our BISON stuff...
> - Include order seems to matter for the grammar's .h file. I didn't
> test if that was the case every time, and after a few miscompiles just
> always made it the last inclusion, but I'm wondering if we should keep
> those inclusions outside %top{} and put it at the start of the next
> %{} ?
I think we have a few of those dependencies already, see e.g.
/*
* NB: include gram.h only AFTER including scanner.h, because scanner.h
* is what #defines YYLTYPE.
*/
> From d723ba14acf56fd432e9e263db937fcc13fc0355 Mon Sep 17 00:00:00 2001
> From: John Naylor <[email protected]>
> Date: Thu, 11 Aug 2022 19:38:37 +0700
> Subject: [PATCH v201 1/9] Build guc-file.c standalone
Might be worth doing some of the moving around here separately from the
parser/scanner specific bits.
> +/* functions shared between guc.c and guc-file.l */
> +extern int guc_name_compare(const char *namea, const char *nameb);
> +extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
> + bool applySettings, int elevel);
> +extern void record_config_file_error(const char *errmsg,
> + const char *config_file,
> + int lineno,
> + ConfigVariable **head_p,
> + ConfigVariable **tail_p);
>
> /*
> * The following functions are not in guc.c, but are declared here to avoid
> --
> 2.36.1
>
I think I prefer your suggestion of a guc_internal.h upthread.
> From 7d4ecfcb3e91f3b45e94b9e64c7c40f1bbd22aa8 Mon Sep 17 00:00:00 2001
> From: John Naylor <[email protected]>
> Date: Fri, 12 Aug 2022 15:45:24 +0700
> Subject: [PATCH v201 2/9] Build booscanner.c standalone
> -# bootscanner is compiled as part of bootparse
> -bootparse.o: bootscanner.c
> +# See notes in src/backend/parser/Makefile about the following two rules
> +bootparse.h: bootparse.c
> + touch $@
> +
> +bootparse.c: BISONFLAGS += -d
> +
> +# Force these dependencies to be known even without dependency info built:
> +bootparse.o bootscan.o: bootparse.h
Wonder if we could / should wrap this is something common. It's somewhat
annoying to repeat this stuff everywhere.
> diff --git a/src/test/isolation/specscanner.l b/src/test/isolation/specscanner.l
> index aa6e89268e..2dc292c21d 100644
> --- a/src/test/isolation/specscanner.l
> +++ b/src/test/isolation/specscanner.l
> @@ -1,4 +1,4 @@
> -%{
> +%top{
> /*-------------------------------------------------------------------------
> *
> * specscanner.l
> @@ -9,7 +9,14 @@
> *
> *-------------------------------------------------------------------------
> */
> +#include "postgres_fe.h"
Miniscule nitpick: I think we typically leave an empty line between header and
first include.
> diff --git a/contrib/cube/cubedata.h b/contrib/cube/cubedata.h
> index dbe7d4f742..0b373048b5 100644
> --- a/contrib/cube/cubedata.h
> +++ b/contrib/cube/cubedata.h
> @@ -67,3 +67,7 @@ extern void cube_scanner_finish(void);
>
> /* in cubeparse.y */
> extern int cube_yyparse(NDBOX **result);
> +
> +/* All grammar constructs return strings */
> +#define YYSTYPE char *
Why does this need to be defined in a semi-public header? If we do this in
multiple files we'll end up with the danger of macro redefinition warnings.
> +extern int scanbuflen;
The code around scanbuflen seems pretty darn grotty. Allocating enough memory
for the entire list by allocating the entire string size... I don't know
anything about contrib/cube, but isn't that in effect O(inputlen^2) memory?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: build remaining Flex files standalone
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 04:07 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-12 06:01 ` build remaining Flex files standalone John Naylor <[email protected]>
2022-08-13 08:39 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-15 18:11 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
@ 2022-08-16 10:41 ` John Naylor <[email protected]>
2022-08-17 01:14 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: John Naylor @ 2022-08-16 10:41 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
For v3, I addressed some comments and added .h files to the
headerscheck exceptions.
On Tue, Aug 16, 2022 at 1:11 AM Andres Freund <[email protected]> wrote:
> On 2022-08-13 15:39:06 +0700, John Naylor wrote:
> > Here are the rest. Most of it was pretty straightforward, with the
> > main exception of jsonpath_scan.c, which is not quite finished. That
> > one passes tests but still has one compiler warning. I'm unsure how
> > much of what is there already is really necessary or was cargo-culted
> > from elsewhere without explanation. For starters, I'm not sure why the
> > grammar has a forward declaration of "union YYSTYPE". It's noteworthy
> > that it used to compile standalone, but with a bit more stuff, and
> > that was reverted in 550b9d26f80fa30. I can hack on it some more later
> > but I ran out of steam today.
I've got it in half-way decent shape now, with an *internal.h header
and some cleanups.
> > - Include order seems to matter for the grammar's .h file. I didn't
> > test if that was the case every time, and after a few miscompiles just
> > always made it the last inclusion, but I'm wondering if we should keep
> > those inclusions outside %top{} and put it at the start of the next
> > %{} ?
>
> I think we have a few of those dependencies already, see e.g.
> /*
> * NB: include gram.h only AFTER including scanner.h, because scanner.h
> * is what #defines YYLTYPE.
> */
Went with something like this in all cases:
/*
* NB: include bootparse.h only AFTER including bootstrap.h, because bootstrap.h
* includes node definitions needed for YYSTYPE.
*/
Future cleanup: I see this in headerscheck:
# We can't make these Bison output files compilable standalone
# without using "%code require", which old Bison versions lack.
# parser/gram.h will be included by parser/gramparse.h anyway.
That directive has been supported in Bison since 2.4.2.
> > From d723ba14acf56fd432e9e263db937fcc13fc0355 Mon Sep 17 00:00:00 2001
> > From: John Naylor <[email protected]>
> > Date: Thu, 11 Aug 2022 19:38:37 +0700
> > Subject: [PATCH v201 1/9] Build guc-file.c standalone
>
> Might be worth doing some of the moving around here separately from the
> parser/scanner specific bits.
Done in 0001/0003.
> > +/* functions shared between guc.c and guc-file.l */
> > [...]
> I think I prefer your suggestion of a guc_internal.h upthread.
Started in 0002, but left open the headerscheck failure.
Also, if such a thing is meant to be #include'd only by two generated
files, maybe it should just live in the directory where they live, and
not in the src/include dir?
> > From 7d4ecfcb3e91f3b45e94b9e64c7c40f1bbd22aa8 Mon Sep 17 00:00:00 2001
> > From: John Naylor <[email protected]>
> > Date: Fri, 12 Aug 2022 15:45:24 +0700
> > Subject: [PATCH v201 2/9] Build booscanner.c standalone
>
> > -# bootscanner is compiled as part of bootparse
> > -bootparse.o: bootscanner.c
> > +# See notes in src/backend/parser/Makefile about the following two rules
> > +bootparse.h: bootparse.c
> > + touch $@
> > +
> > +bootparse.c: BISONFLAGS += -d
> > +
> > +# Force these dependencies to be known even without dependency info built:
> > +bootparse.o bootscan.o: bootparse.h
>
> Wonder if we could / should wrap this is something common. It's somewhat
> annoying to repeat this stuff everywhere.
I haven't looked at the Meson effort recently, but if the build rule
is less annoying there, I'm inclined to leave this as a wart until
autotools are retired.
> > diff --git a/src/test/isolation/specscanner.l b/src/test/isolation/specscanner.l
> > index aa6e89268e..2dc292c21d 100644
> > --- a/src/test/isolation/specscanner.l
> > +++ b/src/test/isolation/specscanner.l
> > @@ -1,4 +1,4 @@
> > -%{
> > +%top{
> > /*-------------------------------------------------------------------------
> > *
> > * specscanner.l
> > @@ -9,7 +9,14 @@
> > *
> > *-------------------------------------------------------------------------
> > */
> > +#include "postgres_fe.h"
>
> Miniscule nitpick: I think we typically leave an empty line between header and
> first include.
In a small unscientific sample it seems like the opposite is true
actually, but I'll at least try to be consistent within the patch set.
> > diff --git a/contrib/cube/cubedata.h b/contrib/cube/cubedata.h
> > index dbe7d4f742..0b373048b5 100644
> > --- a/contrib/cube/cubedata.h
> > +++ b/contrib/cube/cubedata.h
> > @@ -67,3 +67,7 @@ extern void cube_scanner_finish(void);
> >
> > /* in cubeparse.y */
> > extern int cube_yyparse(NDBOX **result);
> > +
> > +/* All grammar constructs return strings */
> > +#define YYSTYPE char *
>
> Why does this need to be defined in a semi-public header? If we do this in
> multiple files we'll end up with the danger of macro redefinition warnings.
I tried to put all the Flex/Bison stuff in another *_internal header,
but that breaks the build. Putting just this one symbol in a header is
silly, but done that way for now. Maybe two copies of the symbol?
Another future cleanup: "%define api.prefix {cube_yy}" etc would cause
it to be spelled CUBE_YYSTYPE (other macros too), sidestepping this
problem (requires Bison 2.6). IIUC, doing it our way has been
deprecated for 9 years.
> > +extern int scanbuflen;
>
> The code around scanbuflen seems pretty darn grotty. Allocating enough memory
> for the entire list by allocating the entire string size... I don't know
> anything about contrib/cube, but isn't that in effect O(inputlen^2) memory?
Neither do I.
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[text/x-patch] v3-0001-Preparatory-refactoring-for-compiling-guc-file.c-.patch (26.0K, ../../CAFBsxsEospoUX=QYkfC=WcJqNB+iZtBf=BaRwn-zbHa48X0NKQ@mail.gmail.com/2-v3-0001-Preparatory-refactoring-for-compiling-guc-file.c-.patch)
download | inline diff:
From 6e780de69ba59e3d921c3115de920de1f98994cd Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 16 Aug 2022 10:42:19 +0700
Subject: [PATCH v3 01/11] Preparatory refactoring for compiling guc-file.c
standalone
Mostly this involves moving ProcessConfigFileInternal() to guc.c
and fixing the shared API to match.
---
src/backend/utils/misc/guc-file.l | 360 +-----------------------------
src/backend/utils/misc/guc.c | 360 +++++++++++++++++++++++++++++-
src/include/utils/guc.h | 9 +
3 files changed, 364 insertions(+), 365 deletions(-)
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index ce5633844c..b4fa09749b 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -48,12 +48,6 @@ static sigjmp_buf *GUC_flex_fatal_jmp;
static void FreeConfigVariable(ConfigVariable *item);
-static void record_config_file_error(const char *errmsg,
- const char *config_file,
- int lineno,
- ConfigVariable **head_p,
- ConfigVariable **tail_p);
-
static int GUC_flex_fatal(const char *msg);
/* LCOV_EXCL_START */
@@ -159,358 +153,6 @@ ProcessConfigFile(GucContext context)
MemoryContextDelete(config_cxt);
}
-/*
- * This function handles both actual config file (re)loads and execution of
- * show_all_file_settings() (i.e., the pg_file_settings view). In the latter
- * case we don't apply any of the settings, but we make all the usual validity
- * checks, and we return the ConfigVariable list so that it can be printed out
- * by show_all_file_settings().
- */
-static ConfigVariable *
-ProcessConfigFileInternal(GucContext context, bool applySettings, int elevel)
-{
- bool error = false;
- bool applying = false;
- const char *ConfFileWithError;
- ConfigVariable *item,
- *head,
- *tail;
- int i;
-
- /* Parse the main config file into a list of option names and values */
- ConfFileWithError = ConfigFileName;
- head = tail = NULL;
-
- if (!ParseConfigFile(ConfigFileName, true,
- NULL, 0, 0, elevel,
- &head, &tail))
- {
- /* Syntax error(s) detected in the file, so bail out */
- error = true;
- goto bail_out;
- }
-
- /*
- * Parse the PG_AUTOCONF_FILENAME file, if present, after the main file to
- * replace any parameters set by ALTER SYSTEM command. Because this file
- * is in the data directory, we can't read it until the DataDir has been
- * set.
- */
- if (DataDir)
- {
- if (!ParseConfigFile(PG_AUTOCONF_FILENAME, false,
- NULL, 0, 0, elevel,
- &head, &tail))
- {
- /* Syntax error(s) detected in the file, so bail out */
- error = true;
- ConfFileWithError = PG_AUTOCONF_FILENAME;
- goto bail_out;
- }
- }
- else
- {
- /*
- * If DataDir is not set, the PG_AUTOCONF_FILENAME file cannot be
- * read. In this case, we don't want to accept any settings but
- * data_directory from postgresql.conf, because they might be
- * overwritten with settings in the PG_AUTOCONF_FILENAME file which
- * will be read later. OTOH, since data_directory isn't allowed in the
- * PG_AUTOCONF_FILENAME file, it will never be overwritten later.
- */
- ConfigVariable *newlist = NULL;
-
- /*
- * Prune all items except the last "data_directory" from the list.
- */
- for (item = head; item; item = item->next)
- {
- if (!item->ignore &&
- strcmp(item->name, "data_directory") == 0)
- newlist = item;
- }
-
- if (newlist)
- newlist->next = NULL;
- head = tail = newlist;
-
- /*
- * Quick exit if data_directory is not present in file.
- *
- * We need not do any further processing, in particular we don't set
- * PgReloadTime; that will be set soon by subsequent full loading of
- * the config file.
- */
- if (head == NULL)
- goto bail_out;
- }
-
- /*
- * Mark all extant GUC variables as not present in the config file. We
- * need this so that we can tell below which ones have been removed from
- * the file since we last processed it.
- */
- for (i = 0; i < num_guc_variables; i++)
- {
- struct config_generic *gconf = guc_variables[i];
-
- gconf->status &= ~GUC_IS_IN_FILE;
- }
-
- /*
- * Check if all the supplied option names are valid, as an additional
- * quasi-syntactic check on the validity of the config file. It is
- * important that the postmaster and all backends agree on the results of
- * this phase, else we will have strange inconsistencies about which
- * processes accept a config file update and which don't. Hence, unknown
- * custom variable names have to be accepted without complaint. For the
- * same reason, we don't attempt to validate the options' values here.
- *
- * In addition, the GUC_IS_IN_FILE flag is set on each existing GUC
- * variable mentioned in the file; and we detect duplicate entries in the
- * file and mark the earlier occurrences as ignorable.
- */
- for (item = head; item; item = item->next)
- {
- struct config_generic *record;
-
- /* Ignore anything already marked as ignorable */
- if (item->ignore)
- continue;
-
- /*
- * Try to find the variable; but do not create a custom placeholder if
- * it's not there already.
- */
- record = find_option(item->name, false, true, elevel);
-
- if (record)
- {
- /* If it's already marked, then this is a duplicate entry */
- if (record->status & GUC_IS_IN_FILE)
- {
- /*
- * Mark the earlier occurrence(s) as dead/ignorable. We could
- * avoid the O(N^2) behavior here with some additional state,
- * but it seems unlikely to be worth the trouble.
- */
- ConfigVariable *pitem;
-
- for (pitem = head; pitem != item; pitem = pitem->next)
- {
- if (!pitem->ignore &&
- strcmp(pitem->name, item->name) == 0)
- pitem->ignore = true;
- }
- }
- /* Now mark it as present in file */
- record->status |= GUC_IS_IN_FILE;
- }
- else if (!valid_custom_variable_name(item->name))
- {
- /* Invalid non-custom variable, so complain */
- ereport(elevel,
- (errcode(ERRCODE_UNDEFINED_OBJECT),
- errmsg("unrecognized configuration parameter \"%s\" in file \"%s\" line %d",
- item->name,
- item->filename, item->sourceline)));
- item->errmsg = pstrdup("unrecognized configuration parameter");
- error = true;
- ConfFileWithError = item->filename;
- }
- }
-
- /*
- * If we've detected any errors so far, we don't want to risk applying any
- * changes.
- */
- if (error)
- goto bail_out;
-
- /* Otherwise, set flag that we're beginning to apply changes */
- applying = true;
-
- /*
- * Check for variables having been removed from the config file, and
- * revert their reset values (and perhaps also effective values) to the
- * boot-time defaults. If such a variable can't be changed after startup,
- * report that and continue.
- */
- for (i = 0; i < num_guc_variables; i++)
- {
- struct config_generic *gconf = guc_variables[i];
- GucStack *stack;
-
- if (gconf->reset_source != PGC_S_FILE ||
- (gconf->status & GUC_IS_IN_FILE))
- continue;
- if (gconf->context < PGC_SIGHUP)
- {
- /* The removal can't be effective without a restart */
- gconf->status |= GUC_PENDING_RESTART;
- ereport(elevel,
- (errcode(ERRCODE_CANT_CHANGE_RUNTIME_PARAM),
- errmsg("parameter \"%s\" cannot be changed without restarting the server",
- gconf->name)));
- record_config_file_error(psprintf("parameter \"%s\" cannot be changed without restarting the server",
- gconf->name),
- NULL, 0,
- &head, &tail);
- error = true;
- continue;
- }
-
- /* No more to do if we're just doing show_all_file_settings() */
- if (!applySettings)
- continue;
-
- /*
- * Reset any "file" sources to "default", else set_config_option will
- * not override those settings.
- */
- if (gconf->reset_source == PGC_S_FILE)
- gconf->reset_source = PGC_S_DEFAULT;
- if (gconf->source == PGC_S_FILE)
- gconf->source = PGC_S_DEFAULT;
- for (stack = gconf->stack; stack; stack = stack->prev)
- {
- if (stack->source == PGC_S_FILE)
- stack->source = PGC_S_DEFAULT;
- }
-
- /* Now we can re-apply the wired-in default (i.e., the boot_val) */
- if (set_config_option(gconf->name, NULL,
- context, PGC_S_DEFAULT,
- GUC_ACTION_SET, true, 0, false) > 0)
- {
- /* Log the change if appropriate */
- if (context == PGC_SIGHUP)
- ereport(elevel,
- (errmsg("parameter \"%s\" removed from configuration file, reset to default",
- gconf->name)));
- }
- }
-
- /*
- * Restore any variables determined by environment variables or
- * dynamically-computed defaults. This is a no-op except in the case
- * where one of these had been in the config file and is now removed.
- *
- * In particular, we *must not* do this during the postmaster's initial
- * loading of the file, since the timezone functions in particular should
- * be run only after initialization is complete.
- *
- * XXX this is an unmaintainable crock, because we have to know how to set
- * (or at least what to call to set) every non-PGC_INTERNAL variable that
- * could potentially have PGC_S_DYNAMIC_DEFAULT or PGC_S_ENV_VAR source.
- */
- if (context == PGC_SIGHUP && applySettings)
- {
- InitializeGUCOptionsFromEnvironment();
- pg_timezone_abbrev_initialize();
- /* this selects SQL_ASCII in processes not connected to a database */
- SetConfigOption("client_encoding", GetDatabaseEncodingName(),
- PGC_BACKEND, PGC_S_DYNAMIC_DEFAULT);
- }
-
- /*
- * Now apply the values from the config file.
- */
- for (item = head; item; item = item->next)
- {
- char *pre_value = NULL;
- int scres;
-
- /* Ignore anything marked as ignorable */
- if (item->ignore)
- continue;
-
- /* In SIGHUP cases in the postmaster, we want to report changes */
- if (context == PGC_SIGHUP && applySettings && !IsUnderPostmaster)
- {
- const char *preval = GetConfigOption(item->name, true, false);
-
- /* If option doesn't exist yet or is NULL, treat as empty string */
- if (!preval)
- preval = "";
- /* must dup, else might have dangling pointer below */
- pre_value = pstrdup(preval);
- }
-
- scres = set_config_option(item->name, item->value,
- context, PGC_S_FILE,
- GUC_ACTION_SET, applySettings, 0, false);
- if (scres > 0)
- {
- /* variable was updated, so log the change if appropriate */
- if (pre_value)
- {
- const char *post_value = GetConfigOption(item->name, true, false);
-
- if (!post_value)
- post_value = "";
- if (strcmp(pre_value, post_value) != 0)
- ereport(elevel,
- (errmsg("parameter \"%s\" changed to \"%s\"",
- item->name, item->value)));
- }
- item->applied = true;
- }
- else if (scres == 0)
- {
- error = true;
- item->errmsg = pstrdup("setting could not be applied");
- ConfFileWithError = item->filename;
- }
- else
- {
- /* no error, but variable's active value was not changed */
- item->applied = true;
- }
-
- /*
- * We should update source location unless there was an error, since
- * even if the active value didn't change, the reset value might have.
- * (In the postmaster, there won't be a difference, but it does matter
- * in backends.)
- */
- if (scres != 0 && applySettings)
- set_config_sourcefile(item->name, item->filename,
- item->sourceline);
-
- if (pre_value)
- pfree(pre_value);
- }
-
- /* Remember when we last successfully loaded the config file. */
- if (applySettings)
- PgReloadTime = GetCurrentTimestamp();
-
-bail_out:
- if (error && applySettings)
- {
- /* During postmaster startup, any error is fatal */
- if (context == PGC_POSTMASTER)
- ereport(ERROR,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors",
- ConfFileWithError)));
- else if (applying)
- ereport(elevel,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
- ConfFileWithError)));
- else
- ereport(elevel,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors; no changes were applied",
- ConfFileWithError)));
- }
-
- /* Successful or otherwise, return the collected data list */
- return head;
-}
-
/*
* Given a configuration file or directory location that may be a relative
* path, return an absolute one. We consider the location to be relative to
@@ -659,7 +301,7 @@ cleanup:
* Capture an error message in the ConfigVariable list returned by
* config file parsing.
*/
-static void
+void
record_config_file_error(const char *errmsg,
const char *config_file,
int lineno,
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 9fbbfb1be5..66ab3912a0 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -243,10 +243,6 @@ static void assign_recovery_target_lsn(const char *newval, void *extra);
static bool check_primary_slot_name(char **newval, void **extra, GucSource source);
static bool check_default_with_oids(bool *newval, void **extra, GucSource source);
-/* Private functions in guc-file.l that need to be called from guc.c */
-static ConfigVariable *ProcessConfigFileInternal(GucContext context,
- bool applySettings, int elevel);
-
/*
* Track whether there were any deferred checks for custom resource managers
* specified in wal_consistency_checking.
@@ -5160,8 +5156,8 @@ static bool report_needed; /* true if any GUC_REPORT reports are needed */
static int GUCNestLevel = 0; /* 1 when in main transaction */
+static struct config_generic *find_option(const char *name, bool create_placeholders, bool skip_errors, int elevel);
static int guc_var_compare(const void *a, const void *b);
-static int guc_name_compare(const char *namea, const char *nameb);
static void InitializeGUCOptionsFromEnvironment(void);
static void InitializeOneGUCOption(struct config_generic *gconf);
static void push_old_value(struct config_generic *gconf, GucAction action);
@@ -5180,7 +5176,359 @@ static bool validate_option_array_item(const char *name, const char *value,
static void write_auto_conf_file(int fd, const char *filename, ConfigVariable *head_p);
static void replace_auto_config_value(ConfigVariable **head_p, ConfigVariable **tail_p,
const char *name, const char *value);
+static bool valid_custom_variable_name(const char *name);
+
+/*
+ * This function handles both actual config file (re)loads and execution of
+ * show_all_file_settings() (i.e., the pg_file_settings view). In the latter
+ * case we don't apply any of the settings, but we make all the usual validity
+ * checks, and we return the ConfigVariable list so that it can be printed out
+ * by show_all_file_settings().
+ */
+ConfigVariable *
+ProcessConfigFileInternal(GucContext context, bool applySettings, int elevel)
+{
+ bool error = false;
+ bool applying = false;
+ const char *ConfFileWithError;
+ ConfigVariable *item,
+ *head,
+ *tail;
+ int i;
+
+ /* Parse the main config file into a list of option names and values */
+ ConfFileWithError = ConfigFileName;
+ head = tail = NULL;
+
+ if (!ParseConfigFile(ConfigFileName, true,
+ NULL, 0, 0, elevel,
+ &head, &tail))
+ {
+ /* Syntax error(s) detected in the file, so bail out */
+ error = true;
+ goto bail_out;
+ }
+
+ /*
+ * Parse the PG_AUTOCONF_FILENAME file, if present, after the main file to
+ * replace any parameters set by ALTER SYSTEM command. Because this file
+ * is in the data directory, we can't read it until the DataDir has been
+ * set.
+ */
+ if (DataDir)
+ {
+ if (!ParseConfigFile(PG_AUTOCONF_FILENAME, false,
+ NULL, 0, 0, elevel,
+ &head, &tail))
+ {
+ /* Syntax error(s) detected in the file, so bail out */
+ error = true;
+ ConfFileWithError = PG_AUTOCONF_FILENAME;
+ goto bail_out;
+ }
+ }
+ else
+ {
+ /*
+ * If DataDir is not set, the PG_AUTOCONF_FILENAME file cannot be
+ * read. In this case, we don't want to accept any settings but
+ * data_directory from postgresql.conf, because they might be
+ * overwritten with settings in the PG_AUTOCONF_FILENAME file which
+ * will be read later. OTOH, since data_directory isn't allowed in the
+ * PG_AUTOCONF_FILENAME file, it will never be overwritten later.
+ */
+ ConfigVariable *newlist = NULL;
+
+ /*
+ * Prune all items except the last "data_directory" from the list.
+ */
+ for (item = head; item; item = item->next)
+ {
+ if (!item->ignore &&
+ strcmp(item->name, "data_directory") == 0)
+ newlist = item;
+ }
+ if (newlist)
+ newlist->next = NULL;
+ head = tail = newlist;
+
+ /*
+ * Quick exit if data_directory is not present in file.
+ *
+ * We need not do any further processing, in particular we don't set
+ * PgReloadTime; that will be set soon by subsequent full loading of
+ * the config file.
+ */
+ if (head == NULL)
+ goto bail_out;
+ }
+
+ /*
+ * Mark all extant GUC variables as not present in the config file. We
+ * need this so that we can tell below which ones have been removed from
+ * the file since we last processed it.
+ */
+ for (i = 0; i < num_guc_variables; i++)
+ {
+ struct config_generic *gconf = guc_variables[i];
+
+ gconf->status &= ~GUC_IS_IN_FILE;
+ }
+
+ /*
+ * Check if all the supplied option names are valid, as an additional
+ * quasi-syntactic check on the validity of the config file. It is
+ * important that the postmaster and all backends agree on the results of
+ * this phase, else we will have strange inconsistencies about which
+ * processes accept a config file update and which don't. Hence, unknown
+ * custom variable names have to be accepted without complaint. For the
+ * same reason, we don't attempt to validate the options' values here.
+ *
+ * In addition, the GUC_IS_IN_FILE flag is set on each existing GUC
+ * variable mentioned in the file; and we detect duplicate entries in the
+ * file and mark the earlier occurrences as ignorable.
+ */
+ for (item = head; item; item = item->next)
+ {
+ struct config_generic *record;
+
+ /* Ignore anything already marked as ignorable */
+ if (item->ignore)
+ continue;
+
+ /*
+ * Try to find the variable; but do not create a custom placeholder if
+ * it's not there already.
+ */
+ record = find_option(item->name, false, true, elevel);
+
+ if (record)
+ {
+ /* If it's already marked, then this is a duplicate entry */
+ if (record->status & GUC_IS_IN_FILE)
+ {
+ /*
+ * Mark the earlier occurrence(s) as dead/ignorable. We could
+ * avoid the O(N^2) behavior here with some additional state,
+ * but it seems unlikely to be worth the trouble.
+ */
+ ConfigVariable *pitem;
+
+ for (pitem = head; pitem != item; pitem = pitem->next)
+ {
+ if (!pitem->ignore &&
+ strcmp(pitem->name, item->name) == 0)
+ pitem->ignore = true;
+ }
+ }
+ /* Now mark it as present in file */
+ record->status |= GUC_IS_IN_FILE;
+ }
+ else if (!valid_custom_variable_name(item->name))
+ {
+ /* Invalid non-custom variable, so complain */
+ ereport(elevel,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("unrecognized configuration parameter \"%s\" in file \"%s\" line %d",
+ item->name,
+ item->filename, item->sourceline)));
+ item->errmsg = pstrdup("unrecognized configuration parameter");
+ error = true;
+ ConfFileWithError = item->filename;
+ }
+ }
+
+ /*
+ * If we've detected any errors so far, we don't want to risk applying any
+ * changes.
+ */
+ if (error)
+ goto bail_out;
+
+ /* Otherwise, set flag that we're beginning to apply changes */
+ applying = true;
+
+ /*
+ * Check for variables having been removed from the config file, and
+ * revert their reset values (and perhaps also effective values) to the
+ * boot-time defaults. If such a variable can't be changed after startup,
+ * report that and continue.
+ */
+ for (i = 0; i < num_guc_variables; i++)
+ {
+ struct config_generic *gconf = guc_variables[i];
+ GucStack *stack;
+
+ if (gconf->reset_source != PGC_S_FILE ||
+ (gconf->status & GUC_IS_IN_FILE))
+ continue;
+ if (gconf->context < PGC_SIGHUP)
+ {
+ /* The removal can't be effective without a restart */
+ gconf->status |= GUC_PENDING_RESTART;
+ ereport(elevel,
+ (errcode(ERRCODE_CANT_CHANGE_RUNTIME_PARAM),
+ errmsg("parameter \"%s\" cannot be changed without restarting the server",
+ gconf->name)));
+ record_config_file_error(psprintf("parameter \"%s\" cannot be changed without restarting the server",
+ gconf->name),
+ NULL, 0,
+ &head, &tail);
+ error = true;
+ continue;
+ }
+
+ /* No more to do if we're just doing show_all_file_settings() */
+ if (!applySettings)
+ continue;
+
+ /*
+ * Reset any "file" sources to "default", else set_config_option will
+ * not override those settings.
+ */
+ if (gconf->reset_source == PGC_S_FILE)
+ gconf->reset_source = PGC_S_DEFAULT;
+ if (gconf->source == PGC_S_FILE)
+ gconf->source = PGC_S_DEFAULT;
+ for (stack = gconf->stack; stack; stack = stack->prev)
+ {
+ if (stack->source == PGC_S_FILE)
+ stack->source = PGC_S_DEFAULT;
+ }
+
+ /* Now we can re-apply the wired-in default (i.e., the boot_val) */
+ if (set_config_option(gconf->name, NULL,
+ context, PGC_S_DEFAULT,
+ GUC_ACTION_SET, true, 0, false) > 0)
+ {
+ /* Log the change if appropriate */
+ if (context == PGC_SIGHUP)
+ ereport(elevel,
+ (errmsg("parameter \"%s\" removed from configuration file, reset to default",
+ gconf->name)));
+ }
+ }
+
+ /*
+ * Restore any variables determined by environment variables or
+ * dynamically-computed defaults. This is a no-op except in the case
+ * where one of these had been in the config file and is now removed.
+ *
+ * In particular, we *must not* do this during the postmaster's initial
+ * loading of the file, since the timezone functions in particular should
+ * be run only after initialization is complete.
+ *
+ * XXX this is an unmaintainable crock, because we have to know how to set
+ * (or at least what to call to set) every non-PGC_INTERNAL variable that
+ * could potentially have PGC_S_DYNAMIC_DEFAULT or PGC_S_ENV_VAR source.
+ */
+ if (context == PGC_SIGHUP && applySettings)
+ {
+ InitializeGUCOptionsFromEnvironment();
+ pg_timezone_abbrev_initialize();
+ /* this selects SQL_ASCII in processes not connected to a database */
+ SetConfigOption("client_encoding", GetDatabaseEncodingName(),
+ PGC_BACKEND, PGC_S_DYNAMIC_DEFAULT);
+ }
+
+ /*
+ * Now apply the values from the config file.
+ */
+ for (item = head; item; item = item->next)
+ {
+ char *pre_value = NULL;
+ int scres;
+
+ /* Ignore anything marked as ignorable */
+ if (item->ignore)
+ continue;
+
+ /* In SIGHUP cases in the postmaster, we want to report changes */
+ if (context == PGC_SIGHUP && applySettings && !IsUnderPostmaster)
+ {
+ const char *preval = GetConfigOption(item->name, true, false);
+
+ /* If option doesn't exist yet or is NULL, treat as empty string */
+ if (!preval)
+ preval = "";
+ /* must dup, else might have dangling pointer below */
+ pre_value = pstrdup(preval);
+ }
+
+ scres = set_config_option(item->name, item->value,
+ context, PGC_S_FILE,
+ GUC_ACTION_SET, applySettings, 0, false);
+ if (scres > 0)
+ {
+ /* variable was updated, so log the change if appropriate */
+ if (pre_value)
+ {
+ const char *post_value = GetConfigOption(item->name, true, false);
+
+ if (!post_value)
+ post_value = "";
+ if (strcmp(pre_value, post_value) != 0)
+ ereport(elevel,
+ (errmsg("parameter \"%s\" changed to \"%s\"",
+ item->name, item->value)));
+ }
+ item->applied = true;
+ }
+ else if (scres == 0)
+ {
+ error = true;
+ item->errmsg = pstrdup("setting could not be applied");
+ ConfFileWithError = item->filename;
+ }
+ else
+ {
+ /* no error, but variable's active value was not changed */
+ item->applied = true;
+ }
+
+ /*
+ * We should update source location unless there was an error, since
+ * even if the active value didn't change, the reset value might have.
+ * (In the postmaster, there won't be a difference, but it does matter
+ * in backends.)
+ */
+ if (scres != 0 && applySettings)
+ set_config_sourcefile(item->name, item->filename,
+ item->sourceline);
+
+ if (pre_value)
+ pfree(pre_value);
+ }
+
+ /* Remember when we last successfully loaded the config file. */
+ if (applySettings)
+ PgReloadTime = GetCurrentTimestamp();
+
+bail_out:
+ if (error && applySettings)
+ {
+ /* During postmaster startup, any error is fatal */
+ if (context == PGC_POSTMASTER)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors",
+ ConfFileWithError)));
+ else if (applying)
+ ereport(elevel,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
+ ConfFileWithError)));
+ else
+ ereport(elevel,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors; no changes were applied",
+ ConfFileWithError)));
+ }
+
+ /* Successful or otherwise, return the collected data list */
+ return head;
+}
/*
* Some infrastructure for checking malloc/strdup/realloc calls
@@ -5737,7 +6085,7 @@ guc_var_compare(const void *a, const void *b)
/*
* the bare comparison function for GUC names
*/
-static int
+int
guc_name_compare(const char *namea, const char *nameb)
{
/*
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index e734493a48..aae071cd82 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -442,6 +442,15 @@ extern void GUC_check_errcode(int sqlerrcode);
pre_format_elog_string(errno, TEXTDOMAIN), \
GUC_check_errhint_string = format_elog_string
+/* functions shared between guc.c and guc-file.l */
+extern int guc_name_compare(const char *namea, const char *nameb);
+extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
+ bool applySettings, int elevel);
+extern void record_config_file_error(const char *errmsg,
+ const char *config_file,
+ int lineno,
+ ConfigVariable **head_p,
+ ConfigVariable **tail_p);
/*
* The following functions are not in guc.c, but are declared here to avoid
--
2.36.1
[text/x-patch] v3-0005-Build-repl_scanner.c-standalone.patch (6.1K, ../../CAFBsxsEospoUX=QYkfC=WcJqNB+iZtBf=BaRwn-zbHa48X0NKQ@mail.gmail.com/3-v3-0005-Build-repl_scanner.c-standalone.patch)
download | inline diff:
From 3d6fc71eaf615bee5a8789f3fdec9cb361b711ff Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Fri, 12 Aug 2022 17:09:45 +0700
Subject: [PATCH v3 05/11] Build repl_scanner.c standalone
---
src/backend/Makefile | 3 ++-
src/backend/replication/.gitignore | 1 +
src/backend/replication/Makefile | 11 +++++++--
src/backend/replication/repl_gram.y | 2 --
src/backend/replication/repl_scanner.l | 31 +++++++++++++++++---------
src/tools/pginclude/headerscheck | 1 +
6 files changed, 33 insertions(+), 16 deletions(-)
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 5a12666918..f527659a7b 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -183,7 +183,7 @@ distprep:
$(MAKE) -C bootstrap bootparse.c bootparse.h bootscanner.c
$(MAKE) -C catalog distprep
$(MAKE) -C nodes distprep
- $(MAKE) -C replication repl_gram.c repl_scanner.c syncrep_gram.c syncrep_scanner.c
+ $(MAKE) -C replication repl_gram.c repl_gram.h repl_scanner.c syncrep_gram.c syncrep_scanner.c
$(MAKE) -C storage/lmgr lwlocknames.h lwlocknames.c
$(MAKE) -C utils distprep
$(MAKE) -C utils/adt jsonpath_gram.c jsonpath_scan.c
@@ -304,6 +304,7 @@ maintainer-clean: distclean
parser/gram.h \
parser/scan.c \
replication/repl_gram.c \
+ replication/repl_gram.h \
replication/repl_scanner.c \
replication/syncrep_gram.c \
replication/syncrep_scanner.c \
diff --git a/src/backend/replication/.gitignore b/src/backend/replication/.gitignore
index d1df6147bd..a5f600232f 100644
--- a/src/backend/replication/.gitignore
+++ b/src/backend/replication/.gitignore
@@ -1,3 +1,4 @@
+/repl_gram.h
/repl_gram.c
/repl_scanner.c
/syncrep_gram.c
diff --git a/src/backend/replication/Makefile b/src/backend/replication/Makefile
index 2bffac58c0..bc8170418f 100644
--- a/src/backend/replication/Makefile
+++ b/src/backend/replication/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
OBJS = \
repl_gram.o \
+ repl_scanner.o \
slot.o \
slotfuncs.o \
syncrep.o \
@@ -28,8 +29,14 @@ SUBDIRS = logical
include $(top_srcdir)/src/backend/common.mk
-# repl_scanner is compiled as part of repl_gram
-repl_gram.o: repl_scanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+repl_gram.h: repl_gram.c
+ touch $@
+
+repl_gram.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+repl_gram.o repl_scanner.o: repl_gram.h
# syncrep_scanner is compiled as part of syncrep_gram
syncrep_gram.o: syncrep_scanner.c
diff --git a/src/backend/replication/repl_gram.y b/src/backend/replication/repl_gram.y
index 4cf087e602..b343f108d3 100644
--- a/src/backend/replication/repl_gram.y
+++ b/src/backend/replication/repl_gram.y
@@ -416,5 +416,3 @@ ident_or_keyword:
;
%%
-
-#include "repl_scanner.c"
diff --git a/src/backend/replication/repl_scanner.l b/src/backend/replication/repl_scanner.l
index 586f0d3a5c..23fcb2a11d 100644
--- a/src/backend/replication/repl_scanner.l
+++ b/src/backend/replication/repl_scanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* repl_scanner.l
@@ -18,6 +18,15 @@
#include "utils/builtins.h"
#include "parser/scansup.h"
+/*
+ * NB: include repl_gram.h only AFTER including walsender_private.h, because
+ * walsender_private includes headers that define XLogRecPtr.
+ */
+#include "replication/walsender_private.h"
+#include "repl_gram.h"
+}
+
+%{
/* Avoid exit() on fatal scanner errors (a bit ugly -- see yy_fatal_error) */
#undef fprintf
#define fprintf(file, fmt, msg) fprintf_to_ereport(fmt, msg)
@@ -130,7 +139,7 @@ WAIT { return K_WAIT; }
{space}+ { /* do nothing */ }
{digit}+ {
- yylval.uintval = strtoul(yytext, NULL, 10);
+ replication_yylval.uintval = strtoul(yytext, NULL, 10);
return UCONST;
}
@@ -138,8 +147,8 @@ WAIT { return K_WAIT; }
uint32 hi,
lo;
if (sscanf(yytext, "%X/%X", &hi, &lo) != 2)
- yyerror("invalid streaming start location");
- yylval.recptr = ((uint64) hi) << 32 | lo;
+ replication_yyerror("invalid streaming start location");
+ replication_yylval.recptr = ((uint64) hi) << 32 | lo;
return RECPTR;
}
@@ -151,7 +160,7 @@ WAIT { return K_WAIT; }
<xq>{quotestop} {
yyless(1);
BEGIN(INITIAL);
- yylval.str = litbufdup();
+ replication_yylval.str = litbufdup();
return SCONST;
}
@@ -173,9 +182,9 @@ WAIT { return K_WAIT; }
yyless(1);
BEGIN(INITIAL);
- yylval.str = litbufdup();
- len = strlen(yylval.str);
- truncate_identifier(yylval.str, len, true);
+ replication_yylval.str = litbufdup();
+ len = strlen(replication_yylval.str);
+ truncate_identifier(replication_yylval.str, len, true);
return IDENT;
}
@@ -186,7 +195,7 @@ WAIT { return K_WAIT; }
{identifier} {
int len = strlen(yytext);
- yylval.str = downcase_truncate_identifier(yytext, len, true);
+ replication_yylval.str = downcase_truncate_identifier(yytext, len, true);
return IDENT;
}
@@ -195,7 +204,7 @@ WAIT { return K_WAIT; }
return yytext[0];
}
-<xq,xd><<EOF>> { yyerror("unterminated quoted string"); }
+<xq,xd><<EOF>> { replication_yyerror("unterminated quoted string"); }
<<EOF>> {
@@ -231,7 +240,7 @@ addlitchar(unsigned char ychar)
}
void
-yyerror(const char *message)
+replication_yyerror(const char *message)
{
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 96a95cad9b..c102adf434 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -118,6 +118,7 @@ do
test "$f" = src/include/parser/gram.h && continue
test "$f" = src/backend/parser/gram.h && continue
test "$f" = src/backend/bootstrap/bootparse.h && continue
+ test "$f" = src/backend/replication/repl_gram.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
test "$f" = src/interfaces/ecpg/preproc/preproc.h && continue
--
2.36.1
[text/x-patch] v3-0002-Move-private-declarations-shared-between-guc.c-an.patch (3.2K, ../../CAFBsxsEospoUX=QYkfC=WcJqNB+iZtBf=BaRwn-zbHa48X0NKQ@mail.gmail.com/4-v3-0002-Move-private-declarations-shared-between-guc.c-an.patch)
download | inline diff:
From 0d33e92fcd15b57977d8eb9d021996912e69ab81 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 16 Aug 2022 12:01:41 +0700
Subject: [PATCH v3 02/11] Move private declarations shared between guc.c and
guc-file.l to new header
FIXME: fails headerscheck
---
src/backend/utils/misc/guc-file.l | 1 +
src/backend/utils/misc/guc.c | 1 +
src/include/utils/guc.h | 10 ----------
src/include/utils/guc_internal.h | 24 ++++++++++++++++++++++++
4 files changed, 26 insertions(+), 10 deletions(-)
create mode 100644 src/include/utils/guc_internal.h
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index b4fa09749b..843838b1df 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/fd.h"
#include "utils/guc.h"
+#include "utils/guc_internal.h"
/*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 66ab3912a0..293834fc13 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -100,6 +100,7 @@
#include "utils/builtins.h"
#include "utils/bytea.h"
#include "utils/float.h"
+#include "utils/guc_internal.h"
#include "utils/guc_tables.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index aae071cd82..45ae1b537f 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -442,16 +442,6 @@ extern void GUC_check_errcode(int sqlerrcode);
pre_format_elog_string(errno, TEXTDOMAIN), \
GUC_check_errhint_string = format_elog_string
-/* functions shared between guc.c and guc-file.l */
-extern int guc_name_compare(const char *namea, const char *nameb);
-extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
- bool applySettings, int elevel);
-extern void record_config_file_error(const char *errmsg,
- const char *config_file,
- int lineno,
- ConfigVariable **head_p,
- ConfigVariable **tail_p);
-
/*
* The following functions are not in guc.c, but are declared here to avoid
* having to include guc.h in some widely used headers that it really doesn't
diff --git a/src/include/utils/guc_internal.h b/src/include/utils/guc_internal.h
new file mode 100644
index 0000000000..5d5db6bdce
--- /dev/null
+++ b/src/include/utils/guc_internal.h
@@ -0,0 +1,24 @@
+/*--------------------------------------------------------------------
+ * guc_internals.h
+ *
+ * Declarations shared between backend/utils/misc/guc.c and
+ * backend/utils/misc/guc-file.l
+ *
+ * Copyright (c) 2000-2022, PostgreSQL Global Development Group
+ *
+ * src/include/utils/guc_internals.h
+ *--------------------------------------------------------------------
+ */
+#ifndef GUC_INTERNALS_H
+#define GUC_INTERNALS_H
+
+extern int guc_name_compare(const char *namea, const char *nameb);
+extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
+ bool applySettings, int elevel);
+extern void record_config_file_error(const char *errmsg,
+ const char *config_file,
+ int lineno,
+ ConfigVariable **head_p,
+ ConfigVariable **tail_p);
+
+#endif /* GUC_INTERNALS_H */
--
2.36.1
[text/x-patch] v3-0004-Build-bootscanner.c-standalone.patch (7.4K, ../../CAFBsxsEospoUX=QYkfC=WcJqNB+iZtBf=BaRwn-zbHa48X0NKQ@mail.gmail.com/5-v3-0004-Build-bootscanner.c-standalone.patch)
download | inline diff:
From ba67b6b8e5c26631d89d4a188a6b06c24bcc5f39 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Fri, 12 Aug 2022 15:45:24 +0700
Subject: [PATCH v3 04/11] Build bootscanner.c standalone
---
src/backend/Makefile | 3 +-
src/backend/bootstrap/.gitignore | 1 +
src/backend/bootstrap/Makefile | 11 +++++-
src/backend/bootstrap/bootparse.y | 2 -
src/backend/bootstrap/bootscanner.l | 60 ++++++++++++++++-------------
src/tools/pginclude/headerscheck | 1 +
6 files changed, 46 insertions(+), 32 deletions(-)
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 3f01c65592..5a12666918 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -180,7 +180,7 @@ utils/probes.o: utils/probes.d $(SUBDIROBJS)
# Be sure that these files get removed by the maintainer-clean target
distprep:
$(MAKE) -C parser gram.c gram.h scan.c
- $(MAKE) -C bootstrap bootparse.c bootscanner.c
+ $(MAKE) -C bootstrap bootparse.c bootparse.h bootscanner.c
$(MAKE) -C catalog distprep
$(MAKE) -C nodes distprep
$(MAKE) -C replication repl_gram.c repl_scanner.c syncrep_gram.c syncrep_scanner.c
@@ -298,6 +298,7 @@ maintainer-clean: distclean
$(MAKE) -C nodes $@
$(MAKE) -C utils $@
rm -f bootstrap/bootparse.c \
+ bootstrap/bootparse.h \
bootstrap/bootscanner.c \
parser/gram.c \
parser/gram.h \
diff --git a/src/backend/bootstrap/.gitignore b/src/backend/bootstrap/.gitignore
index 1ffe8ca39e..6351b920fd 100644
--- a/src/backend/bootstrap/.gitignore
+++ b/src/backend/bootstrap/.gitignore
@@ -1,2 +1,3 @@
+/bootparse.h
/bootparse.c
/bootscanner.c
diff --git a/src/backend/bootstrap/Makefile b/src/backend/bootstrap/Makefile
index 6421efb227..606c8021e7 100644
--- a/src/backend/bootstrap/Makefile
+++ b/src/backend/bootstrap/Makefile
@@ -14,12 +14,19 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
OBJS = \
bootparse.o \
+ bootscanner.o \
bootstrap.o
include $(top_srcdir)/src/backend/common.mk
-# bootscanner is compiled as part of bootparse
-bootparse.o: bootscanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+bootparse.h: bootparse.c
+ touch $@
+
+bootparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+bootparse.o bootscanner.o: bootparse.h
# bootparse.c and bootscanner.c are in the distribution tarball, so
# they are not cleaned here.
diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y
index 7d7655d295..c45ddde67f 100644
--- a/src/backend/bootstrap/bootparse.y
+++ b/src/backend/bootstrap/bootparse.y
@@ -488,5 +488,3 @@ boot_ident:
| XNULL { $$ = pstrdup($1); }
;
%%
-
-#include "bootscanner.c"
diff --git a/src/backend/bootstrap/bootscanner.l b/src/backend/bootstrap/bootscanner.l
index 3094ccb93f..d6eae84816 100644
--- a/src/backend/bootstrap/bootscanner.l
+++ b/src/backend/bootstrap/bootscanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* bootscanner.l
@@ -15,11 +15,17 @@
*/
#include "postgres.h"
+/*
+ * NB: include bootparse.h only AFTER including bootstrap.h, because bootstrap.h
+ * includes node definitions needed for YYSTYPE.
+ */
#include "bootstrap/bootstrap.h"
+#include "bootparse.h"
#include "utils/guc.h"
-/* Not needed now that this file is compiled as part of bootparse. */
-/* #include "bootparse.h" */
+}
+
+%{
/* LCOV_EXCL_START */
@@ -52,7 +58,7 @@ id [-A-Za-z0-9_]+
sid \'([^']|\'\')*\'
/*
- * Keyword tokens return the keyword text (as a constant string) in yylval.kw,
+ * Keyword tokens return the keyword text (as a constant string) in boot_yylval.kw,
* just in case that's needed because we want to treat the keyword as an
* unreserved identifier. Note that _null_ is not treated as a keyword
* for this purpose; it's the one "reserved word" in the bootstrap syntax.
@@ -60,23 +66,23 @@ sid \'([^']|\'\')*\'
* Notice that all the keywords are case-sensitive, and for historical
* reasons some must be upper case.
*
- * String tokens return a palloc'd string in yylval.str.
+ * String tokens return a palloc'd string in boot_yylval.str.
*/
%%
-open { yylval.kw = "open"; return OPEN; }
+open { boot_yylval.kw = "open"; return OPEN; }
-close { yylval.kw = "close"; return XCLOSE; }
+close { boot_yylval.kw = "close"; return XCLOSE; }
-create { yylval.kw = "create"; return XCREATE; }
+create { boot_yylval.kw = "create"; return XCREATE; }
-OID { yylval.kw = "OID"; return OBJ_ID; }
-bootstrap { yylval.kw = "bootstrap"; return XBOOTSTRAP; }
-shared_relation { yylval.kw = "shared_relation"; return XSHARED_RELATION; }
-rowtype_oid { yylval.kw = "rowtype_oid"; return XROWTYPE_OID; }
+OID { boot_yylval.kw = "OID"; return OBJ_ID; }
+bootstrap { boot_yylval.kw = "bootstrap"; return XBOOTSTRAP; }
+shared_relation { boot_yylval.kw = "shared_relation"; return XSHARED_RELATION; }
+rowtype_oid { boot_yylval.kw = "rowtype_oid"; return XROWTYPE_OID; }
-insert { yylval.kw = "insert"; return INSERT_TUPLE; }
+insert { boot_yylval.kw = "insert"; return INSERT_TUPLE; }
_null_ { return NULLVAL; }
@@ -90,25 +96,25 @@ _null_ { return NULLVAL; }
^\#[^\n]* ; /* drop everything after "#" for comments */
-declare { yylval.kw = "declare"; return XDECLARE; }
-build { yylval.kw = "build"; return XBUILD; }
-indices { yylval.kw = "indices"; return INDICES; }
-unique { yylval.kw = "unique"; return UNIQUE; }
-index { yylval.kw = "index"; return INDEX; }
-on { yylval.kw = "on"; return ON; }
-using { yylval.kw = "using"; return USING; }
-toast { yylval.kw = "toast"; return XTOAST; }
-FORCE { yylval.kw = "FORCE"; return XFORCE; }
-NOT { yylval.kw = "NOT"; return XNOT; }
-NULL { yylval.kw = "NULL"; return XNULL; }
+declare { boot_yylval.kw = "declare"; return XDECLARE; }
+build { boot_yylval.kw = "build"; return XBUILD; }
+indices { boot_yylval.kw = "indices"; return INDICES; }
+unique { boot_yylval.kw = "unique"; return UNIQUE; }
+index { boot_yylval.kw = "index"; return INDEX; }
+on { boot_yylval.kw = "on"; return ON; }
+using { boot_yylval.kw = "using"; return USING; }
+toast { boot_yylval.kw = "toast"; return XTOAST; }
+FORCE { boot_yylval.kw = "FORCE"; return XFORCE; }
+NOT { boot_yylval.kw = "NOT"; return XNOT; }
+NULL { boot_yylval.kw = "NULL"; return XNULL; }
{id} {
- yylval.str = pstrdup(yytext);
+ boot_yylval.str = pstrdup(yytext);
return ID;
}
{sid} {
/* strip quotes and escapes */
- yylval.str = DeescapeQuotedString(yytext);
+ boot_yylval.str = DeescapeQuotedString(yytext);
return ID;
}
@@ -121,7 +127,7 @@ NULL { yylval.kw = "NULL"; return XNULL; }
/* LCOV_EXCL_STOP */
void
-yyerror(const char *message)
+boot_yyerror(const char *message)
{
elog(ERROR, "%s at line %d", message, yyline);
}
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 3f8640a03d..96a95cad9b 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -117,6 +117,7 @@ do
# parser/gram.h will be included by parser/gramparse.h anyway.
test "$f" = src/include/parser/gram.h && continue
test "$f" = src/backend/parser/gram.h && continue
+ test "$f" = src/backend/bootstrap/bootparse.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
test "$f" = src/interfaces/ecpg/preproc/preproc.h && continue
--
2.36.1
[text/x-patch] v3-0003-Build-guc-file.c-standalone.patch (2.5K, ../../CAFBsxsEospoUX=QYkfC=WcJqNB+iZtBf=BaRwn-zbHa48X0NKQ@mail.gmail.com/6-v3-0003-Build-guc-file.c-standalone.patch)
download | inline diff:
From 68becb3064c8c573212033339735b3e335eb3af6 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Thu, 11 Aug 2022 19:38:37 +0700
Subject: [PATCH v3 03/11] Build guc-file.c standalone
The proposed Meson build system will need a way to ignore certain
generated files in order to coexist with the autoconf build system,
and #include'd C files generated by Flex make this more difficult.
Build guc-file.c separately from guc.c, as was done in 72b1e3a21.
Reviewed by Andres Freund
Discussion: https://www.postgresql.org/message-id/20220810171935.7k5zgnjwqzalzmtm%40awork3.anarazel.de
---
src/backend/utils/misc/Makefile | 5 +----
src/backend/utils/misc/guc-file.l | 9 +++++----
src/backend/utils/misc/guc.c | 2 --
3 files changed, 6 insertions(+), 10 deletions(-)
diff --git a/src/backend/utils/misc/Makefile b/src/backend/utils/misc/Makefile
index 1d5327cf64..cf7ce9bc83 100644
--- a/src/backend/utils/misc/Makefile
+++ b/src/backend/utils/misc/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
OBJS = \
guc.o \
+ guc-file.o \
help_config.o \
pg_config.o \
pg_controldata.o \
@@ -37,10 +38,6 @@ endif
include $(top_srcdir)/src/backend/common.mk
-# guc-file is compiled as part of guc
-guc.o: guc-file.c
-
# Note: guc-file.c is not deleted by 'make clean',
# since we want to ship it in distribution tarballs.
clean:
- @rm -f lex.yy.c
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index 843838b1df..9aa3abe1ba 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -1,4 +1,4 @@
-/* -*-pgsql-c-*- */
+%top{
/*
* Scanner for the configuration file
*
@@ -7,8 +7,6 @@
* src/backend/utils/misc/guc-file.l
*/
-%{
-
#include "postgres.h"
#include <ctype.h>
@@ -17,10 +15,13 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "storage/fd.h"
+#include <sys/stat.h>
#include "utils/guc.h"
#include "utils/guc_internal.h"
+#include "utils/memutils.h"
+}
-
+%{
/*
* flex emits a yy_fatal_error() function that it calls in response to
* critical errors like malloc failure, file I/O errors, and detection of
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 293834fc13..92b5b18c8f 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -13333,5 +13333,3 @@ check_default_with_oids(bool *newval, void **extra, GucSource source)
return true;
}
-
-#include "guc-file.c"
--
2.36.1
[text/x-patch] v3-0007-Build-specscanner.c-standalone.patch (5.3K, ../../CAFBsxsEospoUX=QYkfC=WcJqNB+iZtBf=BaRwn-zbHa48X0NKQ@mail.gmail.com/7-v3-0007-Build-specscanner.c-standalone.patch)
download | inline diff:
From 829925b21f94ed70c33882490e6fe3351f16697e Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 09:34:17 +0700
Subject: [PATCH v3 07/11] Build specscanner.c standalone
---
src/test/isolation/.gitignore | 1 +
src/test/isolation/Makefile | 15 +++++++++++----
src/test/isolation/specparse.y | 2 --
src/test/isolation/specscanner.l | 28 +++++++++++++++++++---------
src/tools/pginclude/headerscheck | 1 +
5 files changed, 32 insertions(+), 15 deletions(-)
diff --git a/src/test/isolation/.gitignore b/src/test/isolation/.gitignore
index 870dac4d28..2c13b4bf98 100644
--- a/src/test/isolation/.gitignore
+++ b/src/test/isolation/.gitignore
@@ -3,6 +3,7 @@
/pg_isolation_regress
# Local generated source files
+/specparse.h
/specparse.c
/specscanner.c
diff --git a/src/test/isolation/Makefile b/src/test/isolation/Makefile
index 0d452c89d4..b8738b7c1b 100644
--- a/src/test/isolation/Makefile
+++ b/src/test/isolation/Makefile
@@ -15,7 +15,8 @@ override CPPFLAGS := -I. -I$(srcdir) -I$(libpq_srcdir) \
OBJS = \
$(WIN32RES) \
isolationtester.o \
- specparse.o
+ specparse.o \
+ specscanner.o
all: isolationtester$(X) pg_isolation_regress$(X)
@@ -44,8 +45,14 @@ isolationtester$(X): $(OBJS) | submake-libpq submake-libpgport
distprep: specparse.c specscanner.c
-# specscanner is compiled as part of specparse
-specparse.o: specscanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+specparse.h: specparse.c
+ touch $@
+
+specparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+specparse.o specscanner.o: specparse.h
# specparse.c and specscanner.c are in the distribution tarball,
# so do not clean them here
@@ -55,7 +62,7 @@ clean distclean:
rm -rf $(pg_regress_clean_files)
maintainer-clean: distclean
- rm -f specparse.c specscanner.c
+ rm -f specparse.h specparse.c specscanner.c
installcheck: all
$(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule
diff --git a/src/test/isolation/specparse.y b/src/test/isolation/specparse.y
index eb368184b8..657285cc23 100644
--- a/src/test/isolation/specparse.y
+++ b/src/test/isolation/specparse.y
@@ -276,5 +276,3 @@ blocker:
;
%%
-
-#include "specscanner.c"
diff --git a/src/test/isolation/specscanner.l b/src/test/isolation/specscanner.l
index aa6e89268e..b04696f52d 100644
--- a/src/test/isolation/specscanner.l
+++ b/src/test/isolation/specscanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* specscanner.l
@@ -9,7 +9,17 @@
*
*-------------------------------------------------------------------------
*/
+#include "postgres_fe.h"
+/*
+ * NB: include specparse.h only AFTER including isolationtester.h, because
+ * isolationtester.h includes node definitions needed for YYSTYPE.
+ */
+#include "isolationtester.h"
+#include "specparse.h"
+}
+
+%{
static int yyline = 1; /* line number for error reporting */
#define LITBUF_INIT 1024 /* initial size of litbuf */
@@ -75,7 +85,7 @@ teardown { return TEARDOWN; }
/* Plain identifiers */
{identifier} {
- yylval.str = pg_strdup(yytext);
+ spec_yylval.str = pg_strdup(yytext);
return(identifier);
}
@@ -87,13 +97,13 @@ teardown { return TEARDOWN; }
<qident>\"\" { addlitchar(yytext[0]); }
<qident>\" {
litbuf[litbufpos] = '\0';
- yylval.str = pg_strdup(litbuf);
+ spec_yylval.str = pg_strdup(litbuf);
BEGIN(INITIAL);
return(identifier);
}
<qident>. { addlitchar(yytext[0]); }
-<qident>\n { yyerror("unexpected newline in quoted identifier"); }
-<qident><<EOF>> { yyerror("unterminated quoted identifier"); }
+<qident>\n { spec_yyerror("unexpected newline in quoted identifier"); }
+<qident><<EOF>> { spec_yyerror("unterminated quoted identifier"); }
/* SQL blocks: { UPDATE ... } */
/* We trim leading/trailing whitespace, otherwise they're unprocessed */
@@ -104,7 +114,7 @@ teardown { return TEARDOWN; }
}
<sql>{space}*"}" {
litbuf[litbufpos] = '\0';
- yylval.str = pg_strdup(litbuf);
+ spec_yylval.str = pg_strdup(litbuf);
BEGIN(INITIAL);
return(sqlblock);
}
@@ -116,12 +126,12 @@ teardown { return TEARDOWN; }
addlitchar(yytext[0]);
}
<sql><<EOF>> {
- yyerror("unterminated sql block");
+ spec_yyerror("unterminated sql block");
}
/* Numbers and punctuation */
{digit}+ {
- yylval.integer = atoi(yytext);
+ spec_yylval.integer = atoi(yytext);
return INTEGER;
}
@@ -150,7 +160,7 @@ addlitchar(char c)
}
void
-yyerror(const char *message)
+spec_yyerror(const char *message)
{
fprintf(stderr, "%s at line %d\n", message, yyline);
exit(1);
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 77b697e293..cf75e93943 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -120,6 +120,7 @@ do
test "$f" = src/backend/bootstrap/bootparse.h && continue
test "$f" = src/backend/replication/repl_gram.h && continue
test "$f" = src/backend/replication/syncrep_gram.h && continue
+ test "$f" = src/test/isolation/specparse.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
test "$f" = src/interfaces/ecpg/preproc/preproc.h && continue
--
2.36.1
[text/x-patch] v3-0008-Build-exprscan.c-standalone.patch (4.1K, ../../CAFBsxsEospoUX=QYkfC=WcJqNB+iZtBf=BaRwn-zbHa48X0NKQ@mail.gmail.com/8-v3-0008-Build-exprscan.c-standalone.patch)
download | inline diff:
From 4b9237ae205a4eb817124d0347c985de7eb194e0 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 13:35:14 +0700
Subject: [PATCH v3 08/11] Build exprscan.c standalone
---
src/bin/pgbench/.gitignore | 1 +
src/bin/pgbench/Makefile | 13 ++++++++++---
src/bin/pgbench/exprparse.y | 15 ---------------
src/bin/pgbench/exprscan.l | 12 +++++++++++-
src/tools/pginclude/headerscheck | 1 +
5 files changed, 23 insertions(+), 19 deletions(-)
diff --git a/src/bin/pgbench/.gitignore b/src/bin/pgbench/.gitignore
index 983a3cd7a6..07492a993c 100644
--- a/src/bin/pgbench/.gitignore
+++ b/src/bin/pgbench/.gitignore
@@ -1,3 +1,4 @@
+/exprparse.h
/exprparse.c
/exprscan.c
/pgbench
diff --git a/src/bin/pgbench/Makefile b/src/bin/pgbench/Makefile
index f402fe7b91..6647c9fe97 100644
--- a/src/bin/pgbench/Makefile
+++ b/src/bin/pgbench/Makefile
@@ -10,6 +10,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
$(WIN32RES) \
exprparse.o \
+ exprscan.o \
pgbench.o
override CPPFLAGS := -I. -I$(srcdir) -I$(libpq_srcdir) $(CPPFLAGS)
@@ -26,8 +27,14 @@ all: pgbench
pgbench: $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-# exprscan is compiled as part of exprparse
-exprparse.o: exprscan.c
+# See notes in src/backend/parser/Makefile about the following two rules
+exprparse.h: exprparse.c
+ touch $@
+
+exprparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+exprparse.o exprscan.o: exprparse.h
distprep: exprparse.c exprscan.c
@@ -45,7 +52,7 @@ clean distclean:
rm -rf tmp_check
maintainer-clean: distclean
- rm -f exprparse.c exprscan.c
+ rm -f exprparse.h exprparse.c exprscan.c
check:
$(prove_check)
diff --git a/src/bin/pgbench/exprparse.y b/src/bin/pgbench/exprparse.y
index b5592d4b97..ade2ecdaab 100644
--- a/src/bin/pgbench/exprparse.y
+++ b/src/bin/pgbench/exprparse.y
@@ -526,18 +526,3 @@ make_case(yyscan_t yyscanner, PgBenchExprList *when_then_list, PgBenchExpr *else
find_func(yyscanner, "!case_end"),
make_elist(else_part, when_then_list));
}
-
-/*
- * exprscan.l is compiled as part of exprparse.y. Currently, this is
- * unavoidable because exprparse does not create a .h file to export
- * its token symbols. If these files ever grow large enough to be
- * worth compiling separately, that could be fixed; but for now it
- * seems like useless complication.
- */
-
-/* First, get rid of "#define yyscan_t" from pgbench.h */
-#undef yyscan_t
-/* ... and the yylval macro, which flex will have its own definition for */
-#undef yylval
-
-#include "exprscan.c"
diff --git a/src/bin/pgbench/exprscan.l b/src/bin/pgbench/exprscan.l
index 4f63818606..fe8e32838a 100644
--- a/src/bin/pgbench/exprscan.l
+++ b/src/bin/pgbench/exprscan.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* exprscan.l
@@ -22,9 +22,19 @@
*
*-------------------------------------------------------------------------
*/
+#include "postgres_fe.h"
+/*
+ * NB: include exprparse.h only AFTER including pgbench.h, because pgbench.h
+ * contains definitions needed for YYSTYPE. Likewise, pgbench.h must come after
+ * psqlscan_int.h for yyscan_t.
+ */
#include "fe_utils/psqlscan_int.h"
+#include "pgbench.h"
+#include "exprparse.h"
+}
+%{
/* context information for reporting errors in expressions */
static const char *expr_source = NULL;
static int expr_lineno = 0;
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index cf75e93943..e2c4ba9ae4 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -121,6 +121,7 @@ do
test "$f" = src/backend/replication/repl_gram.h && continue
test "$f" = src/backend/replication/syncrep_gram.h && continue
test "$f" = src/test/isolation/specparse.h && continue
+ test "$f" = src/bin/pgbench/exprparse.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
test "$f" = src/interfaces/ecpg/preproc/preproc.h && continue
--
2.36.1
[text/x-patch] v3-0006-Build-syncrep_scanner.c-standalone.patch (5.2K, ../../CAFBsxsEospoUX=QYkfC=WcJqNB+iZtBf=BaRwn-zbHa48X0NKQ@mail.gmail.com/9-v3-0006-Build-syncrep_scanner.c-standalone.patch)
download | inline diff:
From 5cff476dd0588bc6c1a641990ea5050f0998130b Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 15:02:30 +0700
Subject: [PATCH v3 06/11] Build syncrep_scanner.c standalone
---
src/backend/Makefile | 3 ++-
src/backend/replication/.gitignore | 1 +
src/backend/replication/Makefile | 11 +++++++++--
src/backend/replication/syncrep_gram.y | 2 --
src/backend/replication/syncrep_scanner.l | 21 +++++++++++++++------
src/tools/pginclude/headerscheck | 1 +
6 files changed, 28 insertions(+), 11 deletions(-)
diff --git a/src/backend/Makefile b/src/backend/Makefile
index f527659a7b..86cbe03677 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -183,7 +183,7 @@ distprep:
$(MAKE) -C bootstrap bootparse.c bootparse.h bootscanner.c
$(MAKE) -C catalog distprep
$(MAKE) -C nodes distprep
- $(MAKE) -C replication repl_gram.c repl_gram.h repl_scanner.c syncrep_gram.c syncrep_scanner.c
+ $(MAKE) -C replication repl_gram.c repl_gram.h repl_scanner.c syncrep_gram.c syncrep_gram.h syncrep_scanner.c
$(MAKE) -C storage/lmgr lwlocknames.h lwlocknames.c
$(MAKE) -C utils distprep
$(MAKE) -C utils/adt jsonpath_gram.c jsonpath_scan.c
@@ -307,6 +307,7 @@ maintainer-clean: distclean
replication/repl_gram.h \
replication/repl_scanner.c \
replication/syncrep_gram.c \
+ replication/syncrep_gram.h \
replication/syncrep_scanner.c \
storage/lmgr/lwlocknames.c \
storage/lmgr/lwlocknames.h \
diff --git a/src/backend/replication/.gitignore b/src/backend/replication/.gitignore
index a5f600232f..77d5a51068 100644
--- a/src/backend/replication/.gitignore
+++ b/src/backend/replication/.gitignore
@@ -1,5 +1,6 @@
/repl_gram.h
/repl_gram.c
/repl_scanner.c
+/syncrep_gram.h
/syncrep_gram.c
/syncrep_scanner.c
diff --git a/src/backend/replication/Makefile b/src/backend/replication/Makefile
index bc8170418f..23f29ba545 100644
--- a/src/backend/replication/Makefile
+++ b/src/backend/replication/Makefile
@@ -21,6 +21,7 @@ OBJS = \
slotfuncs.o \
syncrep.o \
syncrep_gram.o \
+ syncrep_scanner.o \
walreceiver.o \
walreceiverfuncs.o \
walsender.o
@@ -38,8 +39,14 @@ repl_gram.c: BISONFLAGS += -d
# Force these dependencies to be known even without dependency info built:
repl_gram.o repl_scanner.o: repl_gram.h
-# syncrep_scanner is compiled as part of syncrep_gram
-syncrep_gram.o: syncrep_scanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+syncrep_gram.h: syncrep_gram.c
+ touch $@
+
+syncrep_gram.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+syncrep_gram.o syncrep_scanner.o: syncrep_gram.h
# repl_gram.c, repl_scanner.c, syncrep_gram.c and syncrep_scanner.c
# are in the distribution tarball, so they are not cleaned here.
diff --git a/src/backend/replication/syncrep_gram.y b/src/backend/replication/syncrep_gram.y
index d932f2cda3..4fc3647da1 100644
--- a/src/backend/replication/syncrep_gram.y
+++ b/src/backend/replication/syncrep_gram.y
@@ -112,5 +112,3 @@ create_syncrep_config(const char *num_sync, List *members, uint8 syncrep_method)
return config;
}
-
-#include "syncrep_scanner.c"
diff --git a/src/backend/replication/syncrep_scanner.l b/src/backend/replication/syncrep_scanner.l
index 1952c8c6e0..bdb1a3391c 100644
--- a/src/backend/replication/syncrep_scanner.l
+++ b/src/backend/replication/syncrep_scanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* syncrep_scanner.l
@@ -17,6 +17,15 @@
#include "lib/stringinfo.h"
+/*
+ * NB: include syncrep_gram.h only AFTER including syncrep.h, because syncrep.h
+ * includes node definitions needed for YYSTYPE.
+ */
+#include "replication/syncrep.h"
+#include "syncrep_gram.h"
+}
+
+%{
/* Avoid exit() on fatal scanner errors (a bit ugly -- see yy_fatal_error) */
#undef fprintf
#define fprintf(file, fmt, msg) fprintf_to_ereport(fmt, msg)
@@ -82,28 +91,28 @@ xdinside [^"]+
appendStringInfoString(&xdbuf, yytext);
}
<xd>{xdstop} {
- yylval.str = xdbuf.data;
+ syncrep_yylval.str = xdbuf.data;
xdbuf.data = NULL;
BEGIN(INITIAL);
return NAME;
}
<xd><<EOF>> {
- yyerror("unterminated quoted identifier");
+ syncrep_yyerror("unterminated quoted identifier");
return JUNK;
}
{identifier} {
- yylval.str = pstrdup(yytext);
+ syncrep_yylval.str = pstrdup(yytext);
return NAME;
}
{digit}+ {
- yylval.str = pstrdup(yytext);
+ syncrep_yylval.str = pstrdup(yytext);
return NUM;
}
"*" {
- yylval.str = "*";
+ syncrep_yylval.str = "*";
return NAME;
}
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index c102adf434..77b697e293 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -119,6 +119,7 @@ do
test "$f" = src/backend/parser/gram.h && continue
test "$f" = src/backend/bootstrap/bootparse.h && continue
test "$f" = src/backend/replication/repl_gram.h && continue
+ test "$f" = src/backend/replication/syncrep_gram.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
test "$f" = src/interfaces/ecpg/preproc/preproc.h && continue
--
2.36.1
[text/x-patch] v3-0009-Build-cubescan.c-standalone.patch (5.9K, ../../CAFBsxsEospoUX=QYkfC=WcJqNB+iZtBf=BaRwn-zbHa48X0NKQ@mail.gmail.com/10-v3-0009-Build-cubescan.c-standalone.patch)
download | inline diff:
From e38e77bd77c803acfc66de8c76798d9d7be67c42 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 11:18:06 +0700
Subject: [PATCH v3 09/11] Build cubescan.c standalone
---
contrib/cube/.gitignore | 1 +
contrib/cube/Makefile | 16 +++++++++-----
contrib/cube/cubedata.h | 1 +
contrib/cube/cubeparse.y | 10 +++------
contrib/cube/cubescan.l | 38 ++++++++++++++++++++------------
src/tools/pginclude/headerscheck | 1 +
6 files changed, 40 insertions(+), 27 deletions(-)
diff --git a/contrib/cube/.gitignore b/contrib/cube/.gitignore
index cb4c989fff..f788440c79 100644
--- a/contrib/cube/.gitignore
+++ b/contrib/cube/.gitignore
@@ -1,3 +1,4 @@
+/cubeparse.h
/cubeparse.c
/cubescan.c
# Generated subdirectories
diff --git a/contrib/cube/Makefile b/contrib/cube/Makefile
index cf195506c7..4fd19aac35 100644
--- a/contrib/cube/Makefile
+++ b/contrib/cube/Makefile
@@ -4,7 +4,8 @@ MODULE_big = cube
OBJS = \
$(WIN32RES) \
cube.o \
- cubeparse.o
+ cubeparse.o \
+ cubescan.o
EXTENSION = cube
DATA = cube--1.2.sql cube--1.2--1.3.sql cube--1.3--1.4.sql cube--1.4--1.5.sql \
@@ -15,8 +16,6 @@ HEADERS = cubedata.h
REGRESS = cube cube_sci
-EXTRA_CLEAN = y.tab.c y.tab.h
-
SHLIB_LINK += $(filter -lm, $(LIBS))
ifdef USE_PGXS
@@ -30,11 +29,16 @@ include $(top_builddir)/src/Makefile.global
include $(top_srcdir)/contrib/contrib-global.mk
endif
+# See notes in src/backend/parser/Makefile about the following two rules
+cubeparse.h: cubeparse.c
+ touch $@
+
+cubeparse.c: BISONFLAGS += -d
-# cubescan is compiled as part of cubeparse
-cubeparse.o: cubescan.c
+# Force these dependencies to be known even without dependency info built:
+cubeparse.o cubescan.o: cubeparse.h
distprep: cubeparse.c cubescan.c
maintainer-clean:
- rm -f cubeparse.c cubescan.c
+ rm -f cubeparse.h cubeparse.c cubescan.c
diff --git a/contrib/cube/cubedata.h b/contrib/cube/cubedata.h
index dbe7d4f742..a04fc504ad 100644
--- a/contrib/cube/cubedata.h
+++ b/contrib/cube/cubedata.h
@@ -67,3 +67,4 @@ extern void cube_scanner_finish(void);
/* in cubeparse.y */
extern int cube_yyparse(NDBOX **result);
+extern int scanbuflen;
diff --git a/contrib/cube/cubeparse.y b/contrib/cube/cubeparse.y
index 7577c4515c..8e210b49ec 100644
--- a/contrib/cube/cubeparse.y
+++ b/contrib/cube/cubeparse.y
@@ -7,11 +7,9 @@
#include "postgres.h"
#include "cubedata.h"
+#include "cube_internal.h"
#include "utils/float.h"
-/* All grammar constructs return strings */
-#define YYSTYPE char *
-
/*
* Bison doesn't allocate anything that needs to live across parser calls,
* so we can easily have it use palloc instead of malloc. This prevents
@@ -23,8 +21,8 @@
#define YYMALLOC palloc
#define YYFREE pfree
-static char *scanbuf;
-static int scanbuflen;
+/* TODO: get rid of global variable */
+int scanbuflen;
static int item_count(const char *s, char delim);
static NDBOX *write_box(int dim, char *str1, char *str2);
@@ -265,5 +263,3 @@ write_point_as_box(int dim, char *str)
return bp;
}
-
-#include "cubescan.c"
diff --git a/contrib/cube/cubescan.l b/contrib/cube/cubescan.l
index bd400e3684..569a4e992e 100644
--- a/contrib/cube/cubescan.l
+++ b/contrib/cube/cubescan.l
@@ -1,9 +1,21 @@
-%{
+%top{
/*
* A scanner for EMP-style numeric ranges
* contrib/cube/cubescan.l
*/
+#include "postgres.h"
+
+/*
+ * NB: include cubeparse.h only AFTER including cube_internal.h for YYSTYPE
+ * and cubedata.h for NDBOX.
+ */
+#include "cubedata.h"
+#include "cube_internal.h"
+#include "cubeparse.h"
+}
+
+%{
/* LCOV_EXCL_START */
/* No reason to constrain amount of data slurped */
@@ -21,9 +33,7 @@ fprintf_to_ereport(const char *fmt, const char *msg)
/* Handles to the buffer that the lexer uses internally */
static YY_BUFFER_STATE scanbufhandle;
-/* this is now declared in cubeparse.y: */
-/* static char *scanbuf; */
-/* static int scanbuflen; */
+static char *scanbuf;
%}
%option 8bit
@@ -45,14 +55,14 @@ NaN [nN][aA][nN]
%%
-{float} yylval = yytext; return CUBEFLOAT;
-{infinity} yylval = yytext; return CUBEFLOAT;
-{NaN} yylval = yytext; return CUBEFLOAT;
-\[ yylval = "("; return O_BRACKET;
-\] yylval = ")"; return C_BRACKET;
-\( yylval = "("; return O_PAREN;
-\) yylval = ")"; return C_PAREN;
-\, yylval = ","; return COMMA;
+{float} cube_yylval = yytext; return CUBEFLOAT;
+{infinity} cube_yylval = yytext; return CUBEFLOAT;
+{NaN} cube_yylval = yytext; return CUBEFLOAT;
+\[ cube_yylval = "("; return O_BRACKET;
+\] cube_yylval = ")"; return C_BRACKET;
+\( cube_yylval = "("; return O_PAREN;
+\) cube_yylval = ")"; return C_PAREN;
+\, cube_yylval = ","; return COMMA;
[ \t\n\r\f]+ /* discard spaces */
. return yytext[0]; /* alert parser of the garbage */
@@ -62,7 +72,7 @@ NaN [nN][aA][nN]
/* result is not used, but Bison expects this signature */
void
-yyerror(NDBOX **result, const char *message)
+cube_yyerror(NDBOX **result, const char *message)
{
if (*yytext == YY_END_OF_BUFFER_CHAR)
{
@@ -89,7 +99,7 @@ yyerror(NDBOX **result, const char *message)
void
cube_scanner_init(const char *str)
{
- Size slen = strlen(str);
+ Size slen = strlen(str);
/*
* Might be left over after ereport()
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index e2c4ba9ae4..a778067570 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -115,6 +115,7 @@ do
# We can't make these Bison output files compilable standalone
# without using "%code require", which old Bison versions lack.
# parser/gram.h will be included by parser/gramparse.h anyway.
+ test "$f" = contrib/cube/cubeparse.h && continue
test "$f" = src/include/parser/gram.h && continue
test "$f" = src/backend/parser/gram.h && continue
test "$f" = src/backend/bootstrap/bootparse.h && continue
--
2.36.1
[text/x-patch] v3-0010-Build-segscan.c-standalone.patch (4.5K, ../../CAFBsxsEospoUX=QYkfC=WcJqNB+iZtBf=BaRwn-zbHa48X0NKQ@mail.gmail.com/11-v3-0010-Build-segscan.c-standalone.patch)
download | inline diff:
From d687b826a302c0aa7123b1f6bebdef3b9a75d443 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 12:00:33 +0700
Subject: [PATCH v3 10/11] Build segscan.c standalone
---
contrib/seg/.gitignore | 1 +
contrib/seg/Makefile | 15 +++++++++++----
contrib/seg/segparse.y | 3 ---
contrib/seg/segscan.l | 28 ++++++++++++++++++----------
src/tools/pginclude/headerscheck | 1 +
5 files changed, 31 insertions(+), 17 deletions(-)
diff --git a/contrib/seg/.gitignore b/contrib/seg/.gitignore
index 69e73d2096..fa247a4e67 100644
--- a/contrib/seg/.gitignore
+++ b/contrib/seg/.gitignore
@@ -1,3 +1,4 @@
+/segparse.h
/segparse.c
/segscan.c
# Generated subdirectories
diff --git a/contrib/seg/Makefile b/contrib/seg/Makefile
index bb63e83506..c6c134b8f1 100644
--- a/contrib/seg/Makefile
+++ b/contrib/seg/Makefile
@@ -4,7 +4,8 @@ MODULE_big = seg
OBJS = \
$(WIN32RES) \
seg.o \
- segparse.o
+ segparse.o \
+ segscan.o
EXTENSION = seg
DATA = seg--1.1.sql seg--1.1--1.2.sql seg--1.2--1.3.sql seg--1.3--1.4.sql \
@@ -29,10 +30,16 @@ include $(top_srcdir)/contrib/contrib-global.mk
endif
-# segscan is compiled as part of segparse
-segparse.o: segscan.c
+# See notes in src/backend/parser/Makefile about the following two rules
+segparse.h: segparse.c
+ touch $@
+
+segparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+segparse.o segscan.o: segparse.h
distprep: segparse.c segscan.c
maintainer-clean:
- rm -f segparse.c segscan.c
+ rm -f segparse.h segparse.c segscan.c
diff --git a/contrib/seg/segparse.y b/contrib/seg/segparse.y
index 33e3a9f35f..637eacd1a6 100644
--- a/contrib/seg/segparse.y
+++ b/contrib/seg/segparse.y
@@ -160,6 +160,3 @@ seg_atof(const char *value)
datum = DirectFunctionCall1(float4in, CStringGetDatum(value));
return DatumGetFloat4(datum);
}
-
-
-#include "segscan.c"
diff --git a/contrib/seg/segscan.l b/contrib/seg/segscan.l
index 5f6595e9eb..4744fd5e9e 100644
--- a/contrib/seg/segscan.l
+++ b/contrib/seg/segscan.l
@@ -1,8 +1,18 @@
-%{
+%top{
/*
* A scanner for EMP-style numeric ranges
*/
+#include "postgres.h"
+
+/*
+ * NB: include segparse.h only AFTER including segdata.h, because segdata.h
+ * contains the definition for SEG.
+ */
+#include "segdata.h"
+#include "segparse.h"
+}
+%{
/* LCOV_EXCL_START */
/* No reason to constrain amount of data slurped */
@@ -21,7 +31,6 @@ fprintf_to_ereport(const char *fmt, const char *msg)
/* Handles to the buffer that the lexer uses internally */
static YY_BUFFER_STATE scanbufhandle;
static char *scanbuf;
-static int scanbuflen;
%}
%option 8bit
@@ -42,12 +51,12 @@ float ({integer}|{real})([eE]{integer})?
%%
-{range} yylval.text = yytext; return RANGE;
-{plumin} yylval.text = yytext; return PLUMIN;
-{float} yylval.text = yytext; return SEGFLOAT;
-\< yylval.text = "<"; return EXTENSION;
-\> yylval.text = ">"; return EXTENSION;
-\~ yylval.text = "~"; return EXTENSION;
+{range} seg_yylval.text = yytext; return RANGE;
+{plumin} seg_yylval.text = yytext; return PLUMIN;
+{float} seg_yylval.text = yytext; return SEGFLOAT;
+\< seg_yylval.text = "<"; return EXTENSION;
+\> seg_yylval.text = ">"; return EXTENSION;
+\~ seg_yylval.text = "~"; return EXTENSION;
[ \t\n\r\f]+ /* discard spaces */
. return yytext[0]; /* alert parser of the garbage */
@@ -56,7 +65,7 @@ float ({integer}|{real})([eE]{integer})?
/* LCOV_EXCL_STOP */
void
-yyerror(SEG *result, const char *message)
+seg_yyerror(SEG *result, const char *message)
{
if (*yytext == YY_END_OF_BUFFER_CHAR)
{
@@ -94,7 +103,6 @@ seg_scanner_init(const char *str)
/*
* Make a scan buffer with special termination needed by flex.
*/
- scanbuflen = slen;
scanbuf = palloc(slen + 2);
memcpy(scanbuf, str, slen);
scanbuf[slen] = scanbuf[slen + 1] = YY_END_OF_BUFFER_CHAR;
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index a778067570..203703afa7 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -116,6 +116,7 @@ do
# without using "%code require", which old Bison versions lack.
# parser/gram.h will be included by parser/gramparse.h anyway.
test "$f" = contrib/cube/cubeparse.h && continue
+ test "$f" = contrib/seg/segparse.h && continue
test "$f" = src/include/parser/gram.h && continue
test "$f" = src/backend/parser/gram.h && continue
test "$f" = src/backend/bootstrap/bootparse.h && continue
--
2.36.1
[text/x-patch] v3-0011-Build-jsonpath_scan.c-standalone.patch (7.8K, ../../CAFBsxsEospoUX=QYkfC=WcJqNB+iZtBf=BaRwn-zbHa48X0NKQ@mail.gmail.com/12-v3-0011-Build-jsonpath_scan.c-standalone.patch)
download | inline diff:
From ecf3e99df1c71a579f307a9a7be06e06a88687b5 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 12:35:55 +0700
Subject: [PATCH v3 11/11] Build jsonpath_scan.c standalone
---
src/backend/utils/adt/.gitignore | 1 +
src/backend/utils/adt/Makefile | 11 ++++++--
src/backend/utils/adt/jsonpath_gram.y | 27 +------------------
src/backend/utils/adt/jsonpath_internal.h | 32 +++++++++++++++++++++++
src/backend/utils/adt/jsonpath_scan.l | 29 +++++++++++++-------
src/tools/pginclude/headerscheck | 1 +
6 files changed, 63 insertions(+), 38 deletions(-)
create mode 100644 src/backend/utils/adt/jsonpath_internal.h
diff --git a/src/backend/utils/adt/.gitignore b/src/backend/utils/adt/.gitignore
index 48cf941a52..7fab054407 100644
--- a/src/backend/utils/adt/.gitignore
+++ b/src/backend/utils/adt/.gitignore
@@ -1,2 +1,3 @@
+/jsonpath_gram.h
/jsonpath_gram.c
/jsonpath_scan.c
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 7c722ea2ce..0de0bbb1b8 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -57,6 +57,7 @@ OBJS = \
jsonpath.o \
jsonpath_exec.o \
jsonpath_gram.o \
+ jsonpath_scan.o \
like.o \
like_support.o \
lockfuncs.o \
@@ -119,11 +120,17 @@ OBJS = \
xid8funcs.o \
xml.o
+# See notes in src/backend/parser/Makefile about the following two rules
+jsonpath_gram.h: jsonpath_gram.c
+ touch $@
+
+jsonpath_gram.c: BISONFLAGS += -d
+
jsonpath_scan.c: FLEXFLAGS = -CF -p -p
jsonpath_scan.c: FLEX_NO_BACKUP=yes
-# jsonpath_scan is compiled as part of jsonpath_gram
-jsonpath_gram.o: jsonpath_scan.c
+# Force these dependencies to be known even without dependency info built:
+jsonpath_gram.o jsonpath_scan.o: jsonpath_gram.h
# jsonpath_gram.c and jsonpath_scan.c are in the distribution tarball,
# so they are not cleaned here.
diff --git a/src/backend/utils/adt/jsonpath_gram.y b/src/backend/utils/adt/jsonpath_gram.y
index f903dba3e3..e7027595ac 100644
--- a/src/backend/utils/adt/jsonpath_gram.y
+++ b/src/backend/utils/adt/jsonpath_gram.y
@@ -18,26 +18,11 @@
#include "catalog/pg_collation.h"
#include "fmgr.h"
+#include "jsonpath_internal.h"
#include "miscadmin.h"
#include "nodes/pg_list.h"
#include "regex/regex.h"
#include "utils/builtins.h"
-#include "utils/jsonpath.h"
-
-/* struct JsonPathString is shared between scan and gram */
-typedef struct JsonPathString
-{
- char *val;
- int len;
- int total;
-} JsonPathString;
-
-union YYSTYPE;
-
-/* flex 2.5.4 doesn't bother with a decl for this */
-int jsonpath_yylex(union YYSTYPE *yylval_param);
-int jsonpath_yyparse(JsonPathParseResult **result);
-void jsonpath_yyerror(JsonPathParseResult **result, const char *message);
static JsonPathParseItem *makeItemType(JsonPathItemType type);
static JsonPathParseItem *makeItemString(JsonPathString *s);
@@ -593,13 +578,3 @@ jspConvertRegexFlags(uint32 xflags)
return cflags;
}
-
-/*
- * jsonpath_scan.l is compiled as part of jsonpath_gram.y. Currently, this is
- * unavoidable because jsonpath_gram does not create a .h file to export its
- * token symbols. If these files ever grow large enough to be worth compiling
- * separately, that could be fixed; but for now it seems like useless
- * complication.
- */
-
-#include "jsonpath_scan.c"
diff --git a/src/backend/utils/adt/jsonpath_internal.h b/src/backend/utils/adt/jsonpath_internal.h
new file mode 100644
index 0000000000..edfc6191a0
--- /dev/null
+++ b/src/backend/utils/adt/jsonpath_internal.h
@@ -0,0 +1,32 @@
+/*-------------------------------------------------------------------------
+ *
+ * jsonpath_internal.h
+ * Private definitions for jsonpath scanner & parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/utils/adt/jsonpath_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef JSONPATH_INTERNAL_H
+#define JSONPATH_INTERNAL_H
+
+/* struct JsonPathString is shared between scan and gram */
+typedef struct JsonPathString
+{
+ char *val;
+ int len;
+ int total;
+} JsonPathString;
+
+#include "utils/jsonpath.h"
+#include "jsonpath_gram.h"
+
+extern int jsonpath_yylex(YYSTYPE *yylval_param);
+extern int jsonpath_yyparse(JsonPathParseResult **result);
+extern void jsonpath_yyerror(JsonPathParseResult **result, const char *message);
+
+#endif /* JSONPATH_INTERNAL_H */
diff --git a/src/backend/utils/adt/jsonpath_scan.l b/src/backend/utils/adt/jsonpath_scan.l
index 4351f6ec98..ea824bae73 100644
--- a/src/backend/utils/adt/jsonpath_scan.l
+++ b/src/backend/utils/adt/jsonpath_scan.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* jsonpath_scan.l
@@ -17,9 +17,18 @@
#include "postgres.h"
+/*
+ * NB: include jsonpath_gram.h only AFTER including jsonpath_internal.h,
+ * because jsonpath_internal.h contains the declaration for JsonPathString.
+ */
+#include "jsonpath_internal.h"
+#include "jsonpath_gram.h"
+
#include "mb/pg_wchar.h"
#include "nodes/pg_list.h"
+}
+%{
static JsonPathString scanstring;
/* Handles to the buffer that the lexer uses internally */
@@ -142,9 +151,9 @@ hex_fail \\x{hex_dig}{0,1}
<xnq,xq,xvq>{hex_char} { parseHexChar(yytext); }
-<xnq,xq,xvq>{unicode}*{unicodefail} { yyerror(NULL, "invalid unicode sequence"); }
+<xnq,xq,xvq>{unicode}*{unicodefail} { jsonpath_yyerror(NULL, "invalid unicode sequence"); }
-<xnq,xq,xvq>{hex_fail} { yyerror(NULL, "invalid hex character sequence"); }
+<xnq,xq,xvq>{hex_fail} { jsonpath_yyerror(NULL, "invalid hex character sequence"); }
<xnq,xq,xvq>{unicode}+\\ {
/* throw back the \\, and treat as unicode */
@@ -154,9 +163,9 @@ hex_fail \\x{hex_dig}{0,1}
<xnq,xq,xvq>\\. { addchar(false, yytext[1]); }
-<xnq,xq,xvq>\\ { yyerror(NULL, "unexpected end after backslash"); }
+<xnq,xq,xvq>\\ { jsonpath_yyerror(NULL, "unexpected end after backslash"); }
-<xq,xvq><<EOF>> { yyerror(NULL, "unexpected end of quoted string"); }
+<xq,xvq><<EOF>> { jsonpath_yyerror(NULL, "unexpected end of quoted string"); }
<xq>\" {
yylval->str = scanstring;
@@ -178,7 +187,7 @@ hex_fail \\x{hex_dig}{0,1}
<xc>\* { }
-<xc><<EOF>> { yyerror(NULL, "unexpected end of comment"); }
+<xc><<EOF>> { jsonpath_yyerror(NULL, "unexpected end of comment"); }
\&\& { return AND_P; }
@@ -244,10 +253,10 @@ hex_fail \\x{hex_dig}{0,1}
return INT_P;
}
-{realfail} { yyerror(NULL, "invalid numeric literal"); }
-{integer_junk} { yyerror(NULL, "trailing junk after numeric literal"); }
-{decimal_junk} { yyerror(NULL, "trailing junk after numeric literal"); }
-{real_junk} { yyerror(NULL, "trailing junk after numeric literal"); }
+{realfail} { jsonpath_yyerror(NULL, "invalid numeric literal"); }
+{integer_junk} { jsonpath_yyerror(NULL, "trailing junk after numeric literal"); }
+{decimal_junk} { jsonpath_yyerror(NULL, "trailing junk after numeric literal"); }
+{real_junk} { jsonpath_yyerror(NULL, "trailing junk after numeric literal"); }
\" {
addchar(true, '\0');
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 203703afa7..2e90f52c33 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -122,6 +122,7 @@ do
test "$f" = src/backend/bootstrap/bootparse.h && continue
test "$f" = src/backend/replication/repl_gram.h && continue
test "$f" = src/backend/replication/syncrep_gram.h && continue
+ test "$f" = src/backend/utils/adt/jsonpath_gram.h && continue
test "$f" = src/test/isolation/specparse.h && continue
test "$f" = src/bin/pgbench/exprparse.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
--
2.36.1
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: build remaining Flex files standalone
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 04:07 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-12 06:01 ` build remaining Flex files standalone John Naylor <[email protected]>
2022-08-13 08:39 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-15 18:11 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
2022-08-16 10:41 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
@ 2022-08-17 01:14 ` Andres Freund <[email protected]>
2022-08-17 01:47 ` Re: build remaining Flex files standalone Tom Lane <[email protected]>
2022-08-17 02:53 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-18 07:40 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
0 siblings, 3 replies; 139+ messages in thread
From: Andres Freund @ 2022-08-17 01:14 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
Hi,
On 2022-08-16 17:41:43 +0700, John Naylor wrote:
> For v3, I addressed some comments and added .h files to the
> headerscheck exceptions.
Thanks!
> /*
> * NB: include bootparse.h only AFTER including bootstrap.h, because bootstrap.h
> * includes node definitions needed for YYSTYPE.
> */
>
> Future cleanup: I see this in headerscheck:
>
> # We can't make these Bison output files compilable standalone
> # without using "%code require", which old Bison versions lack.
> # parser/gram.h will be included by parser/gramparse.h anyway.
>
> That directive has been supported in Bison since 2.4.2.
2.4.2 is from 2010. So I think we could just start relying on it?
> > > +/* functions shared between guc.c and guc-file.l */
> > > [...]
> > I think I prefer your suggestion of a guc_internal.h upthread.
>
> Started in 0002, but left open the headerscheck failure.
>
> Also, if such a thing is meant to be #include'd only by two generated
> files, maybe it should just live in the directory where they live, and
> not in the src/include dir?
It's not something we've done for the backend afaics, but I don't see a reason
not to start at some point.
> > > From 7d4ecfcb3e91f3b45e94b9e64c7c40f1bbd22aa8 Mon Sep 17 00:00:00 2001
> > > From: John Naylor <[email protected]>
> > > Date: Fri, 12 Aug 2022 15:45:24 +0700
> > > Subject: [PATCH v201 2/9] Build booscanner.c standalone
> >
> > > -# bootscanner is compiled as part of bootparse
> > > -bootparse.o: bootscanner.c
> > > +# See notes in src/backend/parser/Makefile about the following two rules
> > > +bootparse.h: bootparse.c
> > > + touch $@
> > > +
> > > +bootparse.c: BISONFLAGS += -d
> > > +
> > > +# Force these dependencies to be known even without dependency info built:
> > > +bootparse.o bootscan.o: bootparse.h
> >
> > Wonder if we could / should wrap this is something common. It's somewhat
> > annoying to repeat this stuff everywhere.
>
> I haven't looked at the Meson effort recently, but if the build rule
> is less annoying there, I'm inclined to leave this as a wart until
> autotools are retired.
The only complicating thing in the rules there is the dependencies from one .c
file to another .c file.
> > > diff --git a/contrib/cube/cubedata.h b/contrib/cube/cubedata.h
> > > index dbe7d4f742..0b373048b5 100644
> > > --- a/contrib/cube/cubedata.h
> > > +++ b/contrib/cube/cubedata.h
> > > @@ -67,3 +67,7 @@ extern void cube_scanner_finish(void);
> > >
> > > /* in cubeparse.y */
> > > extern int cube_yyparse(NDBOX **result);
> > > +
> > > +/* All grammar constructs return strings */
> > > +#define YYSTYPE char *
> >
> > Why does this need to be defined in a semi-public header? If we do this in
> > multiple files we'll end up with the danger of macro redefinition warnings.
>
> I tried to put all the Flex/Bison stuff in another *_internal header,
> but that breaks the build. Putting just this one symbol in a header is
> silly, but done that way for now. Maybe two copies of the symbol?
The problem is that if it's in a header you can't include another header with
such a define. That's fine if it's a .h that's just intended to be included by
a limited set of files, but for something like a header for a datatype that
might need to be included to e.g. define a PL transform or a new operator or
... This would be solved by the %code requires thing, right?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: build remaining Flex files standalone
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 04:07 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-12 06:01 ` build remaining Flex files standalone John Naylor <[email protected]>
2022-08-13 08:39 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-15 18:11 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
2022-08-16 10:41 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-17 01:14 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
@ 2022-08-17 01:47 ` Tom Lane <[email protected]>
2 siblings, 0 replies; 139+ messages in thread
From: Tom Lane @ 2022-08-17 01:47 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: John Naylor <[email protected]>; pgsql-hackers
Andres Freund <[email protected]> writes:
> On 2022-08-16 17:41:43 +0700, John Naylor wrote:
>> That directive has been supported in Bison since 2.4.2.
> 2.4.2 is from 2010. So I think we could just start relying on it?
Apple is still shipping 2.3. Is this worth enough to make Mac
users install a non-default Bison? I seriously doubt it.
I don't say that there won't be a reason that justifies that
at some point, but letting headerscheck test autogenerated
files seems of only microscopic benefit :-(
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: build remaining Flex files standalone
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 04:07 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-12 06:01 ` build remaining Flex files standalone John Naylor <[email protected]>
2022-08-13 08:39 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-15 18:11 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
2022-08-16 10:41 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-17 01:14 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
@ 2022-08-17 02:53 ` John Naylor <[email protected]>
2 siblings, 0 replies; 139+ messages in thread
From: John Naylor @ 2022-08-17 02:53 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Wed, Aug 17, 2022 at 8:14 AM Andres Freund <[email protected]> wrote:
> > > > +/* functions shared between guc.c and guc-file.l */
> > > > [...]
> > > I think I prefer your suggestion of a guc_internal.h upthread.
> >
> > Started in 0002, but left open the headerscheck failure.
> >
> > Also, if such a thing is meant to be #include'd only by two generated
> > files, maybe it should just live in the directory where they live, and
> > not in the src/include dir?
>
> It's not something we've done for the backend afaics, but I don't see a reason
> not to start at some point.
BTW, I forgot to mention I did this for the json path parser, which
makes the makefile code simpler than what was there before
550b9d26f80fa30. AFAICS, we could also do the same for gramparse.h,
which is internal to parser.c. If I'm not mistaken, the only reason we
symlink gram.h to src/include/* is so that gramparse.h can include it.
So keeping gramparse.h in the backend could allow removing some gram.h
makefile incantations.
> > > Why does this need to be defined in a semi-public header? If we do this in
> > > multiple files we'll end up with the danger of macro redefinition warnings.
> >
> > I tried to put all the Flex/Bison stuff in another *_internal header,
> > but that breaks the build. Putting just this one symbol in a header is
> > silly, but done that way for now. Maybe two copies of the symbol?
>
> The problem is that if it's in a header you can't include another header with
> such a define. That's fine if it's a .h that's just intended to be included by
> a limited set of files, but for something like a header for a datatype that
> might need to be included to e.g. define a PL transform or a new operator or
> ... This would be solved by the %code requires thing, right?
I believe it would.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: build remaining Flex files standalone
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 04:07 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-12 06:01 ` build remaining Flex files standalone John Naylor <[email protected]>
2022-08-13 08:39 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-15 18:11 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
2022-08-16 10:41 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-17 01:14 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
@ 2022-08-18 07:40 ` John Naylor <[email protected]>
2022-08-18 07:43 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2 siblings, 1 reply; 139+ messages in thread
From: John Naylor @ 2022-08-18 07:40 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
On Wed, Aug 17, 2022 at 8:14 AM Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2022-08-16 17:41:43 +0700, John Naylor wrote:
> > For v3, I addressed some comments and added .h files to the
> > headerscheck exceptions.
>
> Thanks!
>
>
> > /*
> > * NB: include bootparse.h only AFTER including bootstrap.h, because bootstrap.h
> > * includes node definitions needed for YYSTYPE.
> > */
> >
> > Future cleanup: I see this in headerscheck:
> >
> > # We can't make these Bison output files compilable standalone
> > # without using "%code require", which old Bison versions lack.
> > # parser/gram.h will be included by parser/gramparse.h anyway.
> >
> > That directive has been supported in Bison since 2.4.2.
>
> 2.4.2 is from 2010. So I think we could just start relying on it?
>
>
> > > > +/* functions shared between guc.c and guc-file.l */
> > > > [...]
> > > I think I prefer your suggestion of a guc_internal.h upthread.
> >
> > Started in 0002, but left open the headerscheck failure.
> >
> > Also, if such a thing is meant to be #include'd only by two generated
> > files, maybe it should just live in the directory where they live, and
> > not in the src/include dir?
>
> It's not something we've done for the backend afaics, but I don't see a reason
> not to start at some point.
>
>
> > > > From 7d4ecfcb3e91f3b45e94b9e64c7c40f1bbd22aa8 Mon Sep 17 00:00:00 2001
> > > > From: John Naylor <[email protected]>
> > > > Date: Fri, 12 Aug 2022 15:45:24 +0700
> > > > Subject: [PATCH v201 2/9] Build booscanner.c standalone
> > >
> > > > -# bootscanner is compiled as part of bootparse
> > > > -bootparse.o: bootscanner.c
> > > > +# See notes in src/backend/parser/Makefile about the following two rules
> > > > +bootparse.h: bootparse.c
> > > > + touch $@
> > > > +
> > > > +bootparse.c: BISONFLAGS += -d
> > > > +
> > > > +# Force these dependencies to be known even without dependency info built:
> > > > +bootparse.o bootscan.o: bootparse.h
> > >
> > > Wonder if we could / should wrap this is something common. It's somewhat
> > > annoying to repeat this stuff everywhere.
> >
> > I haven't looked at the Meson effort recently, but if the build rule
> > is less annoying there, I'm inclined to leave this as a wart until
> > autotools are retired.
>
> The only complicating thing in the rules there is the dependencies from one .c
> file to another .c file.
>
>
> > > > diff --git a/contrib/cube/cubedata.h b/contrib/cube/cubedata.h
> > > > index dbe7d4f742..0b373048b5 100644
> > > > --- a/contrib/cube/cubedata.h
> > > > +++ b/contrib/cube/cubedata.h
> > > > @@ -67,3 +67,7 @@ extern void cube_scanner_finish(void);
> > > >
> > > > /* in cubeparse.y */
> > > > extern int cube_yyparse(NDBOX **result);
> > > > +
> > > > +/* All grammar constructs return strings */
> > > > +#define YYSTYPE char *
> > >
> > > Why does this need to be defined in a semi-public header? If we do this in
> > > multiple files we'll end up with the danger of macro redefinition warnings.
For v4, I #defined YYSTYPE
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: build remaining Flex files standalone
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 04:07 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-12 06:01 ` build remaining Flex files standalone John Naylor <[email protected]>
2022-08-13 08:39 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-15 18:11 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
2022-08-16 10:41 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-17 01:14 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
2022-08-18 07:40 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
@ 2022-08-18 07:43 ` John Naylor <[email protected]>
2022-08-18 08:00 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: John Naylor @ 2022-08-18 07:43 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
> > > > > index dbe7d4f742..0b373048b5 100644
> > > > > --- a/contrib/cube/cubedata.h
> > > > > +++ b/contrib/cube/cubedata.h
> > > > > @@ -67,3 +67,7 @@ extern void cube_scanner_finish(void);
> > > > >
> > > > > /* in cubeparse.y */
> > > > > extern int cube_yyparse(NDBOX **result);
> > > > > +
> > > > > +/* All grammar constructs return strings */
> > > > > +#define YYSTYPE char *
> > > >
> > > > Why does this need to be defined in a semi-public header? If we do this in
> > > > multiple files we'll end up with the danger of macro redefinition warnings.
>
> For v4, I #defined YYSTYPE
Sorry for the misfire. Continuing on, I #defined YYSTYPE in cubescan.l
before #including cubeparse.h.
I also added scanbuflen to the %parse-param to prevent resorting to a
global variable. The rest of the patches are unchanged.
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[text/x-patch] v4-0002-Move-private-declarations-shared-between-guc.c-an.patch (3.2K, ../../CAFBsxsHdAP0U3AjYuJ_SpWBxj2vmpdqXX3Yg=TLBrq0rAWhFgg@mail.gmail.com/2-v4-0002-Move-private-declarations-shared-between-guc.c-an.patch)
download | inline diff:
From 2b95401d925bed67b2cb1eb9e8cdb1f1dd3bcc8e Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 16 Aug 2022 12:01:41 +0700
Subject: [PATCH v4 02/11] Move private declarations shared between guc.c and
guc-file.l to new header
FIXME: fails headerscheck
---
src/backend/utils/misc/guc-file.l | 1 +
src/backend/utils/misc/guc.c | 1 +
src/include/utils/guc.h | 10 ----------
src/include/utils/guc_internal.h | 24 ++++++++++++++++++++++++
4 files changed, 26 insertions(+), 10 deletions(-)
create mode 100644 src/include/utils/guc_internal.h
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index b4fa09749b..843838b1df 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/fd.h"
#include "utils/guc.h"
+#include "utils/guc_internal.h"
/*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 66ab3912a0..293834fc13 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -100,6 +100,7 @@
#include "utils/builtins.h"
#include "utils/bytea.h"
#include "utils/float.h"
+#include "utils/guc_internal.h"
#include "utils/guc_tables.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index aae071cd82..45ae1b537f 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -442,16 +442,6 @@ extern void GUC_check_errcode(int sqlerrcode);
pre_format_elog_string(errno, TEXTDOMAIN), \
GUC_check_errhint_string = format_elog_string
-/* functions shared between guc.c and guc-file.l */
-extern int guc_name_compare(const char *namea, const char *nameb);
-extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
- bool applySettings, int elevel);
-extern void record_config_file_error(const char *errmsg,
- const char *config_file,
- int lineno,
- ConfigVariable **head_p,
- ConfigVariable **tail_p);
-
/*
* The following functions are not in guc.c, but are declared here to avoid
* having to include guc.h in some widely used headers that it really doesn't
diff --git a/src/include/utils/guc_internal.h b/src/include/utils/guc_internal.h
new file mode 100644
index 0000000000..5d5db6bdce
--- /dev/null
+++ b/src/include/utils/guc_internal.h
@@ -0,0 +1,24 @@
+/*--------------------------------------------------------------------
+ * guc_internals.h
+ *
+ * Declarations shared between backend/utils/misc/guc.c and
+ * backend/utils/misc/guc-file.l
+ *
+ * Copyright (c) 2000-2022, PostgreSQL Global Development Group
+ *
+ * src/include/utils/guc_internals.h
+ *--------------------------------------------------------------------
+ */
+#ifndef GUC_INTERNALS_H
+#define GUC_INTERNALS_H
+
+extern int guc_name_compare(const char *namea, const char *nameb);
+extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
+ bool applySettings, int elevel);
+extern void record_config_file_error(const char *errmsg,
+ const char *config_file,
+ int lineno,
+ ConfigVariable **head_p,
+ ConfigVariable **tail_p);
+
+#endif /* GUC_INTERNALS_H */
--
2.36.1
[text/x-patch] v4-0001-Preparatory-refactoring-for-compiling-guc-file.c-.patch (26.0K, ../../CAFBsxsHdAP0U3AjYuJ_SpWBxj2vmpdqXX3Yg=TLBrq0rAWhFgg@mail.gmail.com/3-v4-0001-Preparatory-refactoring-for-compiling-guc-file.c-.patch)
download | inline diff:
From c066efea2193be8e7b21bb44c067383f34f37ec8 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 16 Aug 2022 10:42:19 +0700
Subject: [PATCH v4 01/11] Preparatory refactoring for compiling guc-file.c
standalone
Mostly this involves moving ProcessConfigFileInternal() to guc.c
and fixing the shared API to match.
---
src/backend/utils/misc/guc-file.l | 360 +-----------------------------
src/backend/utils/misc/guc.c | 360 +++++++++++++++++++++++++++++-
src/include/utils/guc.h | 9 +
3 files changed, 364 insertions(+), 365 deletions(-)
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index ce5633844c..b4fa09749b 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -48,12 +48,6 @@ static sigjmp_buf *GUC_flex_fatal_jmp;
static void FreeConfigVariable(ConfigVariable *item);
-static void record_config_file_error(const char *errmsg,
- const char *config_file,
- int lineno,
- ConfigVariable **head_p,
- ConfigVariable **tail_p);
-
static int GUC_flex_fatal(const char *msg);
/* LCOV_EXCL_START */
@@ -159,358 +153,6 @@ ProcessConfigFile(GucContext context)
MemoryContextDelete(config_cxt);
}
-/*
- * This function handles both actual config file (re)loads and execution of
- * show_all_file_settings() (i.e., the pg_file_settings view). In the latter
- * case we don't apply any of the settings, but we make all the usual validity
- * checks, and we return the ConfigVariable list so that it can be printed out
- * by show_all_file_settings().
- */
-static ConfigVariable *
-ProcessConfigFileInternal(GucContext context, bool applySettings, int elevel)
-{
- bool error = false;
- bool applying = false;
- const char *ConfFileWithError;
- ConfigVariable *item,
- *head,
- *tail;
- int i;
-
- /* Parse the main config file into a list of option names and values */
- ConfFileWithError = ConfigFileName;
- head = tail = NULL;
-
- if (!ParseConfigFile(ConfigFileName, true,
- NULL, 0, 0, elevel,
- &head, &tail))
- {
- /* Syntax error(s) detected in the file, so bail out */
- error = true;
- goto bail_out;
- }
-
- /*
- * Parse the PG_AUTOCONF_FILENAME file, if present, after the main file to
- * replace any parameters set by ALTER SYSTEM command. Because this file
- * is in the data directory, we can't read it until the DataDir has been
- * set.
- */
- if (DataDir)
- {
- if (!ParseConfigFile(PG_AUTOCONF_FILENAME, false,
- NULL, 0, 0, elevel,
- &head, &tail))
- {
- /* Syntax error(s) detected in the file, so bail out */
- error = true;
- ConfFileWithError = PG_AUTOCONF_FILENAME;
- goto bail_out;
- }
- }
- else
- {
- /*
- * If DataDir is not set, the PG_AUTOCONF_FILENAME file cannot be
- * read. In this case, we don't want to accept any settings but
- * data_directory from postgresql.conf, because they might be
- * overwritten with settings in the PG_AUTOCONF_FILENAME file which
- * will be read later. OTOH, since data_directory isn't allowed in the
- * PG_AUTOCONF_FILENAME file, it will never be overwritten later.
- */
- ConfigVariable *newlist = NULL;
-
- /*
- * Prune all items except the last "data_directory" from the list.
- */
- for (item = head; item; item = item->next)
- {
- if (!item->ignore &&
- strcmp(item->name, "data_directory") == 0)
- newlist = item;
- }
-
- if (newlist)
- newlist->next = NULL;
- head = tail = newlist;
-
- /*
- * Quick exit if data_directory is not present in file.
- *
- * We need not do any further processing, in particular we don't set
- * PgReloadTime; that will be set soon by subsequent full loading of
- * the config file.
- */
- if (head == NULL)
- goto bail_out;
- }
-
- /*
- * Mark all extant GUC variables as not present in the config file. We
- * need this so that we can tell below which ones have been removed from
- * the file since we last processed it.
- */
- for (i = 0; i < num_guc_variables; i++)
- {
- struct config_generic *gconf = guc_variables[i];
-
- gconf->status &= ~GUC_IS_IN_FILE;
- }
-
- /*
- * Check if all the supplied option names are valid, as an additional
- * quasi-syntactic check on the validity of the config file. It is
- * important that the postmaster and all backends agree on the results of
- * this phase, else we will have strange inconsistencies about which
- * processes accept a config file update and which don't. Hence, unknown
- * custom variable names have to be accepted without complaint. For the
- * same reason, we don't attempt to validate the options' values here.
- *
- * In addition, the GUC_IS_IN_FILE flag is set on each existing GUC
- * variable mentioned in the file; and we detect duplicate entries in the
- * file and mark the earlier occurrences as ignorable.
- */
- for (item = head; item; item = item->next)
- {
- struct config_generic *record;
-
- /* Ignore anything already marked as ignorable */
- if (item->ignore)
- continue;
-
- /*
- * Try to find the variable; but do not create a custom placeholder if
- * it's not there already.
- */
- record = find_option(item->name, false, true, elevel);
-
- if (record)
- {
- /* If it's already marked, then this is a duplicate entry */
- if (record->status & GUC_IS_IN_FILE)
- {
- /*
- * Mark the earlier occurrence(s) as dead/ignorable. We could
- * avoid the O(N^2) behavior here with some additional state,
- * but it seems unlikely to be worth the trouble.
- */
- ConfigVariable *pitem;
-
- for (pitem = head; pitem != item; pitem = pitem->next)
- {
- if (!pitem->ignore &&
- strcmp(pitem->name, item->name) == 0)
- pitem->ignore = true;
- }
- }
- /* Now mark it as present in file */
- record->status |= GUC_IS_IN_FILE;
- }
- else if (!valid_custom_variable_name(item->name))
- {
- /* Invalid non-custom variable, so complain */
- ereport(elevel,
- (errcode(ERRCODE_UNDEFINED_OBJECT),
- errmsg("unrecognized configuration parameter \"%s\" in file \"%s\" line %d",
- item->name,
- item->filename, item->sourceline)));
- item->errmsg = pstrdup("unrecognized configuration parameter");
- error = true;
- ConfFileWithError = item->filename;
- }
- }
-
- /*
- * If we've detected any errors so far, we don't want to risk applying any
- * changes.
- */
- if (error)
- goto bail_out;
-
- /* Otherwise, set flag that we're beginning to apply changes */
- applying = true;
-
- /*
- * Check for variables having been removed from the config file, and
- * revert their reset values (and perhaps also effective values) to the
- * boot-time defaults. If such a variable can't be changed after startup,
- * report that and continue.
- */
- for (i = 0; i < num_guc_variables; i++)
- {
- struct config_generic *gconf = guc_variables[i];
- GucStack *stack;
-
- if (gconf->reset_source != PGC_S_FILE ||
- (gconf->status & GUC_IS_IN_FILE))
- continue;
- if (gconf->context < PGC_SIGHUP)
- {
- /* The removal can't be effective without a restart */
- gconf->status |= GUC_PENDING_RESTART;
- ereport(elevel,
- (errcode(ERRCODE_CANT_CHANGE_RUNTIME_PARAM),
- errmsg("parameter \"%s\" cannot be changed without restarting the server",
- gconf->name)));
- record_config_file_error(psprintf("parameter \"%s\" cannot be changed without restarting the server",
- gconf->name),
- NULL, 0,
- &head, &tail);
- error = true;
- continue;
- }
-
- /* No more to do if we're just doing show_all_file_settings() */
- if (!applySettings)
- continue;
-
- /*
- * Reset any "file" sources to "default", else set_config_option will
- * not override those settings.
- */
- if (gconf->reset_source == PGC_S_FILE)
- gconf->reset_source = PGC_S_DEFAULT;
- if (gconf->source == PGC_S_FILE)
- gconf->source = PGC_S_DEFAULT;
- for (stack = gconf->stack; stack; stack = stack->prev)
- {
- if (stack->source == PGC_S_FILE)
- stack->source = PGC_S_DEFAULT;
- }
-
- /* Now we can re-apply the wired-in default (i.e., the boot_val) */
- if (set_config_option(gconf->name, NULL,
- context, PGC_S_DEFAULT,
- GUC_ACTION_SET, true, 0, false) > 0)
- {
- /* Log the change if appropriate */
- if (context == PGC_SIGHUP)
- ereport(elevel,
- (errmsg("parameter \"%s\" removed from configuration file, reset to default",
- gconf->name)));
- }
- }
-
- /*
- * Restore any variables determined by environment variables or
- * dynamically-computed defaults. This is a no-op except in the case
- * where one of these had been in the config file and is now removed.
- *
- * In particular, we *must not* do this during the postmaster's initial
- * loading of the file, since the timezone functions in particular should
- * be run only after initialization is complete.
- *
- * XXX this is an unmaintainable crock, because we have to know how to set
- * (or at least what to call to set) every non-PGC_INTERNAL variable that
- * could potentially have PGC_S_DYNAMIC_DEFAULT or PGC_S_ENV_VAR source.
- */
- if (context == PGC_SIGHUP && applySettings)
- {
- InitializeGUCOptionsFromEnvironment();
- pg_timezone_abbrev_initialize();
- /* this selects SQL_ASCII in processes not connected to a database */
- SetConfigOption("client_encoding", GetDatabaseEncodingName(),
- PGC_BACKEND, PGC_S_DYNAMIC_DEFAULT);
- }
-
- /*
- * Now apply the values from the config file.
- */
- for (item = head; item; item = item->next)
- {
- char *pre_value = NULL;
- int scres;
-
- /* Ignore anything marked as ignorable */
- if (item->ignore)
- continue;
-
- /* In SIGHUP cases in the postmaster, we want to report changes */
- if (context == PGC_SIGHUP && applySettings && !IsUnderPostmaster)
- {
- const char *preval = GetConfigOption(item->name, true, false);
-
- /* If option doesn't exist yet or is NULL, treat as empty string */
- if (!preval)
- preval = "";
- /* must dup, else might have dangling pointer below */
- pre_value = pstrdup(preval);
- }
-
- scres = set_config_option(item->name, item->value,
- context, PGC_S_FILE,
- GUC_ACTION_SET, applySettings, 0, false);
- if (scres > 0)
- {
- /* variable was updated, so log the change if appropriate */
- if (pre_value)
- {
- const char *post_value = GetConfigOption(item->name, true, false);
-
- if (!post_value)
- post_value = "";
- if (strcmp(pre_value, post_value) != 0)
- ereport(elevel,
- (errmsg("parameter \"%s\" changed to \"%s\"",
- item->name, item->value)));
- }
- item->applied = true;
- }
- else if (scres == 0)
- {
- error = true;
- item->errmsg = pstrdup("setting could not be applied");
- ConfFileWithError = item->filename;
- }
- else
- {
- /* no error, but variable's active value was not changed */
- item->applied = true;
- }
-
- /*
- * We should update source location unless there was an error, since
- * even if the active value didn't change, the reset value might have.
- * (In the postmaster, there won't be a difference, but it does matter
- * in backends.)
- */
- if (scres != 0 && applySettings)
- set_config_sourcefile(item->name, item->filename,
- item->sourceline);
-
- if (pre_value)
- pfree(pre_value);
- }
-
- /* Remember when we last successfully loaded the config file. */
- if (applySettings)
- PgReloadTime = GetCurrentTimestamp();
-
-bail_out:
- if (error && applySettings)
- {
- /* During postmaster startup, any error is fatal */
- if (context == PGC_POSTMASTER)
- ereport(ERROR,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors",
- ConfFileWithError)));
- else if (applying)
- ereport(elevel,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
- ConfFileWithError)));
- else
- ereport(elevel,
- (errcode(ERRCODE_CONFIG_FILE_ERROR),
- errmsg("configuration file \"%s\" contains errors; no changes were applied",
- ConfFileWithError)));
- }
-
- /* Successful or otherwise, return the collected data list */
- return head;
-}
-
/*
* Given a configuration file or directory location that may be a relative
* path, return an absolute one. We consider the location to be relative to
@@ -659,7 +301,7 @@ cleanup:
* Capture an error message in the ConfigVariable list returned by
* config file parsing.
*/
-static void
+void
record_config_file_error(const char *errmsg,
const char *config_file,
int lineno,
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 9fbbfb1be5..66ab3912a0 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -243,10 +243,6 @@ static void assign_recovery_target_lsn(const char *newval, void *extra);
static bool check_primary_slot_name(char **newval, void **extra, GucSource source);
static bool check_default_with_oids(bool *newval, void **extra, GucSource source);
-/* Private functions in guc-file.l that need to be called from guc.c */
-static ConfigVariable *ProcessConfigFileInternal(GucContext context,
- bool applySettings, int elevel);
-
/*
* Track whether there were any deferred checks for custom resource managers
* specified in wal_consistency_checking.
@@ -5160,8 +5156,8 @@ static bool report_needed; /* true if any GUC_REPORT reports are needed */
static int GUCNestLevel = 0; /* 1 when in main transaction */
+static struct config_generic *find_option(const char *name, bool create_placeholders, bool skip_errors, int elevel);
static int guc_var_compare(const void *a, const void *b);
-static int guc_name_compare(const char *namea, const char *nameb);
static void InitializeGUCOptionsFromEnvironment(void);
static void InitializeOneGUCOption(struct config_generic *gconf);
static void push_old_value(struct config_generic *gconf, GucAction action);
@@ -5180,7 +5176,359 @@ static bool validate_option_array_item(const char *name, const char *value,
static void write_auto_conf_file(int fd, const char *filename, ConfigVariable *head_p);
static void replace_auto_config_value(ConfigVariable **head_p, ConfigVariable **tail_p,
const char *name, const char *value);
+static bool valid_custom_variable_name(const char *name);
+
+/*
+ * This function handles both actual config file (re)loads and execution of
+ * show_all_file_settings() (i.e., the pg_file_settings view). In the latter
+ * case we don't apply any of the settings, but we make all the usual validity
+ * checks, and we return the ConfigVariable list so that it can be printed out
+ * by show_all_file_settings().
+ */
+ConfigVariable *
+ProcessConfigFileInternal(GucContext context, bool applySettings, int elevel)
+{
+ bool error = false;
+ bool applying = false;
+ const char *ConfFileWithError;
+ ConfigVariable *item,
+ *head,
+ *tail;
+ int i;
+
+ /* Parse the main config file into a list of option names and values */
+ ConfFileWithError = ConfigFileName;
+ head = tail = NULL;
+
+ if (!ParseConfigFile(ConfigFileName, true,
+ NULL, 0, 0, elevel,
+ &head, &tail))
+ {
+ /* Syntax error(s) detected in the file, so bail out */
+ error = true;
+ goto bail_out;
+ }
+
+ /*
+ * Parse the PG_AUTOCONF_FILENAME file, if present, after the main file to
+ * replace any parameters set by ALTER SYSTEM command. Because this file
+ * is in the data directory, we can't read it until the DataDir has been
+ * set.
+ */
+ if (DataDir)
+ {
+ if (!ParseConfigFile(PG_AUTOCONF_FILENAME, false,
+ NULL, 0, 0, elevel,
+ &head, &tail))
+ {
+ /* Syntax error(s) detected in the file, so bail out */
+ error = true;
+ ConfFileWithError = PG_AUTOCONF_FILENAME;
+ goto bail_out;
+ }
+ }
+ else
+ {
+ /*
+ * If DataDir is not set, the PG_AUTOCONF_FILENAME file cannot be
+ * read. In this case, we don't want to accept any settings but
+ * data_directory from postgresql.conf, because they might be
+ * overwritten with settings in the PG_AUTOCONF_FILENAME file which
+ * will be read later. OTOH, since data_directory isn't allowed in the
+ * PG_AUTOCONF_FILENAME file, it will never be overwritten later.
+ */
+ ConfigVariable *newlist = NULL;
+
+ /*
+ * Prune all items except the last "data_directory" from the list.
+ */
+ for (item = head; item; item = item->next)
+ {
+ if (!item->ignore &&
+ strcmp(item->name, "data_directory") == 0)
+ newlist = item;
+ }
+ if (newlist)
+ newlist->next = NULL;
+ head = tail = newlist;
+
+ /*
+ * Quick exit if data_directory is not present in file.
+ *
+ * We need not do any further processing, in particular we don't set
+ * PgReloadTime; that will be set soon by subsequent full loading of
+ * the config file.
+ */
+ if (head == NULL)
+ goto bail_out;
+ }
+
+ /*
+ * Mark all extant GUC variables as not present in the config file. We
+ * need this so that we can tell below which ones have been removed from
+ * the file since we last processed it.
+ */
+ for (i = 0; i < num_guc_variables; i++)
+ {
+ struct config_generic *gconf = guc_variables[i];
+
+ gconf->status &= ~GUC_IS_IN_FILE;
+ }
+
+ /*
+ * Check if all the supplied option names are valid, as an additional
+ * quasi-syntactic check on the validity of the config file. It is
+ * important that the postmaster and all backends agree on the results of
+ * this phase, else we will have strange inconsistencies about which
+ * processes accept a config file update and which don't. Hence, unknown
+ * custom variable names have to be accepted without complaint. For the
+ * same reason, we don't attempt to validate the options' values here.
+ *
+ * In addition, the GUC_IS_IN_FILE flag is set on each existing GUC
+ * variable mentioned in the file; and we detect duplicate entries in the
+ * file and mark the earlier occurrences as ignorable.
+ */
+ for (item = head; item; item = item->next)
+ {
+ struct config_generic *record;
+
+ /* Ignore anything already marked as ignorable */
+ if (item->ignore)
+ continue;
+
+ /*
+ * Try to find the variable; but do not create a custom placeholder if
+ * it's not there already.
+ */
+ record = find_option(item->name, false, true, elevel);
+
+ if (record)
+ {
+ /* If it's already marked, then this is a duplicate entry */
+ if (record->status & GUC_IS_IN_FILE)
+ {
+ /*
+ * Mark the earlier occurrence(s) as dead/ignorable. We could
+ * avoid the O(N^2) behavior here with some additional state,
+ * but it seems unlikely to be worth the trouble.
+ */
+ ConfigVariable *pitem;
+
+ for (pitem = head; pitem != item; pitem = pitem->next)
+ {
+ if (!pitem->ignore &&
+ strcmp(pitem->name, item->name) == 0)
+ pitem->ignore = true;
+ }
+ }
+ /* Now mark it as present in file */
+ record->status |= GUC_IS_IN_FILE;
+ }
+ else if (!valid_custom_variable_name(item->name))
+ {
+ /* Invalid non-custom variable, so complain */
+ ereport(elevel,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("unrecognized configuration parameter \"%s\" in file \"%s\" line %d",
+ item->name,
+ item->filename, item->sourceline)));
+ item->errmsg = pstrdup("unrecognized configuration parameter");
+ error = true;
+ ConfFileWithError = item->filename;
+ }
+ }
+
+ /*
+ * If we've detected any errors so far, we don't want to risk applying any
+ * changes.
+ */
+ if (error)
+ goto bail_out;
+
+ /* Otherwise, set flag that we're beginning to apply changes */
+ applying = true;
+
+ /*
+ * Check for variables having been removed from the config file, and
+ * revert their reset values (and perhaps also effective values) to the
+ * boot-time defaults. If such a variable can't be changed after startup,
+ * report that and continue.
+ */
+ for (i = 0; i < num_guc_variables; i++)
+ {
+ struct config_generic *gconf = guc_variables[i];
+ GucStack *stack;
+
+ if (gconf->reset_source != PGC_S_FILE ||
+ (gconf->status & GUC_IS_IN_FILE))
+ continue;
+ if (gconf->context < PGC_SIGHUP)
+ {
+ /* The removal can't be effective without a restart */
+ gconf->status |= GUC_PENDING_RESTART;
+ ereport(elevel,
+ (errcode(ERRCODE_CANT_CHANGE_RUNTIME_PARAM),
+ errmsg("parameter \"%s\" cannot be changed without restarting the server",
+ gconf->name)));
+ record_config_file_error(psprintf("parameter \"%s\" cannot be changed without restarting the server",
+ gconf->name),
+ NULL, 0,
+ &head, &tail);
+ error = true;
+ continue;
+ }
+
+ /* No more to do if we're just doing show_all_file_settings() */
+ if (!applySettings)
+ continue;
+
+ /*
+ * Reset any "file" sources to "default", else set_config_option will
+ * not override those settings.
+ */
+ if (gconf->reset_source == PGC_S_FILE)
+ gconf->reset_source = PGC_S_DEFAULT;
+ if (gconf->source == PGC_S_FILE)
+ gconf->source = PGC_S_DEFAULT;
+ for (stack = gconf->stack; stack; stack = stack->prev)
+ {
+ if (stack->source == PGC_S_FILE)
+ stack->source = PGC_S_DEFAULT;
+ }
+
+ /* Now we can re-apply the wired-in default (i.e., the boot_val) */
+ if (set_config_option(gconf->name, NULL,
+ context, PGC_S_DEFAULT,
+ GUC_ACTION_SET, true, 0, false) > 0)
+ {
+ /* Log the change if appropriate */
+ if (context == PGC_SIGHUP)
+ ereport(elevel,
+ (errmsg("parameter \"%s\" removed from configuration file, reset to default",
+ gconf->name)));
+ }
+ }
+
+ /*
+ * Restore any variables determined by environment variables or
+ * dynamically-computed defaults. This is a no-op except in the case
+ * where one of these had been in the config file and is now removed.
+ *
+ * In particular, we *must not* do this during the postmaster's initial
+ * loading of the file, since the timezone functions in particular should
+ * be run only after initialization is complete.
+ *
+ * XXX this is an unmaintainable crock, because we have to know how to set
+ * (or at least what to call to set) every non-PGC_INTERNAL variable that
+ * could potentially have PGC_S_DYNAMIC_DEFAULT or PGC_S_ENV_VAR source.
+ */
+ if (context == PGC_SIGHUP && applySettings)
+ {
+ InitializeGUCOptionsFromEnvironment();
+ pg_timezone_abbrev_initialize();
+ /* this selects SQL_ASCII in processes not connected to a database */
+ SetConfigOption("client_encoding", GetDatabaseEncodingName(),
+ PGC_BACKEND, PGC_S_DYNAMIC_DEFAULT);
+ }
+
+ /*
+ * Now apply the values from the config file.
+ */
+ for (item = head; item; item = item->next)
+ {
+ char *pre_value = NULL;
+ int scres;
+
+ /* Ignore anything marked as ignorable */
+ if (item->ignore)
+ continue;
+
+ /* In SIGHUP cases in the postmaster, we want to report changes */
+ if (context == PGC_SIGHUP && applySettings && !IsUnderPostmaster)
+ {
+ const char *preval = GetConfigOption(item->name, true, false);
+
+ /* If option doesn't exist yet or is NULL, treat as empty string */
+ if (!preval)
+ preval = "";
+ /* must dup, else might have dangling pointer below */
+ pre_value = pstrdup(preval);
+ }
+
+ scres = set_config_option(item->name, item->value,
+ context, PGC_S_FILE,
+ GUC_ACTION_SET, applySettings, 0, false);
+ if (scres > 0)
+ {
+ /* variable was updated, so log the change if appropriate */
+ if (pre_value)
+ {
+ const char *post_value = GetConfigOption(item->name, true, false);
+
+ if (!post_value)
+ post_value = "";
+ if (strcmp(pre_value, post_value) != 0)
+ ereport(elevel,
+ (errmsg("parameter \"%s\" changed to \"%s\"",
+ item->name, item->value)));
+ }
+ item->applied = true;
+ }
+ else if (scres == 0)
+ {
+ error = true;
+ item->errmsg = pstrdup("setting could not be applied");
+ ConfFileWithError = item->filename;
+ }
+ else
+ {
+ /* no error, but variable's active value was not changed */
+ item->applied = true;
+ }
+
+ /*
+ * We should update source location unless there was an error, since
+ * even if the active value didn't change, the reset value might have.
+ * (In the postmaster, there won't be a difference, but it does matter
+ * in backends.)
+ */
+ if (scres != 0 && applySettings)
+ set_config_sourcefile(item->name, item->filename,
+ item->sourceline);
+
+ if (pre_value)
+ pfree(pre_value);
+ }
+
+ /* Remember when we last successfully loaded the config file. */
+ if (applySettings)
+ PgReloadTime = GetCurrentTimestamp();
+
+bail_out:
+ if (error && applySettings)
+ {
+ /* During postmaster startup, any error is fatal */
+ if (context == PGC_POSTMASTER)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors",
+ ConfFileWithError)));
+ else if (applying)
+ ereport(elevel,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
+ ConfFileWithError)));
+ else
+ ereport(elevel,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("configuration file \"%s\" contains errors; no changes were applied",
+ ConfFileWithError)));
+ }
+
+ /* Successful or otherwise, return the collected data list */
+ return head;
+}
/*
* Some infrastructure for checking malloc/strdup/realloc calls
@@ -5737,7 +6085,7 @@ guc_var_compare(const void *a, const void *b)
/*
* the bare comparison function for GUC names
*/
-static int
+int
guc_name_compare(const char *namea, const char *nameb)
{
/*
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index e734493a48..aae071cd82 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -442,6 +442,15 @@ extern void GUC_check_errcode(int sqlerrcode);
pre_format_elog_string(errno, TEXTDOMAIN), \
GUC_check_errhint_string = format_elog_string
+/* functions shared between guc.c and guc-file.l */
+extern int guc_name_compare(const char *namea, const char *nameb);
+extern ConfigVariable *ProcessConfigFileInternal(GucContext context,
+ bool applySettings, int elevel);
+extern void record_config_file_error(const char *errmsg,
+ const char *config_file,
+ int lineno,
+ ConfigVariable **head_p,
+ ConfigVariable **tail_p);
/*
* The following functions are not in guc.c, but are declared here to avoid
--
2.36.1
[text/x-patch] v4-0003-Build-guc-file.c-standalone.patch (2.5K, ../../CAFBsxsHdAP0U3AjYuJ_SpWBxj2vmpdqXX3Yg=TLBrq0rAWhFgg@mail.gmail.com/4-v4-0003-Build-guc-file.c-standalone.patch)
download | inline diff:
From 65c0203bac3abf127a5900f1a0025b7a62a52d70 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Thu, 11 Aug 2022 19:38:37 +0700
Subject: [PATCH v4 03/11] Build guc-file.c standalone
The proposed Meson build system will need a way to ignore certain
generated files in order to coexist with the autoconf build system,
and #include'd C files generated by Flex make this more difficult.
Build guc-file.c separately from guc.c, as was done in 72b1e3a21.
Reviewed by Andres Freund
Discussion: https://www.postgresql.org/message-id/20220810171935.7k5zgnjwqzalzmtm%40awork3.anarazel.de
---
src/backend/utils/misc/Makefile | 5 +----
src/backend/utils/misc/guc-file.l | 9 +++++----
src/backend/utils/misc/guc.c | 2 --
3 files changed, 6 insertions(+), 10 deletions(-)
diff --git a/src/backend/utils/misc/Makefile b/src/backend/utils/misc/Makefile
index 1d5327cf64..cf7ce9bc83 100644
--- a/src/backend/utils/misc/Makefile
+++ b/src/backend/utils/misc/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
OBJS = \
guc.o \
+ guc-file.o \
help_config.o \
pg_config.o \
pg_controldata.o \
@@ -37,10 +38,6 @@ endif
include $(top_srcdir)/src/backend/common.mk
-# guc-file is compiled as part of guc
-guc.o: guc-file.c
-
# Note: guc-file.c is not deleted by 'make clean',
# since we want to ship it in distribution tarballs.
clean:
- @rm -f lex.yy.c
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index 843838b1df..9aa3abe1ba 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -1,4 +1,4 @@
-/* -*-pgsql-c-*- */
+%top{
/*
* Scanner for the configuration file
*
@@ -7,8 +7,6 @@
* src/backend/utils/misc/guc-file.l
*/
-%{
-
#include "postgres.h"
#include <ctype.h>
@@ -17,10 +15,13 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "storage/fd.h"
+#include <sys/stat.h>
#include "utils/guc.h"
#include "utils/guc_internal.h"
+#include "utils/memutils.h"
+}
-
+%{
/*
* flex emits a yy_fatal_error() function that it calls in response to
* critical errors like malloc failure, file I/O errors, and detection of
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 293834fc13..92b5b18c8f 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -13333,5 +13333,3 @@ check_default_with_oids(bool *newval, void **extra, GucSource source)
return true;
}
-
-#include "guc-file.c"
--
2.36.1
[text/x-patch] v4-0005-Build-repl_scanner.c-standalone.patch (6.1K, ../../CAFBsxsHdAP0U3AjYuJ_SpWBxj2vmpdqXX3Yg=TLBrq0rAWhFgg@mail.gmail.com/5-v4-0005-Build-repl_scanner.c-standalone.patch)
download | inline diff:
From 67d63a9e186e6e393e344627c5b70f2dd7b8609e Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Fri, 12 Aug 2022 17:09:45 +0700
Subject: [PATCH v4 05/11] Build repl_scanner.c standalone
---
src/backend/Makefile | 3 ++-
src/backend/replication/.gitignore | 1 +
src/backend/replication/Makefile | 11 +++++++--
src/backend/replication/repl_gram.y | 2 --
src/backend/replication/repl_scanner.l | 31 +++++++++++++++++---------
src/tools/pginclude/headerscheck | 1 +
6 files changed, 33 insertions(+), 16 deletions(-)
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 5a12666918..f527659a7b 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -183,7 +183,7 @@ distprep:
$(MAKE) -C bootstrap bootparse.c bootparse.h bootscanner.c
$(MAKE) -C catalog distprep
$(MAKE) -C nodes distprep
- $(MAKE) -C replication repl_gram.c repl_scanner.c syncrep_gram.c syncrep_scanner.c
+ $(MAKE) -C replication repl_gram.c repl_gram.h repl_scanner.c syncrep_gram.c syncrep_scanner.c
$(MAKE) -C storage/lmgr lwlocknames.h lwlocknames.c
$(MAKE) -C utils distprep
$(MAKE) -C utils/adt jsonpath_gram.c jsonpath_scan.c
@@ -304,6 +304,7 @@ maintainer-clean: distclean
parser/gram.h \
parser/scan.c \
replication/repl_gram.c \
+ replication/repl_gram.h \
replication/repl_scanner.c \
replication/syncrep_gram.c \
replication/syncrep_scanner.c \
diff --git a/src/backend/replication/.gitignore b/src/backend/replication/.gitignore
index d1df6147bd..a5f600232f 100644
--- a/src/backend/replication/.gitignore
+++ b/src/backend/replication/.gitignore
@@ -1,3 +1,4 @@
+/repl_gram.h
/repl_gram.c
/repl_scanner.c
/syncrep_gram.c
diff --git a/src/backend/replication/Makefile b/src/backend/replication/Makefile
index 2bffac58c0..bc8170418f 100644
--- a/src/backend/replication/Makefile
+++ b/src/backend/replication/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
OBJS = \
repl_gram.o \
+ repl_scanner.o \
slot.o \
slotfuncs.o \
syncrep.o \
@@ -28,8 +29,14 @@ SUBDIRS = logical
include $(top_srcdir)/src/backend/common.mk
-# repl_scanner is compiled as part of repl_gram
-repl_gram.o: repl_scanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+repl_gram.h: repl_gram.c
+ touch $@
+
+repl_gram.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+repl_gram.o repl_scanner.o: repl_gram.h
# syncrep_scanner is compiled as part of syncrep_gram
syncrep_gram.o: syncrep_scanner.c
diff --git a/src/backend/replication/repl_gram.y b/src/backend/replication/repl_gram.y
index 4cf087e602..b343f108d3 100644
--- a/src/backend/replication/repl_gram.y
+++ b/src/backend/replication/repl_gram.y
@@ -416,5 +416,3 @@ ident_or_keyword:
;
%%
-
-#include "repl_scanner.c"
diff --git a/src/backend/replication/repl_scanner.l b/src/backend/replication/repl_scanner.l
index 586f0d3a5c..23fcb2a11d 100644
--- a/src/backend/replication/repl_scanner.l
+++ b/src/backend/replication/repl_scanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* repl_scanner.l
@@ -18,6 +18,15 @@
#include "utils/builtins.h"
#include "parser/scansup.h"
+/*
+ * NB: include repl_gram.h only AFTER including walsender_private.h, because
+ * walsender_private includes headers that define XLogRecPtr.
+ */
+#include "replication/walsender_private.h"
+#include "repl_gram.h"
+}
+
+%{
/* Avoid exit() on fatal scanner errors (a bit ugly -- see yy_fatal_error) */
#undef fprintf
#define fprintf(file, fmt, msg) fprintf_to_ereport(fmt, msg)
@@ -130,7 +139,7 @@ WAIT { return K_WAIT; }
{space}+ { /* do nothing */ }
{digit}+ {
- yylval.uintval = strtoul(yytext, NULL, 10);
+ replication_yylval.uintval = strtoul(yytext, NULL, 10);
return UCONST;
}
@@ -138,8 +147,8 @@ WAIT { return K_WAIT; }
uint32 hi,
lo;
if (sscanf(yytext, "%X/%X", &hi, &lo) != 2)
- yyerror("invalid streaming start location");
- yylval.recptr = ((uint64) hi) << 32 | lo;
+ replication_yyerror("invalid streaming start location");
+ replication_yylval.recptr = ((uint64) hi) << 32 | lo;
return RECPTR;
}
@@ -151,7 +160,7 @@ WAIT { return K_WAIT; }
<xq>{quotestop} {
yyless(1);
BEGIN(INITIAL);
- yylval.str = litbufdup();
+ replication_yylval.str = litbufdup();
return SCONST;
}
@@ -173,9 +182,9 @@ WAIT { return K_WAIT; }
yyless(1);
BEGIN(INITIAL);
- yylval.str = litbufdup();
- len = strlen(yylval.str);
- truncate_identifier(yylval.str, len, true);
+ replication_yylval.str = litbufdup();
+ len = strlen(replication_yylval.str);
+ truncate_identifier(replication_yylval.str, len, true);
return IDENT;
}
@@ -186,7 +195,7 @@ WAIT { return K_WAIT; }
{identifier} {
int len = strlen(yytext);
- yylval.str = downcase_truncate_identifier(yytext, len, true);
+ replication_yylval.str = downcase_truncate_identifier(yytext, len, true);
return IDENT;
}
@@ -195,7 +204,7 @@ WAIT { return K_WAIT; }
return yytext[0];
}
-<xq,xd><<EOF>> { yyerror("unterminated quoted string"); }
+<xq,xd><<EOF>> { replication_yyerror("unterminated quoted string"); }
<<EOF>> {
@@ -231,7 +240,7 @@ addlitchar(unsigned char ychar)
}
void
-yyerror(const char *message)
+replication_yyerror(const char *message)
{
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index ce33ec1f68..917ef29c15 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -120,6 +120,7 @@ do
test "$f" = src/include/parser/gram.h && continue
test "$f" = src/backend/parser/gram.h && continue
test "$f" = src/backend/bootstrap/bootparse.h && continue
+ test "$f" = src/backend/replication/repl_gram.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
test "$f" = src/interfaces/ecpg/preproc/preproc.h && continue
--
2.36.1
[text/x-patch] v4-0004-Build-bootscanner.c-standalone.patch (7.4K, ../../CAFBsxsHdAP0U3AjYuJ_SpWBxj2vmpdqXX3Yg=TLBrq0rAWhFgg@mail.gmail.com/6-v4-0004-Build-bootscanner.c-standalone.patch)
download | inline diff:
From e85b6441ca47566f8edd933e4145bfb2c8f553fa Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Fri, 12 Aug 2022 15:45:24 +0700
Subject: [PATCH v4 04/11] Build bootscanner.c standalone
---
src/backend/Makefile | 3 +-
src/backend/bootstrap/.gitignore | 1 +
src/backend/bootstrap/Makefile | 11 +++++-
src/backend/bootstrap/bootparse.y | 2 -
src/backend/bootstrap/bootscanner.l | 60 ++++++++++++++++-------------
src/tools/pginclude/headerscheck | 1 +
6 files changed, 46 insertions(+), 32 deletions(-)
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 3f01c65592..5a12666918 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -180,7 +180,7 @@ utils/probes.o: utils/probes.d $(SUBDIROBJS)
# Be sure that these files get removed by the maintainer-clean target
distprep:
$(MAKE) -C parser gram.c gram.h scan.c
- $(MAKE) -C bootstrap bootparse.c bootscanner.c
+ $(MAKE) -C bootstrap bootparse.c bootparse.h bootscanner.c
$(MAKE) -C catalog distprep
$(MAKE) -C nodes distprep
$(MAKE) -C replication repl_gram.c repl_scanner.c syncrep_gram.c syncrep_scanner.c
@@ -298,6 +298,7 @@ maintainer-clean: distclean
$(MAKE) -C nodes $@
$(MAKE) -C utils $@
rm -f bootstrap/bootparse.c \
+ bootstrap/bootparse.h \
bootstrap/bootscanner.c \
parser/gram.c \
parser/gram.h \
diff --git a/src/backend/bootstrap/.gitignore b/src/backend/bootstrap/.gitignore
index 1ffe8ca39e..6351b920fd 100644
--- a/src/backend/bootstrap/.gitignore
+++ b/src/backend/bootstrap/.gitignore
@@ -1,2 +1,3 @@
+/bootparse.h
/bootparse.c
/bootscanner.c
diff --git a/src/backend/bootstrap/Makefile b/src/backend/bootstrap/Makefile
index 6421efb227..606c8021e7 100644
--- a/src/backend/bootstrap/Makefile
+++ b/src/backend/bootstrap/Makefile
@@ -14,12 +14,19 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
OBJS = \
bootparse.o \
+ bootscanner.o \
bootstrap.o
include $(top_srcdir)/src/backend/common.mk
-# bootscanner is compiled as part of bootparse
-bootparse.o: bootscanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+bootparse.h: bootparse.c
+ touch $@
+
+bootparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+bootparse.o bootscanner.o: bootparse.h
# bootparse.c and bootscanner.c are in the distribution tarball, so
# they are not cleaned here.
diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y
index 7d7655d295..c45ddde67f 100644
--- a/src/backend/bootstrap/bootparse.y
+++ b/src/backend/bootstrap/bootparse.y
@@ -488,5 +488,3 @@ boot_ident:
| XNULL { $$ = pstrdup($1); }
;
%%
-
-#include "bootscanner.c"
diff --git a/src/backend/bootstrap/bootscanner.l b/src/backend/bootstrap/bootscanner.l
index 3094ccb93f..d6eae84816 100644
--- a/src/backend/bootstrap/bootscanner.l
+++ b/src/backend/bootstrap/bootscanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* bootscanner.l
@@ -15,11 +15,17 @@
*/
#include "postgres.h"
+/*
+ * NB: include bootparse.h only AFTER including bootstrap.h, because bootstrap.h
+ * includes node definitions needed for YYSTYPE.
+ */
#include "bootstrap/bootstrap.h"
+#include "bootparse.h"
#include "utils/guc.h"
-/* Not needed now that this file is compiled as part of bootparse. */
-/* #include "bootparse.h" */
+}
+
+%{
/* LCOV_EXCL_START */
@@ -52,7 +58,7 @@ id [-A-Za-z0-9_]+
sid \'([^']|\'\')*\'
/*
- * Keyword tokens return the keyword text (as a constant string) in yylval.kw,
+ * Keyword tokens return the keyword text (as a constant string) in boot_yylval.kw,
* just in case that's needed because we want to treat the keyword as an
* unreserved identifier. Note that _null_ is not treated as a keyword
* for this purpose; it's the one "reserved word" in the bootstrap syntax.
@@ -60,23 +66,23 @@ sid \'([^']|\'\')*\'
* Notice that all the keywords are case-sensitive, and for historical
* reasons some must be upper case.
*
- * String tokens return a palloc'd string in yylval.str.
+ * String tokens return a palloc'd string in boot_yylval.str.
*/
%%
-open { yylval.kw = "open"; return OPEN; }
+open { boot_yylval.kw = "open"; return OPEN; }
-close { yylval.kw = "close"; return XCLOSE; }
+close { boot_yylval.kw = "close"; return XCLOSE; }
-create { yylval.kw = "create"; return XCREATE; }
+create { boot_yylval.kw = "create"; return XCREATE; }
-OID { yylval.kw = "OID"; return OBJ_ID; }
-bootstrap { yylval.kw = "bootstrap"; return XBOOTSTRAP; }
-shared_relation { yylval.kw = "shared_relation"; return XSHARED_RELATION; }
-rowtype_oid { yylval.kw = "rowtype_oid"; return XROWTYPE_OID; }
+OID { boot_yylval.kw = "OID"; return OBJ_ID; }
+bootstrap { boot_yylval.kw = "bootstrap"; return XBOOTSTRAP; }
+shared_relation { boot_yylval.kw = "shared_relation"; return XSHARED_RELATION; }
+rowtype_oid { boot_yylval.kw = "rowtype_oid"; return XROWTYPE_OID; }
-insert { yylval.kw = "insert"; return INSERT_TUPLE; }
+insert { boot_yylval.kw = "insert"; return INSERT_TUPLE; }
_null_ { return NULLVAL; }
@@ -90,25 +96,25 @@ _null_ { return NULLVAL; }
^\#[^\n]* ; /* drop everything after "#" for comments */
-declare { yylval.kw = "declare"; return XDECLARE; }
-build { yylval.kw = "build"; return XBUILD; }
-indices { yylval.kw = "indices"; return INDICES; }
-unique { yylval.kw = "unique"; return UNIQUE; }
-index { yylval.kw = "index"; return INDEX; }
-on { yylval.kw = "on"; return ON; }
-using { yylval.kw = "using"; return USING; }
-toast { yylval.kw = "toast"; return XTOAST; }
-FORCE { yylval.kw = "FORCE"; return XFORCE; }
-NOT { yylval.kw = "NOT"; return XNOT; }
-NULL { yylval.kw = "NULL"; return XNULL; }
+declare { boot_yylval.kw = "declare"; return XDECLARE; }
+build { boot_yylval.kw = "build"; return XBUILD; }
+indices { boot_yylval.kw = "indices"; return INDICES; }
+unique { boot_yylval.kw = "unique"; return UNIQUE; }
+index { boot_yylval.kw = "index"; return INDEX; }
+on { boot_yylval.kw = "on"; return ON; }
+using { boot_yylval.kw = "using"; return USING; }
+toast { boot_yylval.kw = "toast"; return XTOAST; }
+FORCE { boot_yylval.kw = "FORCE"; return XFORCE; }
+NOT { boot_yylval.kw = "NOT"; return XNOT; }
+NULL { boot_yylval.kw = "NULL"; return XNULL; }
{id} {
- yylval.str = pstrdup(yytext);
+ boot_yylval.str = pstrdup(yytext);
return ID;
}
{sid} {
/* strip quotes and escapes */
- yylval.str = DeescapeQuotedString(yytext);
+ boot_yylval.str = DeescapeQuotedString(yytext);
return ID;
}
@@ -121,7 +127,7 @@ NULL { yylval.kw = "NULL"; return XNULL; }
/* LCOV_EXCL_STOP */
void
-yyerror(const char *message)
+boot_yyerror(const char *message)
{
elog(ERROR, "%s at line %d", message, yyline);
}
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index b8419e46a4..ce33ec1f68 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -119,6 +119,7 @@ do
# parser/gram.h will be included by parser/gramparse.h anyway.
test "$f" = src/include/parser/gram.h && continue
test "$f" = src/backend/parser/gram.h && continue
+ test "$f" = src/backend/bootstrap/bootparse.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
test "$f" = src/interfaces/ecpg/preproc/preproc.h && continue
--
2.36.1
[text/x-patch] v4-0008-Build-exprscan.c-standalone.patch (4.1K, ../../CAFBsxsHdAP0U3AjYuJ_SpWBxj2vmpdqXX3Yg=TLBrq0rAWhFgg@mail.gmail.com/7-v4-0008-Build-exprscan.c-standalone.patch)
download | inline diff:
From 8a39bfb75a69847dd6d1839091f71f36fc9579ba Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 13:35:14 +0700
Subject: [PATCH v4 08/11] Build exprscan.c standalone
---
src/bin/pgbench/.gitignore | 1 +
src/bin/pgbench/Makefile | 13 ++++++++++---
src/bin/pgbench/exprparse.y | 15 ---------------
src/bin/pgbench/exprscan.l | 12 +++++++++++-
src/tools/pginclude/headerscheck | 1 +
5 files changed, 23 insertions(+), 19 deletions(-)
diff --git a/src/bin/pgbench/.gitignore b/src/bin/pgbench/.gitignore
index 983a3cd7a6..07492a993c 100644
--- a/src/bin/pgbench/.gitignore
+++ b/src/bin/pgbench/.gitignore
@@ -1,3 +1,4 @@
+/exprparse.h
/exprparse.c
/exprscan.c
/pgbench
diff --git a/src/bin/pgbench/Makefile b/src/bin/pgbench/Makefile
index f402fe7b91..6647c9fe97 100644
--- a/src/bin/pgbench/Makefile
+++ b/src/bin/pgbench/Makefile
@@ -10,6 +10,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = \
$(WIN32RES) \
exprparse.o \
+ exprscan.o \
pgbench.o
override CPPFLAGS := -I. -I$(srcdir) -I$(libpq_srcdir) $(CPPFLAGS)
@@ -26,8 +27,14 @@ all: pgbench
pgbench: $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-# exprscan is compiled as part of exprparse
-exprparse.o: exprscan.c
+# See notes in src/backend/parser/Makefile about the following two rules
+exprparse.h: exprparse.c
+ touch $@
+
+exprparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+exprparse.o exprscan.o: exprparse.h
distprep: exprparse.c exprscan.c
@@ -45,7 +52,7 @@ clean distclean:
rm -rf tmp_check
maintainer-clean: distclean
- rm -f exprparse.c exprscan.c
+ rm -f exprparse.h exprparse.c exprscan.c
check:
$(prove_check)
diff --git a/src/bin/pgbench/exprparse.y b/src/bin/pgbench/exprparse.y
index b5592d4b97..ade2ecdaab 100644
--- a/src/bin/pgbench/exprparse.y
+++ b/src/bin/pgbench/exprparse.y
@@ -526,18 +526,3 @@ make_case(yyscan_t yyscanner, PgBenchExprList *when_then_list, PgBenchExpr *else
find_func(yyscanner, "!case_end"),
make_elist(else_part, when_then_list));
}
-
-/*
- * exprscan.l is compiled as part of exprparse.y. Currently, this is
- * unavoidable because exprparse does not create a .h file to export
- * its token symbols. If these files ever grow large enough to be
- * worth compiling separately, that could be fixed; but for now it
- * seems like useless complication.
- */
-
-/* First, get rid of "#define yyscan_t" from pgbench.h */
-#undef yyscan_t
-/* ... and the yylval macro, which flex will have its own definition for */
-#undef yylval
-
-#include "exprscan.c"
diff --git a/src/bin/pgbench/exprscan.l b/src/bin/pgbench/exprscan.l
index 4f63818606..fe8e32838a 100644
--- a/src/bin/pgbench/exprscan.l
+++ b/src/bin/pgbench/exprscan.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* exprscan.l
@@ -22,9 +22,19 @@
*
*-------------------------------------------------------------------------
*/
+#include "postgres_fe.h"
+/*
+ * NB: include exprparse.h only AFTER including pgbench.h, because pgbench.h
+ * contains definitions needed for YYSTYPE. Likewise, pgbench.h must come after
+ * psqlscan_int.h for yyscan_t.
+ */
#include "fe_utils/psqlscan_int.h"
+#include "pgbench.h"
+#include "exprparse.h"
+}
+%{
/* context information for reporting errors in expressions */
static const char *expr_source = NULL;
static int expr_lineno = 0;
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index c1b1f3bd2a..1a56a74cda 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -123,6 +123,7 @@ do
test "$f" = src/backend/replication/repl_gram.h && continue
test "$f" = src/backend/replication/syncrep_gram.h && continue
test "$f" = src/test/isolation/specparse.h && continue
+ test "$f" = src/bin/pgbench/exprparse.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
test "$f" = src/interfaces/ecpg/preproc/preproc.h && continue
--
2.36.1
[text/x-patch] v4-0006-Build-syncrep_scanner.c-standalone.patch (5.2K, ../../CAFBsxsHdAP0U3AjYuJ_SpWBxj2vmpdqXX3Yg=TLBrq0rAWhFgg@mail.gmail.com/8-v4-0006-Build-syncrep_scanner.c-standalone.patch)
download | inline diff:
From b8f2e6a055af9f2bffd2078364077fe6d2942e53 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 15:02:30 +0700
Subject: [PATCH v4 06/11] Build syncrep_scanner.c standalone
---
src/backend/Makefile | 3 ++-
src/backend/replication/.gitignore | 1 +
src/backend/replication/Makefile | 11 +++++++++--
src/backend/replication/syncrep_gram.y | 2 --
src/backend/replication/syncrep_scanner.l | 21 +++++++++++++++------
src/tools/pginclude/headerscheck | 1 +
6 files changed, 28 insertions(+), 11 deletions(-)
diff --git a/src/backend/Makefile b/src/backend/Makefile
index f527659a7b..86cbe03677 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -183,7 +183,7 @@ distprep:
$(MAKE) -C bootstrap bootparse.c bootparse.h bootscanner.c
$(MAKE) -C catalog distprep
$(MAKE) -C nodes distprep
- $(MAKE) -C replication repl_gram.c repl_gram.h repl_scanner.c syncrep_gram.c syncrep_scanner.c
+ $(MAKE) -C replication repl_gram.c repl_gram.h repl_scanner.c syncrep_gram.c syncrep_gram.h syncrep_scanner.c
$(MAKE) -C storage/lmgr lwlocknames.h lwlocknames.c
$(MAKE) -C utils distprep
$(MAKE) -C utils/adt jsonpath_gram.c jsonpath_scan.c
@@ -307,6 +307,7 @@ maintainer-clean: distclean
replication/repl_gram.h \
replication/repl_scanner.c \
replication/syncrep_gram.c \
+ replication/syncrep_gram.h \
replication/syncrep_scanner.c \
storage/lmgr/lwlocknames.c \
storage/lmgr/lwlocknames.h \
diff --git a/src/backend/replication/.gitignore b/src/backend/replication/.gitignore
index a5f600232f..77d5a51068 100644
--- a/src/backend/replication/.gitignore
+++ b/src/backend/replication/.gitignore
@@ -1,5 +1,6 @@
/repl_gram.h
/repl_gram.c
/repl_scanner.c
+/syncrep_gram.h
/syncrep_gram.c
/syncrep_scanner.c
diff --git a/src/backend/replication/Makefile b/src/backend/replication/Makefile
index bc8170418f..23f29ba545 100644
--- a/src/backend/replication/Makefile
+++ b/src/backend/replication/Makefile
@@ -21,6 +21,7 @@ OBJS = \
slotfuncs.o \
syncrep.o \
syncrep_gram.o \
+ syncrep_scanner.o \
walreceiver.o \
walreceiverfuncs.o \
walsender.o
@@ -38,8 +39,14 @@ repl_gram.c: BISONFLAGS += -d
# Force these dependencies to be known even without dependency info built:
repl_gram.o repl_scanner.o: repl_gram.h
-# syncrep_scanner is compiled as part of syncrep_gram
-syncrep_gram.o: syncrep_scanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+syncrep_gram.h: syncrep_gram.c
+ touch $@
+
+syncrep_gram.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+syncrep_gram.o syncrep_scanner.o: syncrep_gram.h
# repl_gram.c, repl_scanner.c, syncrep_gram.c and syncrep_scanner.c
# are in the distribution tarball, so they are not cleaned here.
diff --git a/src/backend/replication/syncrep_gram.y b/src/backend/replication/syncrep_gram.y
index d932f2cda3..4fc3647da1 100644
--- a/src/backend/replication/syncrep_gram.y
+++ b/src/backend/replication/syncrep_gram.y
@@ -112,5 +112,3 @@ create_syncrep_config(const char *num_sync, List *members, uint8 syncrep_method)
return config;
}
-
-#include "syncrep_scanner.c"
diff --git a/src/backend/replication/syncrep_scanner.l b/src/backend/replication/syncrep_scanner.l
index 1952c8c6e0..bdb1a3391c 100644
--- a/src/backend/replication/syncrep_scanner.l
+++ b/src/backend/replication/syncrep_scanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* syncrep_scanner.l
@@ -17,6 +17,15 @@
#include "lib/stringinfo.h"
+/*
+ * NB: include syncrep_gram.h only AFTER including syncrep.h, because syncrep.h
+ * includes node definitions needed for YYSTYPE.
+ */
+#include "replication/syncrep.h"
+#include "syncrep_gram.h"
+}
+
+%{
/* Avoid exit() on fatal scanner errors (a bit ugly -- see yy_fatal_error) */
#undef fprintf
#define fprintf(file, fmt, msg) fprintf_to_ereport(fmt, msg)
@@ -82,28 +91,28 @@ xdinside [^"]+
appendStringInfoString(&xdbuf, yytext);
}
<xd>{xdstop} {
- yylval.str = xdbuf.data;
+ syncrep_yylval.str = xdbuf.data;
xdbuf.data = NULL;
BEGIN(INITIAL);
return NAME;
}
<xd><<EOF>> {
- yyerror("unterminated quoted identifier");
+ syncrep_yyerror("unterminated quoted identifier");
return JUNK;
}
{identifier} {
- yylval.str = pstrdup(yytext);
+ syncrep_yylval.str = pstrdup(yytext);
return NAME;
}
{digit}+ {
- yylval.str = pstrdup(yytext);
+ syncrep_yylval.str = pstrdup(yytext);
return NUM;
}
"*" {
- yylval.str = "*";
+ syncrep_yylval.str = "*";
return NAME;
}
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 917ef29c15..1e953d503f 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -121,6 +121,7 @@ do
test "$f" = src/backend/parser/gram.h && continue
test "$f" = src/backend/bootstrap/bootparse.h && continue
test "$f" = src/backend/replication/repl_gram.h && continue
+ test "$f" = src/backend/replication/syncrep_gram.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
test "$f" = src/interfaces/ecpg/preproc/preproc.h && continue
--
2.36.1
[text/x-patch] v4-0009-Build-cubescan.c-standalone.patch (7.5K, ../../CAFBsxsHdAP0U3AjYuJ_SpWBxj2vmpdqXX3Yg=TLBrq0rAWhFgg@mail.gmail.com/9-v4-0009-Build-cubescan.c-standalone.patch)
download | inline diff:
From 4ec5efdae1b02b4e0a34d5deffa5dcbb119445be Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 11:18:06 +0700
Subject: [PATCH v4 09/11] Build cubescan.c standalone
Pass scanbuflen as a parameter to yyparse rather than
resorting to a global variable.
---
contrib/cube/.gitignore | 1 +
contrib/cube/Makefile | 16 +++++++-----
contrib/cube/cube.c | 6 ++---
contrib/cube/cubedata.h | 6 ++---
contrib/cube/cubeparse.y | 8 ++----
contrib/cube/cubescan.l | 44 ++++++++++++++++++++------------
src/tools/pginclude/headerscheck | 1 +
7 files changed, 47 insertions(+), 35 deletions(-)
diff --git a/contrib/cube/.gitignore b/contrib/cube/.gitignore
index cb4c989fff..f788440c79 100644
--- a/contrib/cube/.gitignore
+++ b/contrib/cube/.gitignore
@@ -1,3 +1,4 @@
+/cubeparse.h
/cubeparse.c
/cubescan.c
# Generated subdirectories
diff --git a/contrib/cube/Makefile b/contrib/cube/Makefile
index cf195506c7..4fd19aac35 100644
--- a/contrib/cube/Makefile
+++ b/contrib/cube/Makefile
@@ -4,7 +4,8 @@ MODULE_big = cube
OBJS = \
$(WIN32RES) \
cube.o \
- cubeparse.o
+ cubeparse.o \
+ cubescan.o
EXTENSION = cube
DATA = cube--1.2.sql cube--1.2--1.3.sql cube--1.3--1.4.sql cube--1.4--1.5.sql \
@@ -15,8 +16,6 @@ HEADERS = cubedata.h
REGRESS = cube cube_sci
-EXTRA_CLEAN = y.tab.c y.tab.h
-
SHLIB_LINK += $(filter -lm, $(LIBS))
ifdef USE_PGXS
@@ -30,11 +29,16 @@ include $(top_builddir)/src/Makefile.global
include $(top_srcdir)/contrib/contrib-global.mk
endif
+# See notes in src/backend/parser/Makefile about the following two rules
+cubeparse.h: cubeparse.c
+ touch $@
+
+cubeparse.c: BISONFLAGS += -d
-# cubescan is compiled as part of cubeparse
-cubeparse.o: cubescan.c
+# Force these dependencies to be known even without dependency info built:
+cubeparse.o cubescan.o: cubeparse.h
distprep: cubeparse.c cubescan.c
maintainer-clean:
- rm -f cubeparse.c cubescan.c
+ rm -f cubeparse.h cubeparse.c cubescan.c
diff --git a/contrib/cube/cube.c b/contrib/cube/cube.c
index a5d1ba6733..6e01800a4a 100644
--- a/contrib/cube/cube.c
+++ b/contrib/cube/cube.c
@@ -119,11 +119,11 @@ cube_in(PG_FUNCTION_ARGS)
{
char *str = PG_GETARG_CSTRING(0);
NDBOX *result;
+ Size scanbuflen;
- cube_scanner_init(str);
+ cube_scanner_init(str, &scanbuflen);
- if (cube_yyparse(&result) != 0)
- cube_yyerror(&result, "cube parser failed");
+ cube_yyparse(&result, scanbuflen);
cube_scanner_finish();
diff --git a/contrib/cube/cubedata.h b/contrib/cube/cubedata.h
index dbe7d4f742..640a7ca580 100644
--- a/contrib/cube/cubedata.h
+++ b/contrib/cube/cubedata.h
@@ -61,9 +61,9 @@ typedef struct NDBOX
/* in cubescan.l */
extern int cube_yylex(void);
-extern void cube_yyerror(NDBOX **result, const char *message) pg_attribute_noreturn();
-extern void cube_scanner_init(const char *str);
+extern void cube_yyerror(NDBOX **result, Size scanbuflen, const char *message) pg_attribute_noreturn();
+extern void cube_scanner_init(const char *str, Size *scanbuflen);
extern void cube_scanner_finish(void);
/* in cubeparse.y */
-extern int cube_yyparse(NDBOX **result);
+extern int cube_yyparse(NDBOX **result, Size scanbuflen);
diff --git a/contrib/cube/cubeparse.y b/contrib/cube/cubeparse.y
index 7577c4515c..e3b750b695 100644
--- a/contrib/cube/cubeparse.y
+++ b/contrib/cube/cubeparse.y
@@ -7,6 +7,7 @@
#include "postgres.h"
#include "cubedata.h"
+#include "cube_internal.h"
#include "utils/float.h"
/* All grammar constructs return strings */
@@ -23,9 +24,6 @@
#define YYMALLOC palloc
#define YYFREE pfree
-static char *scanbuf;
-static int scanbuflen;
-
static int item_count(const char *s, char delim);
static NDBOX *write_box(int dim, char *str1, char *str2);
static NDBOX *write_point_as_box(int dim, char *str);
@@ -33,7 +31,7 @@ static NDBOX *write_point_as_box(int dim, char *str);
%}
/* BISON Declarations */
-%parse-param {NDBOX **result}
+%parse-param {NDBOX **result} {Size scanbuflen}
%expect 0
%name-prefix="cube_yy"
@@ -265,5 +263,3 @@ write_point_as_box(int dim, char *str)
return bp;
}
-
-#include "cubescan.c"
diff --git a/contrib/cube/cubescan.l b/contrib/cube/cubescan.l
index bd400e3684..6b316f2d54 100644
--- a/contrib/cube/cubescan.l
+++ b/contrib/cube/cubescan.l
@@ -1,9 +1,21 @@
-%{
+%top{
/*
* A scanner for EMP-style numeric ranges
* contrib/cube/cubescan.l
*/
+#include "postgres.h"
+
+/*
+ * NB: include cubeparse.h only AFTER defining YYSTYPE (to match cubeparse.y)
+ * and cubedata.h for NDBOX.
+ */
+#include "cubedata.h"
+#define YYSTYPE char *
+#include "cubeparse.h"
+}
+
+%{
/* LCOV_EXCL_START */
/* No reason to constrain amount of data slurped */
@@ -21,9 +33,7 @@ fprintf_to_ereport(const char *fmt, const char *msg)
/* Handles to the buffer that the lexer uses internally */
static YY_BUFFER_STATE scanbufhandle;
-/* this is now declared in cubeparse.y: */
-/* static char *scanbuf; */
-/* static int scanbuflen; */
+static char *scanbuf;
%}
%option 8bit
@@ -45,14 +55,14 @@ NaN [nN][aA][nN]
%%
-{float} yylval = yytext; return CUBEFLOAT;
-{infinity} yylval = yytext; return CUBEFLOAT;
-{NaN} yylval = yytext; return CUBEFLOAT;
-\[ yylval = "("; return O_BRACKET;
-\] yylval = ")"; return C_BRACKET;
-\( yylval = "("; return O_PAREN;
-\) yylval = ")"; return C_PAREN;
-\, yylval = ","; return COMMA;
+{float} cube_yylval = yytext; return CUBEFLOAT;
+{infinity} cube_yylval = yytext; return CUBEFLOAT;
+{NaN} cube_yylval = yytext; return CUBEFLOAT;
+\[ cube_yylval = "("; return O_BRACKET;
+\] cube_yylval = ")"; return C_BRACKET;
+\( cube_yylval = "("; return O_PAREN;
+\) cube_yylval = ")"; return C_PAREN;
+\, cube_yylval = ","; return COMMA;
[ \t\n\r\f]+ /* discard spaces */
. return yytext[0]; /* alert parser of the garbage */
@@ -60,9 +70,9 @@ NaN [nN][aA][nN]
/* LCOV_EXCL_STOP */
-/* result is not used, but Bison expects this signature */
+/* result and scanbuflen are not used, but Bison expects this signature */
void
-yyerror(NDBOX **result, const char *message)
+cube_yyerror(NDBOX **result, Size scanbuflen, const char *message)
{
if (*yytext == YY_END_OF_BUFFER_CHAR)
{
@@ -87,9 +97,9 @@ yyerror(NDBOX **result, const char *message)
* Called before any actual parsing is done
*/
void
-cube_scanner_init(const char *str)
+cube_scanner_init(const char *str, Size *scanbuflen)
{
- Size slen = strlen(str);
+ Size slen = strlen(str);
/*
* Might be left over after ereport()
@@ -100,7 +110,7 @@ cube_scanner_init(const char *str)
/*
* Make a scan buffer with special termination needed by flex.
*/
- scanbuflen = slen;
+ *scanbuflen = slen;
scanbuf = palloc(slen + 2);
memcpy(scanbuf, str, slen);
scanbuf[slen] = scanbuf[slen + 1] = YY_END_OF_BUFFER_CHAR;
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 1a56a74cda..c3f3a0acee 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -117,6 +117,7 @@ do
# We can't make these Bison output files compilable standalone
# without using "%code require", which old Bison versions lack.
# parser/gram.h will be included by parser/gramparse.h anyway.
+ test "$f" = contrib/cube/cubeparse.h && continue
test "$f" = src/include/parser/gram.h && continue
test "$f" = src/backend/parser/gram.h && continue
test "$f" = src/backend/bootstrap/bootparse.h && continue
--
2.36.1
[text/x-patch] v4-0010-Build-segscan.c-standalone.patch (4.5K, ../../CAFBsxsHdAP0U3AjYuJ_SpWBxj2vmpdqXX3Yg=TLBrq0rAWhFgg@mail.gmail.com/10-v4-0010-Build-segscan.c-standalone.patch)
download | inline diff:
From dac7ffb5dd095c83570b87b47fb0c60869bf7064 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 12:00:33 +0700
Subject: [PATCH v4 10/11] Build segscan.c standalone
---
contrib/seg/.gitignore | 1 +
contrib/seg/Makefile | 15 +++++++++++----
contrib/seg/segparse.y | 3 ---
contrib/seg/segscan.l | 28 ++++++++++++++++++----------
src/tools/pginclude/headerscheck | 1 +
5 files changed, 31 insertions(+), 17 deletions(-)
diff --git a/contrib/seg/.gitignore b/contrib/seg/.gitignore
index 69e73d2096..fa247a4e67 100644
--- a/contrib/seg/.gitignore
+++ b/contrib/seg/.gitignore
@@ -1,3 +1,4 @@
+/segparse.h
/segparse.c
/segscan.c
# Generated subdirectories
diff --git a/contrib/seg/Makefile b/contrib/seg/Makefile
index bb63e83506..c6c134b8f1 100644
--- a/contrib/seg/Makefile
+++ b/contrib/seg/Makefile
@@ -4,7 +4,8 @@ MODULE_big = seg
OBJS = \
$(WIN32RES) \
seg.o \
- segparse.o
+ segparse.o \
+ segscan.o
EXTENSION = seg
DATA = seg--1.1.sql seg--1.1--1.2.sql seg--1.2--1.3.sql seg--1.3--1.4.sql \
@@ -29,10 +30,16 @@ include $(top_srcdir)/contrib/contrib-global.mk
endif
-# segscan is compiled as part of segparse
-segparse.o: segscan.c
+# See notes in src/backend/parser/Makefile about the following two rules
+segparse.h: segparse.c
+ touch $@
+
+segparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+segparse.o segscan.o: segparse.h
distprep: segparse.c segscan.c
maintainer-clean:
- rm -f segparse.c segscan.c
+ rm -f segparse.h segparse.c segscan.c
diff --git a/contrib/seg/segparse.y b/contrib/seg/segparse.y
index 33e3a9f35f..637eacd1a6 100644
--- a/contrib/seg/segparse.y
+++ b/contrib/seg/segparse.y
@@ -160,6 +160,3 @@ seg_atof(const char *value)
datum = DirectFunctionCall1(float4in, CStringGetDatum(value));
return DatumGetFloat4(datum);
}
-
-
-#include "segscan.c"
diff --git a/contrib/seg/segscan.l b/contrib/seg/segscan.l
index 5f6595e9eb..4744fd5e9e 100644
--- a/contrib/seg/segscan.l
+++ b/contrib/seg/segscan.l
@@ -1,8 +1,18 @@
-%{
+%top{
/*
* A scanner for EMP-style numeric ranges
*/
+#include "postgres.h"
+
+/*
+ * NB: include segparse.h only AFTER including segdata.h, because segdata.h
+ * contains the definition for SEG.
+ */
+#include "segdata.h"
+#include "segparse.h"
+}
+%{
/* LCOV_EXCL_START */
/* No reason to constrain amount of data slurped */
@@ -21,7 +31,6 @@ fprintf_to_ereport(const char *fmt, const char *msg)
/* Handles to the buffer that the lexer uses internally */
static YY_BUFFER_STATE scanbufhandle;
static char *scanbuf;
-static int scanbuflen;
%}
%option 8bit
@@ -42,12 +51,12 @@ float ({integer}|{real})([eE]{integer})?
%%
-{range} yylval.text = yytext; return RANGE;
-{plumin} yylval.text = yytext; return PLUMIN;
-{float} yylval.text = yytext; return SEGFLOAT;
-\< yylval.text = "<"; return EXTENSION;
-\> yylval.text = ">"; return EXTENSION;
-\~ yylval.text = "~"; return EXTENSION;
+{range} seg_yylval.text = yytext; return RANGE;
+{plumin} seg_yylval.text = yytext; return PLUMIN;
+{float} seg_yylval.text = yytext; return SEGFLOAT;
+\< seg_yylval.text = "<"; return EXTENSION;
+\> seg_yylval.text = ">"; return EXTENSION;
+\~ seg_yylval.text = "~"; return EXTENSION;
[ \t\n\r\f]+ /* discard spaces */
. return yytext[0]; /* alert parser of the garbage */
@@ -56,7 +65,7 @@ float ({integer}|{real})([eE]{integer})?
/* LCOV_EXCL_STOP */
void
-yyerror(SEG *result, const char *message)
+seg_yyerror(SEG *result, const char *message)
{
if (*yytext == YY_END_OF_BUFFER_CHAR)
{
@@ -94,7 +103,6 @@ seg_scanner_init(const char *str)
/*
* Make a scan buffer with special termination needed by flex.
*/
- scanbuflen = slen;
scanbuf = palloc(slen + 2);
memcpy(scanbuf, str, slen);
scanbuf[slen] = scanbuf[slen + 1] = YY_END_OF_BUFFER_CHAR;
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index c3f3a0acee..7022ac6c39 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -118,6 +118,7 @@ do
# without using "%code require", which old Bison versions lack.
# parser/gram.h will be included by parser/gramparse.h anyway.
test "$f" = contrib/cube/cubeparse.h && continue
+ test "$f" = contrib/seg/segparse.h && continue
test "$f" = src/include/parser/gram.h && continue
test "$f" = src/backend/parser/gram.h && continue
test "$f" = src/backend/bootstrap/bootparse.h && continue
--
2.36.1
[text/x-patch] v4-0007-Build-specscanner.c-standalone.patch (5.3K, ../../CAFBsxsHdAP0U3AjYuJ_SpWBxj2vmpdqXX3Yg=TLBrq0rAWhFgg@mail.gmail.com/11-v4-0007-Build-specscanner.c-standalone.patch)
download | inline diff:
From 9ac807e74024d6c2e989763bb1dda15f1ad5f41a Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 09:34:17 +0700
Subject: [PATCH v4 07/11] Build specscanner.c standalone
---
src/test/isolation/.gitignore | 1 +
src/test/isolation/Makefile | 15 +++++++++++----
src/test/isolation/specparse.y | 2 --
src/test/isolation/specscanner.l | 28 +++++++++++++++++++---------
src/tools/pginclude/headerscheck | 1 +
5 files changed, 32 insertions(+), 15 deletions(-)
diff --git a/src/test/isolation/.gitignore b/src/test/isolation/.gitignore
index 870dac4d28..2c13b4bf98 100644
--- a/src/test/isolation/.gitignore
+++ b/src/test/isolation/.gitignore
@@ -3,6 +3,7 @@
/pg_isolation_regress
# Local generated source files
+/specparse.h
/specparse.c
/specscanner.c
diff --git a/src/test/isolation/Makefile b/src/test/isolation/Makefile
index 0d452c89d4..b8738b7c1b 100644
--- a/src/test/isolation/Makefile
+++ b/src/test/isolation/Makefile
@@ -15,7 +15,8 @@ override CPPFLAGS := -I. -I$(srcdir) -I$(libpq_srcdir) \
OBJS = \
$(WIN32RES) \
isolationtester.o \
- specparse.o
+ specparse.o \
+ specscanner.o
all: isolationtester$(X) pg_isolation_regress$(X)
@@ -44,8 +45,14 @@ isolationtester$(X): $(OBJS) | submake-libpq submake-libpgport
distprep: specparse.c specscanner.c
-# specscanner is compiled as part of specparse
-specparse.o: specscanner.c
+# See notes in src/backend/parser/Makefile about the following two rules
+specparse.h: specparse.c
+ touch $@
+
+specparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+specparse.o specscanner.o: specparse.h
# specparse.c and specscanner.c are in the distribution tarball,
# so do not clean them here
@@ -55,7 +62,7 @@ clean distclean:
rm -rf $(pg_regress_clean_files)
maintainer-clean: distclean
- rm -f specparse.c specscanner.c
+ rm -f specparse.h specparse.c specscanner.c
installcheck: all
$(pg_isolation_regress_installcheck) --schedule=$(srcdir)/isolation_schedule
diff --git a/src/test/isolation/specparse.y b/src/test/isolation/specparse.y
index eb368184b8..657285cc23 100644
--- a/src/test/isolation/specparse.y
+++ b/src/test/isolation/specparse.y
@@ -276,5 +276,3 @@ blocker:
;
%%
-
-#include "specscanner.c"
diff --git a/src/test/isolation/specscanner.l b/src/test/isolation/specscanner.l
index aa6e89268e..b04696f52d 100644
--- a/src/test/isolation/specscanner.l
+++ b/src/test/isolation/specscanner.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* specscanner.l
@@ -9,7 +9,17 @@
*
*-------------------------------------------------------------------------
*/
+#include "postgres_fe.h"
+/*
+ * NB: include specparse.h only AFTER including isolationtester.h, because
+ * isolationtester.h includes node definitions needed for YYSTYPE.
+ */
+#include "isolationtester.h"
+#include "specparse.h"
+}
+
+%{
static int yyline = 1; /* line number for error reporting */
#define LITBUF_INIT 1024 /* initial size of litbuf */
@@ -75,7 +85,7 @@ teardown { return TEARDOWN; }
/* Plain identifiers */
{identifier} {
- yylval.str = pg_strdup(yytext);
+ spec_yylval.str = pg_strdup(yytext);
return(identifier);
}
@@ -87,13 +97,13 @@ teardown { return TEARDOWN; }
<qident>\"\" { addlitchar(yytext[0]); }
<qident>\" {
litbuf[litbufpos] = '\0';
- yylval.str = pg_strdup(litbuf);
+ spec_yylval.str = pg_strdup(litbuf);
BEGIN(INITIAL);
return(identifier);
}
<qident>. { addlitchar(yytext[0]); }
-<qident>\n { yyerror("unexpected newline in quoted identifier"); }
-<qident><<EOF>> { yyerror("unterminated quoted identifier"); }
+<qident>\n { spec_yyerror("unexpected newline in quoted identifier"); }
+<qident><<EOF>> { spec_yyerror("unterminated quoted identifier"); }
/* SQL blocks: { UPDATE ... } */
/* We trim leading/trailing whitespace, otherwise they're unprocessed */
@@ -104,7 +114,7 @@ teardown { return TEARDOWN; }
}
<sql>{space}*"}" {
litbuf[litbufpos] = '\0';
- yylval.str = pg_strdup(litbuf);
+ spec_yylval.str = pg_strdup(litbuf);
BEGIN(INITIAL);
return(sqlblock);
}
@@ -116,12 +126,12 @@ teardown { return TEARDOWN; }
addlitchar(yytext[0]);
}
<sql><<EOF>> {
- yyerror("unterminated sql block");
+ spec_yyerror("unterminated sql block");
}
/* Numbers and punctuation */
{digit}+ {
- yylval.integer = atoi(yytext);
+ spec_yylval.integer = atoi(yytext);
return INTEGER;
}
@@ -150,7 +160,7 @@ addlitchar(char c)
}
void
-yyerror(const char *message)
+spec_yyerror(const char *message)
{
fprintf(stderr, "%s at line %d\n", message, yyline);
exit(1);
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 1e953d503f..c1b1f3bd2a 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -122,6 +122,7 @@ do
test "$f" = src/backend/bootstrap/bootparse.h && continue
test "$f" = src/backend/replication/repl_gram.h && continue
test "$f" = src/backend/replication/syncrep_gram.h && continue
+ test "$f" = src/test/isolation/specparse.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
test "$f" = src/interfaces/ecpg/preproc/preproc.h && continue
--
2.36.1
[text/x-patch] v4-0011-Build-jsonpath_scan.c-standalone.patch (7.8K, ../../CAFBsxsHdAP0U3AjYuJ_SpWBxj2vmpdqXX3Yg=TLBrq0rAWhFgg@mail.gmail.com/12-v4-0011-Build-jsonpath_scan.c-standalone.patch)
download | inline diff:
From 697b7d18f59b51ebc40672e18791b8deccc3733c Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 13 Aug 2022 12:35:55 +0700
Subject: [PATCH v4 11/11] Build jsonpath_scan.c standalone
---
src/backend/utils/adt/.gitignore | 1 +
src/backend/utils/adt/Makefile | 11 ++++++--
src/backend/utils/adt/jsonpath_gram.y | 27 +------------------
src/backend/utils/adt/jsonpath_internal.h | 32 +++++++++++++++++++++++
src/backend/utils/adt/jsonpath_scan.l | 29 +++++++++++++-------
src/tools/pginclude/headerscheck | 1 +
6 files changed, 63 insertions(+), 38 deletions(-)
create mode 100644 src/backend/utils/adt/jsonpath_internal.h
diff --git a/src/backend/utils/adt/.gitignore b/src/backend/utils/adt/.gitignore
index 48cf941a52..7fab054407 100644
--- a/src/backend/utils/adt/.gitignore
+++ b/src/backend/utils/adt/.gitignore
@@ -1,2 +1,3 @@
+/jsonpath_gram.h
/jsonpath_gram.c
/jsonpath_scan.c
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 7c722ea2ce..0de0bbb1b8 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -57,6 +57,7 @@ OBJS = \
jsonpath.o \
jsonpath_exec.o \
jsonpath_gram.o \
+ jsonpath_scan.o \
like.o \
like_support.o \
lockfuncs.o \
@@ -119,11 +120,17 @@ OBJS = \
xid8funcs.o \
xml.o
+# See notes in src/backend/parser/Makefile about the following two rules
+jsonpath_gram.h: jsonpath_gram.c
+ touch $@
+
+jsonpath_gram.c: BISONFLAGS += -d
+
jsonpath_scan.c: FLEXFLAGS = -CF -p -p
jsonpath_scan.c: FLEX_NO_BACKUP=yes
-# jsonpath_scan is compiled as part of jsonpath_gram
-jsonpath_gram.o: jsonpath_scan.c
+# Force these dependencies to be known even without dependency info built:
+jsonpath_gram.o jsonpath_scan.o: jsonpath_gram.h
# jsonpath_gram.c and jsonpath_scan.c are in the distribution tarball,
# so they are not cleaned here.
diff --git a/src/backend/utils/adt/jsonpath_gram.y b/src/backend/utils/adt/jsonpath_gram.y
index ce5d5af891..35a79ca965 100644
--- a/src/backend/utils/adt/jsonpath_gram.y
+++ b/src/backend/utils/adt/jsonpath_gram.y
@@ -18,26 +18,11 @@
#include "catalog/pg_collation.h"
#include "fmgr.h"
+#include "jsonpath_internal.h"
#include "miscadmin.h"
#include "nodes/pg_list.h"
#include "regex/regex.h"
#include "utils/builtins.h"
-#include "utils/jsonpath.h"
-
-/* struct JsonPathString is shared between scan and gram */
-typedef struct JsonPathString
-{
- char *val;
- int len;
- int total;
-} JsonPathString;
-
-union YYSTYPE;
-
-/* flex 2.5.4 doesn't bother with a decl for this */
-int jsonpath_yylex(union YYSTYPE *yylval_param);
-int jsonpath_yyparse(JsonPathParseResult **result);
-void jsonpath_yyerror(JsonPathParseResult **result, const char *message);
static JsonPathParseItem *makeItemType(JsonPathItemType type);
static JsonPathParseItem *makeItemString(JsonPathString *s);
@@ -593,13 +578,3 @@ jspConvertRegexFlags(uint32 xflags)
return cflags;
}
-
-/*
- * jsonpath_scan.l is compiled as part of jsonpath_gram.y. Currently, this is
- * unavoidable because jsonpath_gram does not create a .h file to export its
- * token symbols. If these files ever grow large enough to be worth compiling
- * separately, that could be fixed; but for now it seems like useless
- * complication.
- */
-
-#include "jsonpath_scan.c"
diff --git a/src/backend/utils/adt/jsonpath_internal.h b/src/backend/utils/adt/jsonpath_internal.h
new file mode 100644
index 0000000000..edfc6191a0
--- /dev/null
+++ b/src/backend/utils/adt/jsonpath_internal.h
@@ -0,0 +1,32 @@
+/*-------------------------------------------------------------------------
+ *
+ * jsonpath_internal.h
+ * Private definitions for jsonpath scanner & parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/utils/adt/jsonpath_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef JSONPATH_INTERNAL_H
+#define JSONPATH_INTERNAL_H
+
+/* struct JsonPathString is shared between scan and gram */
+typedef struct JsonPathString
+{
+ char *val;
+ int len;
+ int total;
+} JsonPathString;
+
+#include "utils/jsonpath.h"
+#include "jsonpath_gram.h"
+
+extern int jsonpath_yylex(YYSTYPE *yylval_param);
+extern int jsonpath_yyparse(JsonPathParseResult **result);
+extern void jsonpath_yyerror(JsonPathParseResult **result, const char *message);
+
+#endif /* JSONPATH_INTERNAL_H */
diff --git a/src/backend/utils/adt/jsonpath_scan.l b/src/backend/utils/adt/jsonpath_scan.l
index 4351f6ec98..ea824bae73 100644
--- a/src/backend/utils/adt/jsonpath_scan.l
+++ b/src/backend/utils/adt/jsonpath_scan.l
@@ -1,4 +1,4 @@
-%{
+%top{
/*-------------------------------------------------------------------------
*
* jsonpath_scan.l
@@ -17,9 +17,18 @@
#include "postgres.h"
+/*
+ * NB: include jsonpath_gram.h only AFTER including jsonpath_internal.h,
+ * because jsonpath_internal.h contains the declaration for JsonPathString.
+ */
+#include "jsonpath_internal.h"
+#include "jsonpath_gram.h"
+
#include "mb/pg_wchar.h"
#include "nodes/pg_list.h"
+}
+%{
static JsonPathString scanstring;
/* Handles to the buffer that the lexer uses internally */
@@ -142,9 +151,9 @@ hex_fail \\x{hex_dig}{0,1}
<xnq,xq,xvq>{hex_char} { parseHexChar(yytext); }
-<xnq,xq,xvq>{unicode}*{unicodefail} { yyerror(NULL, "invalid unicode sequence"); }
+<xnq,xq,xvq>{unicode}*{unicodefail} { jsonpath_yyerror(NULL, "invalid unicode sequence"); }
-<xnq,xq,xvq>{hex_fail} { yyerror(NULL, "invalid hex character sequence"); }
+<xnq,xq,xvq>{hex_fail} { jsonpath_yyerror(NULL, "invalid hex character sequence"); }
<xnq,xq,xvq>{unicode}+\\ {
/* throw back the \\, and treat as unicode */
@@ -154,9 +163,9 @@ hex_fail \\x{hex_dig}{0,1}
<xnq,xq,xvq>\\. { addchar(false, yytext[1]); }
-<xnq,xq,xvq>\\ { yyerror(NULL, "unexpected end after backslash"); }
+<xnq,xq,xvq>\\ { jsonpath_yyerror(NULL, "unexpected end after backslash"); }
-<xq,xvq><<EOF>> { yyerror(NULL, "unexpected end of quoted string"); }
+<xq,xvq><<EOF>> { jsonpath_yyerror(NULL, "unexpected end of quoted string"); }
<xq>\" {
yylval->str = scanstring;
@@ -178,7 +187,7 @@ hex_fail \\x{hex_dig}{0,1}
<xc>\* { }
-<xc><<EOF>> { yyerror(NULL, "unexpected end of comment"); }
+<xc><<EOF>> { jsonpath_yyerror(NULL, "unexpected end of comment"); }
\&\& { return AND_P; }
@@ -244,10 +253,10 @@ hex_fail \\x{hex_dig}{0,1}
return INT_P;
}
-{realfail} { yyerror(NULL, "invalid numeric literal"); }
-{integer_junk} { yyerror(NULL, "trailing junk after numeric literal"); }
-{decimal_junk} { yyerror(NULL, "trailing junk after numeric literal"); }
-{real_junk} { yyerror(NULL, "trailing junk after numeric literal"); }
+{realfail} { jsonpath_yyerror(NULL, "invalid numeric literal"); }
+{integer_junk} { jsonpath_yyerror(NULL, "trailing junk after numeric literal"); }
+{decimal_junk} { jsonpath_yyerror(NULL, "trailing junk after numeric literal"); }
+{real_junk} { jsonpath_yyerror(NULL, "trailing junk after numeric literal"); }
\" {
addchar(true, '\0');
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 7022ac6c39..1db0096758 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -124,6 +124,7 @@ do
test "$f" = src/backend/bootstrap/bootparse.h && continue
test "$f" = src/backend/replication/repl_gram.h && continue
test "$f" = src/backend/replication/syncrep_gram.h && continue
+ test "$f" = src/backend/utils/adt/jsonpath_gram.h && continue
test "$f" = src/test/isolation/specparse.h && continue
test "$f" = src/bin/pgbench/exprparse.h && continue
test "$f" = src/pl/plpgsql/src/pl_gram.h && continue
--
2.36.1
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: build remaining Flex files standalone
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 04:07 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-12 06:01 ` build remaining Flex files standalone John Naylor <[email protected]>
2022-08-13 08:39 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-15 18:11 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
2022-08-16 10:41 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-17 01:14 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
2022-08-18 07:40 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-18 07:43 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
@ 2022-08-18 08:00 ` John Naylor <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: John Naylor @ 2022-08-18 08:00 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers
I wrote
> [v4]
This piece is a leftover from the last version, and forgot to remove
it, will fix:
diff --git a/contrib/cube/cubeparse.y b/contrib/cube/cubeparse.y
index 7577c4515c..e3b750b695 100644
--- a/contrib/cube/cubeparse.y
+++ b/contrib/cube/cubeparse.y
@@ -7,6 +7,7 @@
#include "postgres.h"
#include "cubedata.h"
+#include "cube_internal.h"
#include "utils/float.h"
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 00:47 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 01:38 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 02:41 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-26 02:58 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-05-26 08:47 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-05-31 13:49 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-01 09:39 ` Re: [RFC] building postgres with meson Aleksander Alekseev <[email protected]>
2022-06-02 16:39 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-06-02 17:08 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-06-02 17:26 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-10 17:19 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-11 03:33 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
2022-08-11 03:37 ` Re: [RFC] building postgres with meson Tom Lane <[email protected]>
2022-08-11 03:57 ` Re: [RFC] building postgres with meson John Naylor <[email protected]>
@ 2022-08-11 04:32 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 139+ messages in thread
From: Tom Lane @ 2022-08-11 04:32 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Andres Freund <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
John Naylor <[email protected]> writes:
> I'll volunteer to work on this unless an easier solution happens to
> come along in the next couple days. (aside: guc-file.l doesn't have a
> grammar, so not yet sure if that makes the issue easier or harder...)
That one's probably mostly about the issue mentioned in the other
commit you identified. Without %top, it's impossible to make a
standalone flex module honor the rule about thou-shalt-have-no-
other-includes-before-postgres.h. So embedding it in some other
file was originally a necessity for that. Now that we know how
to fix that, it's just a matter of making sure that any other stuff
the scanner needs is available from a .h file.
regards, tom lane
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-08-09 07:10 ` Andres Freund <[email protected]>
2022-08-09 12:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
9 siblings, 1 reply; 139+ messages in thread
From: Andres Freund @ 2022-08-09 07:10 UTC (permalink / raw)
To: pgsql-hackers; Andrew Dunstan <[email protected]>; Magnus Hagander <[email protected]>
Hi,
I was looking at re-unifying gendef2.pl that the meson patchset had introduced
for temporary ease during hacking with gendef.pl. Testing that I noticed that
either I and my machine is very confused, or gendef.pl's check whether it can
skip work is bogus.
I noticed that, despite having code to avoid rerunning when the input files
are older than the .def file, it always runs.
# if the def file exists and is newer than all input object files, skip
# its creation
if (-f $deffile
&& (-M $deffile > max(map { -M } <$ARGV[0]/*.obj>)))
{
print "Not re-generating $defname.DEF, file already exists.\n";
exit(0);
}
My understanding of -M is that it returns the time delta between the file
modification and the start of the script. Which makes the use of max() bogus,
since it'll return the oldest time any input has been modified, not the
newest. And the condition needs to be inverted, because we want to skip the
work if $deffile is *newer*, right?
Am I missing something here?
I'm tempted to just remove the not-regenerating logic - gendef.pl shouldn't
run if there's nothing to do, and it'll e.g. not notice if there's an
additional input that wasn't there during the last invocation of gendef.pl.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-09 07:10 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
@ 2022-08-09 12:37 ` Andrew Dunstan <[email protected]>
2022-08-21 00:42 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
0 siblings, 1 reply; 139+ messages in thread
From: Andrew Dunstan @ 2022-08-09 12:37 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers; Magnus Hagander <[email protected]>
On 2022-08-09 Tu 03:10, Andres Freund wrote:
> Hi,
>
> I was looking at re-unifying gendef2.pl that the meson patchset had introduced
> for temporary ease during hacking with gendef.pl. Testing that I noticed that
> either I and my machine is very confused, or gendef.pl's check whether it can
> skip work is bogus.
>
> I noticed that, despite having code to avoid rerunning when the input files
> are older than the .def file, it always runs.
>
> # if the def file exists and is newer than all input object files, skip
> # its creation
> if (-f $deffile
> && (-M $deffile > max(map { -M } <$ARGV[0]/*.obj>)))
> {
> print "Not re-generating $defname.DEF, file already exists.\n";
> exit(0);
> }
>
> My understanding of -M is that it returns the time delta between the file
> modification and the start of the script. Which makes the use of max() bogus,
> since it'll return the oldest time any input has been modified, not the
> newest. And the condition needs to be inverted, because we want to skip the
> work if $deffile is *newer*, right?
>
> Am I missing something here?
No, you're right, this is bogus. Reversing the test and using min
instead of max is the obvious fix.
> I'm tempted to just remove the not-regenerating logic - gendef.pl shouldn't
> run if there's nothing to do, and it'll e.g. not notice if there's an
> additional input that wasn't there during the last invocation of gendef.pl.
>
Maybe, need to think about that more.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 139+ messages in thread
* Re: [RFC] building postgres with meson
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-09 07:10 ` Re: [RFC] building postgres with meson Andres Freund <[email protected]>
2022-08-09 12:37 ` Re: [RFC] building postgres with meson Andrew Dunstan <[email protected]>
@ 2022-08-21 00:42 ` Andres Freund <[email protected]>
0 siblings, 0 replies; 139+ messages in thread
From: Andres Freund @ 2022-08-21 00:42 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: pgsql-hackers; Magnus Hagander <[email protected]>
Hi,
On 2022-08-09 08:37:16 -0400, Andrew Dunstan wrote:
> On 2022-08-09 Tu 03:10, Andres Freund wrote:
> > Hi,
> >
> > I was looking at re-unifying gendef2.pl that the meson patchset had introduced
> > for temporary ease during hacking with gendef.pl. Testing that I noticed that
> > either I and my machine is very confused, or gendef.pl's check whether it can
> > skip work is bogus.
> >
> > I noticed that, despite having code to avoid rerunning when the input files
> > are older than the .def file, it always runs.
> >
> > # if the def file exists and is newer than all input object files, skip
> > # its creation
> > if (-f $deffile
> > && (-M $deffile > max(map { -M } <$ARGV[0]/*.obj>)))
> > {
> > print "Not re-generating $defname.DEF, file already exists.\n";
> > exit(0);
> > }
> >
> > My understanding of -M is that it returns the time delta between the file
> > modification and the start of the script. Which makes the use of max() bogus,
> > since it'll return the oldest time any input has been modified, not the
> > newest. And the condition needs to be inverted, because we want to skip the
> > work if $deffile is *newer*, right?
> >
> > Am I missing something here?
>
>
> No, you're right, this is bogus. Reversing the test and using min
> instead of max is the obvious fix.
>
>
> > I'm tempted to just remove the not-regenerating logic - gendef.pl shouldn't
> > run if there's nothing to do, and it'll e.g. not notice if there's an
> > additional input that wasn't there during the last invocation of gendef.pl.
> >
>
> Maybe, need to think about that more.
Any thoughts?
I'd like to commit 0003 in
https://postgr.es/m/20220811002012.ju3rrz47i2e5tdha%40awork3.anarazel.de
fairly soon.
I did fix the bogus "die" message I added during some debugging since posting
that...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 139+ messages in thread
end of thread, other threads:[~2022-08-21 00:42 UTC | newest]
Thread overview: 139+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-10-12 08:37 [RFC] building postgres with meson Andres Freund <[email protected]>
2021-10-12 09:08 ` Andres Freund <[email protected]>
2021-10-12 13:30 ` Peter Eisentraut <[email protected]>
2021-10-12 15:00 ` Robert Haas <[email protected]>
2021-10-12 15:47 ` Tom Lane <[email protected]>
2021-10-12 16:15 ` Andres Freund <[email protected]>
2021-10-12 19:01 ` Andres Freund <[email protected]>
2021-10-13 11:54 ` Daniel Gustafsson <[email protected]>
2021-10-12 15:08 ` Andrew Dunstan <[email protected]>
2021-10-12 15:21 ` Josef Šimánek <[email protected]>
2021-10-12 17:17 ` Andres Freund <[email protected]>
2021-10-12 23:19 ` Josef Šimánek <[email protected]>
2021-10-12 23:54 ` Andres Freund <[email protected]>
2021-10-13 21:58 ` Josef Šimánek <[email protected]>
2021-10-14 13:14 ` Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 13:19 ` Josef Šimánek <[email protected]>
2021-10-14 13:29 ` Alvaro Herrera <[email protected]>
2021-10-14 17:24 ` Andres Freund <[email protected]>
2021-10-14 19:36 ` Josef Šimánek <[email protected]>
2021-10-14 22:08 ` Tom Lane <[email protected]>
2021-10-14 23:38 ` Andres Freund <[email protected]>
2021-10-14 23:51 ` Tom Lane <[email protected]>
2021-10-14 13:32 ` Dagfinn Ilmari Mannsåker <[email protected]>
2021-10-14 19:29 ` Josef Šimánek <[email protected]>
2021-10-14 17:26 ` Andres Freund <[email protected]>
2021-10-12 15:28 ` Andrew Dunstan <[email protected]>
2021-10-12 15:50 ` Andrew Dunstan <[email protected]>
2021-10-12 16:59 ` Andres Freund <[email protected]>
2021-10-12 18:09 ` Andres Freund <[email protected]>
2021-10-12 18:37 ` Andrew Dunstan <[email protected]>
2021-10-12 19:16 ` Andrew Dunstan <[email protected]>
2021-10-12 19:29 ` Andres Freund <[email protected]>
2021-10-12 20:02 ` Andrew Dunstan <[email protected]>
2021-10-12 20:42 ` Andres Freund <[email protected]>
2021-10-13 01:03 ` Andres Freund <[email protected]>
2021-10-13 12:55 ` Andrew Dunstan <[email protected]>
2021-10-13 17:26 ` Andres Freund <[email protected]>
2021-10-13 20:06 ` Andrew Dunstan <[email protected]>
2021-10-13 21:46 ` Andres Freund <[email protected]>
2021-10-13 23:11 ` Andrew Dunstan <[email protected]>
2021-10-14 22:19 ` Andrew Dunstan <[email protected]>
2021-10-12 18:11 ` Andrew Dunstan <[email protected]>
2021-10-12 18:23 ` Andres Freund <[email protected]>
2021-10-12 18:42 ` Andrew Dunstan <[email protected]>
2021-10-12 19:55 ` John Naylor <[email protected]>
2021-10-12 20:59 ` Andres Freund <[email protected]>
2021-10-13 15:51 ` John Naylor <[email protected]>
2021-10-13 16:37 ` Andres Freund <[email protected]>
2021-10-13 17:19 ` John Naylor <[email protected]>
2021-10-13 17:42 ` Andres Freund <[email protected]>
2021-10-13 18:40 ` John Naylor <[email protected]>
2021-10-14 19:14 ` John Naylor <[email protected]>
2021-10-14 20:34 ` Andres Freund <[email protected]>
2021-10-14 20:54 ` John Naylor <[email protected]>
2021-10-14 21:16 ` John Naylor <[email protected]>
2021-10-14 21:48 ` Tom Lane <[email protected]>
2021-10-14 21:41 ` Sergey Shinderuk <[email protected]>
2021-10-14 22:00 ` Tom Lane <[email protected]>
2021-10-14 22:23 ` Thomas Munro <[email protected]>
2021-10-14 22:40 ` Tom Lane <[email protected]>
2021-10-14 23:04 ` Tom Lane <[email protected]>
2021-10-15 00:36 ` Thomas Munro <[email protected]>
2021-10-15 02:46 ` Tom Lane <[email protected]>
2021-10-15 03:20 ` Andres Freund <[email protected]>
2021-10-14 23:15 ` Andres Freund <[email protected]>
2021-10-14 23:23 ` Tom Lane <[email protected]>
2021-10-15 18:50 ` Andres Freund <[email protected]>
2021-10-15 22:36 ` Andres Freund <[email protected]>
2021-10-15 22:47 ` Andres Freund <[email protected]>
2021-10-14 22:55 ` Andres Freund <[email protected]>
2021-10-14 23:27 ` John Naylor <[email protected]>
2021-10-15 00:02 ` Andres Freund <[email protected]>
2021-10-14 04:27 ` Thomas Munro <[email protected]>
2021-10-21 21:48 ` Andres Freund <[email protected]>
2021-10-22 15:55 ` John Naylor <[email protected]>
2021-10-19 18:34 ` Andres Freund <[email protected]>
2021-10-19 19:22 ` Tom Lane <[email protected]>
2021-10-19 21:57 ` John Naylor <[email protected]>
2021-10-20 00:31 ` Andres Freund <[email protected]>
2021-10-20 01:10 ` Tom Lane <[email protected]>
2021-10-20 01:35 ` Andres Freund <[email protected]>
2021-10-20 01:08 ` Andres Freund <[email protected]>
2021-10-20 01:26 ` Tom Lane <[email protected]>
2021-10-20 01:49 ` Andres Freund <[email protected]>
2021-10-20 02:04 ` Tom Lane <[email protected]>
2021-10-20 02:41 ` Andres Freund <[email protected]>
2021-10-20 16:01 ` Andres Freund <[email protected]>
2021-11-15 18:34 ` Andres Freund <[email protected]>
2021-11-15 19:11 ` Tom Lane <[email protected]>
2021-11-15 19:23 ` Andres Freund <[email protected]>
2021-11-15 19:48 ` Robert Haas <[email protected]>
2021-11-15 22:08 ` Thomas Munro <[email protected]>
2021-11-15 22:34 ` Tom Lane <[email protected]>
2021-11-15 22:46 ` Andres Freund <[email protected]>
2021-11-15 22:52 ` Thomas Munro <[email protected]>
2021-11-18 20:44 ` Thomas Munro <[email protected]>
2022-05-26 00:47 ` Andres Freund <[email protected]>
2022-05-26 01:38 ` Tom Lane <[email protected]>
2022-05-26 02:41 ` Andres Freund <[email protected]>
2022-05-26 02:58 ` Tom Lane <[email protected]>
2022-05-26 08:47 ` Aleksander Alekseev <[email protected]>
2022-05-28 19:14 ` Andres Freund <[email protected]>
2022-05-31 13:49 ` Aleksander Alekseev <[email protected]>
2022-05-31 19:25 ` Andres Freund <[email protected]>
2022-06-01 09:39 ` Aleksander Alekseev <[email protected]>
2022-06-01 21:05 ` Andres Freund <[email protected]>
2022-06-02 12:34 ` Aleksander Alekseev <[email protected]>
2022-06-02 15:22 ` Andres Freund <[email protected]>
2022-06-02 16:39 ` Andres Freund <[email protected]>
2022-06-02 17:08 ` Tom Lane <[email protected]>
2022-06-02 17:26 ` Andres Freund <[email protected]>
2022-06-02 17:33 ` Tom Lane <[email protected]>
2022-06-02 17:48 ` Andres Freund <[email protected]>
2022-06-02 19:05 ` Tom Lane <[email protected]>
2022-06-02 19:17 ` Andres Freund <[email protected]>
2022-06-02 19:53 ` Tom Lane <[email protected]>
2022-06-02 21:13 ` Andres Freund <[email protected]>
2022-06-03 09:35 ` Aleksander Alekseev <[email protected]>
2022-06-03 16:23 ` Andres Freund <[email protected]>
2022-08-10 17:19 ` Andres Freund <[email protected]>
2022-08-11 03:33 ` John Naylor <[email protected]>
2022-08-11 03:37 ` Tom Lane <[email protected]>
2022-08-11 03:45 ` Tom Lane <[email protected]>
2022-08-11 03:57 ` John Naylor <[email protected]>
2022-08-11 04:07 ` Andres Freund <[email protected]>
2022-08-12 06:01 ` build remaining Flex files standalone John Naylor <[email protected]>
2022-08-13 08:39 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-15 18:11 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
2022-08-16 10:41 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-17 01:14 ` Re: build remaining Flex files standalone Andres Freund <[email protected]>
2022-08-17 01:47 ` Re: build remaining Flex files standalone Tom Lane <[email protected]>
2022-08-17 02:53 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-18 07:40 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-18 07:43 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-18 08:00 ` Re: build remaining Flex files standalone John Naylor <[email protected]>
2022-08-11 04:32 ` Tom Lane <[email protected]>
2022-08-09 07:10 ` Andres Freund <[email protected]>
2022-08-09 12:37 ` Andrew Dunstan <[email protected]>
2022-08-21 00:42 ` Andres Freund <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox