From: Andres Freund <andres@anarazel.de>
To: pgsql-hackers@postgresql.org
Subject: meson: add and use stamp files for generated headers
Date: Fri, 8 Aug 2025 20:05:57 -0400
Message-ID: <cgkdgvzdpinkacf4v33mky7tbmk467oda5dd4dlmucjjockxzi@xkqfvjoq4uiy> (raw)
Hi,
As $subject says, I'd like to introduce stamp files for generated headers. The
immediate motivation is that more than half of build.ninja just consists out
of repetitions of the list of generated headers (listed as order-only
dependencies). That makes invocations of ninja a tad slower and makes meson
take a bit longer to generate build.ninja.
Without the change:
andres@awork3:/srv/dev/build/postgres/m-master-assert$ wc build.ninja
12713 260280 7754708 build.ninja
andres@awork3:/srv/dev/build/postgres/m-master-assert$ numactl --physcpubind 11 time ninja reconfigure
...
3.48user 0.29system 0:03.84elapsed 97%CPU (0avgtext+0avgdata 87640maxresident)k
0inputs+41520outputs (0major+71323minor)pagefaults 0swaps
andres@awork3:/srv/dev/build/postgres/m-master-assert$ numactl --physcpubind 11 time ninja
ninja: no work to do.
0.06user 0.03system 0:00.10elapsed 98%CPU (0avgtext+0avgdata 22496maxresident)k
0inputs+0outputs (0major+5216minor)pagefaults 0swaps
With the change:
andres@awork3:/srv/dev/build/postgres/m-dev-assert$ wc build.ninja
12733 143627 3376899 build.ninja
andres@awork3:/srv/dev/build/postgres/m-dev-assert$ numactl --physcpubind 11 time ninja reconfigure
...
3.14user 0.30system 0:03.53elapsed 97%CPU (0avgtext+0avgdata 74392maxresident)k
0inputs+34576outputs (0major+66026minor)pagefaults 0swaps
andres@awork3:/srv/dev/build/postgres/m-dev-assert$ numactl --physcpubind 11 time ninja
ninja: no work to do.
0.04user 0.02system 0:00.06elapsed 100%CPU (0avgtext+0avgdata 19332maxresident)k
0inputs+0outputs (0major+4389minor)pagefaults 0swaps
My main motivation however is not those, in absolute terms, relatively small
speedups, but that I have some patches (LLVM bitcode generation, headerscheck
support, compile_commands.json targets for headers) that add more targets
depending on all those headers, and that the slowdown becomes more noticeable
with those. The improvements with just this patch seems a sufficient
improvement on its own, though - hence this thread.
It's a bit sad to use python to create the stamp files, it's more than 3x
slower than using 'touch' - but touch doesn't work on windows. It's also very
hard to imagine a case where creating stamp files is a significant bottleneck,
given that e.g. genbki.pl is somewhat expensive.
Greetings,
Andres Freund
Attachments:v2-0001-meson-add-and-use-stamp-files-for-generated-heade.patch (text/x-diff, 5.4K) download
From d845c0d56a0357730a7ec398cd77c6a1ada392fa Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Fri, 8 Aug 2025 19:49:23 -0400
Subject: [PATCH v2] meson: add and use stamp files for generated headers
Without using stamp files, meson lists the generated headers as the dependency
for every .c file, bloating build.ninja by more than 2x. Processing all the
dependencies also increases the time to generate build.ninja.
The immediate benefit is that this makes re-configuring and clean builds a bit
faster. The main motivation however is that I have other patches that
introduce additional build targets that further would increase the size of
build.ninja, making re-configuring more noticeably slower.
---
meson.build | 16 +++++++++-------
src/include/meson.build | 21 +++++++++++++++++++++
src/backend/meson.build | 2 +-
src/fe_utils/meson.build | 2 +-
4 files changed, 32 insertions(+), 9 deletions(-)
diff --git a/meson.build b/meson.buildindex 431fe9ccc9b..30e0edda3e7 100644--- a/meson.build+++ b/meson.build@@ -3133,6 +3133,8 @@ gen_export_kwargs = {
'install': false,
}
+# command to create stamp files on all OSs+stamp_cmd = [python, '-c', 'import sys; open(sys.argv[1], "w")', '@OUTPUT0@']
###
@@ -3250,14 +3252,14 @@ subdir('src/port')
frontend_common_code = declare_dependency(
compile_args: ['-DFRONTEND'],
include_directories: [postgres_inc],
- sources: generated_headers,+ sources: generated_headers_stamp,
dependencies: [os_deps, zlib, zstd, lz4],
)
backend_common_code = declare_dependency(
compile_args: ['-DBUILDING_DLL'],
include_directories: [postgres_inc],
- sources: generated_headers,+ sources: generated_headers_stamp,
dependencies: [os_deps, zlib, zstd],
)
@@ -3272,7 +3274,7 @@ shlib_code = declare_dependency(
frontend_stlib_code = declare_dependency(
include_directories: [postgres_inc],
link_with: [common_static, pgport_static],
- sources: generated_headers,+ sources: generated_headers_stamp,
dependencies: [os_deps, libintl],
)
@@ -3280,7 +3282,7 @@ frontend_stlib_code = declare_dependency(
frontend_shlib_code = declare_dependency(
include_directories: [postgres_inc],
link_with: [common_shlib, pgport_shlib],
- sources: generated_headers,+ sources: generated_headers_stamp,
dependencies: [shlib_code, os_deps, libintl],
)
@@ -3290,7 +3292,7 @@ frontend_shlib_code = declare_dependency(
frontend_no_fe_utils_code = declare_dependency(
include_directories: [postgres_inc],
link_with: [common_static, pgport_static],
- sources: generated_headers,+ sources: generated_headers_stamp,
dependencies: [os_deps, libintl],
)
@@ -3317,7 +3319,7 @@ subdir('src/interfaces/libpq-oauth')
frontend_code = declare_dependency(
include_directories: [postgres_inc],
link_with: [fe_utils, common_static, pgport_static],
- sources: generated_headers,+ sources: generated_headers_stamp,
dependencies: [os_deps, libintl],
)
@@ -3347,7 +3349,7 @@ backend_code = declare_dependency(
include_directories: [postgres_inc],
link_args: ldflags_be,
link_with: [],
- sources: generated_headers + generated_backend_headers,+ sources: generated_backend_headers_stamp,
dependencies: os_deps + backend_both_deps + backend_deps,
)
diff --git a/src/include/meson.build b/src/include/meson.buildindex 2e4b7aa529e..7cb3075da2a 100644--- a/src/include/meson.build+++ b/src/include/meson.build@@ -177,3 +177,24 @@ install_subdir('catalog',
# autoconf generates the file there, ensure we get a conflict
generated_sources_ac += {'src/include': ['stamp-h']}
+++# Instead of having targets depending directly on a list of all generated+# headers, have them depend on a stamp files for all of them. Dependencies on+# headers are implemented as order-only dependencies in meson (and then using+# compiler generated dependencies during incremental rebuilds ). The benefit+# of using a stamp file is that it makes ninja.build considerably smaller and+# meson setup faster, as otherwise the list of headers is repeated for every C+# file, bloating build.ninja by ~2x.+generated_headers_stamp = custom_target('generated-headers-stamp.h',+ output: 'generated-headers-stamp.h',+ input: generated_headers,+ command: stamp_cmd,+)++generated_backend_headers_stamp = custom_target('generated-backend-headers-stamp.h',+ output: 'generated-backend-headers-stamp.h',+ input: generated_backend_headers,+ depends: generated_headers_stamp,+ command: stamp_cmd,+)diff --git a/src/backend/meson.build b/src/backend/meson.buildindex 2b0db214804..b831a541652 100644--- a/src/backend/meson.build+++ b/src/backend/meson.build@@ -169,7 +169,7 @@ backend_mod_code = declare_dependency(
compile_args: pg_mod_c_args,
include_directories: postgres_inc,
link_args: pg_mod_link_args,
- sources: generated_headers + generated_backend_headers,+ sources: generated_backend_headers_stamp,
dependencies: backend_mod_deps,
)
diff --git a/src/fe_utils/meson.build b/src/fe_utils/meson.buildindex a18cbc939e4..5a9ddb73463 100644--- a/src/fe_utils/meson.build+++ b/src/fe_utils/meson.build@@ -29,7 +29,7 @@ generated_sources += psqlscan
fe_utils_sources += psqlscan
fe_utils = static_library('libpgfeutils',
- fe_utils_sources + generated_headers,+ fe_utils_sources,
c_pch: pch_postgres_fe_h,
include_directories: [postgres_inc, libpq_inc],
c_args: host_system == 'windows' ? ['-DFD_SETSIZE=1024'] : [],
--
2.48.1.76.g4e746b1a31.dirty
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: andres@anarazel.de
Subject: Re: meson: add and use stamp files for generated headers
In-Reply-To: <cgkdgvzdpinkacf4v33mky7tbmk467oda5dd4dlmucjjockxzi@xkqfvjoq4uiy>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox