pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedInvalid control file checksum with AVX-512 during initdb on a clang19 -O0 build
7+ messages / 4 participants
[nested] [flat]
* Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build
@ 2025-06-11 18:10 Soumyadeep Chakraborty <soumyadeep2007@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: Soumyadeep Chakraborty @ 2025-06-11 18:10 UTC (permalink / raw)
To: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hello,
I noticed a strange issue that I can only reproduce with clang:
clang version 19.1.7 (RESF 19.1.7-2.module+el8.10.0+1965+112b558b) on
the devel branch (SHA: 137935bd1167a94b0bfea7239033f1ba1a1d95bb).
We are getting a control file checksum mismatch during initdb. I added
some prints in a small debug patch, and recorded the postgres process
using rr. I have uploaded the rr archive (made with rr pack, tar-ed up) [1].
$ initdb -n -d -D /usr/local/pgsql/data &> initdb.out
2025-06-11 16:19:54.343 UTC [3070] LOG: WriteControlFile crc = 3457434907,
algo is avx = 1
...
2025-06-11 16:19:54.343 UTC [3070] LOG: ReadControlFile crc = 3457434907,
ControlFile->crc = 3457434907, algo is avx = 1
...
2025-06-11 16:19:54.346 UTC [3070] LOG: update_controlfile crc =
2065009488, algo is avx = 1
...
2025-06-11 16:20:13.914 UTC [3070] LOG: update_controlfile crc =
3406554082, algo is avx = 1
...
2025-06-11 16:20:13.920 UTC [3070] LOG: update_controlfile crc =
1234673735, algo is avx = 1
...
2025-06-11 16:20:13.923 UTC [3070] NOTICE: database system is shut down
...
2025-06-11 16:20:13.923 UTC [3070] DEBUG: proc_exit(-1): 0 callbacks to
make
ok
performing post-bootstrap initialization ... 2025-06-11 16:20:13.984 UTC
[3072] LOG: ReadControlFile crc = 2925279607, ControlFile->crc =
1234673735, algo is avx = 1
2025-06-11 16:20:13.984 UTC [3072] FATAL: incorrect checksum in control
file
child process exited with exit code 1
initdb: data directory "/usr/local/pgsql/data" not removed at user's request
Note that this only reproduces with clang-19 -O0 and NOT -O3. I haven't
tried with other versions of clang.
OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting picked
for both cases, for CRC instructions.
Environment:
(1) Configure options:
./configure --prefix=/usr/local/pgsql --with-python --enable-depend
--without-icu --enable-debug CFLAGS='-O0 -fno-omit-frame-pointer' CC=clang
(2) Config log shows:
configure:18262: checking for vectorized CRC-32C
configure:18268: result: AVX-512 with runtime check
pgac_cv_avx512_pclmul_intrinsics=yes
pgac_cv_xsave_intrinsics=yes
(3) Confirmation that we have AVX-512 CRC getting selected at runtime:
(rr) f
#0 WriteControlFile () at xlog.c:4386
4386 ControlFile->pg_control_version = PG_CONTROL_VERSION;
(rr) p pg_comp_crc32c
$1 = (pg_crc32c (*)(pg_crc32c, const void *, size_t)) 0xc8b8b0
<pg_comp_crc32c_avx512>
(4) This is running in a VM with:
Rocky Linux release 8.10 (Green Obsidian)
16 vCPUs
Hypervisor: VMware ESXi, 8.0.3, 24022510
Model: PowerEdge R650
Processor Type: Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
vCenter:
Version: 8.0.3
Build: 24322831
I attempted both vSAN and local storage. That didn't influence matters.
There is a known vSAN bug with invalid checksums + AVX-512, but that has
been fixed in an older version [2] (and besides the issue reproes with
local storage too).
Please let me know if there is any other info I can provide.
Regards,
Deep (VMware)
[1]
https://drive.google.com/file/d/15bGN_NlGsYx0lJCMGnCxDnV2avYkujgk/view?usp=sharing
[2]
https://knowledge.broadcom.com/external/article/367589/applications-using-avx512-instructions-i.html
Attachments:
[application/octet-stream] pg_control (8.0K, ../../CAE-ML+-OV6p9uvCFBcSQjZUEh__y0h-KjN+BseyGJHt7u8EP+w@mail.gmail.com/3-pg_control)
download
[text/x-log] config.log (359.5K, ../../CAE-ML+-OV6p9uvCFBcSQjZUEh__y0h-KjN+BseyGJHt7u8EP+w@mail.gmail.com/4-config.log)
download
[application/octet-stream] cpuid.out (557.2K, ../../CAE-ML+-OV6p9uvCFBcSQjZUEh__y0h-KjN+BseyGJHt7u8EP+w@mail.gmail.com/5-cpuid.out)
download
[application/gzip] initdb.out.tar.gz (1.2M, ../../CAE-ML+-OV6p9uvCFBcSQjZUEh__y0h-KjN+BseyGJHt7u8EP+w@mail.gmail.com/6-initdb.out.tar.gz)
download
[text/x-patch] debug.patch (2.2K, ../../CAE-ML+-OV6p9uvCFBcSQjZUEh__y0h-KjN+BseyGJHt7u8EP+w@mail.gmail.com/7-debug.patch)
download | inline diff:
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 1914859b2ee..cd40c3de06c 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -4436,6 +4436,8 @@ WriteControlFile(void)
offsetof(ControlFileData, crc));
FIN_CRC32C(ControlFile->crc);
+ elog(LOG, "WriteControlFile crc = %u, algo is avx = %d", ControlFile->crc, (pg_comp_crc32c == pg_comp_crc32c_avx512));
+
/*
* We write out PG_CONTROL_FILE_SIZE bytes into pg_control, zero-padding
* the excess over sizeof(ControlFileData). This reduces the odds of
@@ -4554,6 +4556,8 @@ ReadControlFile(void)
offsetof(ControlFileData, crc));
FIN_CRC32C(crc);
+ elog(LOG, "ReadControlFile crc = %u, ControlFile->crc = %u, algo is avx = %d", crc, ControlFile->crc, (pg_comp_crc32c == pg_comp_crc32c_avx512));
+
if (!EQ_CRC32C(crc, ControlFile->crc))
ereport(FATAL,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 62bbd08d9f6..69c561f67ad 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -1609,7 +1609,7 @@ bootstrap_template1(void)
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" --boot %s %s", backend_exec, boot_options, extra_options);
+ printfPQExpBuffer(&cmd, "rr record \"%s\" --boot %s %s", backend_exec, boot_options, extra_options);
appendPQExpBuffer(&cmd, " -X %d", wal_segment_size_mb * (1024 * 1024));
if (data_checksums)
appendPQExpBufferStr(&cmd, " -k");
diff --git a/src/common/controldata_utils.c b/src/common/controldata_utils.c
index fa375dc9129..98de2fc9db5 100644
--- a/src/common/controldata_utils.c
+++ b/src/common/controldata_utils.c
@@ -211,6 +211,12 @@ update_controlfile(const char *DataDir,
memset(buffer, 0, PG_CONTROL_FILE_SIZE);
memcpy(buffer, ControlFile, sizeof(ControlFileData));
+#ifndef FRONTEND
+ elog(LOG, "update_controlfile crc = %u, algo is avx = %d", ControlFile->crc, (pg_comp_crc32c == pg_comp_crc32c_avx512));
+#else
+ pg_log_info("update_controlfile crc = %u, algo is avx = %d", ControlFile->crc, (pg_comp_crc32c == pg_comp_crc32c_avx512));
+#endif
+
snprintf(ControlFilePath, sizeof(ControlFilePath), "%s/%s", DataDir, XLOG_CONTROL_FILE);
#ifndef FRONTEND
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build
@ 2025-06-11 21:42 Nathan Bossart <nathandbossart@gmail.com>
parent: Soumyadeep Chakraborty <soumyadeep2007@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: Nathan Bossart @ 2025-06-11 21:42 UTC (permalink / raw)
To: Soumyadeep Chakraborty <soumyadeep2007@gmail.com>; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; johncnaylorls@gmail.com
(I've CC'd the committer of the feature.)
On Wed, Jun 11, 2025 at 11:10:31AM -0700, Soumyadeep Chakraborty wrote:
> We are getting a control file checksum mismatch during initdb.
>
> [...]
>
> Note that this only reproduces with clang-19 -O0 and NOT -O3. I haven't
> tried with other versions of clang.
>
> OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting picked
> for both cases, for CRC instructions.
I'm wondering if there's anything else this could be besides a compiler
bug. It's interesting that -O0 is the broken case. Have you tried looking
for differences in the assembly code?
--
nathan
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build
@ 2025-06-12 06:57 John Naylor <johncnaylorls@gmail.com>
parent: Nathan Bossart <nathandbossart@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: John Naylor @ 2025-06-12 06:57 UTC (permalink / raw)
To: Nathan Bossart <nathandbossart@gmail.com>; +Cc: Soumyadeep Chakraborty <soumyadeep2007@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Jun 12, 2025 at 4:42 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> (I've CC'd the committer of the feature.)
>
> On Wed, Jun 11, 2025 at 11:10:31AM -0700, Soumyadeep Chakraborty wrote:
> > Note that this only reproduces with clang-19 -O0 and NOT -O3. I haven't
> > tried with other versions of clang.
> >
> > OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting picked
> > for both cases, for CRC instructions.
>
> I'm wondering if there's anything else this could be besides a compiler
> bug. It's interesting that -O0 is the broken case. Have you tried looking
> for differences in the assembly code?
Hmm, clang at -O0 is the same in versions 18, 19, and 20. The change
from v17 is easy enough to workaround as in the attached, which
produces identical output as master at -O2. Does this fix it for you?
If that's not it, it may be more difficult, as the change at -O0 from
v16 to v17 is large, so it's hard to see what's going on.
--
John Naylor
Amazon Web Services
Attachments:
[text/x-patch] v1-crc-workaround-clang-18.patch (636B, ../../CANWCAZa9Q4_NcGn5CUHwEvBCN=nZz1_BFfeJEXXon05E_tYi7w@mail.gmail.com/2-v1-crc-workaround-clang-18.patch)
download | inline diff:
diff --git a/src/port/pg_crc32c_sse42.c b/src/port/pg_crc32c_sse42.c
index 9af3474a6ca..ec267384b5d 100644
--- a/src/port/pg_crc32c_sse42.c
+++ b/src/port/pg_crc32c_sse42.c
@@ -122,7 +122,10 @@ pg_comp_crc32c_avx512(pg_crc32c crc, const void *data, size_t len)
y0;
__m512i k;
- k = _mm512_broadcast_i32x4(_mm_setr_epi32(0x740eef02, 0, 0x9e4addf8, 0));
+ k = _mm512_setr_epi32(0x740eef02, 0, 0x9e4addf8, 0,
+ 0x740eef02, 0, 0x9e4addf8, 0,
+ 0x740eef02, 0, 0x9e4addf8, 0,
+ 0x740eef02, 0, 0x9e4addf8, 0);
x0 = _mm512_xor_si512(_mm512_castsi128_si512(_mm_cvtsi32_si128(crc0)), x0);
buf += 64;
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build
@ 2025-06-12 17:30 Soumyadeep Chakraborty <soumyadeep2007@gmail.com>
parent: John Naylor <johncnaylorls@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: Soumyadeep Chakraborty @ 2025-06-12 17:30 UTC (permalink / raw)
To: John Naylor <johncnaylorls@gmail.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Jun 11, 2025 at 11:57PM John Naylor <johncnaylorls@gmail.com> wrote:
>
> On Thu, Jun 12, 2025 at 4:42AM Nathan Bossart <nathandbossart@gmail.com>
wrote:
> >
> > (I've CC'd the committer of the feature.)
> >
> > On Wed, Jun 11, 2025 at 11:10:31AM -0700, Soumyadeep Chakraborty wrote:
> > > Note that this only reproduces with clang-19 -O0 and NOT -O3. I
haven't
> > > tried with other versions of clang.
> > >
> > > OTOH, gcc-14 is cool with both -O0 and -O3, with AVX-512 getting
picked
> > > for both cases, for CRC instructions.
> >
> > I'm wondering if there's anything else this could be besides a compiler
> > bug. It's interesting that -O0 is the broken case. Have you tried
looking
> > for differences in the assembly code?
Unfortunately, I haven't had a chance to dig into the assembly yet.
>
> Hmm, clang at -O0 is the same in versions 18, 19, and 20. The change
> from v17 is easy enough to workaround as in the attached, which
> produces identical output as master at -O2. Does this fix it for you?
> If that's not it, it may be more difficult, as the change at -O0 from
> v16 to v17 is large, so it's hard to see what's going on.
>
It unfortunately doesn't. I still get the same checksum failure. I tried
-O0 clang-19, clang-18 and clang-17. I've uploaded the rr recordings
and initdb output with your patch applied and the debug logs I had.
Regards,
Deep
initdb-clang17.crc.workaround.out
<https://drive.google.com/file/d/1OptFhlWsOqNx4s2fv1tikOhsqjh27u-E/view?usp=drive_web;
initdb-clang18.crc.workaround.out
<https://drive.google.com/file/d/1yXYWmt7cHMXgONMkr9Yl29YU2HXpE8LT/view?usp=drive_web;
initdb-clang19.crc.workaround.out
<https://drive.google.com/file/d/1JmLKQMXOCwx8YFlJ_xpKdniopjwGpcji/view?usp=drive_web;
rr-clang17.crc.workaround.tar.gz
<https://drive.google.com/file/d/1Tnjva4l4w6S6iKn2hJBaTy1NRNHTTsAV/view?usp=drive_web;
rr-clang18.crc.workaround.tar.gz
<https://drive.google.com/file/d/1m7iKJHcAPPcKXet9gKkGEzTiFbJ2C9Ul/view?usp=drive_web;
rr-clang19.crc.workaround.tar.gz
<https://drive.google.com/file/d/1zXzUtvMJbsNMJNQezI5uC3Fb8Y3sUuMs/view?usp=drive_web;
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build
@ 2025-06-13 14:03 John Naylor <johncnaylorls@gmail.com>
parent: Soumyadeep Chakraborty <soumyadeep2007@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: John Naylor @ 2025-06-13 14:03 UTC (permalink / raw)
To: Soumyadeep Chakraborty <soumyadeep2007@gmail.com>; +Cc: Nathan Bossart <nathandbossart@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Jun 13, 2025 at 12:30 AM Soumyadeep Chakraborty <
soumyadeep2007@gmail.com> wrote:
>
> On Wed, Jun 11, 2025 at 11:57PM John Naylor <johncnaylorls@gmail.com>
wrote:
> >
> > Hmm, clang at -O0 is the same in versions 18, 19, and 20. The change
> > from v17 is easy enough to workaround as in the attached, which
> > produces identical output as master at -O2. Does this fix it for you?
> > If that's not it, it may be more difficult, as the change at -O0 from
> > v16 to v17 is large, so it's hard to see what's going on.
> >
>
> It unfortunately doesn't. I still get the same checksum failure. I tried
> -O0 clang-19, clang-18 and clang-17. I've uploaded the rr recordings
> and initdb output with your patch applied and the debug logs I had.
Okay, in that case we'd better just kludge around it by disabling the
runtime check with clang at -O0 (attached). That will at least get these
kind of builds working while I get more info for a bug report.
--
John Naylor
Amazon Web Services
Attachments:
[application/x-patch] v2-crc-workaround-clang-18.patch (654B, ../../CANWCAZbsuavqUK4tg6UF-0-GVRMaq7BafUx4+Dyd12y=-AuFAA@mail.gmail.com/3-v2-crc-workaround-clang-18.patch)
download | inline diff:
diff --git a/src/port/pg_crc32c_sse42_choose.c b/src/port/pg_crc32c_sse42_choose.c
index 74d2421ba2b..802e47788c1 100644
--- a/src/port/pg_crc32c_sse42_choose.c
+++ b/src/port/pg_crc32c_sse42_choose.c
@@ -95,7 +95,9 @@ pg_comp_crc32c_choose(pg_crc32c crc, const void *data, size_t len)
__cpuidex(exx, 7, 0);
#endif
-#ifdef USE_AVX512_CRC32C_WITH_RUNTIME_CHECK
+#if defined(__clang__) && !defined(__OPTIMIZE__)
+ /* Some versions of clang are broken at -O0 */
+#elif defined(USE_AVX512_CRC32C_WITH_RUNTIME_CHECK)
if (exx[2] & (1 << 10) && /* VPCLMULQDQ */
exx[1] & (1 << 31)) /* AVX512-VL */
pg_comp_crc32c = pg_comp_crc32c_avx512;
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build
@ 2025-06-15 01:23 Andy Fan <zhihuifan1213@163.com>
parent: John Naylor <johncnaylorls@gmail.com>
0 siblings, 1 reply; 7+ messages in thread
From: Andy Fan @ 2025-06-15 01:23 UTC (permalink / raw)
To: John Naylor <johncnaylorls@gmail.com>; +Cc: Soumyadeep Chakraborty <soumyadeep2007@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
John Naylor <johncnaylorls@gmail.com> writes:
Hi John,
> Okay, in that case we'd better just kludge around it by disabling the
> runtime check with clang at -O0 (attached). That will at least get
> these kind of builds working while I get more info for a bug report.
I run into the same issue at [1] and verified your patch works. Thanks
for the fix!
[1] https://www.postgresql.org/message-id/87o6uqd3iv.fsf%40163.com
--
Best Regards
Andy Fan
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build
@ 2025-06-16 02:32 John Naylor <johncnaylorls@gmail.com>
parent: Andy Fan <zhihuifan1213@163.com>
0 siblings, 0 replies; 7+ messages in thread
From: John Naylor @ 2025-06-16 02:32 UTC (permalink / raw)
To: Andy Fan <zhihuifan1213@163.com>; +Cc: Soumyadeep Chakraborty <soumyadeep2007@gmail.com>; Nathan Bossart <nathandbossart@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Sun, Jun 15, 2025 at 8:23 AM Andy Fan <zhihuifan1213@163.com> wrote:
>
> John Naylor <johncnaylorls@gmail.com> writes:
> > Okay, in that case we'd better just kludge around it by disabling the
> > runtime check with clang at -O0 (attached). That will at least get
> > these kind of builds working while I get more info for a bug report.
>
> I run into the same issue at [1] and verified your patch works. Thanks
> for the fix!
Pushed, thanks for testing, and thanks Soumyadeep for the report!
--
John Naylor
Amazon Web Services
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2025-06-16 02:32 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-06-11 18:10 Invalid control file checksum with AVX-512 during initdb on a clang19 -O0 build Soumyadeep Chakraborty <soumyadeep2007@gmail.com>
2025-06-11 21:42 ` Nathan Bossart <nathandbossart@gmail.com>
2025-06-12 06:57 ` John Naylor <johncnaylorls@gmail.com>
2025-06-12 17:30 ` Soumyadeep Chakraborty <soumyadeep2007@gmail.com>
2025-06-13 14:03 ` John Naylor <johncnaylorls@gmail.com>
2025-06-15 01:23 ` Andy Fan <zhihuifan1213@163.com>
2025-06-16 02:32 ` John Naylor <johncnaylorls@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox